Add, a fruit is what you get
Remove, fifty-four is what you get
What word am I looking for?
You could be the word:
LIVE
If you add a letter, you can make:
OLIVE - a fruit.
If you remove a letter, you can make:
LIV - i.e. 54 in Roman numerals.