You are
flat.
If you work out with me, you'll quickly tire.
(I'm not really sure here. A reference to a machine in the gym or toIf you work flat tyres?)out, you work as fast as possible.
If you talk with my voice, you've lost your desire.
A flat voice is monotonous and lacks emotion.
When a joke falls to me, I make nobody laugh.
A joke that falls flat isn't funny.
When I annotate a letter, I take away half.
The letters are musical notes, which will be pitched half a note lower when annotated with the flat sign, ♭.
And of course, a friend you could live with is a
flat mate