I Am A Human Riley

Question

My prefix in your room, you don’t talk about

My infix in your pants, you can’t sit still

My suffix turns a perfect memory into a rare condition

And though I’ve raised questions of a man’s humanity,

My whole is just swell.

What am I?

Answer 1

I bet you are

elephantiasis.

my prefix in your room, you don’t talk about

"The elephant in the room".

my infix in your pants, you can’t sit still

Ant(s), as already identified by other solvers.

my suffix turns a perfect memory into a rare condition

"An elephant never forgets". But add -iasis to "elephant" and you get the rare condition that's the subject of this puzzle.

and though I’ve raised questions of a man’s humanity,

Perhaps a reference to The Elephant Man.

my whole is just swell.

Elephantiasis is a condition (in fact, it's a name for a few different conditions) causing body parts to swell up monstrously.

Answer 2

Partial:

My infix in your pants, you can’t sit still:

Ants (from Ants in your pants)

My suffix turns a perfect memory into a rare condition

Hyper, see Hyperthymesia or Highly/Superior, see HSAM

Answer 3

Not quite sure this works, but are you:

Mr. Fantastic?

My prefix in your room, you don’t talk about

?

My infix in your pants, you can’t sit still

"ant" - see WELZ's answer

My suffix turns a perfect memory into a rare condition

Perhaps something to do with a "tic," often associated with rare brain/memory conditions... or "eidetic memory," commonly known as photographic memory?

And though I’ve raised questions of a man’s humanity,

Mr. Fantastic is a superhero, and the humanity of superheroes is often questioned

My whole is just swell.

It's simply fantastic!

And as for the title:

Well, Mr. Fantastic is at least mostly human, despite being a superhero