This might be stretching it but after trying to figure this out for about an hour, I think I just need confirmation that the answer is or is not
Birch
Without my metal, I'm big in Japan
According to wikipedia, there are 11 different species of birch that are native to Japan. However, silver birch is not native to Japan. Silver is a metal.
I'm a rag with power
Birch is one species of tree that is used for wood pulp, which is made into newspaper. The term 'rag' can be used to describe a newspaper, and 'power' can either be talking about the process that converts birch to pulp, or the fact that newspapers can be powerful in terms of communications or propaganda.
A part of me bears fruit
Some (if not all) birch trees produce fruit called samara.
Another part is lighter than a fly
Can be talking about multiple things including samara, leaves, or the weight of a single piece of newspaper.
I can come in very large groups,
A forest of birch trees.
We can be more than exponential in size
From a little seed a mighty birch tree can grow.