People who are ignorant - and I mean simply "without knowledge" of the subject - go back to whatever bits of info they have and put them together in ways that are, well... ignorant! The guy probably has a sterotype of bodybuilders that he's applied to everyone who weight trains and, since he's not interested for whatever reason, he just bad-mouths the entire group. It happens in many areas of life!
You can try to educate him or just accept his ignorance and move on. If all else fails, just
kick his ass!
