What makes enjoying anime nerdy?

I touched on this one a little while back, but never made it the focus of an article all its own.  I really wanted to hit on the topic and dive into it in order to see if we can come up with anything that makes sense.  I've tried figuring it out on my own, but haven't come up with a notion that makes most sense.  Perhaps some ideas worth discussing will come together when sharing with you guys!

What exactly is it about anime that makes people think it's a nerdy activity?  Anime in and of itself has no nerdy qualities that I can think of.  I'm not even sure what a nerdy quality is!  Is the Supernatural live-action TV show just fine, but the anime adaption is for nerd viewers only?  Is X-Men acceptable in comic, movie and American cartoon form, but Wolverine in an anime style is for nerds?  Do you see why I struggle to make any sense of this?

When people are in love with anime, a lot of others are quick to label them nerds.  This isn't something that happens with other content.  People that love sports aren't considered sports nerds.  Those that love American Idol aren't nerds either.  Does it have something to do with the perceived amount of people that openly confess a love for something?  We know more people love sports and American Idol than anime, so there has to be some truth to that.

Does it have something to do with the fact that anime is animation?  Does the idea of animation for adults not factor in?  We have shows like The Simpsons and Family Guy here, but those shows focus in on rather crude, adult humor.  I happen to love those shows as well as anime.  When anime focuses in on adult situations but in a serious way, that seems to be where others have a problem.  Does anime dealing with serious topics not make any sense?  Can people not take animation in general seriously?

What do you think leads people to believe that anime is nerdy, and watching anime is a nerdy activity?  I think it's an unfair label and see no 'fact's to back it up.  Perhaps I'm just blinded because I'm an anime fan, so someone needs to take my blinders off!