For me, the only reason i think people are calling anime fans childish is because anime is targetted exactly at this particular age group (let's ignore the serious genres, people who don't watch anime don't really know about them anyway) where you are considered still immature.
So the logic seems to be that if you share interests with kids, you are probably one yourself, which is just a load of prejudices.
Not much imput from me (and probably not very elaborated), but i hope i at least made you feel better by letting you know that one more person read this wall of text