I find that the definition of a VN and an eroge varies from person to person, so I feel like it's more of an opinion than anything else.
Personally I mark the difference between a VN and eroge as a simple question: if the sex scenes were to be removed from the game / story, will it actually affect enjoyment in any of said game / story in any way?
Most of the good VNs I've read can be read through without the sex scenes, and it would not reduce the enjoyment of reading said VN in any way whatsoever. If anything, having sex scenes is a nice bonus if one happens to be somewhat horny at the particular time of reading. Can't say I ever was, because I'm usually too gripped by the story that I just roll my eyes at the H scenes and skip them if possible.
Eroges, on the other hand, are defined by me as games that cannot survive without the sex scenes. Games like Sengoku Rance and Monster Girl Quest come to mind. Think about it really - without the sex, the plot in these two games really begins to pale in comparison to actual well written stories. However, because of the element of sex, these games become enjoyable in their own right. In the case of MGQ, an average story becomes compelling enough for people to really get into it thanks to the sex.
In the end, it comes down to personal opinion. If people choose to be ignorant and write off well written VNs as nothing more than 'garbage hentai games', then it's their loss.