Around 30 years ago anime was extremely niche, and hardly know about in the western world. Few had fan subs, and most anime was virtually Japanese exclusive (this was Ryoji's day). Now you look at anime and virtually anybody age 20 or younger has at least watched an anime (even if it's just DBZ, Pokemon or Naruto). VN's have a huge potential as an art form, and will eventually become popular. And believe me, when they do become popular (may not happen for another 20-40 years btw) it will suck for people like us who have been around when they were obscure. So my advice, enjoy the obscurity while it lasts. ;D I prefer a niche community filled with people who are deeply passionate about their hobby (fuwanovel), then a huge community with a bunch of casuals (the contemporary anime community).
If I had to respond to that person though (and I've been in that situation before), I would emphasize how VNs are a new and in some ways an "experimental" art form. Honestly, when dealing with these people the more pretentious you sound the better. In these conversations I always either bring up Narcissu, because 1) it is sort of avant garde, and 2) it has no sex scenes which invalidates their accusations. Or I bring up Swan Song or Saya no Uta (two VNs where the sex/rape scenes actually serve to develop characters) and compare it to popular contemporary fiction like Game of Thrones or something. Because honestly, it's not that different. People have no problem with sex and rape in Game of Thrones (neither in the books or the TV series) but for some inexplicable reason they can't stand 2d sex scenes? Also emphasize how if somebody were looking for porn, no sane person would read through a 30+ hour story just for a handful of sex scenes.