The reason why I think VNs (why not? I'm bored and some kind mod will move it eventually) aren't games stems mainly from not considering books to be games. At the very least, kinetic visual novels are not.
Consider a picture book, and fundamentally all parts except the soundtrack (and there are VNs without sound, though rare) of a VN are already included. The main difference is that instead of clicking your mouse, you flip the page. If you consider newfangled electronic reading devices such as Kindles et cetera this should be even clearer. Being on a computer does not a game make.
In the case of visual novels with choices you have somewhat of a point, if you consider branching-only (no stats systems, etc) CYOA books to be games. The main point, to me, though, is that calling visual novels games is utterly misleading as to what they are actually like.
since players' attitudes differ, this is inconsistent. Can't really approve of it.
///
...now, on the question on whether it matters if they are games or not, it really doesn't. Weeeeeeee!