This is really something that would depend on the person. Personally, I consider VNs to be their own form of media which is very close to a book, maybe even one of those choose-your-own-adventure books, but is different enough to warrant a different name. Calling it a game would imply that there is much more of an interactive element, but given that it is computer software it seems rather difficult to separate it from the same category that "games" get lumped into.
I would probably call it a game in the same sense that I would call The Beginner's Guide a game. It has elements of games, but those elements only serve as a mechanism to tell the story, which is the payoff. Contrast this with a game like Antichamber, where the story is just a metaphor for the game being played and is only a part of the experience, with the rest being filled with the logic and puzzle-driven gameplay.
...or we could go that route.