Between "book," and "video game," I regard them and treat them as more akin to a book. They're definitely more than that, though. I think the best way I could describe it would be that they're stories delivered in a unique format and structure. They're stories with an experiential quality to them. Not unlike a movie or TV show, but with the added dimension of having a more immersive, intimate delivery.
I think you could call them a game, though. In particular those that have gameplay elements. With or without that though, VNs do fit the definition of structured play. They have goals, rules, some (albeit very limited) challenge, and interaction. You could probably dispute just how much is needed of each of those qualities to really be a game, but that'll be a matter of semantics.
In regards to the quality of experience VNs offer, it's just different. Different mediums offer different avenues and it's not really far to compare them. Movies have the benefit of live action and cinematography, which provides layers of meaning that visual novels can't offer. Books are like stories told pure. It gets to the essence of what is trying to be conveyed and it delivers it without much noise. Video games offer player agency. They all have their place and different stories/meanings/intentions are better suited to different mediums.