That's actually a very interesting point...
But anyways, I think that Visual Novels lies somewhere in the middle, having the interactiveness of a game, but the longevity and detail of a book. To me, that was all I've ever wanted. For example, almost any VN (obviously excluding nukige) I've ever read have had a better experience overall than most of the Videogames I've played. Visual Novels, like books, ARE story and therefore, very immersive if they're done right.. In most videogames, the story is sadly only framework or filler for the actuall game, but that can be excused if the game is mostly just trying to be fun to play. As you've probably noticed while playing games, it is not often that the story and gameplay is actually integrated with eachother. Mostly, the story has to take a step back in favor of the gameplay itself.
But to further discuss the question of whether or not you're reading or playing a VN, I would say that if it has gameplay, you "play" it. If it doesn't have gameplay and only choices, you read it.
This was probably not very coherent, but these are my thoughts on this topic.