I'm actually quite confused with your idea of "if it's an animated game made in Japan then it's a visual novel". They are allowed to make other types of the game too, you know?
And yeah, I generally do not consider these games to be visual novel - just interactive game/movie. After all, it's a visual NOVEL, so the reading need to be the main part, not the clicking (I have tried a game of this type before - the darn thing made me pick a choice every minute or two) and listening. Other things should only play as support element that even if you remove them, the game will still work just fine. That's why kinetic novel - a type of visual novel - exists - there is no choice in the game, some don't even have voices, all you need to do is read.
Btw, I do think that live-action and other non-traditional stuff are fine - as long as they still keep the elements mentioned above in the game.