I never switch it off. It's way more immersin, at least for me, to actually hear everyone's voice, that's partially the reason why I haven't started hollow/altaraxia yet. Seiyus put a lot of work into it, so it's such a waste to turn it off!
Though it really gets on my nerves when some characters have a very irritating voice (I'm looking at you, Hoshimemo) or some supporting characters are left unvoiced (Rewrite).