Fluff in Visual Novels, anime, other shows, manga, comics, movies, stories, and life, whether an adult area or not serves a purpose. Not every scene has to a direct link/part to the overall main storyline. If it did i doubt jokes would have been thought of, cultural society itself would not be diverse aside from language. In fiction it can and often does pull you into the story, helps bond you to the characters.
No depth, no drama, no embarrassing parts that can freeze you, no touching moments that can make you cry, no music that captures your heart and soul for a little. Your emotions are a type of fluff, the way you react to a story is based on fluff or lack thereof. But stories would be mostly bland without it.
Think if you will, what would the Harry Potter stories or the Chronicles of Narnia be like if you eliminated all the fluff from the stories. We would not have been as captured and enthralled if we were at all. The stories would not have been great to read or listen to on the audio books unabridged versions.
i say keep the fluff that is there in the stories. They would not be as engaging or as great without it.