A novel clearly has roleplay elements. Less obviously, a novel also has game elements integrated with those roleplay elements: it presents the reader with choices regarding what to believe of the characters and their situations, how to interpret the fictional events and the narration itself given those beliefs, and what to predict given those interpretations; and it provides enough of a conclusion for the reader to interpret whether or not they chose correctly. (If it doesn't do all this, then I say it isn't really a literary novel; it's just a story.)

Not that I'd consider D&D literary. There are some indie RPGs which I think are more likely to guide players toward an artistic story-based experience, but there's a difference between them and literary novels. With a novel, the intended result is embedded into the artifact. What you see is what you get, plus (distinct from the literary artifact) whatever you bring yourself. But with most if not all RPGs, it's the process that is primarily embedded in the artifact. They are more like "How to Write a Novel" books than like actual novels.