at the point in late Goblet of Fire where we find out about the stunt the family Crouch pulled to get Barty Jr. out of Azkaban. I seem to recall that there's something about how someone witnessed the burial of "Barty Jr." (in reality his mother) with their own eyes, or some such.

When you think about it, it makes sense that there would have to be human wardens as well as Dementors. Can you see Dementors going around providing food, cleaning the building, and so on? If that's the case, though, those poor humans must have pretty much the worst job in all of the wizarding world...