I kind of do mean that, yes - that he's never had anything even remotely as traumatic as that happen to him (cf. the description of his temper tantrum all the way back in early book one after they drive around in the car for a whole day on the run from Harry's Hogwarts letter, where it explicitly says he'd never had such a bad day in his life). And of course there is the point you've mentioned many times before - that the Dursleys, and Dudley most of all, are rather afraid of magic, partially because they know nothing about it. And let's not forget that since Muggles can't even see Dementors, the attack must become automatically more confusing and disconcerting to Dudley.