Yes, but aren't we told in book 3 that the reason why Dementors affect Harry more than other people is because there are true horrors in his past, unlike his classmates who have only experienced moderately bad things. The effect of a dementor is therefore tied quite explicitly to the past experiences of the person affected. Dudley's reaction is worse even than Harry's, which suggests, to me at least, that there must be something fairly traumatic in his past. Unless you read it as him simply not being used to being without the pampering cloud of Petunia's love, which would be taken away by the dementor attack. But that seems too simple to me.