Up until Hermione was a victim (and they all changed their tunes after that anyway) I could see how the other kids would jump to that conclusion. And I think that one kid, Ernie?, made a good case for it when explaining it to the other Hufflepuffs. He said that maybe Voldemort knew that Harry was going to be a more powerful dark wizard and that's why he went after him in the first place. And how else could a baby survive the killing curse? I could see that from their perspective and given what knowledge of Harry they have available. Plus he revealed himself a parseltongue and add in the hysteria they are probably feeling. They just want to find somebody to blame and make it less scary. If they continued thinking that after Hermione was petrified, then I would have agreed with you.