How the hell did all those bad guys get back to Roman times?
Sontarans, Cyber/cybusmen, Daleks at this point, Nestine, Slytheen and all the other mentioned. None of them had Time travel, did they? The Silurians should still have been hibernating at this point. And there's something that looks like a Zygon that should still be asleep under neath Loch Ness.
SO It seems that the Big bad would need Time travel to get them all there to make the Pandorica work. And I don't see that fleet fitting through the cracks, if that's the reply. I still can't see any way of accurately navigating through them, certainly not to coordinate a fleet that size.
Meaning that the short list is pretty much Time Lords only.

I do wonder if the Dream Lord is behind this. The idea of the Doctor's evil side doing some damage appeals to me as much now as it did during trial of a Time lord all those years ago...

And, as an extra bonus thought, I hope to hell that they don't find some way to make all of this a consequence of the choices made during Amy's Choice and essentially retcon away what has happened...