The major polls aren't going to be right half the time. They are each going to be right more than 95% of the time.

If we knew nothing, we'd say each person has a 50-50 chance of choosing either Romney or Obama. Polling a single individual would be like the flip of our coin. And that is the flip which we are trying to prove is biased: i.e., that it will flip towards Obama more often than it will flip towards Romney.

If a poll only surveyed 100 people, and we were only looking at that one poll, you would have a point. But each poll is by itself surveying enough people to say with 95% confidence (or usually more) that the average is between this and that.

At 95% confidence, there is only a 5% chance the real average is outside of that margin. So let's say there are two polls saying pretty much the same thing. For one to be wrong, they would both have to be wrong. But there's only a .25% chance of them both being wrong. If there are three polls that say pretty much the same thing, then for one of them wrong they would all three have to be wrong, and for that there is only a .0125% chance. In other words, if there is a true consensus, then the possibility of even one of those polls being wrong becomes absurdly small; because for one to be wrong many of the others would also have to be wrong, and that is absurdly improbable. (Unless, of course, there is some industry-wide modeling error going on.)

Of course, there isn't a true, true, true consensus. Not in the sense that you can use that math so simply to get easy numbers. But there is still something of a consensus, between polls that each have a high confidence level, and so the probability of any one of the polls within the consensus being wrong is still going to be far less than 5%. Assuming there are no great model errors going on industry-wide, that is precisely why there is a consensus: because the probability of a poll with a high confidence level falling outside of those bounds is very small.