You're correct about the point of a totem; the idea is that it creates a guaranteed inconsistency for you to be able to tell if you're in a dream or not. If someone puts you in another world, you don't know if you're in reality or not. If your totem has not been re-created accurately by that person though, you will recognize the problem, and realize that you're in a dream.

This means that, yes, IF Cobb was the only other person to know what his wife's totem felt like, then he's safe. But since was his wife's, he doesn't know what's happened to it over the years. He assumes that she kept it safe, but it's possible that someone broke into their house one night to examine it. Or it's possible that after his wife died, someone examined the totem before Cobb was able to get ahold of it (perhaps they intercepted the body on the way to the morgue/hospital).

What I'm saying is, Cobb can't possibly know FOR SURE that the totem is clean, he simply assumes it is, and therein lies the danger.