But I've never seen it affect non-channeling capabilities of channelers.

The only thing it seems to inhibit are behaviors or intentions that would enable them to remove the a'dam or escape, and act as a direct impediment to the ability to channel due to it being an involuntary Link.

Thus, if you didn't know what the collar did, you could reach up and feel it but until you try to remove it you wouldn't start suffering the negative physical reactions.

It acts like a binder in that way - some Aes Sedai can't even bring themselves to think of lying or using the One Power as a weapon, while others interpret it to be an inhibitor on action and not thought.

Rand, for example, could access the True Power to destroy the a'dam around his neck, because it seems the only thing the a'dam affects is your access to saidin or saidar, not other capabilities. He could even form weaves that would directly harm the a'dam...once again, because all the a'dam seems to affect is your bodily control and your access to saidin or saidar. Slightly flawed example, but it does show that the specialized construction of the a'dam and sad bracelets can only limit channelers from using saidin or saidar.