Perhaps it's OP objects he won't touch, indeed. The OP is inimical to the TP, and vice versa.

Another possibility is that it's not picking and touching objects SH can't do, it's bringing them with him when he "shadow-travels" the Myddraal way (some speculate it involves stepping out of the Pattern to re enter it in another "shadow-zone", that Myrddraal can spot these zones of shadow in the pattern. Perhaps objects from the Pattern can't survive this travel, and this is why the Myddraal bind their swords with a human soul...). To steal the DM himself, he may have had to move out of Cadsuane's room and all the way to where Semirhage was held the mundane way (which was not exactly a good plan!). We know he can bring people with him (he did it to Moghedien), but maybe not inanimate objects with no soul. The Myddraal's clothes don't interact with the world the normal way.