They refer to it as a thing of the Shadow because many of the Forsaken would drag people into TAR in the flesh during the War of Power and then (I assume) do bad things to them.

Imagine the ramifications of being nabbed by a Forsaken, dragged through a gateway to who-knows-where, and from there you can be taken anywhere in TAR with a thought. Anything can be done to you from OP weaves to walking nightmares, and there is NO ONE THERE to rescue you. We know how hard it is to find someone specific in TAR unless you have a pre-determined meeting place.

The Wise Ones said that each time you enter TAR in the flesh, you lose a little bit of your human self, but I think that part was derived over a long time from the belief that it was an evil thing to do.

The Shadow Rising - Beyond the Stone - Bair speaking to Egwene

"I will now tell you about dremwalking and Tel'aran'rhiod, in a very general way.... For one who is fully in the dream, dreamwalker or not, death there is death here. To enter the dream too completely, though, is to lose touch with the flesh; there is no way back, and the flesh dies. It is said that once there were those who could enter the dream in the flesh, and no longer be in this world at all. This was an evil thing, for they did evil; it must never be attempted, even if you believe it possible for you, for each time you will lose some part of what makes you human."