They refer to it as a thing of the Shadow because many of the Forsaken would drag people into TAR in the flesh during the War of Power and then (I assume) do bad things to them.

Imagine the ramifications of being nabbed by a Forsaken, dragged through a gateway to who-knows-where, and from there you can be taken anywhere in TAR with a thought. Anything can be done to you from OP weaves to walking nightmares, and there is NO ONE THERE to rescue you. We know how hard it is to find someone specific in TAR unless you have a pre-determined meeting place.

The Wise Ones said that each time you enter TAR in the flesh, you lose a little bit of your human self, but I think that part was derived over a long time from the belief that it was an evil thing to do.