The issue with the Isle of Madmen is not that all of the channelers are insane - it is that the female channelers never consolidated into an organization such as The White Tower so civilization never could progress due to the tainted male channelers running about.

Any Forsaken could have set up a base there, converted female channelers and less tainted male channelers to their cause, and would have had their position strengthened after the taint was removed from saidin.

There is a lot of potential for a Forsaken to be in the position to create the rudiments of a society and then be worshiped as a god by its' people.