Ishamael was not defeated in tGH, he was defeated in book 3, tDR. He, along with Be'lal, was defeated in the battle for the stone. It was clear that Ishamael was mostly insane so him being a major threat is a stretch. Also the series was originally meant as a trilogy, so having big bad Ishamael being finally defeated (after pretending to be defeated in books 1 and 2) in book 3 was probably just part of the conversion from 3 books to 14.

As for the reason he had a sword fight with Rand instead of killing him. Ishamael, and even Moridin, has continuously tried to convert Rand instead of killing him. He believes that if he can convert Rand they together can help finally break the wheel of time. Instead of killing him which would give the Shadow a momentary victory for a few ages until LTT is reborn and the battle is fought again. Ishy is going for a complete victory, instead of one that will only last a few ages on the turning of the wheel.