For example, Logain is just as big as Rand, but is weaker than him in the Power.

Rahvin too, is just as big as Rand, but is weaker than him. In contrast, Sammael seems to be stronger than Rahvin.

I agree that your soul has a certain potential, and that your body can possibly limit that potential.

One thing that fits into this theory is RJ's quote where he states that weaves appear to originate from the channeler's body, and that this is one reason why there are physical limits on how much of the Power one person can channel.

It is also corroborated by RJ's quote that a shield surrounds the body of a person, which also means that a bigger person would require a larger shield if he is to be cut off from the Power.

We also know that Semirhage is as as tall as most men, and that Lanfear is also very tall. Graendal in turn is "fleshy", so her reduced height may be compensated for by her extra "width" if you will.

The link is vague, but I too believe there is a link. I just can't quantify it sufficiently to establish a clear rule.