I agree with you on the points above on the Whites

Here's how I see it when it comes to the Ajah sizes

First consider that you have a group of women who come to the Tower specifically for certain types of education and end up Yellow, Grey, White or Brown. Odds are that most of these women don't ever really consider another Ajah because they are relatively specialized and the Red, Green and Blue won't offer them much. Basically I think of the Yellows as women with a "calling" and the White, Grey and Brown as the more general scholars with personality being the main difference between them (Grey's being practical and worldly and Browns being "in their own heads" and the Whites being logical). I think Jordan did a pretty good job of making this clear through how he portrayed the women he showed us from each Ajah.

Seems, however, that the majority of women come to the Tower for one of two reasons, they wish to fight the Shadow in a direct way (i.e. Green) or they wish to contain the menace of male channelers (i.e Red). This makes sense when you think about how outsiders view the Tower and the function of Aes Sedai, assuming they don't think of Aes Sedai as Darkfriends thus you have a large pool of women predisposed to join the Green or the Red, but there are the handful that don't have the personality to fit in either place and thus find their way to the Blue which caters to a broader world view than either of the others and allows for passions unrelated to the Power making it more akin to the Brown/White/Grey.

Obviously these women could go in a variety of directions once they actually learn about the Tower and what it has to offer, but the mentality is there and barring a great Talent for Healing or a scholarly disposition they probably aren't going to be steered too much toward the Yellow, Grey, White or Brown.

The Blues seem to have the least defined purpose of any Ajah, so it seems like the women who end up there are probably women who, if they disliked men or had some personal reason to want to kill shadowspawn, could easily have been Green or Red. Moiraine even thinks to herself that Siuan might choose the Green in New Spring. We also see that the Blues have a variety of skills that overlap a bit with other Ajah's such as Merana approving of Rand sending Rafela to negotiate with the Sea Folk and Moiraine and Anaiya's clear ability with Healing.

So basically I think the Blue is smaller because it draws in women who could fit into another Ajah but have more general interests so you end up with women who are somewhat combinations of multiple Ajah traits without any one trait taking primacy. Moiraine is sort of Yellow-Grey-Green combo and Siuan, is a Grey-Green-Brown. Leane seemed to be a Grey-White-Green combo before her Stilling/Healing, but after her ordeal the Green side took over leading her down a new path.

I think that is the root of the comment made in New Spring about "the Blue making use of all of it's talents" the recognize that there is more than just one pursuit for an Aes Sedai and encourage their members to become proficient in broad subjects.