Book told Mal exactly how an Operative thinks, how an Operative works. A man like that believes, he said.

I'm quoting from my copy of the script book, so it might not be exact to the movie.

Book: "It's how they think: sideways. It's how they move. Sidle up and smile, hit you where you're weak. Sorta man they like to send believes hard. Kills and never asks why."

Mal: "It's of interest to me how much you seem to know about that world."

Book: "I wasn't born a Shepherd, Mal."

Mal: "Have to tell me about that some time."

Book: "No, I don't."

Book has demonstrated his knowledge of weapons, fighting, tactics, and the criminal underworld. He can lay a stone cold beatdown when called upon, and the perceptive Early the Bounty Hunter goes after Book first ("That ain't no Shepherd"), going straight for the knock-down without asking any questions or taking any other approach (notice he threatens Kaylee and reasons with Simon, but does not even attempt either with Book). He has high level Alliance clearance, or at least enough pull to get immediate medical attention from them. From River reading his thoughts, he doesn't give half a hump if you're innocent or not. He knows how Operatives work and how they think. And he believes.

From later in the script:

Book (to Mal): "I don't care what you believe! Just ... believe it."

Book (and Whedon) doesn't have to "tell us about that some time" because the answers are already all there. Book was an Operative, believing hard, executing missions and people for the Alliance. At some point and for whatever reason, he lost his belief in that life and took a leave of absence, perhaps a permanent one, to sort out his life and find a different path, which is why he spent the time in the abbey and became a Shepherd. Hell, he even gave himself a fake name based on the Bible (the Book).

He left the abbey with no purpose (Kaylee: "You like ships. Don't seem to be looking at the destinations. What you care about is the ships and mine's the nicest.") and found himself caught up with the crew (Book: "I think I'm on the wrong ship."), suddenly getting to see the world from the other side, where the Alliance lawmen are the bad guys.

I don't think Joss would have left things completely hanging on Book if there weren't a solution to be found. That thus believed, the simplest answer that fits the known facts is that he was a former Operative for the Alliance, who lost his belief in them and found it in God instead, but then found himself dragged back into a world where his unique skills and knowledge at times became necessary.

I was only able to come up with that answer after seeing in Serenity how he understood the strong role belief played in the life of an Operative, and how that relates to Book himself. It's possible he was some other type of Alliance officer, but the belief thing cemented it in my mind.

Book: "When I talk about belief, why do you always assume I'm talking about God?"

Also, all the operatives and bounty hunters in the Firefly world are black. Conclusive fact.