I don't remember who said it, but to succeed in the music world as an indie artist you need a dedicated following of 300 fans that will spend $100 dollars a year. If you get these 300 fans you will make a living where you can subsist on these fans for you can make 30k assuming no expenses in the creation of this $100 dollars (such as gas money, t-shirt base cost, creating cds to sell, etc). Furthermore while 30k is livable it is definitely not a great living.

The same logic will apply to this specialty book store, problem is unlike being a lowly person with a passion on music, expenses are everything in this industry.
1) You have to buy the books from someone else, you may only make $1 dollar for every book you sell
2) You have to pay rent
3) Paychecks for any help you have
4) Stolen product, missing money from the till, fraud.
5) etc

How many books will you have to sell to make ends meet? How many book fanatics will you have that will gravitate to the store? The number will have to be in the several thousand and each fan will have to spend at least $100+ and on this $100+ you may only make 5 to 20% of that total, only $5 to $20 dollars.

------------------------------------------------

Furthermore as a sole proprietor you won't have any negotiating power with the publishers of the books.

------------------------------------------------

There is a reason many mama/pop book stores or second hand/used this way they can keep there margins at least in the 33% to 66% area per each book sold.