Can I limit my practice to criminal defense and keep the lights on?

I know the title seems ridiculous, but hear me out....

When I left my job at a large firm, it was with the idea that I was leaving to become a criminal defense attorney. After all, that is why I went to law school in the first place. The whole big law firm thing happened because, after never having made more than $8.00/hour in my life, it was hard to turn down a guaranteed six-figure salary. However, my heart was never really into the corporate practice and I was always looking forward to leaving and becoming a criminal defense attorney.

The ironic thing, however, was that when I left the firm, most attorneys I talked to were telling me that I couldn't sustain a practice limited to criminal defense. They were saying that I would have to do some family law or personal injury to stay afloat. So there I was - I had left the firm to become a criminal defense attorney and now, everyone was telling me that, while I could practice criminal defense, it would have to be as part of a general practice.

I didn't really give the comments much attention at first. I assumed that these attorneys were basically saying that if I wanted to make a lot of money I would have to expand my practice. I wasn't as concerned about making a lot of money, as I was developing as a criminal defense attorney and as such, I tended to brush off such comments. However, as I started making my forays into the courtroom, I noticed that there were quite a few prominent criminal defense attorneys handling cases in other areas of practice (e.g. divorces, PI cases, civil lawsuits, etc.). I began to believe what others had been telling me.

As a result, not too long after I got out on my own, I started taking on some civil litigation. At first, the civil stuff was the only retained work I was getting. My criminal practice was primarily limited to court appointments. Since that time, as my criminal practice has grown to occupy a majority of my time, I have given serious consideration to the idea of eliminating my civil practice. I want to do this for a couple of reasons.

For one, I am somewhat worried about taking on cases in multiple areas of the law. The one thing I did like about big firm practice is that it allowed you to specialize in one area of the law. It gives you a certain confidence to know that you are qualified to handle most issues that will come up in the area you are in. I don't know that I can keep up with all of the goings-on in criminal law and other areas of the law. I also worry that by trying to keep up with these other areas, I will be taking time from developing my criminal practice.

My other primary concern is that I've noticed that I just don't have the same motivation or drive when it comes to my civil cases. I always seem to place my civil cases at the bottom of my priority list. I worry that this lack of motivation will catch up to me eventually and one of my clients will suffer for it. I just don't feel that my civil clients get the same effort that my criminal clients get.

All that being said, I am still worried about getting rid of this part of my practice. I don't know too many attorneys in San Antonio or the surrounding area that have the luxury of maintaining a criminal defense-only practice. There are a few, but all of those I can think of are the prominent attorneys. Did they start out having to take civil cases as well? Am I jumping the gun and trying to specialize too quickly?

Advice? Suggestions?