Nothing wrong with Bakersfield per se. It is in the middle of nowhere and so when you are not flying there is nothing to do.
Personally, I prefer to start from somewhere reasonably nice near the coast.
If you want somewhere small with no frills - take a look at Santa Paula.