I find that I need to give my continental O-300 one shot of primer if it has been sitting for 15 minutes - if I don't do that, it won't start.
Normal shutdown procedure for lycoming and continental engines is to set the mixture to idle cut-off. Therefore when you start there will be no fuel in the carb and you'll need to keep cranking the engine until you suck the fuel back into the system. Using the primer puts the fuel directly into the cylinders, so when you crank it (hot or cold) it should start immediately without having to drain your battery with endless cranking. As long as you don't overdo it there is no chance of backfire. As another poster said - the primer is there to be used, so use it!