In the past few months I've done more flying than in the last 7 years put together (I started 7 years ago). I've finally put some money together and got some time to do it "full time".
I do, however, like the way you stated "it's not really being a pilot, just somebody with the flying license"... As I'm only in lower three digit hours, I do feel competent to perform a safe flight, but it's far from being experienced. Also, I do pick my flights not to be extremely demanding, as I don't want to push too far too soon.
As a point; its probably a good thing that you have concerns as to safety, it's something I'd rather see than a 'kick the tyres, light the fires' type.
Quite often many of us look back at our earlier lower timed flying and wonder how little we knew and how we didn't have a ding.. its quite common that in the post ppl stage, probably around 100-150 hours, people have enough knowledge to kill themselves and/or scare themselves rigid. Often they do.
Once you get through that stage, you start to realise that, no, flying is not easy and if you are not doing something, there's something you probably should be doing.
Its a sign of maturity that you approach the issue with concern, rather than machismo and bravado, don't put yourself off, but take it in a professional manner, allways leave yourself an way out.
as the adage goes; '
there's old pilots and bold pilots, but very, very few old, bold pilots'.
Don't beat yourself up.