In a word ‘YES’!!
As can £5k or even £3K. I'm going to assume that you are an 'a' license holder, not a microlight pilot and so I’m not going to recommend a microlight.
First thing you need to do is join the PFA. Membership is £42PA this will give all the benefits of the PFA (see their website) but also entitle you to copies of 'Popular Flying' - the by monthly mag. Many aircraft are advertised in the classified section at the back... many under £ 10K too.
You need to decide what type of aircraft you want - single or two seat? Monoplane, biplane, open cockpit... it’s a long list.
£10K would get you a Jodel, maybe a Taylorcraft, or an Aeronca Chief etc. It depends what’s on sale at the time really. At the other end of the scale aircraft like the wonderful Isaacs Fury or Pietenpol Aircamper also change hands for this kind of money.
£5K would get you a Turbulent, Taylor Mono, Jodel D9 etc.
£3K may get you a VP1, or a FRED
I actually bought a freshly permitted PFA single seater recently for fifteen hundred quid... but that was a bit of a fluke!
If you don't mind getting your hands dirty once in a while and find a nice grass strip to fly from you'll be amazed just how affordable the whole thing becomes.
Good Luck
Kingy