Yes, you can buy your own aircraft and fly it pretty much wherever you like! The license is what it says it is, private - so you can't profit from it (ie. be paid).
You can also choose to do add-on ratings and qualifications which allow you to fly in the dark, in cloud, with more then one engine, on floaty things instead of wheels etc etc..
Edit: like he said ^