Where to find an aircraft: start off in the back of all the GA magazines. Or at your local flying club. If your local club is full of spam-cans then find another airfield which has more "interesting" aircraft. There are several websites out there for classified aircraft ads. Try the PFA magazine. And so on. No one said it would be easy - but you asked for cheap, not easy!
Hidden costs would be hangarage (price depends very much where you are based) and insurance. Hopefully no major maintenance, but that's always possible with aircraft ownership. You'd probably need a tail-wheel checkout, since most of these types of aircraft are tail-draggers, so budget on 7 or 8 hours of dual on something like a Piper Cub.
Biggest problem would be insurance. Generally, insurance companies are happy to ensure you on anything as long as you get some dual instruction first. Dual on a single-seater obviously isn't possible, so you might find it hard to get insured on a single-seater tail-dragger if you're new to tail-dragging - I don't know. Insurance is not a legal requirement, of course, but it is definitely a good idea.
Can't think of anything that I've forgotten. Hope that helps.
FFF
-----------