Sorry to sound unsympathetic Claire, but it doesn't sound as if you were "fit to fly" and you obviously knew it. The doctor may have been satisfied in issuing a certificate of fitness to fly, but that is largely for the airlines coverage. The certificate is certainly no guarantee that you will not suffer discomfort or pain, and the sensible judgment must rest with you.
The airline premium classes are commercial products that retail at significant additional cost, and it is not simply in the universal gift of the cabin crew to give them away. I am afraid there is no genuine medical reason for a free upgrade. You either pay for one, or if circumstances permit your insurance company may pay for one. The airline doesn't have to, and won't unless it feels it is in it's commercial interests to do so. To that end it is irrelevant if the higher class has only 2 people in it, or indeed is completely empty.
To answer your question when can you get one? That depends on the airline. Usually for operational reasons, for example when the flight is overbooked. Premium frequent flyer cardholders might also be upgraded free if economy is looking full with the possibility of additional ticket sales.
On another flight or another carrier or another day, you might have got lucky, but in most cases I doubt it. It certainly didn't hurt (and possible that was the only thing that didn't,) to ask, but I think your expectation was unrealistic.