sms777 - Really, you have no grounds for complaint. The bloke was carrying out his job which authorises him to inspect aircraft in an aviation environment where the aircraft can be put into the air at any time.
It's no different to you driving your car on a public road and being pulled up and stickered by the cops for a mechanical fault. By driving on a public road, you are essentially putting yourself under the authority and discretion of traffic police.
The same thing applies if I operate a workshop with employees. An authorised Govt safety inspector can arrive at any time and demand to inspect any machinery that requires a safety inspection, such as an air compressor.
If I refuse him entry, he can get a warrant to enter my premises. I can tell him now it not a convenient time and try to arrange an inspection at a more convenient time - but he still has the authority to enter and inspect equipment he is authorised to inspect.
By leaving your aircraft in a hangar on a location defined as an aviation location, you are putting your aircraft up for inspection at any time, just as your car is liable for inspection at any time on a public road.