Dunno about the UK (or others that are regulated by JAA), but in the USA, the regulations state that an FAA inspector shall have a seat available for his use on the flight deck, and must be admitted to same for inspection purposes.
Yes, I suppose the Commander could refuse, but would certainly have to have a valid reason.
To twist the tigers tail ain't a good idea.
Having said this, have on one occasion asked an FAA inspector to leave, when I caught him playing 'round with the landing gear lock lever (DC-3, many years ago).
Never saw him again.