From Fiona Bruce's article on the portrait
Even now Simon told me about an incident in a restaurant not that long ago when the people sitting next to them turned to the waiter and said ‘you need to move that man, he’s putting us off our food and frightening the children’,” she said. “All in his earshot.
It wasn't recorded whether the family were forcibly removed from the restaurant and had their meals chucked after them, but that would certainly have been the result if it had been my place!