I never fly with the GPS off. I do however, on the occasional flight, turn it away so that I can't easily see it.
Any doubt though, and I will refer back to it.
I would not advocate turning it off, but keeping it onboard as a backup. This is for the simple reason, that GPS's find it more difficult to get a fix when moving fast. (This is particularly important with non Sifi III models, which is most of the portable aviation units.)
Combine this with a poor view of the sky in many GA machines, and perhaps the unit not having been switched in in 8 weeks (which means an old satelite position database) means that you might struggle to get a fix when you most need it.
For these reasons, I tend to turn it on, getting a good 3d fix, and then turning it away where I can't see it, if I want to test my dr skills.
dp