essouira,
Because as said, using it reduces the elements life. Now, most aircraft owners will winge when they have to fork out for a new one, most clubs will winge and build it into their hire rates. So whether you are a private owner or a considerate aircraft hiring customer, it possibly is helping you and the club/owner to not put it on unless required.
Then you must consider element of risk by not putting it on and whether that out weighs cost of replacement. I would suggest that most VFR around the houses flying below the freezing level, day VFR chaps, it is possibly worth considering leaving it off till required. And as someone else posted, if you are going IFR, well sodd it, put the thing on and have done with it. You have lots of other things to be doing and are more likely to require it.
As for the compass swing, very true. They are swung with them on and again for "most GA around the houses", is 5-10 degrees going to break the bank? BUt again, if IFR or doing precision flying well then yes, put the Pitot on. Someone said they have not noticed a swing, well I have quite remarkable so - up to 15 degrees. So yes it must be valid.