Perhaps controversially I take the same view of altitude, VFR, I don't get worked up about deviations but just enjoy the ride and keep an eye on CAS etc. unless I've positively agreed with ATC a specific altitude, or I'm IFR.
Round here the lookout is way more important...