I've had a similar experience at Culdrose - at the time it was unproblematic for me, and the reasons were reasonably obvious, so I just did it. I think that they'd sort of decided I was a helicopter (I wasn't, but was flying something similarly slow and ugly).
Legalities? They were basically asking you to do something for their convenience, in VFR ops (as you certainly can't be IFR that low unless on an instrument procedure). If you were unable to do this safely and legally, then you had a duty as captain to state "unable" and ask for alternative clearances. If you breached the ANO, it's your licence, not theirs.
G