Sorry I do not understand the problem here. You were not
obliged to comply if you were not in CAS but I do not see why you should not be able to comply and still maintain your lookout. The only exception here would be if it put yu into some other situation you could not comply with ie. into cloud/CAS without clearance/too low over built up areas etc. Assuming this was not the case you did the correct thing. the person in the "wrong" if anyone would be the airtrafficker who is going beyond his remit to issue it as an instruction not an advisory.