You did the right thing.
You could have, if you wished, continued your approach to 450 feet and then executed the go-arround. However, it was a perfectly sensible decision to go around at the point you did. In this situation the controller is not going to issue a landing clearance or a lower Not Below altitude following his decision to instruct you to go around.