Have you asked your instructor for advice?
What I was taught to do was to hold over a specific point (could be a VFR point or another prominent feature that you can also see on your map).
That way, it's much more difficult to get disorientated because you always have a reference point you can quickly and easily find on your map.
It's also really important that you have the plane trimmed as well as possible to minimise altitude changes. If the plane suddenly veers up or down when you let go, it's not trimmed.
I think it's expected that you may gain a little altitude, speed up/ slow down and not do the perfect orbit when planning a diversion. But you still need to know clearly where you are and the plane still needs to be relatively where it should be in terms of altitude and speed.