I was taught side by side. Some advantages I saw were:
Appreciation of different perspective of the engine coming in relation to the horizon in right & left turns
Instructor could hold a carefully folded map over the instruments to force me to fly by visual reference only (if you recognise yourself, sorry if I swore at that point!)
Instructor simulating electrical failures by pulling circuit breakers