Service levels probably wouldn't suffer. The A319's, in the main, operate flights with a more relaxed cabin service. i.e City destinations and shorter routes. Therefore, people aren't buying as much and hence the service is generally quicker anyway.
The A320's operate the longer routes and these are often very busy service wise and the 4th crew member is obviously very much needed for cabin service.
Easyjet management have been harping on for over a year now about crew costs and ironically the crew at EZY have been screaming at them to reduce 319 capacity in order for them to become 3 man operated.