It would purely be up to the discretion of the crew on the particular flight.
They may not like the idea of people wandering in and out of the premium cabin in the middle of the flight, even if it's just 2 people and just the once. They can be quite particular.
May be worth your while checking if they have any last minute upgrade special deals when you check in. Quite often you may be offered a special reduced upgrade fare a few weeks or days before travel via email if the business class cabin is particularly quiet on your flight.