Over run
Your maths are very good but some key facts have been forgotten.
The lease costs of the 737-800 are significantly higher than the F100 and must fly a great deal more to recover that cost. That means back of the clock flying and I am sure VP would rather commit a new A/C to a schedule on the SYD-MEL route before they commit it to Broome, so at best VP would schedule the broome service as an night flight like ADL-BME.
If this is the case then where are VP going to pick up another 168 people each way to make that profit? it might work for the Dry but it isnt going to work in the wet!
If we use your best guess that VP can get 90 pax at $495 then Skyeast could use the same cost and sit those 90 pax on the F100 that is already based in Perth, at prime time.
That would give Skyeast a reveune of $44550 which equates to a profit of $13157 for the round trip. Do that 7 times a week and they are returning 4.8 Million a year. Not a bad profit for an A/C that would cost less than that to buy.
Whilst the 737 has proved it's self to be the back bone of the low cost operations so far look it only works for cities with a population of greater than 50,000. Broome does not fit that category even in the dry.
Maybe it would be interesting for you to look at the jetblue model or germania becuse both of them have achieved success without the 1960's designed 737, and both seem to be making money.