Why did your instructor call 'Short Final' when you were still on the downwind leg?
"I looked over and there was the airport right next to us..."Ohhh..""
Were you the only a/c in the pattern?
Don't worry, it's a very common situation. At least, after only 4 hours, you had the presence of mind to find a field which you felt 'confident' about getting into and didn't freak out (until you got low enough to see the rocks!!). There will come a point where you will look around a tad more.