Anything on the field has priority over anything coming into land as far as I know (apart from emergencies, I'd have thought!). Although you could turn round and taxi back, why should you? You made you intentions known at the appropriate juncture. They called final after this. Their problem.
The only way I can see you being justifiably sheepish is if you only turned you radio on just prior to lining up, thus missing any previous calls from said craft on final. Otherwise nah, not your fault.
A/G stations rely almost entirely on the sensibilities and communication skills of individual pilots. Sadly not entirely infallible by any means