I dined with an ex-Army guy last night. He spent 6 weeks in Aden, ex-Malaysia, as the incoming battalion was late from UK and they were diverted from their return to UK.
He tells me that there were no "Special Forces" at that time as the only people who knew that term were the Special Forces themselves; it was not a term in general use until later.
I think the public perception of the SAS etc only arose after the Iranian Embassy siege in 1980 though possibly in the later 60s and early 70s as their role in Borneo emerged.