I'm perplexed at many of the responses to the OP.
Non-transponding VFR traffic can, as has been said, enter Class E/E+ airspace only with ATC approval. Transponding VFR traffic can enter without speaking to anyone.
During the hours of radar service at Inverness N560 from RIMOL to NESDI is delegated to Inverness up to Flight Level (FL) 130. Inbound IFR aircraft to Inverness from the south on N560 are transferred to Inverness radar descending to FL130 so may be under the control of Inverness ATC at any point below FL195 and depending on the tactical traffic situation outbound flights may be under the control of Inverness up till they reach FL195. Approach Radar ATCOs have a rating that can only be exercised within 40nm of the airfield and up to FL195. Hence the OP seeing traffic wearing Inverness squawks between FL100 and FL195.
In Class E/E+ airspace there is no separation requirement between IFR and VFR flights. Traffic information is all that is required. Why then would a glider be given a height to maintain, a band of levels or a track to fly? I'm puzzled at why a controller would be trying to separate IFR traffic from VFR in Class E/E+. A call from pilots to ATC with your intentions helps controllers give accurate traffic information to IFR flights and vice versa.
ATC shouldn't be giving any transponding VFR traffic that calls clearance to enter Class E/E+ airspace. If a pilot calls the "controlling authority" in the interests of good airmanship they should simply be asked to report entering Controlled Airspace and upon that call being received requested to maintain VMC and report leaving.
The GUSSI box has been mentioned a fair bit. By the gliding communities' own admission it is only required to be activated in particular weather conditions and even then only when pilots have a high degree of ability and when the LoA was written the gliding club admitted there were very few who were of that standard. I am unaware of the GUSSI box ever having been activated.
Outside Controlled Airspace Inverness ATC can provide IFR traffic with a Deconfliction Service, Traffic Service, Procedural Service or a Basic Service. When IFR inbound flights to Inverness leave Lower ATS Routes N560 or Y906, either laterally or vertically, they are given a Deconfliction Service by default unless they request a lesser service. At that point they must be separated by 3nm or 1000' from other identified aircraft that are displayed on radar as both primary and secondary returns. 5nm and 3000' has to be provided against unknown traffic seen as a primary and secondary return with Mode C. The separation standard against traffic which is not identified and only seen as a primary return, or with secondary but no Mode C, is 5nm. There is a sudden jump from no separation standard within Controlled Class E/E+ Airspace between IFR and VFR to very proscriptive standards within Uncontrolled Class G Airspace when flights are in receipt of a Deconfliction Service. This can present ATCOs with some problems but delaying descent to keep aircraft in CAS till they are clear of any transponding gliders (or other VFR traffic) is one solution.
The ability and experience levels of pilots varies and the same is true of controllers. It may not be the busiest airspace in the world but the complexity of airspace, mix of airspace users and the service provision rules in the general vicinity of Inverness are complex, not always easy to understand and can be frustrating for both pilots and controllers. Familiarisation flights and visits to towers to promote a better understanding of each other's roles were regular occurrences before security issues restricted them.