If we approach this from first principles the first point to think about is what the rules are and ICAO Annex 11 in the Note to para 2.6.3 says...
Where the ATS airspaces adjoin vertically, i.e. one above the other, flights at a common level would comply with requirements of, and be given services applicable to, the less restrictive class of airspace. In applying these criteria, Class B airspace is therefore considered less restrictive than Class A airspace; Class C airspace less restrictive than Class B airspace, etc.
Next comes what traffic has to be separated. Here, PANS-ATM (in para 5.2.1) says
5.2.1 Vertical or horizontal separation shall be provided:
a) between all flights in Class A and B airspaces;
b) between IFR flights in Class C, D and E airspaces;
c) between IFR flights and VFR flights in Class C airspace;
d) between IFR flights and special VFR flights; and
e) between special VFR flights, when so prescribed by the appropriate ATS authority....
Just in case there was any doubt, note that there is no requirement to separate anything from traffic that is flying in class G.
It's a strange thing - and nearly 35 years later I can still clearly remember the college lesson where I learned it - but all traffic inside controlled airspace is separated from anything outside. This is an axiom that appears to be commonly held by controllers wherever I go. Of course, in all my years as an operational controller I usually tried to build in some distance between aircraft in such situations, even if not quite the same distance as standard separation - and, once or twice, I have to admit that I used the inside vs. outside point when things got a bit closer than intended.
So, to answer the OP's question - in both examples a) and b) there is no loss of separation (because there is no separation required). Whether it is a good idea to allow such a situation to occur is a different matter entirely - at the very least I would pass traffic information to the two aircraft involved.
As Plazbot points out, the interpretation of the rules can differ in different countries and I, too, have seen three different ways that this is handled. In each case I have seen the differences are not in what is separated (or not) but rather the national procedures about what levels can be allocated in relation to the airspace boundary (or, horizontally, how close an aircraft can be vectored).
And that's before you consider the UK rules which allow aircraft to be 'controlled' outside CAS.
One last thought - although I'm sure it's fully understood. renard talks about TAs and RAs - ACAS/TCAS has little to do with whether separation exists but is intended only to try to prevent a collision between aircraft which are on converging trajectories.