In simple terms, the vortex is caused by high pressure air under the wing arriving at an 'open end' (wing-tip) where it naturally wishes to make friends with the low pressure air on top. Hence the improved efficiency due to winglets which modify the 'open end'.
If I recall correctly, the boundary layer thickening you mention is only a concern for the upper surface of the wing. I think any BL thickening under the wing is either non-existent or insignificant?
It may be of interest to your thoughts, but a combat jet spends a significant amount of time with the wing near the stall in manoeuvre, so it is vital that it does not 'pitch-up', otherwise control could be lost. I'm sure JF will correct my failing memory, but I recall the Mk 2/6 Lightnings used a leading-edge 'notch' to reduce boundary layer outflow and a wider chord at the tips (extended leading edges). Both of these improved the manoeuvre characteristics of the type which spent a lot of time in stall buffet
(including in the circuit).