Yes, the highly swept wing does suffer from problems at low speeds. Why else would fast jet designers bother to faff about with expensive, complicated and heavy swing-wing designs?
A couple of the problems are the tendency for the centre of pressure to depart significantly forward of the C of G in a stall, due to the tendency for spanwise flow to develop at the tips causing the chordwise flow speed to drop close to, or even below the stall speed for the wing. This has the obvious problem that at low speeds, a stall may develop which results in a tendency for the nose to pitch up, rather than the usual nose drop in the stall.