A fan without a housing is a propeller, and different laws of fluid mechanics apply. Simply put, the air (flow field), when it reaches the ducted fan, slows down or hesitates at the face of the fan. This is part of the reason that the fan remains more efficient as speed increases. The lateral swirl, which is lost energy (slip) from a propeller, is contained by the ducting. It has happened. Apart from being noisy there was a return to the inefficiency of propellers at the air speed required.
But let's for a minute assume that the core of the fanjet contributes very little to the forward motion and assume that the fanjet produces most of the thrust (which I think I've heard is the case).
Now, if that's a "sweet spot" and an efficient way of reaching high cruise speeds and thrust envelopes it begs the inevitable questions: why not use ducted fans on piston and turboprop engine aircraft then? Why is the propeller still around if the fanjet is such a fantastic thing of efficiency and so good at reaching higher speeds?