You started off within the Cardiff CTR, on a clearance given to you by Cardiff Tower. It's perfectly reasonable to expect modifications of that clearance (such as "no altitude restriction") to come from Cardiff Approach or Radar. I wouldn't regard being below the floor of the CTA for a short while on a straight line route as making any significant difference, unless your exit from controlled airspace was explicitly noted by Cardiff.
Bit surprised to hear "no altitude restriction" though, given the overlying airway. Above what level is the airspace controlled by London?