I certainly don't log IFR time and there is no legal requirement to do so
That statement is absolutely correct. The legal requirement is at FCL.050:
The pilot shall keep a reliable record of the details of all flights flown in a form and manner established by the competent authority.
The form and manner that is established by the UK competent authority is detailed in Article 79 of the ANO 2009 and does not require the recording of flight time under IFR. AMC1 FCL.050 is, therefore, in the case of UK licensed pilots at least, irrelevant in the context of this thread.
The fact that the OP, having presumably spent a considerable amount of money and effort gaining a pilot licence, cannot be ar$ed to understand the privileges, restrictions or potential of what he has paid and worked for, whilst mildly disappointing, is hardly relevant.