For the earphones the polarity is not significant, so reversing the connections within the adapter will not matter, but strictly tip of 0.25" connector should connect to pin1 of the XLR.
Consider a sine wave - the signal moves positive then negative to complete the sine wave cycle. All that happens with the reversed wiring case is instead of having the cone of the speaker move outwards initially, it will move inwards - the phase is reversed. This will not affect what you hear to any degree since humans are not phase sensitive.
Phase really only matters when you have multiple speakers driving into a single 'space' - for example if you wire one speaker in-phase and one out-of-phase then you will suffer cancellation, which will be more pronounced at the lower frequencies, perhaps to the point of no bass at all!