It's not only the current employer's view. For all you know the relevant director there may themselves take a new position in future years elsewhere in the industry, above you, and remember the situation.
At least let your new employer know the situation, otherwise they may ask for a reference, this is refused, the new employer's HR policy is not to take people on without an adequate reference, and then you are stuffed.