I had exactly the same problem, vanHorck.
I got legal advice on it. The answer (UK law) was that when the child is with one parent or the other, the other parent has NO say in what the other parent does.
If a divorced woman (with, as usual, child custody) objects to her ex husband taking the kids for a flight, she has two options:
1) p*ss in the wind
2) get a court injunction to prevent the father from taking the kids for a flight
The injunction would almost certainly be refused because it is unreasonable. Taking a kid to a strip club, or watching dogging... (sorry, a popular English past-time) that would be tricky.
Of course she has a 3rd option, frequently exercised by divorced women with sole child custody, which is to get revenge by messing with contact. This is a tough one, because if she is determined there isn't much one can do about it without going for a court order fast. You turn up at the house at the agreed time, ring the bell, and there is nobody there. After a few months of this, it will be hard to get an order because she can claim that since the kids have not seen the father for X months, they would be emotionally disturbed if they saw him. And, hey ho, hey ho, the court always acts in the interests of the children
One has to nip this kind of thing in the bud FAST, before it is too late.
My ex threatened that unless I did as she said, she would move away and I would not see the kids again. I doubted she actually would have done so but she was pretty aggressive and I decided that it was not a battle worth fighting, so I would always have a co-pilot (her demand). This was a huge hassle, of course, because co-pilots are hard to find unless one is going somewhere "nice" and (in some cases) I pay for his food and hotel room as well.
However, very recently, she was evidently desperately keen to go out on a date and asked me if I would have my son (12) for the day, but I was going for a flight, and guess what? The date was more important to her, so she let it go. After that, she could not go back so that battle has been won
So, wait for an opportunity
This seems to be a very common problem for divorced men who fly.