Yep, Brian and Walt were the back-up for the first Libyan job but were not needed. The second mission had Brian and Walt as primary but they had a sensor failure near the target area. The back-up never received the call, so that called for a third mission for which Brian and Walt were back-up, and again not needed. So they probably made it to Libya.
Just to add that it actually says in Sled Driver that Brian and Walt flew in on the night of the raid and passed the F-111s (rocking their wings) as they flew out. Both the Mildenhall jets flew that night so I presume they were primary crew for one of these missions.
Anyway, it's a wonderful book and one that conveys brilliantly what it was like to be involved with the jet. I don't blame him for being dreamy at all!
Gareth