There is a degree of ambiguity; 14 CFR 1.1 doesn't define 'federal airway'. FAA order 7400.9Z does define victor routes etc to be federal airways extending to the base of Class E unless otherwise defined.
So my inference is you effectively have to project 4NM from centre line down to ground level (or 1200' anyway which amounts to the same).
I live near quite a criss-cross of victor routes so stick to an area which meets all those restrictions