.....Can I simply remove the firewall without touching anti-virus ? .....
In a word, as I recall,
No.
My experiences relate to NIS2003 about eighteen months ago, but if you have a later version the principle will be similar I'm sure.
You can't uninstall just the firewall or antivirus individually, although I've got a feeling (maybe wrong) that if you uninstall the whole thing you do have the option to install seperately. However, there would seem to be a simpler solution in your case; Disable it.....
Bring up main NIS Window > 'Options' > 'Internet Security' > 'General' (tab) > under 'Start Norton Internet Security' select 'Manual' > OK. (remember: this is the interface on NIS2003; maybe cosmetically different 2004/5).
Effects:
Will not run automatically on StartUp.
Tray icon will indicate 'Disabled' (a cross over the green globe on '03).
Can be 'started'/'stopped' manually from tray icon, or just left 'off'.
NB: To revert to 'Automatic' at any time, reverse process at the top.