On the DG834G under WEP Security Encryption select Authentication Type "Automatic" and Enryption Strength "64 bit". Enter a passphrase if you've not done so already. The selected key (Key 1 by default) is what you'll need on your Mac. In theory the passphrase should work, but I've never go it to. On your Mac choose WEP 40/128-bit hex under "Wireless Security", select the network name and enter the key mentioned above.
If you store the wrong passphrase on your keychain (where it will be stored by default), choose "Other..." from the wireless menu and enter the detail manually.
I along with various people I know with DG834G's have had to upgrade the firmware to maintain a wireless connection longer than a few hours (to both Mac and Windows machines). Mine is V1.05.00 and that seems okay.