If you're accessing it from the Internet, is that via a VPN?
If so, your "accessing" PC will have an IP address on your network, and will be seen as "internal" to the network. Or that's how mine works, anyway.
It's therefore as secure as your VPN connection, which is pretty good if you set it up correctly.
Just make sure your router's security password is a good one (not the "default" from the manufacturer).
My router won't allow an external VPN connection to access the router control panel anyway. It's easy to get round that - I just VNC to the main desktop, then access the router from there. Two IDs and passwords needed.
All that stuff is in the logs anyway, so if anyone else started peeking inside, I could see who (and the IP address they came from).