Because of software and Internet bandwidth limitations, these devices only monitor constantly for the wake up command. While it is technically possible to monitor all conversations from an individual Google Home device, the same can be said of mobile phones, tablets and computers, together with virtually any internet connected device.
Your personal security is no more compromised than if you had servants or a housekeeper. Conversations are constantly overheard in restaurants, on public transport, or elsewhere. As these devices are progressively adopted, it will be the people who don't have one that draw attention to themselves.
They certainly have a place assisting the elderly or infirm allowing devices such as the lights to be controlled without needing to get up from a chair or bed. Also useful as you enter a dark house with your arms full of shopping. They can also allow help to be summoned in event of a fall or perhaps a long period of inactivity in the house.
All technology can be subverted to bad purposes, but I rather suspect that in a few years they will just be accepted as part of the background technology that already surrounds us.