While it's difficult to defend such poor service from ATC, in fairness to them, even though the freq you are working may not be too busy, he/she may also be covering other much busier sectors/freqs. That being said, I've never heard a controller before try to throw someone off the freq that just wanted to listen. A first.