If you are using an area forecast as there is no TAF available then the criteria required at the destination are VMC. Would it not then be possible to overfly and look at the windsock and isn't that what "they" expect you to do?
The biggest problem with using an area forecast is deciding if your destination is in a particular area used in the forecast eg. on the western slopes or on the ranges south of xxx. Can become sticky.
It's ok to say just plan for an ALT if in doubt but sometimes fuel and weight become a big issue and out west it's a long way to another destination that does not require an ALT. I know of quite a few aerodromes without TAF service but they have an aid or an RNAV approach.