I use flaps when runway exit/length warrants its use,to reduce AB use.
I dont use flaps 40 when airport has close in habitation(noise sensitive),possible bird strikes(engine failure) and gusts...
Yes i feel the ailerons to be much more responsive with flaps 40,a gust correction(wing low)can be corrected with quick input but with flaps 40,a slower gentle input is required to avoid some overcontrolling,the controls do feel mushier with flaps 40.