We try to design for the lesser skilled pilots, but you need to draw the line somewhere or we'll never be able to certify a piloted aircraft.
I agree that "average", "must not require unusual skill" and the like are very difficult to quantify. But there has to be a starting point somewhere. For my experience, if the pilot assessing the handling characteristic has any doubt about compliance, the first thing he/she does, is discuss it with the team, including the certifying authority. This would be the opposite of murmuring to one's self that it's "probably okay" and saying nothing, knowing that doing so steps the certification process along.
I have certainly trained pilots on airplane types which I knew to be totally benign, and the pilot had a terrible time with it - it was the pilot! And, I own a certified single engine GA type, which is so demanding of skill, it is actually actually requires a type rating in other countries. In my opinion, it
does demand unusual skill, and type specific training, but it was certified "normally".
In any case, like anything in certification, if in doubt, report and discuss (then probably have another pilot assess, with the concern in mind).