Simulator Evaluation

[Jetstream Rider]

I realise the answer to this request could occupy many pages and prop up many bars, but here goes.

My former University has asked me to pop along to help evaluate their newly acquired simulator as they know I went into commercial flying after my Aerospace MEng. I am keen to help and think I could make a passable effort at evaluating the sim for the purposes they would like to use it for, however, I would appreciate some advice. Any top tips you have would be very much appreciated and if anyone wants to come along to East London sometime towards the end of next month, I am sure they would be most welcome!

Useful snippets:

Full Moving Base: Base is a Stewart Platform driven by Electric motors

Compliant with JAA STD-1A LEVEL D.

It is conventionally instrumented and does not have an EFIS.

i) Basic Idea is to use it for teaching: Labs on Stability and Control, Demonstrate various aircraft response Modes

ii) To employ it for advanced projects for the M. Eng. course.

iii) Possibly offer a versatile facility to the outside world in the future. To do this we would try and develop the technology driving the simulator using a real flight model in one of our wind tunnels. We then expect the simulator to respond to a number of relatively slow aerodynamic phenomenon experienced by the model flying in the tunnel.

My experience: JAA integrated course after UAS and flying schol, Jetstream 41, 757 and 767 time. Interest in FT and flying qualities (not one of the blokes who plugs in the autopilot and gets the coffee) and would do a lot more little aeroplane flying if it were not for the mortgage....

Regards,
JR

[Mad (Flt) Scientist]

First thing to check is the fidelity of the flight model for the purposes described. If it's the original level D model then it's *supposed* to be able to represent all the proper effects, but that doesn't mean it does. Sim qual test guide points are quite often all at mid c.g. If they're going to use it to teach, one question I'd have would be whether it responds properly to edge-of-envelope conditions - where the aerodynamics might be more interesting, but where the sim is likely not validated (doesn't mean invalid, but it might never have been proven there).

I presume the sim model is one of the types you've flown, otherwise evaluation will be a bitch. But the next step is to go looking for known aircraft vices from real life and see what it does. If you don't have any time on the relevant type, skip that.

Next, try out the manoeuvres they'll want to teach. Get them to run the QTG points for Dutch Roll etc., as appropriate for the planned course material. Then do some more; either to confirm that the sim is like the a/c OR to be able to say "OK, the best place to show spiral instability is ...." which saves people doing it by trial and error in the courses.

AC120-41 has the list of QTG points, and the sim qual documentation should have all the proofs of match.

Just a few ideas. Oh, and (iii) sounds very ambitious. :)

[Genghis the Engineer]

The big issues here are whether it'll do the job for which it was designed - which clearly means understanding the task.

Normally with a sim, you are looking for realism - are the view, visual cues, control mechanical characteristics and handling qualities as near as possible to those of the aircraft being simulated. In this case, that doesn't seem to be what you're looking for - and this does you a favour since true realism is one of the hardest things to achieve in a sim.

Going beyond that, you've identified three primary requirements:-

Basic Idea is to use it for teaching: Labs on Stability and Control, Demonstrate various aircraft response Modes

My approach to evaluation here would be to ask for several models to be set up, which should have particular characteristics set up in terms of basic static and dynamic stability modes. Then to fly the standard academic tests for (say) apparent LSS, manoeuvre stab, (STD1/D should give this level of control force feedback), DR, SPO, Phugoid, Spiral, static lat-dir (steady heading sideslip) and hopefully show similar (±10% ?) characteristics to those expected by analysis.

To employ it for advanced projects for the M. Eng. course

Presumably this means aircraft design projects? So, it's really an evaluation of the ability to meaningfully input aircraft design parameters and reproduce them. This is starting to get closer to a true sim assessment and I'd probably tackle this by asking for characteristics of a known type that you've got handling qualities data for to be input, then doing a cross-check evaluation.


Possibly offer a versatile facility to the outside world in the future. To do this we would try and develop the technology driving the simulator using a real flight model in one of our wind tunnels. We then expect the simulator to respond to a number of relatively slow aerodynamic phenomenon experienced by the model flying in the tunnel.

Whether it's any good for this, should come from the first two assessments - anything else is really down to whether their wind tunnel testing and results analysis is up to snuff, and little to do with the sim's quality.

I could probably spare a free day if you want a hand, and could bring the usual handheld instrumentation for flight testing - give me a shout if i could be useful.


One final thought, the odds are that this will not be an accurately representative sim. What it may well be is a very very useful teaching tool, and what the university really needs (whether they realise it or not!) is a good clear report on how well it does that job, and what the points for attention within that are.

G

[John Farley]

Concentrate of item (i) as unless it scores well for that it will be rubbish for (ii) or (iii) - which topics are well beyond the scope of a few comments here anyhow.

Leave the motion off (more later) and make sure the basics of the control hardware are correct. Does it respond in the correct sense to any input? Likewise any trimmers. Is the breakout sensibly small? (fingertip stuff) Are the control force build-ups reasonably linear with deflection? Friction is OK sticktion (lumpy friction ) is not. Detectable lost motion (or backlash around centre) would make it useless for purpose. In all of this the wheel/stick is clearly more important than the rudder.

Then as Madman says concentrate on the flight model.

To see if the flight model is even sensible go to a few thousand feet and trim it out straight and level in clean config. If you can’t do that there is your first problem as most aircraft YOU have flown would let you do that eh? If you can trim it hands off then put in say half right rudder and make sure it rolls right. I could go on and on but I am sure you get the point YOU have to decide whether it flies like any aeroplane you have flown. Roll in and out of turns and see if the adverse or proverse yaw indications are sensible (negligible for most decent models unless it is a high aspect ratio glider or some such) Does it behave sensibly if you slow down in level flight until it stalls?

If all this seems halfway decent then check out the dutch roll mode, phugoid and so on.

Now motion. I hate the stuff because it takes a real box of money to get even a half way correct seat of the pants sensation during quite mild manoeuvres. My view is that an INCORRECT cue in any sim is very bad news whereas a MISSING cue can be compensated for. But as most people don’t understand this they think a sim without motion has to be worse than one with motion so you will have to see what you think of it. My best advice would be to find some motion off case - phugoid, rolling in and out of a turn, or whatever – that you think is good and then try it with the motion on and make sure that the motion does not obviously spoil things. Sorry, rant over.

The bottom line is YOU deciding whether as a pilot this machine seems a mix of all the stuff you have seen in real aeroplanes (OK) or whether it does some stuff you have never experienced (not OK).

If you decide it has problems deciding the cause of said problem can be a real pain and given that the sim is not going to be used for type conversion you may just have to frig data on a suck it and see basis until it seems like an aeroplane. NB I do not believe this sort of approach is acceptable for a pilot training sim despite the fact that the practice is widespread.

Mark Genghis's last para well. I agree it completely.

Good luck anyhow!

JF

[Milt]

J R

Determining the validity/true to form/likeness/similarity of a modern simulator is no simple task as previous posts are highlighting.

One serious example makes the point.

Had the responsibility once for accepting an F-111C simulator from Singer Link in the US.
Called upon the experience of a TP currently flying with the F-111 Joint Test Force at Edwards AFB.

First take off the thing went longitudinally unstable and it crashed. Reset and try again - same result.

Singer Link non pilot engineers climbed in and much to the embarrassment of the TP flew it with reckless abandon.

TP tried again and crashed but not before he began to sort the problem.

A week of head scratching and analysis revealed that an algorithm to do with long stab had been programmed in to the computer with the wrong sign. Should have been positive but installed negative.

Now the TP could fly it but the engineers were then crashing.

[Jetstream Rider]

Thanks all, very useful. I am not holding out much hope that the sim will be an all singing all dancing machine, but Genghis's last paragraph sums it up well. I think the sim will be a very good tool, providing it is used properly and if I can provide a bit of guidance then all well and good. I do think that teaching things the wrong way, or implying things work in a way in which they don't is very dangerous so will try to highlight those areas. Certainly some of the sims I have trained on and played with are nothing like real aeroplanes (can't be trimmed etc) but they do have their uses.

Thanks for your thoughts on motion John, I didn't notice it was missing in a 777 sim the other day, my brain filled it all in for me.

Milt - I hope I can fly this one! Will be embarassed if I can't!

Genghis - I am sure the university would appreciate your time, if you wouldn't mind giving up a day then I can try to arrange a date. You might bump into an old nemesis from Southampton though! When I get a few more details I can let you know.

[Shawn Coyle]

I would suggest a short course in flight testing to understand some of the flight test techniques (if you don't already). My experience has been that unless you understand the techniques, you'll have a hard time making sense of the QTG or advisory circular.
I've had the experience on a very small, no motion FTD of showing how it was as close to the real aircraft as needed, only to have someone from an authority who, to put it politely, didn't have a clue about what he was doing, come in and not fly according to the test procedure and then say it wasn't right - with no reasons given.
Good luck! If done properly, this could have a lot of use.

[Genghis the Engineer]

Shawn, you should know by now, that's what you take an Engineer for:cool:

JR - glad to help, just keep me posted. I'm guessing it's QMW then, and a certain former airship designer with whom I used to have some long and heated discussions about the validity of Breguet range formulae. I look forward to a good argument :p

I can probably help out with some stability and control flight test data on various small aeroplanes if that's of any use by the way.

G

[Tarnished]

JR

One of the most important (ie potentially intrusive) characteristics of simulation devices is the time delay between the input and the response of the visual system or motion system. As John Farley points out, you can live without motion and still conduct a great number of effective exercises in a simulator. I know you can even live without visuals if all you are looking at is procedural aspects, emergency drills, intercepts etc.

But if you are going to use the device to evaluate or at least demonstrate the evaluation of handling qualities you need to know what extra slug the device itself is putting into the outcome. Modern computers have enabled current development simulators to reduce overall delays down to very small numbers indeed, but if your device is a secondhand I suspect there will be some more significant delays. Even the smallest of lags can greatly affect the outcome of certain high gain, closed loop tasks -AAR, close formation, tracking etc.

I would suspect "line simulators" don't come equipped with any means to measure or gauge internal delays as comes standard in most industry tools these days.

Regards

T

[Jetstream Rider]

Genghis, spot on both counts. The people who have asked me along do not include the chap in question by the way. I'll email you if that is OK.

My main task is to find out exactly what they want me to do, details are a bit sketchy right now as one of the guys is on holiday and I will not get more info until he is back.

As soon as I find out a bit more I will post the information in case anyone has anything else to add. I don't think this is going to be a big simulator evaluation to make the sim a direct model of something (at least not yet). At the moment I see it as a generic sim to help teach and give some students a bit of 'hands on'. Bearing in mind some of the more advanced things mentioned here will help though and are useful in discussions with the department.

Shawn - I had an intro to Flight Testing at Cranfield in their Jetstream, 3 x 20 min flights where we did a bit of phugoid, spo, dutch roll, stall etc and then analysed afterwards. Loved it, but do you have any suggestions for any other similar courses (that are not too expensive...)? Always wanted to be a test pilot or FTE, but when it came to the choice between DERA as FTE, RAF as pilot, or BA, I had to follow my instinct and go flying (decided against RAF). Don't regret my descision, but would love to get more involved with FT and any short courses you know of, I would be pleased to hear about.

Thanks,
JR

[Genghis the Engineer]

I shouldn't worry, we're probably both disfunctional and hard to get on with personalities, but there's no bad blood there.

I'd suggest that the University may not know what they want in terms of assessment and that may be up to you to determine - this is quite common amongst designers (of anything). They know it needs assessment, but have no real idea about what that assessment comprises. The bottom line is that if the assessment is to be meaningful, the TP/FTE has got to develop a good clear understanding of the system and what it's for - and then construct the assessment accordingly. Otherwise it's far too easy to slip into "having a bit of a play" - something that I suspect all FT professionals have seen at some time by untrained persons (often otherwise very competent pilots or engineers) trying to conduct an assessment without real understanding of what they're trying to do.

Incidentally, like Shawn (although in a very different environment) I teach flight testing - in my case for light aircraft. Maybe if we get a chance to have a chat and look at the sim, we can have a talk about it.

G

[farqueue]

Just a quick detail.

I think the earlier reference should be to AC 120-40B, not 41.

[Jetstream Rider]

Things have gone a bit quiet from QM, so no further details just yet. Genghis, did you get my email?

Regards,
JR

[Shawn Coyle]

The late Ralph Smith (of the Smith-Geddes handling qualities criteria) was quite concerned about the various delays that would crop up in simulations.
He was concerned enough by this that he didn't want to use anything but a CRT for visuals (everything else he knew of would introduce a delay that would eventually affect things), and he also wanted a hydrualic force feedback system.
Not wishing to speak ill of the deceased, but I'm not sure if those concerns are still valid.
His software, by the way, was quite good at spotting problems, and I believe is still available at high plains engineering - don't know the web site, sorry.

[Genghis the Engineer]

No Email received, possibly because I've had to change ISP but can't (for reasons currently unclear to me) get the Pprune system to change my forwarding Email.

If you re-send to [email protected] I'll get it.

G

[Jetstream Rider]

Genghis - thanks for the new address, have sent a message.

Shawn - thanks as well, will look up his name and his software and see what I come up with.

Surely if the aeroplane uses other than CRT displays, there must be a delay there? (unless we have got round that with some electronic trickery). If the display is only slightly more delayed in the sim, will it be noticed (even subconsciously?).