Tom Furness is a renowned virtual reality expert – a technology component relevant to the “supercockpit” concept. A 50th anniversary celebration of his work took place in November 2015. More than familiar with data communication rates that might be required of virtual reality demonstrators, Furness once quipped that (virtual) sex itself would require a rate of one billion baud, or one billion bits per second. This was a great deal more than we had grown accustomed to for our 24 baud fax machines and modems way back when. Now that I have your attention…
Furness is acknowledged as one of the most prominent figures to explore how information was to be communicated to the pilot, in future aircraft. The ubiquity of analogue buttons and dials in cockpits up to and including those of the early 1980s, it was agreed, could not be sustained. This article explores some of the technological ideas resulting from early research and development into future cockpits, most notably head-down and helmet-mounted displays. A focus on supercockpit technologies can help us appreciate some of the differences exhibited by the three aircraft Canada is considering; namely, the Lockheed Martin F-35A Lightning II; the Boeing F/A 18E/F Super Hornet; and the SAAB Gripen JAS 39E/F. Our sincere gratitude is extended to all three aerospace companies for their generous help with this complex subject matter.
First Generation
In first generation cockpits, information was presented to the pilot on a small number of dials. This information was mainly engine or aircraft related. The pilot’s situational awareness (SA) was gained almost entirely from outside of the cockpit and was therefore restricted to an area within visual range.
The evolution of the aircraft brought improvements in instrumentation, sensors, and general performance. These improvements, in turn, led to an increase in the number of switches, dials, and displays that presented information to the pilot. By 1985, there were more than 300 in some fighters. In the early 1980s, for example, the CF-18 Hornet cathode ray tubes displayed 675 acronyms, 177 symbols, 73 threat and danger warnings, 40 malfunction display formats, and 22 head-up display configurations. In addition, there were 200 film strips and maps on the horizontal situation indicator, 59 indicator lights, 6 auditory warning tones, 19 pushbuttons on the up-front panel, 9 switches, mostly multi-function, on the throttle, and 7 switches on the stick. The aim of aircraft and helicopter designers was to design future cockpits that would be more manageable. The goal was to improve the pilot’s situational awareness (SA) by reducing their workload.
Later Generations
The most modern fighter technology, namely 5th generation, addresses: stealth (you must appear smaller or further away than everyone else); aerodynamics (you should be able to out-manoeuvre certain of your adversaries if ever they get too close); and, most relevant to our present discussion on SA, knowledge (enabled by integrated avionics, in that your ability to receive, process and act on your information must give you an advantage over everyone else).
There are two components to SA: global and tactical. Global SA generally covers the non-visual, spherical world out to approximately 150 to 200 nautical miles, or about 300 kilometres, corresponding to the distance to the horizon, at an altitude of 30,000 feet (9,230 metres). Tactical SA covers close-in, visual, air-to-air and air-to-ground combat and visual navigation. Situational awareness is gained from information presented to the pilot on cockpit displays, or from the outside world in visual range.
Third and 4th generation cockpits tended to inundate the pilot with tremendous amounts of data from many sources. In a human factors survey of CF-18 pilots conducted in 1990 by the Defence and Civil Institute of Environmental Medicine in Downsview, 59 per cent of the pilots interviewed admitted that task saturation was a problem, and 37 per cent reported losing SA at some point in the mission.
In an interview with recently retired Lockheed Martin Senior F-35 Test Pilot Lieutenant-Colonel Billie Flynn, RCAF (retired), Billie recalled his early experiences as the very first pipe-liner on Canada’s newly acquired CF-18, circa 1984. During his training, on the same day, only minutes apart, two of his fellow students became transfixed by their (multi-function) displays while attempting to decipher what their radars were telling them. When both came out of cloud, each in a very steep dive, one pulled over 11g, nearly losing consciousness while the aircraft bottomed out of the dive at only 192 feet above ground. He lived to fly another day. The other student pilot was not so fortunate, having lost his SA and his life after flying straight into the ground at over 700 Knots (1,296 kilometres per hour).
The multi-function display (MFD) had the effect of increasing workload. These displays conveyed the majority of aircraft sensor information and required a great deal of training and cognitive skill to operate effectively. The pilot spent much of his or her time managing the MFDs. S/he changed display selections frequently to gather as much information as possible and was constantly gathering and evaluating data from the selected sensors, at the ranges presently being displayed, while at the same time, wondering about data that might be available on a different range scale, or from another sensor all-together.
The Supercockpit
In 1985, studies were initiated to identify future cockpit technologies that would address these and other challenges. One of these technologies was dubbed “The Supercockpit.” The concept was initially conceived as a cockpit that was to be implemented via virtual interfaces between the pilot and the aircraft. Light, sound, and tactile patterns were to be transmitted to the senses such that they appear to originate in three-dimensional space surrounding the pilot. The intent was to create a sphere of information through which the aircraft systems and sensors provided the pilot with the required SA. A virtual cockpit was implemented by organizing and fusing data from aircraft avionics, weapons, and sensors and portraying this information within a panoramic visual and auditory display for rapid assimilation by the pilot. The pilot could then direct weapons and command aircraft systems using visual, voice, and other psychomotor responses.
These ideas necessitated the development of a cockpit that would display information in a three-dimensional format, owing to the natural way humans perceive their environment. This three-dimensional presentation would take the form of a virtual world of animation depicting the environment surrounding the cockpit. The use of icons to depict objects such as enemy aircraft and surface-to-air missile sites was expected to allow the information to be more rapidly understood and assimilated than common alphanumeric terms such as “MIG-29” or “SA-14.” To effectively interface with this three-dimensional presentation, the supercockpit had to be able to determine where the pilot was looking and where his head was pointing. To accomplish this, a special helmet-mounted display system employing very accurate head and eye position sensing would be required. This helmet-mounted system was to eventually permit the activation of aircraft systems simply by looking at the appropriate controls and switches.
Senor Fusion
Sensor fusion, when properly implemented, was a major aid to the pilot’s decision-making process and the maintenance of SA. Sensor fusion integrates the information from radar, EW, JTIDS, and other sources, and displays it over a digital map on a screen increasingly larger than MFDs typical of the mid to late 1980s. This would allow the pilot to look at a single source to get the big picture. The critical concern of pilots, though, was that sensor fusion needed to be implemented in a manner such that all inputs incorporate historical and real-time information in an easily digestible, uncluttered, and utilitarian manner. The sheer number of symbols, alone, necessitated very large screens in order that sufficient separation was maintained between symbols. As well, the use of coloured symbology would allow more rapid display interpretation when compared with traditional monochrome displays. The solution was envisaged as a large-area, high-resolution, three-dimensional, colour display.
Cockpit Displays
Cockpit displays were initially of the electro-mechanical vernier type. Increasingly sophisticated sensors and systems required improved displays. With the advent of search-and-track radars, radar warning receivers, and forward-looking infra-red (FLIR) systems, each with their own dedicated CRT display, the cockpit panel became very cluttered. The multi-function CRT provided display improvements over the original dedicated sensor display. The most popular CRT used for MFDs was the shadow-mask CRT. The shadow mask CRT used three separate electron guns, a single vacuum tube and a phosphor coating consisting of a series of arrangements of phosphor colour dots (usually red, green and blue).
As the size of the display was increased, the angles at which the beams transit the shadow mask became greater and more electrons ended up being absorbed than was desirable. This was especially critical in a high ambient light environment since the shadow mask absorption reduced the brightness of the picture, impacting directly on the contrast of the display and overall readability. Therefore, flat panel displays grew ever more popular. These display technologies included: thin film electro-luminescent, plasma display panels, light emitting diodes, and liquid crystal displays. The target technology for supercockpits was the active matrix liquid crystal display (AMLCD).
Early development work on the Hornet 2000 included consideration to equip the aircraft with a 10×10 inch, centrally mounted, tactical situation CRT display, two multi-function, colour CRT displays measuring 6×6 inches, and a fuel and engine situation display (either an LED or plasma display). The tactical situation displays also incorporated a full digital moving map display. The YF-22 Advanced Tactical Fighter and the European Fighter Aircraft were expected to incorporate similar improvements in head-down display technologies.
The heads-up display (HUD) provided vital flight and target tracking information to the pilot while allowing him to retain tactical SA by helping to keep his head out of the cockpit. The major limitation of the HUD was its limited field of view. The helmet mounted display was a logical follow on to the HUD because it could duplicate most, if not all the HUD’s functions but over a much larger field of regard.
Helmet-mounted displays
The use of a helmet-mounted display helps the pilot to designate a target outside the weapons field-of-view and pull lead on to the target or programme the missile to turn onto the target after launch. Furthermore, by interfacing the HMD with radar and (Joint Tactical Information Distribution System) JTIDS, steering information was possible via the use of cuing arrows. These enhancements were expected to allow faster acquisition at greater ranges and faster weapons lock-on. Perhaps the best helmet-mounted-display application for improving situational awareness involved using the device to slave sensors, such as the radar, IR systems or a missile seeker, onto targets that have been acquired. By the early 1990s, ten companies had developed prototypes of functioning helmet-mounted displays.
Helmet-mounted-display optics are very similar to those of HUDs and perform two functions. First, they provide the pilot with an unobstructed view of the outside world and the cockpit. Second, they superimpose typical HUD symbology over the pilot’s view. To achieve the dual display the optics are designed so that they reflect or refract a very narrow waveband of light generated by the helmets’ CRT, while allowing all other wavelengths to pass through. The display optics technologies available are reflective and refractive optical sub-systems coupled with diffractive combiners.
To achieve reduced workload brought about by faster visual acquisition and weapon lock-on, determination of an accurate head position remained paramount. Virtual cockpit criterion for head position sensing systems was 1 milliradian. One degree is equal to 17.54 milliradians therefore a required accuracy of one milliradian is equal to .057 degrees. Manufacturers therefore investigated four head position sensing systems: electro-magnetic, electro-optical, infra-red and acoustic.
The electro-magnetic head position sensing system involved a small transmitter in the top of the canopy which generated a three-axis magnetic field. The relative strengths of the magnetic field were picked up by sensors on the helmet and processed by the helmet tracker. The accuracy of this system was between 4 and 17 milliradians. This proved accurate enough to allow the HMD to be used as a sighting system for missiles but not for guns or unguided munitions.
The electro-optic head position sensing system used a charge coupled device, solid-state camera, to monitor the position of the helmet. Reflectors built into the helmet reflected emissions from light emitting diodes situated around the cockpit. These reflections were detected by the camera and determined the helmet position 50 times per second. Accuracies of 2 milliradians were obtained making this system suitable for air-to-air gunnery.
The infra-red head position sensing system consisted of detectors on both sides of the helmet and two transmitting units fixed to the cockpit. The cockpit units generated a pair of fan-shaped, infra-red beams that swept the helmet mounted detectors at a constant angular velocity. The infra-red beams generated pulses from the helmet detectors. Helmet position and angle was determined by integrating the elapsed time between the initial reference pulse and the known angular velocity of the scanning beams. The infra-red system offered a greater accuracy than electromagnetic systems.
The acoustic head position sensing system was a new technology which operated by timing the arrival of acoustic pulses emitted from a series of transmitters and received by a series of ultrasonic microphones mounted on the cockpit structure. The manufacturers of this system claimed a head position sensing accuracy of 3.43 milliradians pointing and .1-inch positioning accuracy.
The GEC Falcon Eye underwent trials with the USAF as an improvement on the F-16 Falcon. It was linked to a FLIR system and the primary purpose of the display was to provide a FLIR picture to the pilot regardless of where s/he was looking. The GEC Falcon Eye used an electro-magnetic head tracking system and binocular flat eye pieces, like a HUD prism, which could be folded out of the way when not in use. The Kaiser Agile Eye was designed for high performance tactical aircraft in the air-to-air and air-to-ground roles. It had a 20-degree monocular field-of-view and used a Hughes one-half inch CRT. Billie Flynn explained, “Falcon eye and agile eye were predecessors of the Joint Helmet Mounted Cueing System or JHMCS. Billie Flynn recalls, “they were balanced, but bulky, expensive, and had a limited display capability. The JHMCS did a good job at getting air forces to buy into the idea of a production HMD but fast jets [lagged] behind the attack helicopter community when it came to adopting HMDs.
Boeing’s ACS
A visit to boeing.com shows us that, “[in] August, 2003, the U.S. Department of Defense (DOD) awarded Boeing a $24 million contract to produce 100 JHMCS destined for the U.S. Air Force F-15 and F-16 aircraft and the Navy’s F/A-18E/F aircraft. The system’s first operational use was during Operation Iraqi Freedom. In 2012, Boeing validated the integration of the next-generation Joint Helmet Mounted Cueing System II/h (JHMCS II/h) on the company’s F-15 Silent Eagle demonstrator aircraft.”
Boeing has high praise for a contemporary of Thomas Furness’ named Gene Adam. Gene referred to the supercockpit as the “Big Picture.” His work inspired Boeing’s “Advanced Cockpit System (ACS)” which features the “Large Area Display” (LAD) and “Low Profile Hud” (LPHUD). During a 2017 tour of Boeing’s production line for the F/A 18E/F, Boeing representatives conveyed their APG-79 active electronically scanned array radar is key to the Super Hornet’s “continuously improving overall mission capability and supportability.” The radar is integrated with an “Advanced Targeting Forward Looking Infrared system, JHMCS, Multifunctional Information Distribution System, advanced high-capacity computer system, and a state-of-the-art cockpit.” Their relatively new infra-red search-and-track system (IRST) proved successful in flight in February 2014 and won the approval of the US Navy the following year.
Boeing’s Advanced Cockpit System (ACS) is currently represented by a 10×19 inch touchscreen that operates quite similar to the most current mobile (cell) phones and tablets. Website video dated early 2018 shares some important and anticipated capabilities. For example, the software that drives the LAD will be easily upgradeable such that current 2-dimensional depiction of elements by symbols and such, will become 3-dimensional as upgrades progress. Clearly, Boeing is a part of this wide-ranging effort to come ever closer to the 3-dimensional “Supercockpit” world-view men like Adams and Furness conceived of so long ago.
Saab’s Targo
At CANSEC 2018, Saab Canada Inc. shared video of their wide-area display featured in the Gripen “Echo.” This is the version acquired by Brazil, and integrates the 19×8 inch high-resolution WAD with an HMD and a traditional HUD. The HMD-HUD-WAD integrated system is manufactured by the Brazilian company AEL Sistemas. If anything, one cannot help but notice by the example of Saab’s JAS 39 Gripen E how incredibly clean, if not sparse the 21st century fighter cockpit has become, in comparison to some 3rd and 4th generation fighters through the 1980s and 1990s.
AEL Sistemas has helped respond to the Swedish Defence Material Administration’s (FMV) 2016 order for an advanced HMD system valued at SEK 119 million. The system, called Targo, was anticipated to begin deliveries between 2022 and 2026. Promotional information from Saab explains, “Pilots equipped with Targo will be able to better locate, track, identify and engage air and ground targets, as the system provides situational awareness and easier engagement options. A helmet mounted visor displays flight information such as altitude and air speed, as well as cueing targets and supplying supporting tracking data to the pilot.”
Aspects of the integration of the WAD and the HUD have been reported on by Jamie Hunter, of “The Drive.” For example, if the pilot for whatever reason chooses not to make changes to the display by employing the touch-screen methods, s/he can revert to controlling the display via the Hands-on-Throttle-and-Stick or HOTAS methods. Saab’s Gripen E test pilot Robin Nordlander also explains the Gripen designers acknowledged the benefits of including the traditional HUD, in the human-machine-interface (HMI) of the Gripen E as pilot testimonials indicated sometimes they prefer flying using only the HUD, in moments where the HMD isn’t really necessary. Finally, one important feature of the WAD is the pilot’s ability to readily adjust the scale of the subject matter on the display.
Lockheed Martin’s HMD
In an interview with Billie Flynn, this ability to adjust the scale proves to be important, perhaps for the very reasons Furness, Adams and others conceived of the “supercockpit” in the first place. The anecdote shared by Lockheed Martin’s F-35 test pilot also says a lot about what lies at the heart of our discussion – our collective accomplishments that go into the design of these advanced fighter aircraft. I asked Billie if he believes his experiences flying the CF-18A-D Hornets, all models of the F-16 Fighting Falcon, the Eurofighter Typhoon and all three variants of the F-35 Lightning II show us we have achieved the supercockpit envisaged by men like Furness and Adam long ago.
He replied with an anecdote.
“I was invited to an air show in Finland, not long ago, and a reporter’s questions seemed to encourage me to reflect on my early Hornet experiences, which I had first flown 36 years earlier. A much younger USAF F-35 pilot from Hill Air Force Base was next in line. In response to a similar question he said, ‘I don’t know anything about that guy there,’ pointing to me, ‘but I’ve flown the F-35 for the past two years on squadron. I’m 27 years old.’ Thereafter, he pretty much discounted my entire experiences, labelling me a 4th gen baby. The point is this younger generation has a real appetite for the enormous bandwidth of information we can feed them. They are gamers – they love this stuff. When we ask, what we can do to satiate their appetites, and really push the envelope of their cognitive capabilities, the answer comes in the form of the helmet F-35 pilot wears. It also comes from the displays offered to them in a cockpit completely cleaned out of all the needles, dials and other displays that today would be considered a real waste of time, and in some respects serious distractions.
“The F-35 HMD works every single time; it allows them to see everything they need to see, in and out of the cockpit. And the screen – that one big touchscreen – can easily share tons of data. You can put hundreds of different symbols for air-to-air, air-to-ground, and electronic order of battle tasks on the screen. Early in the design and testing process, we planned for certain numbers of symbols for each task in the air. However, after fielding the F-35, we discovered that the users wanted to double up the number of symbols that they could have displayed because these young kids had the appetite and the capacity to take in and see way more than I would have imagined. Can we do better, with this “supercockpit?” Possibly. Why do we need to populate heads-down screens with ‘domestic’ displays such as engine instruments, fuel indicators, weapons loads, when that could all be automated and monitored by the airplane itself and everything else could be projected on my HMD? We can always work on ramping up the computer power – make it even faster than it is. We can then improve incrementally the sensors and their spectrums. We can also work on fusion, making it better and better each time.
“Do we have a super cockpit? Boy, I believe that we are there. I have to believe when you watch what men and women do in the F-35 now, that we really understand this next generation, in terms of what we intended for the pilot to do.”