Automation Airmanship

On the night of July 12, 2006, the crew of Tusker 914, in a CH149 Cormorant search and rescue helicopter, departed 14 Wing Greenwood to conduct practice night boat hoists with the Canadian Coast Guard. Tragically, while approaching the hover in the vicinity of Canso, Nova Scotia, the aircraft impacted the water and three crewmembers lost their lives. The causes of this accident, which have been well documented in the Flight Safety Investigation Report, were directly related to the pilots’ use of the aircraft automation and a loss in situational awareness. In short, the causal factors in the loss of Tusker 914 were directly attributed to human factors.^[2] This accident served as a catalyst for a deliberate effort within the Air Force to assess its ability to safely and effectively operate modern, highly automated and integrated aircraft like the Cormorant. This effort has been made even more relevant by the significant investment in modern aircraft for the Air Force by the Government of Canada. Created in conjunction with the development of the 1 Canadian Air Division Automation Philosophy, the Automation Policy and Planning Development (APPD) Project was initiated in 2008. The APPD Project is complete and its findings, conclusions and recommendations have been accepted by the Air Force. They indicate a requirement for significant cultural change within the organization and will require a protracted and deliberate effort to address. At its base level, the APPD Project is primarily concerned with developing a robust human factors program that optimizes aircrew performance to fully exploit the new technologies being delivered to the Air Force. This optimal level of performance can best be described as “automation airmanship,” and its development and sustainment are critical to preventing further accidents like Tusker 914. Achieving optimum levels of performance is entirely dependent upon the ability of the Air Force to focus (lead, coordinate and advocate) efforts aimed at delivering automation airmanship.

“ The quality of the box matters little. Success depends upon the man who sits in it. - Baron Manfred von Richtofen (“The Red Baron”), 1918 ^[1]

The methodology applied to automation efforts within the Air Force has been the “Four P” model, a strategy developed by the National Aeronautics and Space Administration (NASA) Ames Research Center. The four Ps are philosophy, policies, procedures and practices and it is only through the deliberate development of the first three in sequential order that the desired practices are achieved on the aircraft.^[3] With the creation of the 1 Canadian Air Division Automation Philosophy in 2007, the APPD Project was the next logical step. Conducted primarily by contracted personnel, with significant developmental and operational human factors experience in military and civil automated aircraft, it included coordination and liaison visits with senior Air Force leadership; fact-finding visits with staffs, projects and engineering organizations; a review of all flying publications and manuals; and fly-along observations of aircrew in several different fleets across Canada. The project was intended to provide a foundation for subsequent activities in the development of both policies and procedures. It also analysed in detail the gaps that exist between current operations and operations that fully reflect the objectives of the Automation Philosophy. It is not the physical ability or inability of the aviator to fly an instrument approach to minimums and execute a landing or to execute an approach to the hover for a night hoist that causes the vast majority of civil and military accidents. Rather, accidents are usually the result of the mental ability or inability of the crew while flying that approach or any other manoeuvre. In other words, the causes of the vast majority of accidents in aviation are related to human factors, just as they were in Tusker 914. It is the ability of the aviator to use all available resources to their maximum potential, including the aircraft systems and crew (or the wingman), to “trap” errors and to operate within the limits of cognitive effectiveness that allows them to develop a high level of situational awareness and determines whether or not the approach is successful. For this reason, automation efforts are in fact targeted at the entire organization. As the aircraft continues to evolve, and the ways in which it interacts with the crew continues to evolve, so too must the way in which the Air Force approaches the training and qualification of its aircrew to operate that aircraft. Therefore, current automation efforts are concerned with the human-factors aspects of modern aircraft and ensuring the Air Force has properly positioned itself to safely and effectively operate them.

The APPD Project observed the following three distinct areas within aircrew that were best able to achieve the Automation Philosophy: Clear aircrew automation task definition throughout all flight documentation supported by clearly-defined automation performance measures and standards. Flight procedures and documentation that support the development and maintenance of a robust automation culture. Aviator flight discipline is incorporated into procedures and automation policy.^[4] Aircrew on aircraft procured and operated on the basis of civil-compliance certification, such as the Airbus A310, generally performed at a high level due to the reliance on industry operating methodologies and documentation to achieve operational readiness. The analysis also revealed that overall aviator experience is not necessarily an accurate predictor or indicator of the flight crew’s automated flight performance. One of the most effective crews observed consisted of two recent graduates of the new multi-engine syllabus at Southport that fully incorporates industry best practice automated training methodologies and flight procedures. The exposure to appropriate automation training and the measurement of performance with respect to automation that are embedded in the syllabus enabled this relatively inexperienced crew to perform at a high level. There were several cause factors attributed to crews who did not exhibit a high level of automated flight performance. In all cases, these crews were missing at least one or more of the three identified areas. When examining the overall effectiveness of these crews, it became necessary to consider not just their flight performance, but all aspects of the Air Force that directly or indirectly contribute to that performance, or “practice,” in operations. As a result, the scope of the findings within the APPD Report^[5] is significant.

The APPD Project concluded that current orders and manuals lack coherence and do not contain policy identified within industry, both civil and military, to support the operation of modern aircraft. Several changes and additions to 1 Canadian Air Division Orders are required to implement the Automation Philosophy. In concurrence with the development of new policy, the APPD Report recommends that the Air Force considers developing a flight operations manual. This manual would replace the current 1 Canadian Air Division Orders Volume 3 and would become a single reference document to replace the multiple, and often contradictory, references aircrew are currently required to use in the conduct of operations. The APPD Report also concluded that flight procedures and documents currently in use within the Canadian Forces (CF) are not adequate to support the Automation Philosophy. Critical flight documents such as aircraft operating instructions, flight manuals, standard manoeuvre manuals and aircraft operating checklists reflect a wide variety of guidance, in some cases contradicting themselves within the same fleet. They differ in terms of content, language, terminology and organization and do not reflect industry best practices with regard to the operation of modern, highly automated aircraft. Deliberate effort is required to standardize and harmonize flight procedures and documentation within and across all fleets where able, while concurrently incorporating the changes to procedures and documents mandated by the introduction of new technologies. Not only will this significantly increase overall standardization, effectiveness and safety, it will directly contribute to reducing, or “streamlining,” training requirements as individual aircrew transition from ab-initio training into and between the various operational communities. As an example, it was observed that there are multiple methods of calling a rejected take-off within the multi-engine communities that include the use of the terms reject, abort and malfunction. While each term is valid and trained to a high level, the fundamental question becomes, “What term is taught to the pilot trainees in the multi-engine school and why do those individuals then need to learn something new depending on which operational fleet they join?” In addition to the efficiencies gained in the pilot training continuum by using the same term, there is a fundamental human-factors issue to consider that determines whether or not the aircrew can communicate critical safety of flight information. This is entirely dependent on the ability of successive training systems to ensure the method used to perform the task in previous fleets has in fact been replaced with the method of performing the task in the new fleet.^[6]

Fundamental to the shortcomings observed during the APPD Project are the lack of clearly defined automation performance measures and standards, both from an Air Force perspective and within individual fleets. The analysis revealed that they simply do not exist and as a result, aircrew performance in all areas of Air Force training and operations is negatively affected. The Air Force does not define the level of automated performance it expects from its crews and does not, therefore, measure whether or not its crews achieve that level. This also directly impacts the ability of the Air Force to effectively create a common culture and language that permits the organization to efficiently describe and communicate issues associated with automated flight performance in current and future aircraft. Issues that affect all aircraft, regardless of type or level of complexity, “are not able to be fully described through daily operations, simulator and aircraft training, maintenance, staff support requirements, lifecycle [sic] issues, and designing the requirements for future aircraft acquisitions, upgrades and support.”^[7]

The APPD Project has concluded that current CF aircrew training and evaluation methods are not capable of supporting the implementation and sustainment of the Automation Philosophy. They rely on individual proficiency in performing technical skills largely attained through a prescribed “hours” based program. This is manifested in the current practice of delivering most training and evaluation through single-pilot, manoeuvres-based events in all multi-crew fleets. Many of the skills required to safely and effectively fly complex multi-crew aircraft are neither defined in current training guidance nor are they evaluated. It represents an incomplete assessment of the crew’s ability to effectively operate the aircraft in line operations, where they are expected to function as a crew. Basic “hands and feet” skills are fundamental to all successful aviators, but those skills are pursued and evaluated to the detriment of other critical flying skills required in an automated aircraft. The style of flight evaluation currently employed across the Air Force, in which a training pilot/evaluator occupies one of the crew seats, does not promote the requirement for close coordination of tasks between all crew members in all phases of flight in order to safely and effectively fly modern, automated aircraft.^[8]

The reliance on legacy training and evaluation criteria is manifested in the use of simulation. The APPD Report describes the current attitude towards simulation in the Air Force as “Sim-Phobic.”^[9] Emphasis is placed on actual aircraft utilization and current policies rank the aircraft as a more effective training aid than simulators. Modern simulators are designed to replicate the task demands on an aviator in terms of perception, attention, decision making, memory and action while allowing the crew to juggle multiple tasks, supervise automated subsystems, maintain situational awareness and develop an accurate mental model of aircraft dynamics in order to achieve mission success. While fullmotion, high-fidelity training devices have the potential to create an extremely realistic training environment, research shows that most of the procedures and cognitive emphasis required to train safe and effective aircrew can be delivered via less costly means.^[10] In many fleets, the Air Force already possesses the simulators required to begin developing a robust automation culture but is failing to use them. Current practice also fails to recognize that replicating the warnings, cautions and advisories associated with the failure of multiple interconnected subsystems on modern aircraft, and the required crew reactions/ interactions to effectively deal with those situations, can only be done in a simulator. On a CC130H there are approximately 60 aural warnings or annunciations and warning, cautionary and advisory lights that the aircrew are expected to react to during operations. On a C130J, there are in excess of 780.^[11] The ability to successfully resolve an abnormality or malfunction on a modern aircraft requires the crew to function at a very high level and to deal effectively with a cascading and sometimes contradictory information flow, while still adhering to the fundamental principles of “aviate, navigate and communicate.” Teaching and evaluating these skill sets in the actual aircraft, in which none of the annunciations or warnings are present due to the event being “simulated,” with an instructor or evaluator occupying one of the crew seats is not effective. Using simulators to teach and evaluate these new, fundamental skill sets requires fundamentally different methodologies from those currently being used in the majority of Air Force simulators.^[12]

Although the Air Force has invested significant time and effort into the Human Performance in Military Aviation (HPMA) Program, designed to replace the traditional Crew Resource Management Program, the APPD Project found little evidence of its use in operations. There are two fundamental reasons for this. The first is that the Air Force has not created HPMA performance measures and standards. Aircrew are educated in the concepts of HPMA but they are not trained to use them nor evaluated on that use in the aircraft. This is reinforced by the culture of single-pilot, vice crew, training and evaluation. The second reason is that HPMA concepts have not been designed and integrated into Air Force normal and abnormal operating procedures. Aircraft operating procedures which have been developed and designed with careful consideration and integration of HPMA concepts can have powerful positive results for the disciplined aircrew. Even the “most difficult HPMA converts” can perform at very high levels of HPMA by virtue of well designed operating procedures that they are evaluated against. Attitudes and personality differences mean less when operating with procedures created with an embedded HPMA strategy. In fact, HPMA and automation airmanship become the same measures and standards in a well-developed automation culture.^[13] The relative failure of the HPMA Program is revealing and provides a valuable lesson about cultural change. Regardless of good intentions and a great deal of effort by a dedicated, skilled staff, meaningful change in behaviour on the flight deck does not take place until it is first legislated through orders and regulations and then assessed for compliance. Until these two principles are applied, widespread behavioural change will not occur. Failure to recognize the shortcomings in the successful implementation of the HPMA Program must be learned and applied to the development of an automated culture if it is to be successful.^[14]

The analysis conducted by the APPD Project included the flight safety system and an examination of flight safety occurrences since 1998, the year the CC130 Avionics Update Project came on-line. It concluded that due to the absence of both automation and HPMA performance measures and standards, the flight safety system is challenged to identify and report automation and human factors related issues. There is a flight safety investigation report for a CH149 Cormorant incident dated July 2004 contained within the APPD Report. The human factors and automation causes in the 2004 report are virtually identical to the cause factors surrounding the loss of Tusker 914. The July 2004 incident also predates the maintenance issues with the Cormorant helicopter, and associated loss of aircrew proficiency, that were cited as latent causal factors in Tusker 914. In both cases, the automated systems on the aircraft were used inappropriately resulting in a failure, which was further exacerbated by the crew’s reaction to that failure. The only significant difference between the July 2004 incident and Tusker 914 was that in the first incident the aircraft pitched up and away from the water instead of down and towards the water. The failure to create a common language and culture as it pertains to automation is hampering the ability of the Air Force to both identify and learn from automation related incidents. Incidents are instead generally treated as “one-of” events with little or no real corrective measures adopted either within or amongst fleets.

The APPD Project concluded that the Air Force structure has evolved into strong vertical organizations that operate as “stovepipes of excellence” along fleet lines, with little or no transfer of information between the stovepipes. Whether it is the shortcomings of the HPMA Program, the wide variance in standard manoeuvre manuals, or the differences in basic flight deck terminology, the Air Force is not achieving levels of standardization and synchronization that would allow it to effectively implement and sustain the Automation Philosophy. It is expending effort on basic, common problems that are being solved with multiple independent solutions within different fleets requiring multiple support systems to sustain. This is having a significant impact on already overstretched resources, both in terms of personnel and funding. In other cases, such as the CH146 Griffon simulator program or Wolf Net, solutions have been developed that are a model to emulate but which others have not adopted. The APPD Report recommends that an Air Force standards organization be created. This organization would develop and implement the required Air Force policy statements, create the required performance measures and standards that directly contribute to the development and implementation of procedures to support those policy statements, and then spearhead their implementation to achieve the desired practices. This standardization effort, and the development of a robust automation training system with performance measures and standards like those found in the new multi-engine syllabus at Southport, will allow the CF to transition to a “pipeline” Air Force. This standardization effort will also directly facilitate the introduction of upgraded or new aircraft fleets. The Air Force standards organization would be the means by which the Air Force creates a common language and culture of automation and begins to effectively communicate across all functional areas.

“While the most common explanation for an accident is operator error, a more frequent cause is faulty design of the sociotechnical system (that is, people and technology in combination) in which the operator is embedded.” - Marc Gerstein^[15]

The crew of Tusker 914 was armed with everything the Air Force viewed as essential to the safe and effective operation of a CH149 Cormorant in terms of training, qualifications and equipment. There was a qualified pilot instructor onboard and the total flight time of all three pilots was in excess of 10,000 hours. It was an experienced CH149 crew supported by a chain of command that was confident in their ability to execute the assigned mission. Significantly, almost all of the findings of the APPD Project were described in the Tusker 914 Flight Safety Investigation Report. Cormorant pilots were being trained using single-pilot, manoeuvres-based methodologies; HPMA and automation performance measures were non-existent; standardization was an issue; and the changes required in training and operating methodologies as a result of the introduction of new technologies had not been captured. British simulator instructors who observed Cormorant training stated that “in comparison to other EH 101 operators, the CH149 pilots were permitted too much variability in how they performed their procedures, set up their displays and handled malfunctions.”^[16] In general, it was their opinion that better and more detailed descriptions of standard operating procedures in the CH149 Standard Manoeuvre Manual would be beneficial. Finally, they remarked that they commonly saw CH149 pilots using techniques that they felt were a carry-over from the non-automated, manual flying procedures used in the CH113/A Labrador.^[17] The fundamental determinant of success in the application of aerospace power, regardless of the mission, is the way in which the aviator manages competing task demands, ambiguity and operational pressures. The training and tools that the Air Force gives to warfighters to execute their assigned missions are critical in determining whether or not the mission is successful. All aviators, regardless of experience, are prone to error and vulnerable to characteristic forms of error based on the limits of cognitive ability that all share, whether civil or military. More importantly, individual actions and errors are not the source of accidents but more often a result of systemic causes.^[18] For much of the last 40 years, the Air Force has struggled with systemic problems that have impeded change efforts and resulted in ad hoc responses to change requirements. This has included the lack of an effective lessons learned capability. The disjointed and often dysfunctional nature of Air Force command and control has significantly impacted the ability of the Air Force to formulate policy and deal effectively with recent challenges.^[19] The loss of Tusker 914, and the findings within the APPD Report, is a manifestation of these systemic problems starting with a failure to develop policy appropriate to the operation of modern aircraft. The “Swiss Cheese Model” of error, developed by James Reason, has been widely accepted in aviation as a means of explaining human performance within the broader context of the system in which the humans are operating. Simply put, if the slices of cheese represent the layers of defence against error, the holes represent potential shortcomings in that layer. Accidents are prevented when the organization is able to prevent the holes from lining up.^[20] As confirmed in the APPD Report, and depicted in Figure 1, the systemic failings of the Air Force to deliver policy, procedures and training relevant to automated aircraft, coupled with the weaknesses in the HPMA Program, left the crew of Tusker 914 with only basic aircraft handling skills as a defence. When those were used in a manner incompatible with the automation, the “holes” aligned and the accident was the result.

The Air Force must develop and implement a robust culture of automation airmanship to optimize aircrew performance in the 21st century and must use 21st century methods to achieve it. Automation airmanship will demand new skills, knowledge and attitudes to safely and effectively achieve mission success. It must be a standardized, disciplined and integrated operating strategy that uses all available resources on an aircraft, including the crew and aircraft systems. Automation airmanship will integrate traditional technical skills, automation skills and human factors skills to achieve optimum situational awareness and mission effectiveness. Its development will permit the Air Force to be responsive to, and exploit to the maximum extent possible, advances in aircraft technology, operations and training methodologies. Delivering it will require the deliberate, coordinated and systematic development of policies and procedures across the Air Force to support the Automation Philosophy.

The critical first step in developing automation airmanship is to address what Allan English has properly identified as the disjointed and dysfunctional nature of current Air Force command and control relationships. These shortcomings directly contributed to the systemic failings evident in the loss of Tusker 914 and subsequently confirmed in the APPD Report. This mandates the creation of an Air Force standards organization, through a rationalization or realignment of current structures, which is able to lead, advocate and coordinate the various components essential to the development of automation airmanship. Concurrently, the Air Force must create the overall governance, or doctrine, necessary to give the new Air Force standards organization the foundation it requires. The ongoing debate on how best to structure Canadian aerospace forces and exercise command and control must consider the deeper, latent flight safety factors brought about by a failure to effectively develop, implement and coordinate policy on a wide range of issues. The potential stand-up of other command functions must ensure the development of decision rights and information flows that do not currently exist within the strong vertical stovepipes identified within the APPD Report. Future Air Force-wide initiatives will find themselves attaining the same level of success as the HPMA Program while incurring the risk of losing additional, hard to replace aircrew and aircraft, unless these systemic failings are addressed. As the issues of structure are resolved, concurrent activity can begin on the development of policy and procedures in coordination with the various projects, operational communities and individual fleets. There exists an immediate need to address automation airmanship issues in transitional fleets such as the Aurora upgrade project, the Maritime Helicopter Project and the C130J project. The required levels of expertise do not currently exist within the CF and it is expected that continued contracted assistance will be necessary in the short to medium term to begin addressing the issues identified in the APPD Report, to include direct developmental assistance to the initial operations of the Air Force standards organization. Achieving and maintaining the desired levels of automation airmanship across the Air Force is readily within reach provided the requisite doctrine, structure and focus are dedicated to the effort.

There are several examples of coordinated and highly effective training and operating systems within the aviation industry that the Air Force can use as a model. The most promising one is the Advanced Qualification Program (AQP) currently in use, to varying levels, by several civil and military organizations. The primary goal of AQP is to achieve the highest possible standard of individual and crew performance.^[21] In order to achieve this goal, AQP seeks to reduce the probability of crew-related errors by aligning training and evaluation requirements more closely with the known causes of human error. It recognizes that the capabilities and use of simulators and other computer-based training devices in training and qualification activities have changed dramatically and allows operators to develop innovative training and qualification programs that incorporate the most recent advances in training methods and techniques. Achieving these benefits requires the deliberate and coordinated development of policies to support wide-scale implementation. The direct benefits of AQP are as follows:

Crew Performance

Current Air Force training programs focus on individual training and evaluation. Under AQP, the focus is on individual and crew performance in both training and evaluation.

HPMA

Most accidents are caused by errors of judgement, communication and crew coordination. Current training programs focus primarily on flying skills and systems knowledge. Under AQP, competence in flying skills and systems knowledge are integrated with HPMA skills in training and evaluated throughout the curriculum.

Scenario-Based Training and Evaluation

Most accidents are caused by a chain of errors that build up over the course of a flight and which, if undetected or unresolved, result in a final, fatal error. Traditional CF training, with its manoeuvre-based training and evaluation, artificially segments training events in such a way as to prevent the realistic buildup of the error chain. Under AQP, both training and evaluation are scenariobased and simulate more closely the actual flight conditions known to cause most fatal aviation accidents.

There are additional well-documented benefits to AQP that would lend themselves to automation airmanship and have a direct and positive impact on other Air Force initiatives such as increasing pilot production and absorption. These include the ability to rapidly modify training curricula, media and intervals; improved standardization across fleets and aircrew; and the movement from programmed flying hours to proficiencybased training. Finally, the development of a successful “closed-loop” AQP requires a robust data management system in the background that directly facilitates the implementation of other critical flight safety and standardization functions already deemed to be essential within the Air Force, but currently not in existence, such as flight data monitoring.^[22] Many of the functional elements required to achieve an “AQP-like” system already exist. However, they have either failed to live up to their potential or will not deliver the promised benefits due to the systemic leadership, coordination and advocacy issues that plague the Air Force today. AQP compatible elements include the well-developed Instructional System Design model (in the CF it is known as the Canadian Forces Individual Training and Education System [CFITES]), the Canadian Aviation Synthetic Environment (CASE) Project and the Air Force Individual Integrated Learning Environment (AFIILE) Project. Achieving and maintaining a high level of automation airmanship will require a deliberate and coordinated approach to their implementation and use.

The failings in automation airmanship that directly contributed to the loss of Tusker 914 were, in fact, the result of systemic Air Force failings and are a call to action. Addressing the disjointed and dysfunctional nature of current Air Force structures and processes is critical to solving the issues identified in the APPD Report. Dealing with the vertical stovepipes and implementing Air Force solutions to identified shortcomings in areas such as 1 Canadian Air Division Orders, standards development and training methodologies will quite likely address other problems the Air Force finds itself struggling with. The attainment and sustainment of a high level of automation airmanship that supports the Automation Philosophy, or the desired “practices” in operations, will require a deliberate and focused effort to deliver complementary and coordinated policies and procedures. Failure to implement and sustain a high level of automation airmanship will prevent the Air Force from being able to fully exploit both the technical and human potential it either currently possesses or will take delivery of in the future. The message of the Red Baron from over 90 years ago rings as true today as it did then. The Air Force must continue to develop and implement the means to achieve a high level of aircrew performance if it is to safely and effectively exploit the capabilities of the “box” our aircrew find themselves sitting in. That demands that it find the means to achieve a high level of automation airmanship.

The author wishes to acknowledge the generous contributions of Lieutenant-Colonel Darryl Shyiak, Major Don Barnby and Captain Tim Rawlings to the development of this paper. Their steadfast and selfless dedication to the attainment of automation airmanship within the Air Force is a force to be reckoned with.

While on exchange from 2001–2004, Lieutenant-Colonel Colin Keiver was the Director of Safety and Standardization at the first United States Marine Corps KC-130 squadron to convert to the KC-130J. His “love affair” with the field of human factors and the impact of automation on aviation was born during that time. In 2004 he was posted to A3 Transport in 1 Canadian Air Division Headquarters where he was heavily involved with introducing the C17 into service and developing the C130J project. In addition to functioning as the Project Authority for the Automation Policy and Planning Development Project, he is attending classes at the University of Manitoba with the aim of completing that which he should have completed 20 years ago—a degree.