Royal NLR has been working on applications for the use of HMDs that provide an AR experience in the aerodrome tower ATC environment for more than a decade now. The first device of such kind was tested in 2010 on the NARSIM Tower platform, the NLR in-house developed environment for highly realistic real-time simulations of tower operations.
More recently, NLR participated in a consortium for a project that is part of the SESAR 2020 Programme called Digital Technologies for Tower (DTT) conducting real-time simulation experiments with a focus on the use of an AR device for Attention Capturing and Attention Guidance for tower controllers.
During the experiments we used two Microsoft HoloLens 2™ devices in a simulated aerodrome control tower environment for Amsterdam Airport Schiphol (EHAM), and we demonstrated an operational concept for tower controller attention capturing and guidance based on visual and auditory cues. Existing Schiphol runway controller alerting systems triggered the attention capturing and guidance process by providing alerts with different priorities and in different operational contexts. Potentially, these alerts could occur at the same time.
A team of Simulation and Human Performance experts at NLR elaborated the basic sequence of operational steps for guidance of controller attention during the safety-critical events and designed the necessary cues inside the AR device. The presentation of the cues, in combination with aircraft labels containing information provided by the surveillance and flight data processing system, increased the Situational Awareness of tower controllers.
In a typical sequence of attention capturing and guidance, the safety net tools detected an event, relayed that information to an attention guidance logic component, and a non-intrusive element was displayed in the centre of the Tower controllers field-of-view indicating the type of alert and the most relevant information, including a pointer symbol towards the location where the event took place. This attention capturing activity had to be acknowledged by the user. Also the callsigns of the aircraft or vehicles were highlighted. They resembled radar labels and appeared as cues inside the field-of-view of the AR device once the user did not look in the desired direction. They would cling and snap to the actual labels of the relevant aircraft and vehicles when the area of the safety-critical event was in view. Thus, they also guided the user towards the area of interest. 3D spatial auditory cues (a voice indicating the type of event) were added as another guidance element.
The test programme consisted of different events and combinations of events that happened while two experienced tower controllers carried out routine work in the NARSIM environment for Schiphol airport. Pseudo-pilots were in control of aircraft movements and communicated with the tower controllers. Similar traffic scenarios were used to compare situations of working with and without the AR device. Results were gathered by using questionnaires after each test run and performing dedicated debriefing sessions.
Our experiment showed that the developed operational concept for capturing and guiding the attention of aerodrome control towers with an AR device can be considered feasible, despite a reduced operational scope and the fact that feedback for improvement of elements of the chosen concept was given. These improvements mainly concerned the symbology and timing of attention guidance cues. In general, though, this result also means that we can continue working on the concept from a solid basis.
The information in the AR device correlated accurately with the objects in the simulated outside view and tracking labels followed the aircraft without noticeable deviations. We assume that this may be different in a real tower environment with less perfect surveillance information, but we also know from our experience with the device that there are methods to improve such imperfections. Further, visibility of the symbology was sometimes competing with reflections of light coming from the surroundings, but it was also considered that such issues might be more prominent in a simulator due to the low light intensity and contrast in the out-the-window view. Finally, the AR attention guidance module received information from the alerting system inside the NARSIM environment and communicated with the AR device as expected.
The two experienced tower controllers participating in our experiment described the device in combination with the concept as a favourable addition to the controller working environment. While desired technical performance improvements (mostly related to user comfort and general adjustments) will depend on vendor development, the Microsoft HoloLens 2™ used was considered a technically useful device for implementing prototypes for Attention Capturing and Guidance with aural and visual cues. A major recommendation was to elaborate detailed guidelines for the use the system and to adapt it to the different roles in the control tower.