©SHUTTERSTOCK.COM/RAULALMU
Olivier Clément, Stephane Rondeau
©SHUTTERSTOCK.COM/DESIGN PROJECTS
In the last decade, we have witnessed rapid technology advancements associated with AR, VR, and MR devices. This has led to several business- and consumer-oriented products becoming available on the market. Over the years, both start-ups and major companies have developed fascinating and accessible devices, opening many new possibilities in various domains. The new possibilities for the training and education industry are particularly compelling. Consequently, it is no surprise that these technologies are actively being evaluated for various training contexts (Xie et al., 2021) and compared to their traditional equivalent solutions (Gross et al., 2023). In a near future, these innovations will directly impact our professional and personal environments, from providing new simple collaboration tools (Wang et al., 2019) up to how our health-care specialists are being trained (Schild et al., 2022).
Concurrently, the COVID-19 pandemic hit the world hard and forced us to rethink the way we work. Problems associated with travel restrictions and teleworking arrangements limiting onsite attendance put a spotlight on virtual environments as suitable alternatives for various training scenarios. At the CSA, these evolving technologies and new restrictions combined sparked several discussions and initiatives, leading to a concept study around remotely training astronauts to perform robotic operations using MR technologies. Is it possible to carry out a training program that achieves a comparable level of quality? What are the upsides and considerations with such an approach?
The main objective of this concept study is to develop and test an MR environment leveraging modern technologies to provide astronauts with a remote training program within a shared virtual environment that yields equivalent results when compared to the current traditional approach. This prototype application will be used to evaluate the feasibility of such an endeavor and to identify its advantages and drawbacks. The current International Space Station (ISS) program supports and funds this study, but this effort has the potential to influence how training could be performed on upcoming programs, such as Canadarm3 on the Lunar Gateway.
Astronauts on a mission to the ISS undergo an extensive and rigorous training program, spread over several years and covering a wide variety of subjects. One of these subjects is to learn how to perform various robotic operations using the mobile servicing system (MSS) components shown in Fig. 1.
Fig 1 An on-orbit picture of the MSS components. MBS: mobile base system; SPDM: special-purpose dexterous manipulator; SSRMS: space station remote manipulator system. (Source: NASA; used with permission.)
The MSS represents Canada’s main contribution to the ISS and includes three components:
The MBS serves as a mobile base for the other two components (the SSRMS and SPDM). It moves along the truss segments on a rail-like structure to reach various sections of the ISS. The SSRMS, also known as Canadarm2, can attach to the MBS or one of the many grapple fixtures spread across the station. It can grapple large objects and was used to assemble the various nodes and components now forming the ISS. It can also be used to capture and dock free-flyer spacecraft. The SPDM, commonly known as Dextre, is used for tasks involving smaller objects and requiring more precision. For example, it can interact with scientific payloads and orbital replacement units. Astronauts can operate the MSS through one of the robotic workstations (RWSs) located within either the Destiny module or the Cupola module, both located on the ISS. Figure 2 shows the distribution of the RWS components on the Destiny module.
Fig 2 The RWS installed in the Destiny module of the ISS. DCP: display and control panel; PCS: portable computer system; RHC: rotational hand controller; THC: translational hand controller. (Source: NASA; used with permission.)
The RWS is composed of several components, including the following:
The CSA is responsible for providing relevant training to astronauts and ground operators to enable them to perform the required robotic operations, including capturing and docking free-flyer spacecraft and replacing faulty components across the ISS. The current training program favors a classic approach in which the astronauts are required to travel to the CSA to access a mock-up of the RWS within a replica of the Destiny module driven by the MSS operations and training simulator (MOTS). Figure 3 exhibits this mock-up at the CSA’s training facilities.
Fig 3 The RWS mock-up in the CSA’s training facilities.
This physical mock-up consists of
Using this mock-up environment driven by the MOTS, a certified instructor goes through the training program with the trainee. Typically, it consists of approximately eight sessions split over five days, during which the instructor starts by explaining relevant concepts required to operate the MSS components. Afterward, the trainee is required to perform some realistic robotic operations while the instructor is monitoring his actions and inputs to the system. Comments and guidelines are provided through debrief sessions, and the training program ends with a certification examination.
At the core of this training program is the MOTS, a complex simulator system composed of several subsystems and components. Its primary role is to gather inputs from the various available controls, including the THC, RHC, DCP, and PCS, and then simulate the system responses using an accurate robotic model. These responses are then conveyed back to the users through the regular RWS interface (e.g., the PCS and DCP) and using a rendering engine to generate applicable camera views to be displayed on the monitors. Several additional subsystems are involved to handle other tasks, such as contacts between objects, ISS component reconfiguration (e.g., solar panels, radiators, and free flyers), or ground commanding. Figure 4 presents a simplified MOTS architecture with its main components and their typical interactions.
Fig 4 A simplified MOTS architecture. UI: user interface; I/Os: inputs/outputs.
In this context, it is not realistic to expect to be able to encapsulate this complex simulation system within a single MR application running on a given headset. Consequently, the first constraint of our prototype application was to build it as an interface interacting with the actual MOTS rather than trying to implement a simplified simulation model. Subsequently, our training department defined its own limitations so that our prototype is able to provide valuable and representative training. These additional constraints mainly revolve around requirements for the astronauts to interact with the actual physical hand controllers rather than a virtual equivalent since their tangible interactions and feedback are critical to get an accurate understanding of their operation. Finally, it was also imperative to develop this prototype in a timely manner with a limited team and budget, so emphasis was put into using mature technologies and leveraging as many off-the-shelf software products as possible.
With all of this in mind, the Microsoft HoloLens 2 headset was selected for this concept study, particularly since the application required a perfect pass-through for the astronauts to interact with the RWS physical devices. Development was done within Unity using the MR toolkit. Our multiuser framework was built using the Photon engine, which is the recommended option from Microsoft tutorials. Figure 5 presents the application’s overall architecture and its interactions with the MOTS.
Fig 5 The prototype MR application architecture with relevant MOTS interactions.
The prototype MR application environment is built within a holographic virtual Destiny module created from images captured onboard the ISS for increased immersiveness compared to our current physical mock-up. The RWS used to interact with the MOTS is a combination of virtual and physical devices configured depending on your role in the session (trainee or instructor) and your preferences. Typically, the three monitors are always virtual and are streaming video generated by the MOTS rendering engine since these components do not require any manual handling. To ensure representative interactions, the THC, RHC, and PCS are physical devices used by the trainee to send commands to the MSS. The DCP is a fully functional virtual device also used by the trainee to send commands to the system. This approach was privileged in our study for the DCP to evaluate virtual interactions for simple controls, such as buttons and switches. Simplified avatars are instantiated locally to depict every other active user in the session. The HoloLens headset tracks the position and orientation of the user’s head, wrists, and fingers. These data are broadcasted to other users using the Photon engine and are used locally to animate the corresponding avatar through an inverse kinematic solver.
The instructor has a slightly different setup in which everything is virtual and animated from the simulator status. The instructor has access to the regular MOTS user interface (UI) through his laptop and screens at his desk and uses it to configure and follow through his training scenario. Finally, a virtual holographic ISS 3D model animated from the simulation status replaces the traditional static model positioned behind the participants.
In the physical environment, the astronaut starts by positioning the physical PCS, THC, and RHC on a desk and follows a setup sequence to align the virtual environment appropriately. The MR application running on the HoloLens adds the virtual immersive environment (Destiny module) and the RWS monitors. The astronaut is able to interact with the instructor positioned on his left. Figure 6 shows the MR application running from the trainee’s perspective in front of the RWS.
Fig 6 The prototype MR application from the trainee’s perspective.
Typically, the instructor can work at a regular desk with a laptop and screens connected to the MOTS UIs and goes through a similar setup sequence to align the virtual environment appropriately. The MR application running on the HoloLens adds the virtual immersive environment (Destiny module) and a completely virtual RWS driven by the status of the simulation on the right-hand side. The instructor is able to observe the astronaut operating the various RWS components through the simplified avatar movements. Figure 7 shows the MR application running from the instructor’s perspective.
Fig 7 The prototype MR application from the instructor’s perspective.
The dynamic ISS 3D model is located behind the participants. It is animated in sync with the MOTS while the astronaut is performing its operations at the RWS and can be completely reconfigured through simple clicks within the MOTS UIs. Throughout the training scenarios, the instructor can use it to explain various concepts using a representative reference view rather than just trying to describe a specific setup as (shown in Fig. 8). The MOTS framework also enables the instructor to save and recall predefined ISS and MSS arrangements for very quick transitions between explanations.
Fig 8 The holographic ISS 3D model driven by the MOTS.
Regarding performance, our current implementation runs at approximately 25 frames per second. The prototype is functional and completely integrated with the MOTS. The prototype was presented to the training staff and a few astronauts. Preliminary discussions helped to identify some of advantages and shortcomings, detailed in the next sections. Our training department has developed a complete validation plan, and instructors will soon go through the training program using this alternative training environment. These dry runs will be used to thoroughly evaluate its capacity to yield equivalent training results when compared to the current traditional approach.
Astronauts have very busy training schedules often resulting in complex travel arrangements. In addition, they tend to be accompanied by a relatively large entourage (a personal trainer, interpreter, and so on), so a one-week training program in Canada can end up being quite costly. Adding an option to perform this training remotely is a game changer regarding astronaut training, and this advantage alone makes it worth exploring the project feasibility.
Complex travel arrangements tend to force the training to happen within a static one-week block. Even though that could be beneficial for reasons such as training continuity, the proposed MR application prototype adds flexibility within training program because it does not require any travel and can be set up quickly wherever and whenever needed. It also offers the possibility for last-minute refresher courses.
The current MOTS training facilities support training two astronauts simultaneously and uses a large, 2,200-ft2 dedicated space. The Destiny module and RWS mock-ups constitute a permanent fixture, and the room cannot be repurposed easily. It is not really a problem if this dedicated space is to be used very frequently, but, in our training context, the usage tends to be sporadic. The proposed MR application prototype can easily be set up in a regular meeting room with little effort.
The level of immersion provided by the MR environment can be very high. Since our prototype uses images captured from the actual ISS, the astronauts will be able to immerse themselves into the on-orbit environment. While physical mock-ups can also be very realistic, they can be harder to maintain accurately over the years while their on-orbit equivalents evolve. Digital counterparts can be updated easily without requiring any downtime to the training infrastructure.
The technologies behind such MR applications open new possibilities regarding tools for the instructor. For example, the HoloLens headset is able to track the astronaut’s gaze while operating the RWS. This can be displayed to the instructor so that he can identify problems with the trainee’s scan pattern (e.g., the astronaut did not look at the left monitor for several minutes). One could also envision useful debrief capabilities through a record-and-play-back system.
From our experience, the battery life of the HoloLens headset running a graphic-intense MR application is approximately 2.5 h. A typical training day on the MOTS is composed of two sessions of 3.5 h each. Even if we swap headsets between the sessions, it is very unlikely that one charge will be sufficient. Depending on the training context, this could be mitigated by running the exercise with the headset plugged in or by wearing an additional battery pack attached to a belt.
Real-time simulators like the MOTS put in place latency requirements to get a representative operation. Latency is defined as the delay between an input to the system and the associated response provided to the operator. In our case, this means the astronaut should see the SSRMS moving on the monitors in a realistic timeframe after some inputs on the THC or RHC. If this is not achieved, the operator might alter the commands, leading to unrepresentative negative training. With the MOTS being located in Canada, astronauts connecting through the MR application from Houston will experience additional delays. We measured this additional delay within our current IT infrastructure to approximately 200 ms, which is significant considering the original MOTS latency is around 350 ms. Unfortunately, the acceptable additional delay should not exceed 50 ms. This could be mitigated by using a dedicated communication channel between the CSA and NASA (that is typically used for on-orbit operations) instead of going through the Internet. Another interesting approach would be to decentralize the simulator architecture to have one simulator instance close to the potential trainees. This could mean having one simulator instance at the NASA training facilities and having the instructor connect from Canada instead of the other way around.
Current untethered mature technologies are somewhat limited regarding their performance capacities. High-resolution integration of complex avatars is not quite possible yet when combined with other required items, such as a large immersive environment, video streaming, and so on. Unfortunately, this results in very limited facial expression cues from simplified avatars and makes it impossible for the instructor to identify relevant information such as stress or confusion. Until technologies such as facial motion capture or holoportation enable the capturing and rendering of complete facial expressions, this issue can be mitigated by ensuring that participants are aware of this limitation and make sure to verbalize their emotions.
Wearing the AR, VR, or MR headsets for a long period can become uncomfortable and strenuous. The additional weight put on the neck can be fatiguing when a headset is worn for an entire day. Moreover, some users report dizziness or disorientation after longer use. Having this in mind, the HoloLens headset was specifically selected because it is relatively lightweight and because MR devices tend to cause less dizziness than VR devices. From our experience, these discomforts are likely to vanish after an acclimation period of a few weeks. In that regard, a training program using such technologies should plan preliminary offline sessions for the astronauts to get used to wearing the headset. Additionally, several upcoming products, most notably the Apple Pro Vision, should be evaluated, as they could provide more comfortable experiences.
This concept study proposed an alternative to the CSA’s traditional training approach for astronauts using a shared remote MR environment depicting the RWS aboard the ISS. There are still several shortcomings to be worked on, but the advantages introduced by this approach are substantial. Apart from the remote capabilities, the added flexibility regarding schedule and dedicated physical training space could have a major impact on training operations. In contrast, our specific context involving a real-time simulator accessed remotely adds some latency constraints that can be hard to achieve and imply appropriate infrastructure planification (such as dedicated communication channels or decentralized simulators). The associated technologies still need to be improved, but, based on this experience, it is safe to say that training operations for future space programs at the CSA will integrate MR components.
Aside from that, several additional initiatives emerged from this concept study. While its original focus was to ensure equivalent training capabilities, it is apparent that these technologies could be utilized even more to provide new and better support tools for instructors. Gaze-tracking capability was built in our prototype, but several other support tools could be envisioned. For example, facial expression analysis could be added to identify stress level, or metrics could be computed based on tracked movements to evaluate operation sequence effectiveness. Artificial intelligence technologies could also be integrated as an alternative to the instructor to dispense training material, answer questions, and even objectively evaluate the trainee’s performance. Finally, holoportation technologies (Orts-Escolano et al., 2016) could be looked at to replace the simple avatars from our prototype with accurate participant representations to increase immersion and provide more realistic human interactions.
• B. Xie et al., “A review on virtual reality skill training applications,†Frontiers Virtual Reality, vol. 2, Apr. 2021, Art. no. 645153, doi: 10.3389/frvir.2021.645153.
• G. Gross, R. Ling, B. Richardson, and N. Quan, “In-person or virtual training?: Comparing the effectiveness of community-based training,†Amer. J. Distance Educ., vol. 37, no. 1, pp. 66–77, 2023, doi: 10.1080/08923647.2022.2029090.
• P. Wang et al., “An MR remote collaborative platform based on 3D CAD models for training in industry,†in Proc. IEEE Int. Symp. Mixed Augmented Reality Adjunct (ISMAR-Adjunct), Beijing, China, 2019, pp. 91–92, doi: 10.1109/ISMAR-Adjunct.2019.00038.
• J. Schild, G. Carbonell, A. Tarrach, and M. Ebeling, “ViTAWiN – Interprofessional medical mixed reality training for paramedics and emergency nurses,†in Proc. IEEE 10th Int. Conf. Serious Games Appl Health (SeGAH), Sydney, Australia, 2022, pp. 1–8, doi: 10.1109/SEGAH54908.2022.9978566.
• S. Orts-Escolano et al., “Holoportation: Virtual 3D teleportation in real-time,†in Proc. 29th Annu. Symp. User Interface Softw. Technol. (UIST), New York, NY, USA: Association for Computing Machinery, 2016, pp. 741–754, doi: 10.1145/2984511.2984517.
Olivier Clément (olivier.clement@asc-csa.gc.ca) earned his B.S. and Ph.D. degrees in software engineering, specialized in computer graphics, from the École de Technologie Supérieure in Montréal, QC, Canada. He worked in the civil aviation industry at CAE Inc. on real-time full flight simulators for eight years before moving to the Canadian Space Agency (CSA), Saint-Hubert, QC J3Y 8Y9, Canada. At the CSA, he works on 3D applications applied to robotic simulations used to train astronauts and ground operators.
Stephane Rondeau (stephane.rondeau@asc-csa.gc.ca) received his M.Sc. degree in aerospace engineering from Polytechnique Montréal, QC, Canada. He has 33 years of working experience in software development and system engineering in both the aeronautics and aerospace industries. He is currently working at the Canadian Space Agency, Saint-Hubert, QC J3Y 8Y9, Canada, as system lead engineer for the Canadarm2 simulator used to train astronauts and ground operators in how to operate the Canadarm2 onboard the International Space Station. He is an inquiry-based learning enthusiast and strongly believes that combining augmented reality and artificial intelligence has the potential to increase crew expertise and knowledge while reducing their workload.
Digital Object Identifier 10.1109/MPOT.2023.3318929
