Haoxin Wang, Ziran Wang, Dawei Chen, Qiang Liu, Hongyu Ke, Kyungtae (KT) Han
©SHUTTERSTOCK.COM/SUWIN
The metaverse is a perpetual, immersive, and shared digital universe that is linked to but is beyond the physical reality, and this emerging technology is attracting enormous attention from different industries. In this article, we define the first holistic realization of the metaverse in the mobility domain, coined as metamobility. We present our vision of what metamobility will be and describe its basic architecture. We also propose two use cases, tactile live maps and metaempowered advanced driver-assistance systems (ADASs), to demonstrate how metamobility will benefit and reshape future mobility systems. Each use case is discussed from the perspective of the technology evolution, future vision, and critical research challenges. Finally, we identify multiple concrete open research issues.
In less than half a decade, the expeditious evolution of wireless communications, artificial intelligence (AI), and high-performance computing has been reinventing the mobility concept and systems. For example, in 2019, Toyota announced a profound transformation from being an automaker to becoming a mobility company, with an emphasis on connectivity, autonomous driving, shared mobility, and the electrification of vehicles (CASE) [1]. Meanwhile, other major automotive original equipment manufacturers (OEMs), including Volkswagen, Audi, BMW, etc., are investing heavily in future mobility solutions to enhance their core competencies and to ingratiate themselves with customers. Although OEMs may frame their own blueprints for the future mobility, they share the same ambition of creating a mobility society in which safe, sustainable, frictionless, fun, and personalized transportation is universal by leveraging CASE technologies. Figure 1 illustrates a timeline of selected OEMs’ research activities and product/service rollouts in terms of connectivity, one of the cornerstones of future mobility. A trendy shift in OEM strategies is partnering with technology giants, such as Google, Amazon, and Microsoft, to develop cloud platforms, automotive operating systems, AI, etc.
Figure 1 A timeline of selected OEMs’ research activities and product/service rollouts from 2016 to 2026 in terms of car connectivity. IoT: Internet of Things.
Metaverse, as an emerging representation of the immersive Internet, has attracted enormous attention in both academia and industry. Figure 2 illustrates the timeline of the metaverse development from 1992 (the year that the term metaverse was coined in a science fiction novel, Snow Crash [2]) to 2022, including remarkable concepts, technologies, prototypes, products, and applications. Although the definitions of the metaverse vary in different versions [3], [4], [5] [6], the basic concepts are essentially the same: the metaverse is a perpetual, immersive, and shared digital universe (“verse”) that is linked to but is beyond (“meta”) the physical reality. Recently, several prominent prototypes and applications have been proposed by forerunners. The Chinese University of Hong Kong, Shenzhen Metaverse [3], for instance, is a campus metaverse prototype that allows students and faculty to perform immersive interactions in a real–virtual mixed world. Along with its rapid development in gaming [5], social media [3], and manufacturing [6], the metaverse retains potential in future mobility as well.
Figure 2 The evolution of the metaverse from 1992 to 2022, including the development of products and applications and the expansion of concepts, technologies, and prototypes (information source partially from [4]); automatic avatar creation with GANs [7]; human–machine interface (HMI) for augmented/virtual reality (AR/VR) [8]; holographic-type communication [9]; The Chinese University of Hong Kong, Shenzhen campus metaverse prototype [3]; Toyota InfoTech Labs mobility digital twin prototype [10]; and Generating Object-interActing whoLe-body motions (GOAL 4D) avatar [11]. ETSI: European Telecommunications Standards Institute.
Designing a metaverse solution for future mobility is strongly motivated by the ambition of achieving a frictionless, fun, and personalized mobility society. Additionally, the emergence of CASE technologies provides a catalyst for connecting the metaverse with future mobility. However, to the best of our knowledge, there has been no effort to discuss the confluence of the metaverse and future mobility technologies. In this article, a terminology metamobility, is coined and defined as a holistic realization of the metaverse (“meta”) in the mobility domain (“mobility”) with the support of CASE technologies. Metamobility is capable of driving customers in both physical and digital spaces, where connected and automated vehicles (CAVs) or other mobile entities, such as urban air mobility (UAM), will be physical carriers for customers to access and interact with both real and virtual worlds. For example, during the COVID-19 pandemic, many international travels were suspended due to the pandemic prevention requirements, but metamobility could provide great accessibility to allow people quarantining in Shanghai to enjoy an immersive Tokyo city tour in the cyberworld by remotely interacting with a car (e.g., a Toyota e-Palette self-driving car) in the physical world. Two essential features of metamobility could be observed in the presented example:
This article serves as the first effort to offer a comprehensive vision for building up a perpetual, synchronous, and shared metamobility in terms of its system architecture, use cases, and research opportunities.
This section presents the basic architecture of metamobility. It consists of three main parts: facility, technology, and ecosystem, as illustrated in Figure 3. The facility includes static entities, e.g., intelligent transportation infrastructures, and mobile entities, e.g., CAVs, wearable extended reality (XR) devices, and UAM. These physical entities perform as data generators as well as service requesters.
Figure 3 The architecture of the metamobility system. XR: extended reality.
From the technology part, we have eight pillars. Physical world data acquisition can be performed in a crowdsourcing manner by leveraging the ubiquitous smart sensors deployed in static and mobile entities. When the number of physical entities is sufficiently large, the size of the collected data will be exponentially increased. Edge–cloud storage and computing can relieve the pressure of processing the big data and enhance the performance of latency-sensitive and computation-intensive mobility applications. To enable the edge–cloud storage and computing, heterogeneous communication technologies, such as 5G, 6G, and cellular vehicle to everything (C-V2X), are necessary for supporting high-speed data transmissions. Furthermore, to handle the concurrent data transmission from numerous physical entities, network management strategies, e.g., adaptive data offloading, are required to enhance the data transmission efficiency.
AI is delivering on its promise of learning complicated attributes behind the data and performing precise and fast prediction. In metamobility, trustworthy AI is indispensable due to AI’s high involvement in driving safety. In other words, it is a question of not just what can be done with AI but how it should be done. A digital twin can use the historical data and learned behavior models to conduct scalable simulations in the cyberworld and mimic what might happen in the next stage. As these data and digital twin models usually have associated privacy concerns, blockchain can be an effective tool to prevent privacy leakage. Last but not least, XR techniques make human users accessible and manipulable to the cyberworld.
The ecosystem delineates a perpetual and shared virtual world, a digitized counterpart of the real world. With the assistance of the presented technologies, each physical entity is able to have a unique digital replica or digital avatar. Any changes to the physical entity will result in a real-time digital avatar update accordingly. With the continuous input of real-world data, the corresponding digital assets, e.g., personalized digital content and virtual driving scenes, can be created according to the needs of mobility applications and services.
In this section, we present two metamobility-empowered use cases to demonstrate how metamobility will benefit and reshape future mobility systems. Each use case is elaborated from the perspective of the technology evolution, future vision, and key research challenges.
Car navigation systems have already been shaping the driving experience in an unprecedented way. Most drivers today rely heavily on the modern navigation systems, such as Google Maps, which has grown into a multibillion dollar industry. However, it all started with the Iter Avto, the first dedicated car navigation system created in the 1930s. As illustrated in Figure 4, it is striking how far we have come with car navigation systems in the last 100 years. Generally, the evolution of automotive navigation has passed through three key stages as follows:
Figure 4 The evolution of car navigation systems from 1932 to 2030, including four stages: paper maps, digital maps with GPS, high-definition (HD) maps with onboard sensors, and tactile live maps with metamobility.
What could be the next item on the agenda? As listed, either conventional digital or HD maps mainly target the enhancement of car localization precision. Could next-generation car navigation systems serve to make driving more accessible, interactive, and entertaining? Metamobility can play a critical role.
The era of autonomous driving is approaching, where driving will be optional for human drivers as full driving automation matures. Nonetheless, this does not signal a termination of human access to the steering wheel or other driver controls. Today, cars with an automatic transmission still provide a manual mode for drivers who want to shift for themselves. Similarly, OEMs may retain manned driving features in some models to ingratiate themselves with specific customer segments. What, then, are the essential reasons that attract people to drive by themselves? One of the red-hot responses is to savor a more accessible, interactive, and entertaining driving experience, which self-driving alone will never be able to provide. Tactile live maps, empowered by metamobility, will be the tacit complement to autonomous driving techniques and the key enabler to enhance the accessibility, interactability, and entertainability of CAVs.
The architecture of the tactile live map ecosystem is illustrated in Figure 5, which consists of tactile live maps, digital twins, and CAVs. Specifically, a tactile live map is classified into five layers according to the time intervals at which map information changes (i.e., dynamic or static) and the function of either augmenting the perception of the real environment or enabling the immersive interaction with virtual scenes and content (i.e., reality or virtual). These five layers are perfectly aligned with each other and indexed to allow for efficient parallel information deliveries to corresponding CAV system components, such as the perception system, localization system, planning and control, and holographic display. Detailed attributes of each layer are as follows:
Figure 5 The architecture and ecosystem of the proposed tactile live maps. GNSS: global navigation satellite systems; L: layer.
Layers 1–3 in tactile live maps provide information about the static and dynamic parts of the physical world and are critical to the autonomous driving systems. They are generated and maintained at significantly high fidelity, and there is very little ambiguity about what the ground truth is.
Layers 4 and 5 are the keys for tactile live maps to evolve from CAV oriented to both human and CAV oriented as well as to have human-like consciousness. The tactile live maps can be utilized by not only drivers physically sitting in cars (i.e., option 1 in Figure 5) but also by qualified users who are driving remotely with XR remote control systems through holographic communications (i.e., option 2 in Figure 5).
Furthermore, as the dynamic layers are time sensitive and require real-time maintenance to ensure the data freshness and precision, it is untenable to gather and integrate the offloaded sensor data from CAVs on a city scale. An applicable solution is to segment a city into corridors and to deploy a designated local edge server at each. Hence, tactile live maps could be built and maintained at the scale of the corridor by leveraging edge computing. The key performance indicators (KPIs) for evaluating tactile live maps include end-to-end latency, scalability, accuracy, data efficiency, download/upload data size, and maintenance frequency. Table 1 summarizes the KPIs, features, requirements, and technologies of human-oriented digital maps, CAV-oriented HD maps, and everything-oriented tactile live maps.
Table 1 A comparison of digital maps, HD maps, and tactile live maps.
ADASs have matured during the last decade, with an estimated global market size of US$25.92 billion in 2021 [15]. They are designed by automotive OEMs and their tier 1 suppliers to either inform drivers or directly engage vehicle actions in driving and parking scenarios. Since human errors play a big role in traffic accidents, the emergence of ADASs significantly eases the burden on drivers, making driving more relaxed and safer at the same time.
Most commercially available ADASs on the current market rely on the onboard perception of real-time data and conduct onboard computing based on these perceived data. For example, adaptive cruise control (ACC) systems use the radar equipped on the front bumper of the vehicle (mostly behind the vehicle badge), together with the camera on the windshield, to identify the preceding vehicle and measure its relative speed and distance compared to the ego vehicle. The onboard computer of the vehicle will then calculate its acceleration and braking inputs to adjust its speed and maintain a safe preset distance from the preceding vehicle. Another widely used ADAS on current vehicles is the precollision assist system with automatic emergency braking (AEB), which uses the camera on the windshield to continuously detect a potential collision with a vehicle or pedestrian directly ahead of the ego vehicle and produces visual and audio warning messages for the driver (and applies brakes automatically when necessary).
Although existing ADASs are useful in certain traffic scenarios, they are limited to real-time, short-range information perceived by onboard perception sensors of the ego vehicle. Actions made by the ADAS are carried out by vehicle onboard computations, where the behavior-prediction and decision-making processes rely on the perceived data without any access to historical, large-region information. Moreover, existing ADASs always come with a handful of factory settings, which leave very few options for human drivers to customize to satisfy their personalized preferences.
Traditional ADASs can be transformed into meta-empowered ADASs by leveraging emerging technologies, such as XR, edge–cloud storage and computing, and heterogeneous communication technologies (e.g., 5G, 6G, and C-V2X). More importantly, meta-empowered ADASs can take advantage of multiple sources of data beyond the ego vehicle, which are capable of making more informed decisions than simply relying on a single data source.
The architecture of the meta-empowered ADAS is illustrated in Figure 6, where three data layers complement each other by bringing valuable data that can contribute to the construction of digital twins. Each layer contains not only real-time data that can be sampled from hardware sensors on vehicles or traffic infrastructures but also historical data that are previously sampled and stored for future reference.
Figure 6 The architecture and ecosystem of the meta-empowered ADAS.
Once the aforementioned data are retrieved from these three data layers and digital twins, the planning and control component of the ego vehicle applies advanced algorithms to process them and generate guidance to the driver through human–machine interfaces. For meta-empowered ADASs, the human–machine interface can be designed as an augmented reality-based HUD (illustrated in Figure 7). Guidance information can be visualized to the driver by the projection on the windshield, where neighboring vehicles’ driver inputs are overlaid on top of each vehicle. This design outperforms traditional ADASs by enabling the driver to make more informed driving decisions (e.g., decelerate or conduct a lane change to avoid rear-end crashes before the preceding vehicle has a hard braking).
Figure 7 An example illustration of a meta-empowered ADAS. Information is displayed on the ego vehicle’s windshield as the AR-based HUD, which includes neighboring drivers’ 1) proficiency scores and their trends, 2) possibilities of certain potential actions (e.g., hard braking and lane change), and 3) current mood score.
In the future development and deployment of the metamobility technology in both academia and industry, together with the involvement of CAVs and digital twins, numerous challenges need to be tackled from the perspectives of both research and engineering.
The metaverse has gained momentum in multiple domains through initiatives led by major industry players. Meta has released its metaverse ecology for social networks, while Nvidia announced a new collaboration with BMW on creating future manufacturing solutions by leveraging Omniverse [6], Nvidia’s metaverse platform. Similar to social networks and manufacturing, joint initiatives for future mobility with the metaverse need to be investigated with the help from both industry and academia. These initiatives can help regulate the development of metamobility, such as defining standard APIs for data access across various platforms or building safety layers to address potential cyberattacks. However, achieving a consensus among diverse sectors (e.g., telecommunication companies, car manufacturers, transportation agencies, and customers), predictably, might be arduous.
Edge AI, the confluence of edge computing and AI, will be the base to support various features of metamobility, such as the autonomy of an avatar, data interoperability, scene understanding, and distributed learning. Hence, human involvement in metamobility will be minimized where edge AI should accomplish these features in a proactive fashion. On the other hand, the main characteristics of metamobility—the immeasurable source of sophisticated data and high user engagement—would provide both challenges and opportunities for AI techniques to achieve efficient data processing, analysis, and training.
The metamobility applications require high throughput (e.g., to upload multifold onboard sensor data in real time), ultralow motion-to-photon latency (i.e., the delay between a user’s action and the corresponding reaction on display), and pervasive network access while physical objects, such as CAVs, are moving. For example, an unsatisfied motion-to-photon latency might cause car sickness and, thus, degrade metamobility experiences. To tackle these issues, two essential techniques should be studied and implemented in metamobility: 1) context-aware data offloading to adaptively adjust the data collection (i.e., spatial–temporal), perception, and transmission with high mobility and 2) application-oriented network resource provisioning to reduce cross-domain resource usage while meeting the strict latency requirement.
In metamobility, cybersecurity and user privacy are the most crucial issues that should be investigated to protect legitimate entities, such as physical CAVs or their corresponding digital assets, against attacks from both physical and digital spaces. Compared to the physical entity, the digital asset will be more vulnerable, as it usually contains sufficient information, such as the driver’s biometric data, to mimic or even clone the entity. Furthermore, due to the nature of metamobility, any entities may monitor others’ activities in the digital space (e.g., layer 5 in tactile live maps). Numerous records of behaviors, user interaction traces, and digital replicas will dwell in metamobility. Hence, how to prevent eavesdropping, continuous monitoring, and privacy leakages is the key to secure the metamobility natives.
In this article, metamobility was first coined and defined to connect future mobility systems with the metaverse. The breakthrough lying before us is to create a frictionless, fun, and personalized mobility society by enabling metamobility. We illustrated an example architecture of metamobility that integrates key technologies and facilities them to enable a plurality of new mobility applications, products, and services. The exploration of metamobility will drive innovations in future mobility technologies, industries, and economies, which, in turn, will help to make the world a better place to live.
Haoxin Wang (haoxinwang@gsu.edu) is an assistant professor with the Department of Computer Science, Georgia State University, Atlanta, GA 30303 USA, and leads the Advanced Mobility and Augmented Intelligence Lab. He received his Ph.D. degree from the University of North Carolina at Charlotte in 2020. From 2020 to 2022, he was a research scientist at Toyota Motor North America, InfoTech Labs. His current research interests include mobile augmented/virtual reality, autonomous driving, holographic communications, and edge intelligence. He is a Member of IEEE.
Ziran Wang (ryanwang11@hotmail.com) received his Ph.D. degree from the University of California, Riverside in 2019. He is an assistant professor in the College of Engineering at Purdue University, West Lafayette, IN 47907 USA, and was a principal researcher at Toyota Motor North America. He serves as founding chair of the IEEE Technical Committee on Internet of Things in Intelligent Transportation Systems and associate/guest editor of five academic journals. His research interests include automated driving, human–autonomy teaming, and digital twins. He is a Member of IEEE.
Dawei Chen (dawei.chen1@toyota.com) received his B.S. degree in telecommunication engineering from Huazhong University of Science and Technology, Wuhan, China, and his Ph.D. degree in electrical and computer engineering from the University of Houston, Houston, TX, in 2015 and 2021, respectively. From 2015 to 2016, he was a network engineer at Ericsson. Currently, he is a research scientist at Toyota Motor North America R&D, InfoTech Labs, Mountain View, CA 94043 USA. His research interests include deep learning, edge/cloud computing, federated learning/analytics, and wireless networks. He is a Member of IEEE.
Qiang Liu (qiang.liu@unl.edu) is an assistant professor with the School of Computing, University of Nebraska–Lincoln, Lincoln, NE 68588 USA. He received his Ph.D. degree in electrical engineering from the University of North Carolina at Charlotte in 2020. His papers won the IEEE Communications Society’s TAOS Best Paper Award 2019 and IEEE International Conference on Communications Best Paper Award 2019 and 2022. His research interests include the broad fields of edge computing, wireless communication, computer networking, and machine learning. He is a Member of IEEE.
Hongyu Ke (hke3@student.gsu.edu) received his bachelor’s degree from StonyBrook University in 2021 and his master’s degree from the University of Buffalo in 2022. He is pursuing his Ph.D. degree with the Department of Computer Science, Georgia State University, Atlanta, GA 30303 USA. His current research areas include mobile augmented reality, efficient artificial intelligence, and edge computing. He is a Student Member of IEEE.
Kyungtae (KT) Han (kyungtae.han@toyota.com) received his Ph.D. degree in electrical and computer engineering from the University of Texas at Austin in 2006. He is currently a senior principal researcher at Toyota Motor North America R&D, InfoTech Labs, Mountain View, CA 94043 USA. Prior to joining Toyota, he was a research scientist at Intel Labs and a director at Locix Inc. He is an associate editor of SAE International Journal of Connected and Automated Vehicles. His research interests include cyberphysical systems, connected and automated vehicle techniques, and intelligent transportation systems. He is a Senior Member of IEEE.
[1] “Reforming our company to become a ‘mobility company’.” Toyota. Accessed: Sep. 2022. [Online] . Available: https://global.toyota/en/company/messages-from-executives/details/
[2] N. Stephenson, Snow Crash: A Novel. New York, NY, USA: Random House Publishing Group, 2003.
[3] H. Duan, J. Li, S. Fan, Z. Lin, X. Wu, and W. Cai, “Metaverse for social good: A university campus prototype,” in Proc. 29th ACM Int. Conf. Multimedia, 2021, pp. 153–161, doi: 10.1145/3474085.3479238.
[4] L.-H. Lee et al., “All one needs to know about metaverse: A complete survey on technological singularity, virtual ecosystem, and research agenda,” 2021, arXiv:2110.05352.
[5] “Meta horizon worlds.” Meta. Accessed: Sep. 2022. [Online] . Available: https://www.oculus.com/horizon-worlds/
[6] “Nvidia, BMW blend reality, virtual worlds to demonstrate factory of the future.” Nvidia. Accessed: Sep. 2022. [Online] . Available: https://blogs.nvidia.com/blog/2021/04/13/nvidia-bmw-factory-future/
[7] Y. Jin, J. Zhang, M. Li, Y. Tian, H. Zhu, and Z. Fang, “Towards the automatic anime characters creation with generative adversarial networks,” 2017, arXiv:1708.05509.
[8] M. Zhu et al., “Haptic-feedback smart glove as a creative human-machine interface (HMI) for virtual/augmented reality applications,” Sci. Adv., vol. 6, no. 19, May 2020, Art. no. eaaz8693, doi: 10.1126/sciadv.aaz8693.
[9] A. Clemm, M. T. Vega, H. K. Ravuri, T. Wauters, and F. De Turck, “Toward truly immersive holographic-type communication: Challenges and solutions,” IEEE Commun. Mag., vol. 58, no. 1, pp. 93–99, Jan. 2020, doi: 10.1109/MCOM.001.1900272.
[10] Z. Wang et al., “Mobility digital twin: Concept, architecture, case study, and future challenges,” IEEE Internet Things J., vol. 9, no. 18, pp. 17,452–17,467, Sep. 2022, doi: 10.1109/JIOT.2022.3156028.
[11] O. Taheri, V. Choutas, M. J. Black, and D. Tzionas, “GOAL: Generating 4D whole-body motion for hand-object grasping,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition (CVPR), 2022, pp. 13,263–13,273, doi: 10.1109/CVPR52688.2022.01291.
[12] “TomTom HD maps.” TomTom. Accessed: Sep. 2022. [Online] . Available: https://www.tomtom.com/en_us/
[13] “HERE HD live map.” HERE. Accessed: Sep. 2022. [Online] . Available: https://www.here.com/platform/HD-live-map
[14] “Deep reality display.” WayRay. Accessed: Sep. 2022. [Online] . Available: https://wayray.com/deep-reality-display/#about
[15] “Advanced driver assistance systems (ADAS) market size and forecast.” Verified Market Research. Accessed: Sep. 2022. [Online] . Available: https://www.verifiedmarketresearch.com/product/global-advanced-driver-assistance-systems-adas-market-size-and-forecast-to-2025/
Digital Object Identifier 10.1109/MVT.2023.3263330