Imagine an office meeting where everyone is present in one space and interacting, but in reality, each person is either a holographic version of themselves or represented as a virtual avatar. It might sound like science fiction, but Microsoft’s Mesh platform aims to bring this kind of mixed reality to life.
Mesh is the company’s new mixed reality platform, which promises a powerful experience, including the ability for holoportation for individuals. Holoportation would mean that a holographic version of one’s self ports to another space for a collaborative meeting or interaction.
Mesh can beam a lifelike image of the person into a virtual scene with the help of 3D capture technology, according to the company. Microsoft Mesh was showcased at the company’s Ignite conference. In fact, the entire keynote was designed for mixed reality with “people attending the conference from living rooms and home offices around the world” as avatars.
Mesh is built on Microsoft’s Azure cloud platform and can allow developers to create apps which will support mixed reality for multiple users. Mesh would allow for collaboration at a mixed reality level and Microsoft wants to bring multi-device support to the same.
“This has been the dream for mixed reality, the idea from the very beginning. You can actually feel like you’re in the same place with someone sharing content or you can teleport from different mixed reality devices and be present with people even when you’re not physically together,” Microsoft Technical Fellow Alex Kipman said, according to an official blog post. In fact, Kipman was present on the Ignite stage as a holographic version of himself, transported to the stage from another location.
Others who joined Kipman to discuss the Mesh experience included renowned filmmaker James Cameron, and John Hanke, CEO and founder of leading augmented reality company Niantic, Inc.
What is Microsoft Mesh ?
Mesh isn’t one app or device; rather it is a platform from Microsoft and will support multiple devices and not just the HoloLens. Microsoft says Mesh will support fully immersive head-mounted displays (HMDs) such as its HoloLens, HP Reverb G2, or Oculus Quest 2 to even phones and tablets on iOS or Android to PCs and Mac for 2-dimensional viewpoint. Users would be able to connect to Mesh from anywhere.
According to Microsoft, Mesh “will also enable geographically distributed teams to have more collaborative meetings, conduct virtual design sessions, assist others, learn together and host virtual social meetups,” thanks to the powerful mixed reality experience. People’s avatars will be present in the virtual experience or their holographic selves, depending on what a user opts for.
Over time, they will be able to use holoportation to project themselves as their most lifelike, photorealistic selves, the company said.
Microsoft says the technology is the results of years of “research and development in areas ranging from hand and eye-tracking and HoloLens development to creating persistent holograms and artificial intelligence models that can create expressive avatars.” Mesh will also have Azure’s enterprise-grade security and privacy features.
What are the possible use cases of Mesh?
Given that social distancing and work from home are here to stay, Mesh could take long-distance collaboration to the next level. For engineers, designers, who need to work together, especially 3D physical models for cars or high-end furniture, Mesh could provide the perfect means of collaborating. They could all be in different places or even time zones, and yet collaborate on holographic models of future products without worrying about the location.
According to Microsoft, Mesh could allow architects and engineers to “physically walk through a holographic model of a factory floor under construction, seeing how all the pieces of equipment fit together in three dimensions, potentially avoiding costly mistakes.”
When is Mesh coming?
Microsoft says that in the coming months, Mesh will offer a “full suite of AI-powered tools for avatars, session management, spatial rendering, synchronization across multiple users and holoportation to build collaborative solutions in mixed reality.”
It will have open standards and will give developers the freedom to build solutions that will work across many different devices: HoloLens 2, a range of virtual reality headsets, smartphones, tablets and PCs.
Microsoft has announced two apps built on the Microsoft Mesh platform. One is a preview of the Microsoft Mesh app for HoloLens, which allows team members to remotely collaborate and is available for download. Customers can also request access to a new version of Mesh-enabled AltspaceVR, which will allow companies to hold meetings and work gatherings in virtual reality with enterprise-grade security features including secure sign-ins, session management and privacy compliance.
Microsoft Mesh- integration is also coming to products such as Microsoft Teams and Dynamics 365.
Niantic demonstrated at Ignite a proof-of-concept Pokémon GO demo experience that runs on HoloLens 2, which includes some of the mixed reality capabilities. However, this is not a consumer ready product and will likely take some time before it becomes available to more and more people.