AI & XR: Super power in the workplace

Why we need "Spatial Copilot", what it has to do with augmented and mixed reality and why "AI for the frontline" is so important.  

Artificial intelligence (AI) and Copilot, as recently introduced by Microsoft, will massively increase employee productivity in the coming years. The better a copilot knows the user's context and "intent", the better it can provide support and the greater the productivity gain. However, the context of many employees is not limited to a display or an office, but is located in the three-dimensional world.

These so-called "frontline workers" work only occasionally or not at all on a computer or display. They play a crucial role in healthcare, manufacturing, construction, logistics, retail, public transportation - in short, almost everywhere. It is therefore crucial for companies to support these employees with AI and Copilot.

«AI for the frontline» 

At Ignite 2023, Microsoft used the example of Dynamics Guides to impressively demonstrate what a co-pilot can do for "frontline workers" and how the real world "becomes a user interface", as Satya Nadella put it: 

In particular, the video shows that specific questions can be asked, the co-pilot gives specific answers and can thus provide targeted support. This means that users are not simply shown a PDF document, a plan or a 3D model and then have to search for answers themselves, but receive specific answers directly.

Spatial context 

As already mentioned, the context and the "intention" of the employee is very important for the support provided by AI and Copilot. Mixed reality devices such as the HoloLens 2 or the soon-to-be-released Apple Vision Pro not only capture the space, movement and hands of employees, but also what they are looking at (eye tracking).

Used well, this tracking information can enable almost "magical" interaction. In particular, the context can also be used as input for the (spatial) co-pilot. This makes it possible to look at a spare part or component, ask questions about it, update documentation or place an order.

«Spatial Copilot»  

A co-pilot is particularly important and helpful in the room, as the hands of "frontline workers" are often not free and it is not possible to operate a (graphic) user interface. In particular, there is no mouse or keyboard, but touchscreens on smartphones and tablets are often difficult to use and "rob" employees of a hand.

A "Spatial Copilot" solves this problem by allowing you to talk to it and effortlessly capture text, images and videos. As the devices also scan and "recognize" the room, even 3D models can be captured and analyzed.  

Copilot in action  

To get an idea of what working with a copilot can look like, we recommend taking a look at our OnSiteXR project. An application that is already in productive use on timber construction sites.

Our founder Paul Affentranger spoke about working in the metaverse at the asut IoT conference at the beginning of the year.

You'll find his presentation on the topic in the video below:
The related article can be found here: Arbeiten im Metaverse – Augmented Reality im Holzbau