Increasing Quality in Construction With Mixed Reality

Project Idea

schaerholzbau ag in Altbüron is a modern timber construction company that builds houses according to ecological, economic and social principles. The planning of the building projects is done completely digitally with the software cadwork. In the in-house production, building elements are manufactured using the timber frame construction method. During the construction of the building, these elements are installed as wall, floor or roof components.

The ElementBau app supports the employees in production hall with Mixed Reality technology. In a subsequent step, a companion app for work on the construction site was created, which covers two crucial sub-steps in the construction phase: "OnSiteXR" assists in fastening the prefabricated building elements and recording the work done for construction documentation.

How does the app work?

The employees can overlay the 3D construction plan onto the building shell. All sub-areas where the assembly of struts and angles is required are highlighted. For each position, additional information (type of wall bracket) can be retrieved by eye tracking. The operation of the app is deliberately kept minimalist, so that the employee's hands are free to do the work and that the HoloLens wearer's field of view is never unnecessarily obstructed.

In a further step, the work must be thoroughly reviewed and precisely documented. Since the installed angles are no longer visible in the finished building, the building code stipulates that the work must be thoroughly documented. This step is also supported by the OnSiteXR app: The supervisor takes photos with the HoloLens 2 and comments in natural language on the work that was performed. The spoken audio is converted into text and is automatically stored in the digital construction documentation catalog. The content is immediately available as a PDF.

In developing this Mixed Reality system, we focused strongly on providing all front-line staff with helpful computing solutions that give them independence from the printed building plan and free up their hands for the actual work.

Technical Background

The two apps rely heavily on an Azure PaaS infrastructure on the backend and enable a fully digitized construction process for the first time: From the drawing board to handing over the keys to the happy homeowner.

Both apps leverage the wide range of capabilities of HoloLens to recognize the user's intentions, taking into account the surrounding environment. The intuitive experience is enabled by the MRTKv2 library and Azure Cognitive Services for speech recognition. The fully automated data exchange with the CAD system is powered by a dedicated plugin (implemented by Design-to-Production) and an Azure PaaS solution.