Augmented Reality Applications Testing 101: A Practical Guide

When product owners work on an AR application creation, they should be aware that quality assurance plays a crucial role in the product’s success. Poor AR software quality leads to poor user experience, and this, in turn, to unavoidable cost losses.  Having extensive experience in augmented reality applications testing on long-term projects, I’ll explain why quality assurance in AR is important, and cover various aspects of AR testing, including methodologies, environmental considerations, challenges, and key performance criteria. Based on the experience of the MobiDev QA team, as well as the success stories of our clients, I’ll explain to product owners who are going to develop an AR application how to conduct AR application testing in practice.

Augmented Reality Testing: What We Do First 

Augmented reality is the real environment with an additional layer of information. This augmented layer is put on top of the environment viewed through the camera and is meant to enhance the real world with relevant text, images, or 3D objects. 

Before starting the testing process, the QA team examines product requirements first to see the conditions under which the product will be used. It includes specified devices and types of interaction with the product. The selected AR development environment — whether it is based on Apple ARKit, Vuforia, or Unity 3D — is monitored as well. This analysis helps to develop an effective software testing strategy.

After those necessary steps, it’s time to create a storyboard of use cases that should be tested in a real environment. Use cases help QA engineers cover all the potential scenarios and provide a holistic view of the product — far more thoroughly than a simple review of what wireframes might provide.

Source: Storyboard

The process involves setting up specific environments and exposing the app to various physical objects, scenes, and lighting conditions. While it aligns with the traditional testing pyramid (User Interface/Integration/Unit Testing), AR testing requires additional specifics. Let’s take a detailed look at them. 

Choosing the Right Testing Environment

Proper environment emulation for AR app testing is crucial because augmented reality is tied to real-world interactions. AR apps are designed to overlay virtual elements onto the physical environment. Choosing the appropriate testing environments is essential to ensure the optimal performance of AR apps when it comes to:

• measurements
• design
• in-store shopping experiences
• navigation
• AR-enhanced maps
• healthcare
• retail
• training
• gaming
• tourism
• and exploration

During the testing process, we specifically create diverse scenes, examining the app’s functionality across varied conditions. This helps to ensure that the app works well in real-world scenarios.

The scope of AR interactions varies, including strictly indoor, exclusively outdoor, or a blend of both. The choice depends on the goals and use cases of the AR application. Let’s examine all the possible scenarios in depth. 

Indoor AR testing

Firstly, let’s give a short characteristic of the indoor testing environment.  It is strictly limited to indoor spaces like homes, offices, malls, galleries, or museums and there definitely will be interactions with indoor objects, surfaces, and features. Applications for interior design, indoor navigation, training, or virtual try-on experiences often fall into this category.

Here are the main six environmental properties we have to consider and include in indoor AR testing.

1. Varied lighting

To examine the app’s adaptability to different indoor lighting conditions, it’s necessary to consider different options, including natural light, various types of artificial lighting, and low-light scenarios during testing. Vary light sources by including overhead lighting, ambient lighting, or direct lighting setups. Additionally, consider placing light sources at different heights and angles to simulate real-life conditions. 

2. Specific conditions like small and confined spaces

Testing the app’s performance in compact and confined indoor spaces helps to ensure a comprehensive assessment of its spatial adaptability. I recommend including testing scenarios in diverse environments, such as small offices, narrow hallways, or compact storage rooms, to simulate common spatial constraints.

3. Furniture and decor interaction assessment

Testing app capabilities in varied scenarios, including settings with different furniture types, layouts, and decorations, ensures that the app adeptly recognizes these real-world elements and seamlessly allows the placement and manipulation of virtual objects around them. 

4. Various surface recognition and interactions

It is important to consider this when we examine how the specific app’s features can identify and respond to different surfaces commonly found in indoor environments like carpets, textured walls, and wooden floors. Also, this can be reflective surfaces such as glass surfaces (including transparent ones), polished metal, glossy, or mirrored surfaces. 

5. Moving objects (testing in the dynamic environment)

QA assesses how well the AR app deals with dynamic elements in indoor environments, considering factors like pets or people moving around, changing lighting conditions, mirror reflections, open/closed doors and windows, digital screens, and any added or removed decor, etc.

6. Architectural complexity, building size, and indoor multi-level structures

Testing in such spaces is especially crucial for AR indoor navigation applications designed to guide users within complex structures such as huge shopping malls, airport terminals, university campuses, workshops, business centers, or any other large buildings. 

Outdoor AR testing

The environment might not be limited to evaluating AR app performance in different indoor spaces and lighting, it extends far beyond, introducing new challenges. For instance, we had a case of non-static scenes, where an AR-enhanced mobile application would allow passengers in moving transport to augment the outside reality. The application was tested in moving vehicles, with a detailed comparison of results in various use cases and the evaluation of whether it achieved the desired precision or not. In such cases, factors like potential disruptions in GPS signals, varying speeds, and changing scenery usually add layers of testing complexity. 

The outdoor environment is always more dynamic and less controlled than the indoors. Therefore, let’s check the five outdoor environment properties QA engineers should consider in this case.

1. Light conditions

Similar to indoor AR testing, we can face various situations. There can be different outdoor lighting conditions: direct sunlight that may give intense and harsh lighting, shade or partial shade, and more dark lightning scenarios with low light intensity. 

2. Dynamic environments and crowded spaces

Testing the app in crowded outdoor spaces to evaluate how well it handles a high density of people and dynamic elements is key. Our goal is to verify that it maintains accurate tracking and object placement in such conditions.  

3. Variable terrain and uneven ground

Unlike indoor surfaces that are typically flat and even, outdoor environments introduce challenges such as irregularities, bumps, and changes in elevation. Testing on variable terrain and uneven ground focuses on the app’s ability to handle these outdoor conditions and accurately place and interact with virtual objects even when the ground is not uniform.

4. Outdoor objects and structures

These objects can vary significantly from indoor objects like furniture and decorations in terms of size, shape, scale, and material, and they are subject to changing environmental conditions. Testing the interaction with outdoor objects like trees, rocks, statues, signs, etc ensures that the AR app adapts effectively.

5. The complexity of navigation in large environments

Similar to indoor AR testing, navigation functionality poses a few challenges here. In case the AR app depends on GPS or other location-based services, QA engineers provide the integration testing to verify that the service delivers the information with minimal drift and lag, especially in high-density urban areas. 

The testing focuses on how the application mitigates GPS drifting and on checking whether the virtual overlay elements stay aligned with the user’s actual position over time. Depending on application use cases, QA engineers can test it in urban conditions with a high concentration of landmarks, between tall buildings, skyscrapers, street intersections, parks, or other open landscapes.

Mixed environment AR testing 

The final physical environment that I would like to talk about is a mixed one, which combines interactions with both outdoor and indoor spaces. In this case, the testing focus will be on the app’s adaptability capabilities. Here are the two unique properties we have to consider in this case. 

1. Transition

Transitions between indoor and outdoor environments in AR applications involve adapting to changes in various factors: sunlight to artificial light, switching of used navigation technology, the appearance of dynamic objects and obstacles, etc. It’s important to evaluate all of them during the testing process. 

2. Network switching

Users can change the type of network connection almost everywhere, but it happens more often during transitions from indoor to outdoor. Wi-Fi can be switched to mobile networks and vice versa. In case of network coverage gaps or weak signals, the app must handle the transition to offline mode and connection restoration without data loss. In such a case, the app should provide relevant cached content, ideally. 

Evaluation of user experience quality while testing AR apps

Once the appropriate testing environment aligns with specific use cases, the next critical aspect involves evaluating the quality of the user experience in AR apps. Let’s take a look at four aspects that can be useful here. 

1. Guideline adherence

The influence of guideline adherence on UX and user satisfaction is significant. When users interact with a familiar and consistent interface, they feel more comfortable navigating the app. Consistency contributes to a positive learning curve, and promotes a sense of trust, as users are more likely to trust an app that behaves as expected based on platform conventions. 

So, the application design must comply with platform-specific guidelines (e.g., Apple ARKit, Google ARCore, Unity guidelines, Kudan, DeepAR, etc). In the preliminary testing phase, we usually conduct a checklist-based assessment to verify compliance and to confirm that AR features are implemented correctly.

2. User interactions in AR

AR user interactions refer to how users engage with augmented reality apps. These interactions involve real and virtual elements blending. We can divide interactions into implicit and explicit.

Implicit interactions leverage various cues and inputs, such as gestures, head movements, location-based interactions, and real object recognition, to enable the system to autonomously understand and respond to the user’s intentions. 

Explicit interactions involve direct and intentional input from the user to interact with AR elements or perform specific actions within the AR environment. Examples are tapping, touching, pressing the physical button, or swiping. 

3. Accessibility testing 

The purpose of mobile accessibility testing is to make sure your app is equally usable for as many different people as possible. To confirm the app is accessible (usable and inclusive), QA engineers evaluate the app’s compatibility with accessibility features, such as screen readers, and ensure that AR content is perceivable and operable for users with disabilities. As examples of quality criteria, we can consider the integration with screen readers, such as VoiceOver (iOS), and TalkBack (Android), the presence of contrast and color settings, and the presence of the ability to adjust the text size to improve readability. 

4. Working with feedback

Frank Chimero, a renowned designer and author of The Shape of Design says, “People ignore design that ignores people.”  For me, this is an important aspect. Collected feedback, crash reports, and any statistical data have to be analyzed and used as a source of ideas for future user experience improvements and necessary optimizations. To reach this goal (to collect data) we can use in-app feedback forms, crash reporting tools, complex analytical tools like Firebase and Mixpanel, AR-specific metrics (custom logging, ARKit/ARCore diagnostic tools), beta-testing (TestFlight, Google Play Console), etc.

Compatibility and performance of AR applications

Compatibility and performance are primarily technical characteristics, focusing on how the app functions across various devices and the efficiency of its underlying processes. Testing for compatibility and performance often requires a deep understanding of hardware configurations, operating systems, and technical optimizations. What do you need to pay attention to here?

Compatibility testing in the AR context

Keeping in mind the diversity of AR-supporting smartphones, tablets, and AR headsets, testing on all devices listed in the product requirements is essential. While emulators and cloud-based devices are valuable tools, they fall short in comparison to testing on actual physical devices when we talk about AR. 

The specificity of AR testing lies in the unique interaction of AR apps with the real-world environment, and only real devices can accurately replicate the myriad conditions users might encounter. That’s why testing on all devices outlined in the product requirements becomes more than a checkbox exercise. 

Performance testing in the AR сontext 

Performance analysis of the AR app is important, especially considering its resource-intensive character. Performance testing and analysis help to minimize the risk of app crashes or slowdowns during resource-intensive tasks for end users because such bugs will be found during the testing phase. 

Based on identified performance issues, it’s possible to choose the best performance optimization strategies for the app. By doing so,  delivered AR experiences can not only meet but exceed user expectations in terms of visual fidelity, responsiveness, and overall immersion. AR apps performance testing can also be divided into several parts. 

Our main goal here, what we want to achieve by performing performance analysis, is the efficient use of device resources. It means that the AR app runs effectively across a range of devices, supporting both high-end and lower-end hardware. 

Limitations of AR applications

It’s important to remember that AR apps, like any technology, have limitations and may encounter known issues. Let’s get familiar with some of them.

1. Tracking limitations: Issues related to tracking accuracy and stability may be observed during dynamic movements or in environments with poor lighting. ARCore and ARKit both rely on visual tracking features, and Unity may integrate them. Older devices with less advanced camera capabilities may experience tracking limitations.
2. Environmental sensitivity: Environmental factors such as complex or reflective surfaces, lack of visual features, or extreme lighting conditions can impact AR app performance. AR technologies are continually improving environmental awareness, but challenges persist in areas with limited distinctive features. High-end devices with advanced sensors and cameras may provide a more robust AR experience in diverse environments.
3. Limited Field of View (FoV): Users may notice a restricted field of view, limiting the area where AR objects can be placed or interacted with. FoV limitations are often inherent to the device’s hardware and may not be solely related to the AR technology that we are using. Smart glasses and some AR-enabled devices may have a narrower FoV compared to smartphones.
4. Depth perception challenges: Users may experience issues with accurate depth perception, leading to virtual objects appearing disconnected from the real world. AR technologies/cores have depth-sensing capabilities, but challenges may arise in certain scenarios, affecting depth accuracy. Devices equipped with advanced depth sensors may mitigate some challenges, while others may rely on stereoscopic cameras.
5. Real-time occlusion: Virtual objects may not consistently interact realistically with physical objects in real-time, leading to improper occlusion.
6. Battery consumption: AR apps may drain device batteries quickly, affecting the overall user experience.

By considering these limitations, AR developers and QA engineers can set realistic expectations, work towards continuous improvement, and deliver a more reliable and enjoyable AR experience for users across different devices and platforms.

Testing Augmented Reality Apps: Valuable Practical Insights 

Here are some QA insights based on our testing experience of several AR apps.

1. Communication

The user has to know how the app works and how to use it effectively. Also, the best choice is user-friendly communication, without difficult tech terms — the instructions shouldn’t be confusing to support different types of users. Necessary information should be displayed on the screen when needed. Unnecessary UI elements should be hidden to help focus on what is important.

2. Interactivity

This criteria in the context of AR applications, refers to the level of engagement and responsiveness that users experience when interacting with virtual objects (AR objects) within an augmented reality environment. It encompasses how users can engage with, manipulate, and receive feedback from virtual models or objects overlaid in the real-world environment. Interactions with virtual objects like 3D models of furniture, decorations, and areas, have to be simple and intuitive. 

What can we analyze? 

• The time taken for users to initiate interactions, and what number of actions performed to achieve the desired activity/goal 
• User satisfaction level (Is the product an aesthetically pleasing and sensually satisfying one?)
• The app’s responsiveness to different user gestures, such as taps, swipes, and gestures
• Compatibility of interactive features with accessibility settings
• A presence of the ability to use the entire display during interactions with AR objects
• If the interface elements for indirect manipulations have a fixed place on the screen during the interactions with objects
• Objects continue to be visible during interactions with them like scaling, rotating, position/placement changes, etc

The quality of the object interactions directly impacts the usability of the app, ensuring that virtual objects align precisely with the physical surroundings.

3. Integrity and Reliability/ Measurement Accuracy

During the testing, QAs document the minimum and maximum error of measurements if the app processes quantitative data obtained from outside with the help of augmented reality technologies.

What can influence the measurement’s accuracy? 

• The shape/form of an area
• The area was modified — adjusted, rotated layouts
• Lightning conditions
• Incorrect calculation logic was used
• Distance to the object

This involves using standardized markers or known objects. QA engineers measure known real-world distances using physical tools to verify the app’s accuracy. The evaluation extends to diverse surfaces — flat, inclined, or irregular terrains — to measure how surface variations impact measurement precision.

4. Presence

“Presence” refers to the extent to which users feel physically and emotionally connected to the augmented reality environment. It assesses how convincingly virtual elements are integrated into the real world, creating a sense of coexistence.

 What do we pay attention to?

1. AR objects do not fall through the surface and do not go beyond the boundaries of the room model.
2. The object stays in the user-selected location.
3. Change of scale of the object is possible. 
4. Object retains its shape and texture when the device camera changes location, at different angles
5. Realistic rendering and visual consistency — 3D models and assets look like a part of the real world and have a realistic design. 

5. Depth

“Depth” in AR applications refers to the perception of distance and three-dimensional space. 

It involves how well virtual models are presented in terms of depth and spatial relationships within the augmented environment.

What do we consider?

1. User interaction depth: Users should be able to interact with the AR model by rotating, scaling, and moving items in three-dimensional space. The depth of these interactions should feel natural and responsive, contributing to a sense of depth and control.
2. Occlusion realism: When placing a virtual sofa, it should realistically appear partially hidden behind a real coffee table, demonstrating accurate occlusion. This enhances the user’s perception of depth and the physical presence of virtual objects.

6. Compatibility

Apple devices running iOS 11 or higher are natively compatible with AR applications, so that could be the entry point to decide from what OS version we start to support. 

In practice, compatibility criteria included device (+ screen orientation, screen size, LiDar, and resolution adaptability) and iOS and Android compatibility. How well the AR app is optimized for both iPad and iPhone devices also impacts leveraging ARKit features for a consistent augmented reality experience. 

7. Environment adaptability

Environment adaptability refers to the system’s/application’s ability to intelligently respond and optimize its performance in diverse physical surroundings. Through the physical context, we understand the conditions in which the app will be used. This can be just an indoor space limited by rooms with all its surfaces and indoor objects, or an outdoor living space, such as a backyard.  Usage models refer to patterns of interactions. That is surface measurements and room planning, AR objects catalog integration and placement, materials selections, etc. Consideration of how the device is held or positioned also affects the user’s field of view and interaction dynamics. 

What do we consider?

1. How does the app work during the user movements and in static conditions?
2. Do shadows, landmarks, or other physical objects interfere with accurate measurements, or surface detection and scanning?
3. If the light conditions change, will the digital objects render the same way as before, or will the image be adapted to the environment? 

WHY CHOOSE MOBIDEV TO BUILD AND TEST YOUR AR PRODUCT

If you have a product idea that requires AR features, MobiDev is here to make it real. As part of our AR consulting services, we will be able to examine the specific requirements of your project, sync them with the needs of the market, and offer a roadmap for the technical implementation of the most effective solution. 

Having vast experience with innovative technologies, our AR experts know how to overcome the limits of existing AR frameworks to create more effective solutions. If you already have an AR product, MobiDev is ready to provide you with qualified QA engineers experienced in testing AR apps to ensure that you give your users the best possible AR experience ever. 

A combination of cross-domain multi-platform AR expertise and quality assurance services is the key to success, so contact us to start a conversation!