Augmented Reality in iOS Apps: ARKit Development Guide
Augmented reality was considered science fiction in the past. However, it has evolved to become an integral part of the digital experience that we live in today. Brought to fruition with the power of iPhone hardware and software technologies, AR has entered the mainstream for many different real world applications. Thanks to Apple’s software development tools in ARKit, development of advanced mobile AR solutions has never been easier.
Despite unprecedented accessibility and power at the hands of developers, there are still challenges that must be overcome. Let’s talk about how we can use non-standard solutions for real world projects in order to get around technological limitations. By working around these constraints, we can achieve tech solutions for the businesses of our clients.
How ARKit Works
To create an augmented reality experience on an iOS device, three steps must be done: tracking, scene analysis, and rendering. In more detail, AR applications require input from sensors like cameras, accelerometers, and gyroscopes. This data then must be processed in order to determine motion of the camera in the real world. Once this is complete, 3D virtual objects can be drawn on top of the image and displayed to the user.
The most optimal environment for ARKit apps is a well-textured and lit area, a flat surface for visual odometry, and a static scene of motion odometry. When the environment does not meet these requirements, ARKit provides users with information on the tracking state. The three possible states are: not available, normal, and limited.
ARKit has advanced far beyond simply displaying objects on flat surfaces. Vertical surfaces, image recognition, and objects placed with geographic coordinates are all possible now with the latest features of ARKit 5. However, it’s important to consider that every virtual object requires a frame of reference. Without a reference point, virtual objects cannot be displayed in a static location on a user’s screen.
Standard solutions that are available to developers make it easy to add AR experiences to any app. This has resulted in a great deal of AR apps on the App Store. This is a good thing, as it increases user exposure to the technology and helps businesses gain more engagement for their services and products.
However, AR is capable of much more than these standard solutions. To be competitive, your business must be unique. Developing a brand new experience that offers the user something new paves the way to innovation and cornering of the market. Let’s talk about some example cases to demonstrate this in action.
Case Study #1: Augmented Reality in Moving Vehicles
When ARKit was first announced, we began developing a product that would allow users in moving vehicles to transform the environment outside their windows. However, initially, ARKit was not designed to support experiences in motion. Naturally, we ran into some serious roadblocks as visual and motion data didn’t match up. Without a static scene, AR experiences wouldn’t work properly.
This was a very complicated issue to compensate for. Instead of constantly dealing with the motion data, we opted for a simpler solution. With geographic coordinates we could render an object more accurately. For example, while looking out the window of a moving vehicle, AR pins can be displayed showing the names of various locations as you move past. These could be distant cities, buildings, or other landmarks.
Our solution then relied on periodic GPS location updates. We could use routing to calculate intermediate positions and combine this with compass data to correctly place these pins in the scene. This was simpler than what ARKit provided, but nonetheless it worked to our advantage.
Vibration is another obstacle that we encountered in this example. As the image shook, so too did virtual content. In order to correct this, we came up with an image stabilization algorithm. By taking the position calculations from the previous task and adding a phase of data smoothing with the moving average, we could improve the stability of the system. However, this has a major issue: these averages may result in a completely different direction.
If the average of 90 degrees and 92 degrees is 91, the difference in direction is marginal. However, the average of 360 degrees and 2 degrees is 181. This is a very different result that can throw off the correct rendering of virtual objects in ARKit. This also resulted in some lag whenever the user quickly moved their device.
The solution to this issue required conditional adaptation. We interpreted small vibrations to be shaking, while large vibrations meant that the camera had been moved deliberately by the user. By adjusting the smoothing of the data with this in mind, we were able to improve the experience of displaying objects outside the window of a moving vehicle.
This method could be improved for vehicles closer to the ground with ARKit 5’s enhanced location anchors. Introduced with ARKit 4, location anchors use Apple Maps Look Around data to identify landmarks like buildings. This allows for better visual positioning of the device and enables more accurate placement of virtual objects with geographic coordinates. By using landmarks as reference points, virtual objects like text, pins, and other elements can be placed in supported areas with unprecedented accuracy.
Given good enough network infrastructure with ultra wideband technology, it may be possible to use this technology to better determine one’s dynamic position and display AR elements accordingly in dense city areas where GPS coverage is limited. This also would reduce the need for image processing with VPS solutions.
Case Study #2: Scaling Scenes For Augmented Reality Experiences
In order to best optimize AR experiences for users, ARKit must use a scale that matches the real world. This helps to immerse the user and help them believe that the virtual objects on their display could actually exist. Realistic scaling is also helpful for placing objects on surfaces near users.
However, we run into some challenges when we want to place virtual elements above buildings or near surfaces, not directly on those surfaces and buildings. For example, you may want to display a pin above a destination in a navigation app, or some text above a popular landmark. In the past, this required a more complex solution. However, with the latest version of ARKit 5 it’s easy!
The latest versions of ARKit have location anchors, a feature which was also helpful in the moving vehicle example. If we wanted to place text or a pin above a destination, we could simply give the app the geographic coordinates of the location and where the virtual element should be displayed. When the user’s camera spots the landmark and compares it to Apple Maps Look Around data, the virtual object can be displayed on their screen correctly.
However, Location Anchors are not available in all cities and are not yet available in non urban areas, so alternative methods of placement may be necessary like GPS, UWB, or BLE beacons.
No matter what method is employed for placement, the scale of the virtual object must always be proportional to the distance from the user. Naturally, virtual objects that are farther away must be made smaller in the user’s view, while closer objects are larger.
However, when solving this problem for ourselves, we decided to transform coordinates with a projection on an invisible sphere. This helped when displaying coordinates that were very close to each other on the map.
Case Study #3: Indoor Navigation with Augmented Reality
Getting around has never been easier with the technologies that we have available to us. When outside in cities, GPS technology’s unprecedented accessibility has made getting lost almost impossible. Augmented reality experiences have been crafted so that users can hold up their mobile device and get on-screen directions in the form of virtual elements drawn over the real world.
However, indoor navigation with AR is not so simple. Without GPS as widely available, it can become easy to lose one’s place inside a large indoor environment like a shopping mall, convention center, or airport terminal.
Although major mapping apps primarily focused on outdoor GPS navigation have begun to create extensive indoor maps of major areas, there is still a major problem: scene positioning. GPS isn’t accurate enough for indoor environments to make it reliable for positioning a user for augmented reality navigation. Without an accurate position, drawing AR directions in the real world is much more difficult.
Using Innovative Technologies for Indoor Positioning
In order to get around this problem, we cannot rely on GPS. Other technologies, like Bluetooth Low Energy (BLE), Wi-Fi Round-Trip Time (RTT), and Ultra-Wide Band (UWB), can help us achieve a far more precise location for augmented reality navigation. Once a user’s position can be found, ARKit and the rendering engine can take care of the rest.
By embracing an IoT solution with BLE beacons, regardless of technology chosen, AR navigation becomes possible. By triangulating the position of the device through use of beacons, users can find their way through a building. However, there are some obstacles to success. Developers must ensure that these beacons are placed in such a way that they can best serve users. Depending on the technology used, interference can still be a problem. For example, UWB signals cannot pass through walls, people, plants, and more.
Triangulation is necessary for indoor positioning. As long as the user is within range of three beacons, the user’s position can be ascertained. However, this can be challenging given that there may be obstacles that can impede the system. Maintaining each beacon is necessary as well.
Image Recognition & Machine Vision
Beacon technology can be expensive in some conditions. It requires an investment in infrastructure that may not be desired for a project. To solve this problem, we created a solution that did not use any kind of beacon technology.
Our solution works similar to how ARKit location anchors query a database of Apple Maps Look Around images to find matches. Instead of looking for buildings as landmarks, we used visual markers. These could be QR codes or any other kind of marking easily identified by a computer. These markers communicate location metadata to the mobile device when scanned. These may be placed on walls or floors.
Once a marker is scanned, the mobile device understands its 3D coordinates. When used in this way, visual markers eliminate the need for expensive equipment and routine machine maintenance. They may need to be cleaned and objects may need to be moved away from them to not obscure them from view.
Rendering Paths with Augmented Reality: The Importance of Occlusion
In most ARKit applications, rendering the content is the easy part. There is one important consideration in indoor navigation, and that is route occlusion. For example, when a drawn route goes around a corner, we would expect that the path would be occluded by the wall. When it is not occluded, users can become confused.
Showing the route with an arrow instead of a drawn path to guide the user is a possible solution to this issue, but not as convenient. We could also simply show the route within a certain distance from the user so that it does not extend too far away around corners. Both solutions are much easier to implement but may not be the best options.
To take this a step further, we chose to develop a very low-polygon 3D model of the building. By overlaying this model over the top of the real world as seen through the user’s camera, occlusion can be better supported. Although this requires more time and effort in development, it allowed for a much more natural and understandable indoor navigation solution with ARKit. We believe that this is the most cost effective solution while providing the highest quality for users.
Best of all, creating the 3D model of the building is easier than ever with advancing technology. The iPhone 12 Pro’s LiDAR scanner can be used for this purpose as well, allowing for the scan of entire rooms into 3D environments.
Check out the video below to see indoor navigation routing technology in action!
Case Study #4: Face Based Augmented Reality
One of the advantages that ARKit has over ARCore is hardware compatibility and quality. iPhone TrueDepth cameras since the iPhone X have been capable of a very consistent and high level of quality for face-based augmented reality. Similar to other kinds of AR, this technology works in three stages: tracking, scene analysis, and rendering.
ARKit’s data processing results in tracking data, face mesh, and blend shapes. Tracking data gives us a better idea of where to display the content as the subject face moves. Face mesh generates geometry of the face. Blend shapes are parts of the face that have analog information. For example, if the eyes were half open, the result would be 50% for that blend shape.
Face-based AR is helpful for face masks that are commonly used in apps like Snapchat, as well as animated virtual faces like Animoji. However, we can take this a step further. Face-based AR can help users track their weight loss.
It may seem like a stretch, but it is a viable possibility. When a person gains and loses weight, their face varies in width. An iPhone True Depth camera can detect these variations and track them over time with an app. This can allow users to track changes in their weight loss efforts over time.
One solution that is rising in popularity are virtual fitting rooms. Face-based AR can help consumers try on products at home like sunglasses and makeup without having to leave the house. Also, face-based AR can be helpful for controlling on-screen elements through using blend shapes like eye motions. With a little imagination, face-based AR opens up a great deal of possibilities with ARKit solutions.
Future of ARKit Development
Imagination is really the most important part of developing augmented reality solutions with ARKit. Although its commercial, functional, and business applications may not be immediately obvious, its applications in indoor navigation, facial recognition, and display of information based on geographic coordinates shows us that AR has incredible potential. Many useful applications of AR may not have even been thought of yet. Time will tell what the future of AR will look like.
It’s up to you to decide what you want to explore to keep your business competitive in an innovative market. Augmented reality services by MobiDev are here to help you find the solutions that you need to gain that edge.