Immersive Swift with ARKit and Augmented Reality - Surface detection with ARKit

In this post, I want to talk about ARKit. let’s explore how to create engaging augmented reality (AR) experiences using Swift and ARKit. ARKit, Apple’s framework for building AR applications, provides powerful tools and features to integrate AR into iOS apps.
Imagine being able to visualize data, images, 3D objects, or effects in tandem with the world around you. It would be short of those experiences happening right before you, but when blended in with your environment in real time, it can seem like it is happening before your eyes.

I’m going to create a simple ARKit app that detects horizontal surfaces (or planes). Then, having that detection in place, I will use that information to display a 3D object in our AR world!
I will also do this in SwiftUI with ARView, which builds a view for our AR world to be displayed through. ARView is primarily a UIKit component. But with a little help from UIViewRepresentable, we can still take advantage of SwiftUI’s goodness (and with less code overall).

Understanding ARKit and RealityKit:

• ARKit integrates hardware sensing features to produce AR apps and games. It combines device motion tracking, world tracking, scene understanding, and display conveniences to simplify building AR experiences1.
• RealityKit, on the other hand, is an AR-first 3D framework that leverages ARKit. It provides high-performance 3D simulation and rendering capabilities for creating visionOS apps or AR apps on Apple devices.

How to do?

Create a new project in Xcode. and choose Augmented Reality App, with this option, Xcode will create the basic setup for you.

Let’s explain what Xcode did.
In the Info.plist, it declared a few attributes for the app. First is Privacy, Camera Usage Description, and then Required Device Capabilities (Item 0 = ARKit).

Then in the ContentView, imported RealityKit and have created a struct conforming to UIViewRepresentable that will wrap an ARView:

import RealityKit
struct ARViewContainer: UIViewRepresentable {
...
}

And the ARView configuration isin the makeUIView.

func makeUIView(context: Context) -> ARView {
    let arView = ARView(frame: .zero)

    // Create a cube model
    let mesh = MeshResource.generateBox(size: 0.1, cornerRadius: 0.005)
    let material = SimpleMaterial(color: .red, roughness: 0.15, isMetallic: true)
    let model = ModelEntity(mesh: mesh, materials: [material])
    model.transform.translation.y = 0.05

    // Create horizontal plane anchor for the content
    let anchor = AnchorEntity(.plane(.horizontal, classification: .any, minimumBounds: SIMD2<Float>(0.2, 0.2)))
    anchor.children.append(model)

    // Add the horizontal plane anchor to the scene
    arView.scene.anchors.append(anchor)

    return arView
}

To create a simple box. In three lines of Swift, Xcode create a box (MeshResource.generateBox), define its surface (SimpleMaterial), and put it together as a prepared ModelEntity, ready to be inserted in ARView.

let mesh = MeshResource.generateBox(size: 0.1, cornerRadius: 0.005)
let material = SimpleMaterial(color: .red, roughness: 0.15, isMetallic: true)
let model = ModelEntity(mesh: mesh, materials: [material])

AnchorEntity will tap into any horizontal plane detected during our session. This will be used for any objects that we want to display in our AR world to decide where it’s located relative to reality.

let anchor = AnchorEntity(.plane(.horizontal, classification: .any, minimumBounds: SIMD2<Float>(0.2, 0.2)))

then attach the new object to the AnchorEntity so it now knows where to rest in our AR world.

anchor.children.append(model)

Lastly, it attached our AnchorEntity to our ARView:

arView.scene.anchors.append(anchor)

One thing about this implementation is, if someone were to walk in front of the camera, our box would still show up in front of the person — even if, distance-wise, the person were standing closer to the camera than the box.

We can account for this difference with a simple configuration!

import ARKit

let configuration = ARWorldTrackingConfiguration()
configuration.planeDetection = [.horizontal]
configuration.frameSemantics.insert(.personSegmentationWithDepth)
arView.session.run(configuration)

ARKit provides the option to include person occlusion in the configuration of ARView. This means that it will detect a person’s presence (could be a whole body or just part) and calculate the distance and location relative to the AR world. It will then compare it to any AR objects, determine which is closer to the camera, and decide what should be shown.

That’s it for this post. in the next post I will try to use a 3D model with ARKit.

To access the code discussed in this post, visit here.