Hand tracking in the visionOS simulator, driven by your webcam

ARKit hand tracking does not run in the visionOS simulator. HandTrackingProvider returns nothing, so any app whose core interaction is hands has to go on device for every change. That is a slow loop for a feature you are iterating on dozens of times an hour.

DicyaninMockHandTracking fills the gap. It publishes mock hand poses that your app reads exactly where it would read ARKit anchors, steers them from an on-screen control panel, and (as of 3.0) streams your actual hands in from a Mac webcam over the local network.

One pose source, two backends

The package is built around a single shared controller. Read from it instead of ARKit in simulator builds:

import DicyaninMockHandTracking
 
let controller = MockHandTrackingController.shared
 
controller.leftHandPosition   // SIMD3<Float>, head-relative
controller.rightHandPosition
controller.rightHandYaw       // Float, radians
controller.isPinching         // Bool
 
for await _ in controller.updates() {
    // 60 fps tick, read positions here
}

Put the environment switch behind one type so the rest of the app never branches:

import simd
import DicyaninMockHandTracking
#if !targetEnvironment(simulator)
import ARKit
#endif
 
struct HandPose {
    var position: SIMD3<Float>
    var isPinching: Bool
}
 
@MainActor
final class HandSource {
    #if targetEnvironment(simulator)
    private let mock = MockHandTrackingController.shared
 
    func rightHand() -> HandPose {
        HandPose(position: mock.rightHandPosition, isPinching: mock.isPinching)
    }
    #else
    private let session = ARKitSession()
    private let provider = HandTrackingProvider()
 
    func start() async throws {
        try await session.run([provider])
    }
 
    func rightHand() -> HandPose {
        guard let anchor = provider.latestAnchors.rightHand,
              anchor.isTracked,
              let wrist = anchor.handSkeleton?.joint(.wrist) else {
            return HandPose(position: .zero, isPinching: false)
        }
        let m = anchor.originFromAnchorTransform * wrist.anchorFromJointTransform
        return HandPose(
            position: SIMD3<Float>(m.columns.3.x, m.columns.3.y, m.columns.3.z),
            isPinching: detectPinch(anchor)
        )
    }
    #endif
}

Call sites use handSource.rightHand() and never know which backend produced the pose.

The control panel

Drop the overlay into a window in simulator builds only. It gives you two joysticks, rotation sliders, and a pinch button:

#if targetEnvironment(simulator)
MockHandControlView()
#endif

That covers deterministic testing: park a hand at an exact position, fire a pinch, verify the hit.

Webcam poses

Dragging joysticks is fine for reproducing a case and bad for feeling out an interaction. The WebcamHandRunner macOS app in Examples/WebcamHandRunner estimates hand poses with Vision's VNDetectHumanHandPoseRequest and broadcasts them over _dicyaninhands._tcp. Your app subscribes with one call:

ContentView()
    .task {
        #if targetEnvironment(simulator)
        MockHandTrackingController.shared.connectToWebcamRunner() // localhost:50673
        #endif
    }

The simulator shares your Mac's network, so localhost works with no configuration. On a real Vision Pro on the same Wi-Fi, use Bonjour discovery instead:

MockHandTrackingController.shared.connectToWebcamRunner(bonjourName: nil)

Call disconnectWebcamRunner() to hand control back to the joysticks. Because the webcam writes into the same published state the joysticks do, every existing consumer works unchanged.

Tip: a webcam is one 2D view. Depth is approximated from apparent hand size, and yaw comes from the wrist to knuckle direction. Use it for iteration speed, not metric precision, and validate against ARKit on device.

Gloves

DicyaninHandGlove vendors Apple's Tracking and visualizing hand movement sample (a HandTrackingComponent plus HandTrackingSystem mapping all 27 skeleton joints per frame) as one view, with a simulator bridge to the mock controller:

import DicyaninHandGlove
 
ImmersiveSpace(id: "Gloves") {
    HandGloveView()
}

Configure the look, or load your own rigged USDZ:

HandGloveView(configuration: .init(style: .joints))          // Apple's spheres
HandGloveView(configuration: .init(tracksLeftHand: false, color: .orange))
HandGloveView(configuration: .init(
    style: .model(left: "LeftGlove_v001", right: "RightGlove_v001")
))

On device the gloves follow the real skeleton joint for joint. In the simulator they follow the joysticks or the webcam. Same code path.

Next steps

Add the package at https://github.com/hunterh37/DicyaninMockHandTracking.git (3.0.0+), then run Examples/HandTrackingDemo on a simulator to see gloves, control panel, and webcam bridge wired together.