Moving through immersive scenes on Vision Pro without controllers

Movement is one of the hardest UX problems on Apple Vision Pro. There are no controllers, no thumbsticks, and no established convention for covering distance in an immersive scene. Get it wrong and users get disoriented within seconds. Get it right and the scene stops feeling like a diorama you are standing outside of.

Three approaches are now open source: laser teleport and pinch to walk in DicyaninSceneMovement, a world-anchored joystick rig in DicyaninVirtualJoystick, and hand-tracked thumb input in DicyaninThumbController.

Move the scene, not the camera

The rule underneath all three: visionOS owns the camera. You do not get to move it, and trying to fight that is the first thing that goes wrong.

Instead, parent the entire scene to a single root entity and translate that root. The user stays where they physically are and the world slides around them.

RealityView { content in
    let world = Entity()   // everything in the scene goes under this
    content.add(world)
 
    movement.attach(root: world, in: content)
    Task { await movement.start() }
}

Physical walking still works normally on top of this, because the OS keeps tracking the user inside their real room. You are only offsetting where the virtual world sits relative to that room.

Tip: one root entity, translated. Nesting movement offsets at multiple levels makes the math unreadable and the motion inconsistent.

Laser teleport (DicyaninSceneMovement)

github.com/hunterh37/DicyaninSceneMovement

Point a hand forward, a laser and reticle track the floor, pinch to teleport. Teleport remains the most comfortable option because the transition is instant, and instant motion produces no vestibular conflict at all.

Surfaces opt in. Mark anything walkable with a component, and give it a CollisionComponent so the ray has something to hit:

floor.components.set(TeleportSurfaceComponent())

Use it when the scene has defined ground and the user is covering real distance. It is the safest default for anyone who has not worn a headset much.

Pinch to walk (DicyaninSceneMovement)

github.com/hunterh37/DicyaninSceneMovement

Look at a spot, pinch to drop a glowing orb, and the scene slides you there. This is continuous motion, which reads as more natural and preserves the user's sense of where things are relative to each other, at the cost of being the option most likely to cause discomfort.

Keep the speed conservative and keep it constant. Acceleration and deceleration are what actually trigger discomfort, more than velocity itself.

Registration for both modes happens once at launch:

import DicyaninSceneMovement
 
SceneMovementManager.registerSystems()

Both need hand tracking and world sensing authorization in the immersive space. Neither works in the simulator without mocked hand input.

Joystick rigs (DicyaninVirtualJoystick)

github.com/hunterh37/DicyaninVirtualJoystick

When the experience is a game rather than a walkthrough, a visible control surface beats an invisible gesture. DicyaninVirtualJoystick gives you two grabbable joysticks, mounted on either a flat hand-held pad (Gamepad3DEntity) or a floor-standing arcade pillar (GamepadPillarEntity). Tilt is read out as normalized two-stick input.

The package never reaches into your app. It talks through one seam:

VirtualJoystickBridge.isEnabled = { myControlScheme.usesJoystickRig }
 
VirtualJoystickBridge.output = { input in
    myController.apply(
        leftDirection: input.leftDirection,  leftMagnitude: input.leftMagnitude,
        rightDirection: input.rightDirection, rightMagnitude: input.rightMagnitude
    )
}

The sticks are driven by a targetedToEntity pinch-drag, which means the same input path works in the Simulator and on device with no hand-tracking wiring. That alone makes it the fastest of the three to iterate on.

Thumb as a joystick (DicyaninThumbController)

github.com/hunterh37/DicyaninThumbController

DicyaninThumbController converts thumb position from hand tracking into a virtual joystick: a direction vector plus a magnitude, with a configurable deadzone and max distance. No visible rig, nothing to grab.

import DicyaninThumbController
 
let thumbController = ThumbController(handSide: .right)
try await thumbController.start()

Output is a SIMD3 direction and a magnitude you route into the same movement pipeline as anything else. It depends on DicyaninARKitSession for hand session management.

The tradeoff is discoverability. Nothing on screen tells the user this control exists, so it works best as a secondary scheme or in an app where you can teach it once.

Picking one

• Users new to headsets, or scenes with real distance to cover: laser teleport.
• Short, deliberate repositioning where spatial continuity matters: pinch to walk.
• Games, or anything wanting a tactile control surface: joystick rig.
• Experienced users and hands-free contexts: thumb controller.

Shipping more than one and letting the user switch is usually the right call. Comfort thresholds vary more between people than any single default can cover.

Next steps

Repos: DicyaninSceneMovement, DicyaninVirtualJoystick, DicyaninThumbController, DicyaninARKitSession.

All four are MIT licensed and listed with the rest of the packages on the open source page. DicyaninSceneMovement ships a SceneMovementModePicker if you want mode switching in your UI without building it yourself.