Drive a scene in 10 minutes

The lowest-friction path: the MCP server against Claude Desktop or Cursor. No C++ SDK installation required to start.

1

Clone and set up the MCP server

The MCP server lives at src/mcp/ in the raku-runtime repo. It is a Python server — no C++ toolchain needed to start talking to a running instance.

# one-time setup
git clone https://github.com/RakuXR/raku-runtime
cd raku-runtime/src/mcp
pip install -r requirements.txt

2

Point Claude Desktop at the server

Add the MCP server entry to your claude_desktop_config.json. Claude Desktop discovers it automatically on restart.

{
  "mcpServers": {
    "raku-runtime": {
      "command": "python",
      "args": ["path/to/raku-runtime/src/mcp/server.py"]
    }
  }
}

3

Ask Claude to drive your scene

Claude now has 17 native MCP tools and a passthrough to ~9,500 REST endpoints. Spawn objects, query the scene graph, run physics — all from natural language.

# example Claude prompt
"Spawn a rigid-body cube at position (0, 2, 0),
apply an upward impulse of 5 N, and list
the scene nodes after 1 second."

4

Go deeper: link a DLL directly

When you outgrow MCP, link RakuXR.dll or RakuRenderer.dll directly. All 30,738 exports are public symbols — dumpbin /exports shows them all, no black box.

Full Claude Desktop integration guide →

17 native MCP tools — bring your own model

The MCP server exposes a curated set of scene-aware tools plus a universal api_call passthrough that surfaces the full ~9,500-endpoint REST API. Bring Claude, GPT, Gemini, Copilot, or a local model — no model lock-in.

scene_query

Read the scene graph — nodes, transforms, properties

scene_spawn

Spawn entities with components into the live scene

scene_transform

Get or set world-space transforms on any node

physics_apply

Apply forces, impulses, or constraints at runtime

animation_play

Trigger or blend animation states on skeletal rigs

asset_load

Stream assets from disk or URL into the scene

anchor_create

Pin world-locked spatial anchors across sessions

audio_play

Trigger spatial audio at a 3D position

render_capture

Capture a frame or stream from the renderer

script_run

Execute Lua scripting in the scene context

xr_session

Manage OpenXR session, head pose, tracking state

input_inject

Inject synthetic input events for test automation

debug_inspect

Read runtime diagnostics, memory, and frame stats

ui_update

Mutate spatial UI panels and overlay content

network_sync

Push or query multiplayer scene-delta state

splat_load

Load a 3D Gaussian splat as a first-class scene object

api_call

Universal passthrough to all ~9,500 REST endpoints

Full agent-driven scene authorship (spawn, animate, script, anchor — the full loop) is the Phase 2 roadmap target. Today’s 17 tools cover query, spawn, transform, physics, and asset operations. MCP server docs →

Splat-native: captured reality as a scene object

The engine is the substrate for Raku Capture. A 3D Gaussian splat captured with a phone camera becomes a first-class scene object — positioned, anchored, and addressable by any LLM via the same MCP tools as any other entity. The same runtime path runs on XR, desktop, and glasses.

Capture → scene in one pipeline

splat_load streams a captured .spz file into the live scene. The splat inherits the scene graph, physics, and anchor subsystems immediately.

LLM-addressable from day one

Ask Claude to move a captured object, query what’s in the splat, or anchor it to a real-world position — the MCP surface treats a splat like any other node.

OpenXR display path included

RakuXR renders the splat through the same foveated 120 Hz-capable pipeline used for all scene content — no separate viewer.

Learn about Raku Capture →

World-locked, to the sub-millimeter

Sensor fusion feeds scene understanding, which pins anchors that hold their place across sessions and devices.

Inside the runtime

Swipe through the pipeline and the 18 native DLLs that make up the engine — grouped by what they do.

The pipeline

Scan → Simulate → Render → LLM drives

One runtime carries a space from a raw capture all the way to a world-locked, physically simulated render — the same path on XR, desktop, and glasses. An LLM can inject at every stage.

01

Scan

Sensor fusion + reconstruction build a metric scene from camera, depth, and IMU.

02

Simulate

Deterministic physics, particles, and animation run against the reconstructed scene.

03

Render

Foveated, 120 Hz-capable render with HRTF spatial audio, world-locked to anchors.

04

LLM drives

17 MCP tools surface the live scene to any model — spawn, query, anchor, mutate.

Simulation & physics

Make the world behave

The subsystems that turn a static scene into something that moves, collides, and reacts.

RakuPhysics

Rigid bodies, constraints, raycast (1,278 exports)

RakuAnimation

Skeletal blend trees, IK, retargeting (1,900 exports)

RakuAudio

HRTF binaural spatial audio (1,654 exports)

RakuScene

Scene graph, transforms, culling (2,530 exports)

RakuCSG

Constructive solid geometry (61 exports)

RakuGameplay

Gameplay systems & state (2,012 exports)

Spatial & XR

Lock it to the world

Sensor fusion, scene understanding, and the OpenXR layer that keeps content sub-millimeter accurate across sessions.

RakuXR

OpenXR 1.1 session, anchors, head pose (2,704 exports)

RakuRenderer

DX12 + Vulkan, foveated render pipeline (3,677 exports)

RakuSLM

On-device inference, glasses thermal target (666 exports)

RakuAI

Scene reasoning & agent hooks (740 exports)

RakuInput

Hands, controllers, gaze (586 exports)

RakuVoice

STT / TTS / VAD (258 exports)

Platform & interface

Drive it from anywhere

The DLLs that expose the runtime to apps and LLMs — the foundation of 30,738 exports across all 18 modules.

RakuCore

Memory, jobs, math, platform shims (3,818 exports)

RakuNetwork

Scene-delta sync, multiplayer state (1,359 exports)

RakuAssets

Streaming, packing, content pipeline (1,604 exports)

RakuUI

Spatial UI & 2D overlays (1,147 exports)

RakuScripting

Lua scripting & behavior glue (3,649 exports)

RakuEditor

Authoring & inspection tooling (1,141 exports)

PHYSICS — RakuPhysics

Physics Engine

Production-grade 3D physics with rigid bodies, constraints, raycasting, and collision detection. Deterministic simulation with sub-millisecond step times even on mobile XR hardware. Fully addressable via the physics_apply MCP tool — an LLM can apply forces, query collision state, and mutate constraints from a natural-language prompt.

Rigid bodies, joints, constraints, raycasting
Deterministic step — reproducible replays and agent-driven simulation
Sub-ms step on mobile XR hardware
1,278 exported functions in RakuPhysics.dll

Key benefit: Consistent, reliable physics across VR, desktop, and mobile — no per-platform tuning. The same step that runs in headset runs headless in an agent eval.

VFX — RakuRenderer

Particle System & VFX

GPU-accelerated particle system with per-genre VFX presets. Supports emitters, attractors, collision, and LOD scaling. Automatic budget management keeps frame rates locked at 72–120 Hz on XR hardware. The renderer uses DirectX 12 with a Vulkan back-end (RakuVulkan) — both paths export through RakuRenderer.dll (3,677 exports).

Key benefit: Cinematic VFX that scale to the hardware — no manual optimization. The same budget manager that governs particles governs agent-injected scene mutations.

ANIMATION — RakuAnimation

Animation System

Skeletal animation with blend trees, state machines, IK, and retargeting. Supports concurrent animated characters with smooth transitions and runtime blending. The animation_play MCP tool lets an LLM trigger or blend animation states directly — no custom integration required.

Blend trees, state machines, IK, retargeting
Runtime blending — LLM-driven transitions
1,900 exported functions in RakuAnimation.dll

Key benefit: Natural, responsive character animation with zero per-character setup. An agent can direct character behavior the same way a director would — in natural language.

AUDIO — RakuAudio

Spatial Audio

HRTF-based 3D spatial audio with binaural rendering, occlusion, reverb zones, and adaptive music layers. Tuned for XR headset speakers and headphones with minimal CPU overhead. The audio_play MCP tool allows an LLM to place sounds at any 3D position in the scene.

HRTF binaural rendering — natural directional perception
Occlusion, reverb zones, adaptive music layers
1,654 exported functions in RakuAudio.dll

Key benefit: Players locate sounds naturally in 3D space — critical for immersive XR. Agent-placed audio anchors follow the scene graph just like visual objects.

Honest comparison: RakuAI vs Unity / Unreal

We are not a Unity or Unreal replacement. We win where they are weak. This table stays public because developer trust matters more than marketing spin.

Capability	RakuAI	Unity	Unreal Engine
LLM drives live scene (MCP)	Native — 17 tools + ~9,500 endpoint passthrough	Not built-in; requires custom plugin	Not built-in; requires custom plugin
Transparent native exports	30,738 public symbols — link against any DLL directly	Closed runtime; IL2CPP / managed layer	Closed runtime; UObject / Blueprint layer
Glasses-class thermal budget	RakuSLM on-device inference; tuned for sub-5W envelope	Not a design target	Not a design target
OpenXR 1.1 (hand/eye tracking, anchors)	RakuXR — 2,704 exports, full OpenXR surface	Supported via OpenXR plugin	Supported via OpenXR plugin
Splat / 3DGS as a first-class scene object	Native — `splat_load` MCP tool, scene-graph integrated	Third-party packages; not scene-graph native	Third-party packages; not scene-graph native
Editor tooling & asset store	Early-stage — RakuEditor DLL; no graphical editor yet	Industry-standard editor; large asset store	Industry-standard editor; Fab marketplace
AAA console pipeline	Not a current target	Full console certification support	Full console certification support
Documentation breadth	Sparse — Phase 0 (per-DLL references are owed)	Extensive	Extensive
Bring-your-own LLM (no lock-in)	Any MCP-compatible model — Claude, GPT, local	No native equivalent	No native equivalent
Export count verifiability	dumpbin output checked into repo — auditable	N/A — closed runtime	N/A — closed runtime

Full honest comparison (more subsystems, licensing, pricing) → compare.html

Where the runtime stands today

We publish the honest state so you can evaluate the runtime on real facts, not marketing.

Phase 0 — Making the runtime credible & consumable

Shipping and credible: 18 C++ DLLs with 30,738 exported functions (verified via dumpbin /exports, tracked in raku-runtime/dll_exports/ as a CI artifact). MCP server at src/mcp/ with 17 native tools live. OpenXR 1.1 support in RakuXR. NVIDIA Inception member (May 2026). 5 granted US patents (priority July 2012).

Active defect: RakuAssets has an in-progress LNK2001 linker fix. The other 17 DLLs are CI-green; a clean 18/18 build is imminent but not yet confirmed. This notice will be removed once the fix lands.

Still owed (Phase 0 tracks): per-DLL reference documentation; a published DLL capability matrix; the “drive your scene in 10 minutes” quickstart video. First embedded OEM shippers are Phase 1.

Export count note: 30,738 is the documented/main baseline. A local Release build of an unmerged WIP branch shows +46 (30,784) — only the 30,738 figure is canonical for public use.

Optimized for XR — and for the glasses thermal envelope

Every subsystem is optimized for the demanding frame budgets of VR and AR headsets — and for the tighter constraints of smart-glasses hardware where Unity and Unreal do not target.

120 Hz XR Rendering

Foveated rendering pipeline with automatic LOD keeps frame budgets on every headset class.

On-Device SLM Inference

RakuSLM targets the sub-5 W thermal envelope of smart-glasses hardware — not tethered GPU rigs.

Transparent Internals

All 30,738 exports are public symbols. Profile and optimize at the C++ level — no black box.

Start with the MCP quickstart XR Features AI Systems

The engine that powers Raku Capture is built by an NVIDIA Inception member. We use the program’s NGC catalog, partner cloud credits, and DLI training to accelerate the spatial pipeline. Read more →

RakuAI is built on AWS and an AWS Activate participant. Production reconstruction runs on Azure today; AWS GPU capacity (G5 / G6e) is on our roadmap, honestly gated until quota lands.

“AWS” and “AWS Activate” are trademarks of Amazon.com, Inc. or its affiliates. RakuAI’s participation does not imply that AWS endorses RakuAI’s products.

The C++ Spatial Runtime an LLM Can Drive