1
Capture
Scan a room with your phone.
What you saw
You pointed a phone camera at a room and captured it. No app install, no depth sensor, no rig — it runs in the browser.
The component
Raku Capture — a Progressive Web App that does the capture-side camera and frame work on your device. Open it directly at rakuai.com/capture-app/.
Build on it
Turn any camera into an input to a world model — the phone in your pocket today, smart glasses and robots on the same runtime next.
2
Reconstruct
The room comes back as a 3D scene — drawn on your device.
What you saw
Your capture reappears as a 3D Gaussian-splat scene you can orbit and inspect, rendered live in the browser.
The component
On-device splat rendering. WebGPU by default, WebGL2 fallback — the client tier is compute. No video is streamed from a server; the pixels are drawn on your own GPU.
Build on it
Ship spatial scenes to any WebGPU / WebGL2 device without a render farm or a per-frame streaming bill.
3
Understand
The scene is parsed into objects — now with names.
What you saw
The splat resolves into discrete objects — the table you shoot over, the wall behind the hoop. The in-game HUD reads it back honestly:
OBJECT COLLISION · N cells · N obj
The component
HP-6 object collision builds voxel occupancy from the splat centers and clusters it into per-object 3D bounding boxes. On top of it, a template-driven scene-object detector matches those clusters against class templates — ground, table-like surfaces, wall planes, screen/TV rectangles, columns, seat-height objects — using connected components, robust plane fits, and occupancy statistics. Purely geometric: no learned model, deterministic, testable.
Honest labels
Every label claims only what geometry can establish. The detector says "seat-height surface", never "couch" — a storage crate at seat height matches identically, and pretending otherwise would be fake success. A cluster that matches no class is labelled exactly that: an unclassified object. Nothing is ever invented; an empty room detects nothing.
Build on it
Ask a scene what's in it: occupancy grids, per-object bounding boxes, and class-tagged detections an app or an agent can query.
4
Simulate
Real physics against your room — in three playable games.
What you saw
The ball bounces off the actual furniture you captured. This isn't a canned level — the level is your capture.
The component
Physics runs against the object map derived from your scene, using one versioned collision model — the RakuSplat contract v1. A class→physics bindings registry maps each detected class to a collider shape and material feel, so a ball rebounds differently off a detected table than off a wall plane. A class the registry doesn't know gets a fallback marked as generic — never passed off as class-specific tuning.
The games
Three games ship in the capture app today: RAKU HOOPS (with smart hoop detection that finds a real hoop in your scan), SPLAT SHOT, and STACK. All three now target the real furniture the detector found — a detected table or seat is a game surface, not just an anonymous voxel cluster.
Build on it
Games, training simulations, and robot planning that run against real captured spaces instead of hand-authored levels.
5
Drive with AI
Any LLM can drive the world over MCP.
What you saw
The world is drivable by an AI agent — MCP-first and BYO-LLM. An agent connects its own model to mcp.rakuai.com and acts through tools, not a bespoke integration.
The component
Live MCP server with a versioned 17-tool native core contract (5 read-only + 12 mutation) plus a wider tool registry. The capture app now publishes every detection into the scene store that the list_scene_objects tool reads — so an MCP-connected agent sees the real table, TV, or seat from your capture, class-tagged with its label and confidence, instead of an empty scene. When nothing was detected, it reads back an honest empty list.
Teach it
The key unlock: agents can bring their own object classes. Via set_scene_object_library / get_scene_object_library, an LLM describes a class as a geometry template — a "crate", a "planter" — against the same frozen template schema the built-in library uses. The capture runs the new template and finds a match only if one is really there; then the agent reads back what's actually in the room. Templates are validated on arrival, and a malformed library fails loudly instead of being silently skipped.
Build on it
Claude, ChatGPT, Gemini, Copilot — bring your own model and drive the same spatial tools. No server-side LLM, vendor-neutral by design.
Where MCP control stands, honestly: the full loop — capture detects
real objects → the browser publishes them → an agent reads them via
list_scene_objects, including the agent-provided-template round-trip —
is merged and was proven end-to-end at the real-local tier before shipping:
a real detector, a real store, real MCP reads, verified on a local stack rather than
assumed from the code. Two limits we state plainly: detection is geometric, so labels
claim shape classes ("seat-height surface"), never semantic identity ("couch"); and if
nothing matches, the agent reads an empty list — a detection is never invented to make
the demo look better. Fake success is forbidden here.
6
Native core
The same math, browser and native — proven identical.
What you saw
The collision math you just played runs in the browser — and the exact same contract runs in a native C++ core.
The component
RakuSplat contract v1 — one JSON-Schema contract, three tiers. The browser JS reference and a native C++/Linux core (libraku_splat.so) are proven bit-exact by a 74-check conformance suite (integer outputs exact, floats within 1e-6). The header is the single source of truth, ABI drift-gated.
Build on it
Pin to one versioned schema and get identical spatial semantics from browser to native — port a tier without silent divergence.
Where the native core stands, honestly: the RakuSplat v1 contract, the
browser JS reference, and the native C++/Linux core are shipped in the open repos, and the
conformance suite proves they agree bit-for-bit. Native activation is rolling out tier by
tier (Rung 1) — we are not claiming production endpoints are native-served yet.
When a stage isn't ready, it says so and fails loudly. Fake success is forbidden here.