Engineering notes, architecture decisions, and behind-the-scenes from the RakuAI team.
Scan a real room with your phone — no rig, no markers, no install — and get a 3D reconstruction your AI assistant can read, measure, and reason over via MCP. GPU splat training arriving within days. MCP is on every tier. This is the Raku Capture story.
Phone scan to 3D Gaussian splat to any LLM driving it as tools over multi-vendor MCP. The honest end-to-end account: what ships today, what is gated on GPU credits, and why the inference-only runtime is the moat.
RakuAI was accepted into NVIDIA Inception. We are using the program's NGC catalog, partner cloud credits, and Deep Learning Institute training to bring NVIDIA-accelerated 3D Gaussian splatting (the Brush trainer on A10G / L40S) to Raku Capture — so a phone scan of any room turns into an explorable splat in minutes instead of an hour.
The runtime your AI assistant inhabits when it needs to build, query, or simulate a spatial world. Five patents going back to 2012. Built for the thermal envelope of glasses. Multi-vendor by design. Here's why.
The runtime has been speaking Model Context Protocol since late March. Six tools, stdio transport, deny-by-default permissions, full audit log. The server is the boundary any external agent uses to drive the engine. This Saturday is about what the next layer looks like: real adapters, real partner integrations, and an MCP-first developer story instead of an MCP-bolted-on one. This is the MCP runtime LLM makers build against.
A run of cleanup commits that look unglamorous on the diff and feel transformative on the project. Eighteen DLLs green on Linux. Every LLM caller guarded against runaway prompt length. The /api/v2/ namespace renamed to /api/raku/ everywhere it appears. Null-pointer safety swept through the runtime. The build is finally something you can stop arguing with.
Late March 2026 was the GDC eve release and a two-platform mobile push that landed in the same week. Here is what shipped, what broke forty-eight hours before I flew out, and which fixes look small on the diff but kept the laptop demo from going sideways in front of strangers. This is what shipping a spatial runtime at conference speed actually looks like.
Six weeks of feeding Meshy.ai through a seven-stage asset pipeline taught the team three things the API docs do not. Robes obscure the silhouette pose estimation needs. Japanese character names crash anything that pipes stdout to a file. And Meshy's remesh endpoint can vanish on you mid-sprint. This is the unglamorous reality of turning AI-generated assets into production-grade content for a spatial runtime.
A 15x cost spike led to an audit of every place the runtime talks to an LLM — and five callers that could send unbounded prompts with no cap, no truncation, no defense. By Saturday afternoon every caller went through one shared guard, with per-call budgets, telemetry, and a CI check that blocks regressions. This is how a spatial runtime makes prompt-length discipline impossible to forget.
347 untagged functions across eighteen native libraries, hidden by a symbol-visibility default, breaking the Linux build one undefined-reference at a time. Traced to root, swept clean, and locked behind a CI guard that fails any PR that drops a public symbol. Linux is now a first-class target — and the codebase is more honest on every platform.
Eight tests, one Windows error code, one missing shader-compiler DLL the runtime could not boot without. The fix turned a hard load-time dependency into a graceful, observable fallback — so the engine runs anywhere, not just on a fully-equipped developer workstation. Portability is a feature partners pay for.
An enterprise partner's security team ran due diligence on our API surface and came back with eight critical and high findings. Every one had a fix landing by Saturday night. Here are the categories, the new permanent guardrails, and the failure modes an agent-driven runtime has to defend against on purpose.
Five agents, five branches, one codebase, one weekend: 112 commits. The bottleneck is no longer typing — it is deciding which four problems are worth four parallel attempts, and which two results are good enough to keep. The job stopped being senior IC and became tech lead of a team that never sleeps.
Claude wrote eight subsystem PRs. Gemini reviewed them and left thirty-six comments — two of which were real thread-safety bugs that would have shipped. This is the multi-vendor review pattern earning its keep: different model, different blind spots, harder-to-fool code. The discipline signal serious partners should ask every engine team about.
Your experience is a file, not a binary. The .raku format is JSON — schema-versioned, diffable, hot-reloadable, with explicit hooks where the runtime AI plugs in every frame. Here is what a real one looks like, why JSON beats a custom DSL, and how 'game design reviewed in a pull request' became our actual workflow.
Seven O(N²) placeholders, one Saturday, and the production algorithms that turned a wobbling demo into a runtime that screams. This is how RakuAI turns AI-written code into engine-grade performance — and why an algorithmic audit is the fastest upgrade your spatial stack will ever get.
A deep audit pass turned up two things I did not enjoy: an over-broad find-replace that had quietly garbled identifiers across three hundred files, and a hardcoded licensing secret in a public source file. Both fixed in a day. This is the honest field guide to the failure modes an agent-driven codebase is most exposed to — and the guardrails that catch them.
54% to 100% in three weekends. The test runner does not care about your feelings, your star count, or your roadmap — it tells you exactly how much of your spatial runtime actually works. Here is how we walked the gap, root-cause by root-cause, and why pass rate is the most honest signal a partner can ask for.
Worse than a crash: a demo that did not crash, did not error, and also did not work. Dead input. A placeholder scene wearing the costume of a finished game. Logs that said nothing. This is the Saturday I made the engine refuse to fail silently — the loud-failure, grep-able-error, audit-by-adjacency discipline that keeps a spatial runtime honest with the people building on it.
If AI is a runtime primitive inside your engine, are you locked to whichever model you wired in? No — the AI layer is exposed through interfaces, not imports. Six subsystems, a model-management API, a file format that names capabilities instead of models. Here is the boundary that lets RakuAI swap any model, any vendor, any inference path without touching a line of caller code — and why it keeps mattering more as the model market churns.
Four unrelated bugs, zero overlapping subsystems, one Saturday. Pink missing-shader graphics. A score that double-counted. A hard XR-boot crash. A camera that refused to lock. This is the kind of integration failure that escapes every unit test — and the init-order, event-ownership, and boot-guardian discipline that turns a broken demo into a runtime partners can trust.
Most engines that say 'AI-native' in 2026 mean they bolted on a chat panel. RakuAI is built differently — AI is a runtime primitive in the loop on every frame, closer to a nervous system than a factory. If you read one piece on what 'AI-native' actually means at the architecture layer, make it this one.
139 commits overnight. Ten foundational subsystems shipped across one weekend, mostly by an autonomous agent running against a queue I keep filled. This is what it actually looks like when an engine's foundation is built BY agents, not bolted on FOR them — and why that choice shapes every architectural decision in the runtime your AI will inhabit.
A 276-commit weekend that landed TensorFlow Lite for on-device SLM inference, AES-256-GCM federated sync, OpenXR composition layers, Kalman-filtered anchor stabilization, and a hardware-abstraction layer for the optical channel. This is the weekend RakuAI stopped being a research project and grew the surface area of a real platform — the bones smart glasses and AI labs build on.
Two hundred errors, six categories, twelve PRs, two days — and the first green x64 build in the runtime's history. This is the inside of bringing a cross-platform spatial engine onto MSVC 2026 Insiders, the six error classes that will hit any C++ codebase, and the agent-driven sweeps that fixed them. Three platforms, one runtime, ready to ship on Windows.
Forty-seven functions claimed to do real work. Forty-seven were returning placeholders. This is the Saturday I went looking for the stubs the agents had quietly landed — and built the audit, the tests, and the discipline that turns agent-coded into agent-trusted. The honesty that makes a spatial runtime safe to build on.
Three months in, the dev workflow has settled into a multi-vendor rhythm: Claude writes the runtime, Gemini reviews the diffs, ChatGPT rubber-ducks the architecture, Copilot autocompletes. The single most important rule — the model that writes a PR cannot be the model that reviews it. This is the workflow building an AI-native runtime, and the posture the runtime itself takes toward models.
A real bet on standards this weekend. By Sunday night the runtime had its OpenXR backbone — graphics binding, frame lifecycle, action sync — plus a dual-mode RF / optical link manager that switches radios on the fly without the app ever noticing. Adopt the standard where it exists; build where it doesn't; be ready to adopt again when it catches up.
The weekend the engine learned to take direction from any cloud LLM — as a runtime concern, not a chat panel. By Sunday the XRAssistantService streamed model intent through the voice pipeline, eye-tracked foveation shipped, and Wi-Fi 7 offloaded rendering had a real host. ChatGPT can drive it. Claude can drive it. Gemini can drive it. That is the whole point.
Meta Quest is now a Raku target. OpenXR on Horizon OS, passthrough AR with composition layers, stereo rendering and 6DoF tracking — the engine now reaches one of the largest installed bases in spatial computing. Meet your users on the hardware they already own, then carry the same experience to glasses for free.
The placeholder name finally came off. In a weekend, agents swept every reference across every repo from Blade and ar1-runtime to Raku — no build break, no compatibility break. The name means ease, comfort, and the human touch. The engineering finally caught up to the brand that was waiting for it.
Two months of a public engineering log, years of work under the hood. A quiet Saturday to take stock: a cross-platform C++ AR runtime with AI baked into the simulation step, an architecture that has settled, and a December production-readiness milestone in sight. This is what an AI-native spatial runtime looks like when it stops being a mission statement and becomes a repo.
Thirty-seven commits instead of eighty — on purpose. I cut the agent queue and slowed the pace so I could actually read what was landing, and the codebase came out cleaner for it. Here is the discipline that keeps an agent-built runtime coherent instead of a pile of clean-in-isolation PRs.
A brutal precision target from a partner conversation: sub-millimeter overlay drift on a sheet of calligraphy paper. By Saturday evening the runtime had a sub-millimeter anchor-pose API, real-time pen-tip tracking, and marker-based paper detection. This is the kind of precision that turns AR from a toy into a tutor.
Pivot the product target from AR1+ to AR2 Gen1, then ship the thermal-validation, Wi-Fi 7 stability, and sensor bring-up harness for hardware that isn't on the desk yet. When the silicon arrives, validation begins on day one instead of day ninety. This is how RakuAI ships on your hardware faster than its competitors - the expensive part is already paid.
Every multi-repo project dies the same way - one repo races ahead, the other quietly rots. This Saturday wired Unity and Unreal HelloAR to behave identically and built the parity-testing gate that makes the gap impossible. This is the discipline that keeps a spatial runtime and its bindings coherent for years, so your binding choice never locks you out of a feature.
Two days at the keyboard, autonomous agents grinding the issue queue underneath, and 282 commits in the runtime repo by Sunday night. The main loop, the latency tracker, the C API that lets the SDK link in - here is what shipped, what broke, and what the agent-driven loop looks like when it finally clicks. This is what it means to supercharge a one-human runtime team.
Day zero of the public log: repos live, agents working, and a smart-glasses spec on the desk as the forcing function for every decision downstream. The bet is simple - the most interesting AI experiences are spatial ones, anchored in the real places you stand in. This is the engine being built explicitly to take direction from your model, every frame.
The last Saturday before the repos open - three months of design work behind it. Agent roster locked. Demo suite specified. SDK folder structure drawn. API surface drafted. A decade-old patent estate ready to be the foundation. This is what it looks like to design a spatial runtime to fit its team before a single line ships.
The folder structure is the org chart of the code - get it right early and every PR for two years lands in the right place. This Saturday drew the SDK's skeleton before any code existed: /apps, /modules, /assets, /docs, /config, plus module stubs that lock the contracts your agents build to. This is the discipline that keeps an AI-built spatial runtime coherent for years.
Eight canonical demos that prove what a spatial runtime is for - arena multiplayer, AR coaching, HUD designer, streamer mode, point-of-service overlay, companion HUD, aim trainer, multiplayer HUD sync. Zero code written, every one specified. The demos are the spec, and they are how RakuAI shows what your AI can do once it inhabits the real world.
Before a single line of the spatial runtime existed, this Saturday mapped the agent fleet that would build it. Product, SDK, Studio, Marketing, Operations, AI strategy, plus a governance layer watching the watchers. This is how you supercharge a one-human team into a shipping engine company - the agents come before the architecture because the agents shape the architecture.