Agents need maps too

The demo is always the same: an AI agent plans your evening, finds a restaurant, books the table. The demo always ends there, because the next step is the hard one. Which entrance? Which floor? Where does the delivery robot wait, and where does your AR headset anchor the name tag? Language gets you to the decision. Space gets you to the door.

Language models don't know where things are

This isn't a rhetorical jab; it's measured. Yamada et al. built natural-language navigation tasks over simple spatial structures — grids, rings, trees — and found that LLM performance swings wildly with the shape of the space, with errors reflecting genuinely spatial failures.^[1] GPT4GEO showed that GPT-4 carries an impressive amount of factual geography — it can sketch coastlines and rank rivers — but breaks down on fine-grained, application-grade detail.^[2] And MapEval, a benchmark built specifically around map-based reasoning, found that even frontier foundation models trail human performance on tasks a map app user would consider routine.^[3]

The pattern across all three: models absorb a fuzzy prior of the world from text, and a fuzzy prior is exactly what you can't navigate with.

Give an agent a map and it gets better

The counterfactual has also been measured. MapGPT took a zero-shot GPT-4 navigation agent and added one thing: an online, language-formed map of the environment — nodes, topology, relationships — injected into the prompt, plus a planning loop over it. Success rates on the R2R and REVERIE benchmarks jumped by roughly 10 and 12 points.^[4] R2R itself — photorealistic buildings, natural language instructions — has been the standard testbed for this since 2018, and its lesson has been consistent: agents that reason over explicit spatial structure beat agents that don't.^[5]

None of this should be surprising. We don't ask humans to navigate cities from memory of prose descriptions either. We hand them a map.

The last 50 meters

Even with a map, there's a resolution problem. A smartphone's GPS is accurate to about 4.9 meters under open sky — and materially worse next to buildings, which is where doors live.^[6] Five meters is the difference between the café and the bank next to it, between the loading dock and the bike lane. Coordinates end where products actually operate.

The industry's answer is visual positioning: localize a camera frame against a reconstructed model of the place itself. Google's Geospatial API does this against tens of billions of Street View images and returns pose in under a second.^[7] The underlying technique — hierarchical, coarse-to-fine visual localization — has been robust at city scale since at least 2019.^[8] What's still missing is coverage of the places Street View cars can't go: interiors, campuses, courtyards, the last 50 meters.

Scenes agents can read

The representation side has moved fast. 3D Gaussian Splatting renders photoreal reconstructions in real time,^[9] which means the ground layer no longer has to choose between looking like the world and running at 60fps. Pair that with semantics — entrances, storefronts, floors, curbs — and a scene stops being a picture and becomes something a robot can localize in and an agent can query.

Tools, not tokens

The delivery mechanism arrived in late 2024, when Anthropic open-sourced the Model Context Protocol — a standard way to hand agents tools instead of hoping they memorized the world.^[10] Geocode, route, locate, query-a-scene: these belong in the tool call, not in the weights. The model supplies judgment; the map supplies ground truth.

That's the bet behind Rayway. Not a smarter model of the world — a real one, continuously rebuilt from the street up, served to people as pixels, to robots as poses, and to agents as tools. The next demo shouldn't end at the booking. It should end at the door.