The Intelligence Layer: EmbodiedAgents Roadmap#

Shipped in 2026

Agentic Layer — Cortex & Memory

  • v0.5.0 (Jan 2026) — Agentic Semantic Router — LLM-based routing, runtime-switchable. Event-driven model fallbacks. Generic HTTP STT/TTS.

  • v0.5.1 (Feb 2026) — Native Vision Actions (record_video, take_picture). Router extended to route lists of topics and Actions. Ollama think toggle.

  • Cortex — planner-executor agent (PR #25, Apr 2026, shipped via EMOS 0.7.0) — natural-language task composition with Action lookup and chaining. Consumes the roadmap’s “NL Task Composition” bucket entirely.

  • Memory component (PR #28, May 2026) — spatio-temporal graph indexed by meaning, location, and time. Single-file persistence. Consolidates episodes into long-term gists. Deprecates the earlier MapEncoding approach. This one component absorbed the roadmap’s three separate items on state aggregation, memory, and hierarchical spatio-temporal organization.

  • Cortex ↔ Robot-Plugin integration (PR #32, Jun 2026) — plugin actions become LLM tools; robot identity injected into prompts.

  • MemLayers (PR #35, Jul 2026) — prior memories exposed via a layered API; UI/API adapted to Sugarcoat’s front-end-agnostic API.

Manipulation Stack (VLA)

  • v0.5.0 — VLA components — LeRobot integration (LeRobotClient, LeRobotPolicy), SmolVLA checkpoints from HuggingFace, URDF-driven joint-limit safety, MoveIt Servo + ROS2 Control action-server.

  • VLA stack refresh (PR #37, Aug 2026) — support for latest LeRobot policies, local-models stack modernization.

Local Model Layer

  • Built-in local models across LLM / VLM / STT / TTS / Vision (Qwen3, Moondream2, Whisper, Kokoro, DEIM) with llama-cpp-python, sherpa-onnx, and onnxruntime backends. Shipped via EMOS 0.6.0. Zero-config fallback_to_local().

Perception

  • The agent stack consumes rich perception streams — RGBD, Detections, POI — with Detection2D and object tracking wired directly into LLM/VLM components, and supports StreamingString for progressive text and UDP audio streaming, all exposed through the auto-generated web UI.

  • MotionDetector component (PR #36, Jul 2026) — motion detection from images and point clouds.

In Flight — Q4 2026 → Q1 2027

Manipulation Expansion (new lane — reflects the shipped VLA momentum)

  • Additional VLA policies and dexterous grasping recipes.

  • More embodiments beyond SmolVLA — arm + mobile-manipulator combos and humanoid platforms.

  • Deeper integration between Cortex agentic planning and VLA execution.

Data Collection & Compute

  • Universal Data Collection — standardized “Collect Data” hooks on all components to facilitate dataset generation for future model fine-tuning. (Vision partial — record_video, take_picture already shipped.)

  • NPU Deployment Harness — auto-compilation harness for NPUs on components that run local models (Vision, STT). The onnxruntime backend is already shipped as prerequisite.

Q2 2027 → Q4 2027

  • Isaac Sim Extension — dedicated extension for NVIDIA Isaac Sim, allowing developers to build, test, and validate EmbodiedAgents logic in high-fidelity virtual worlds.

  • Structured Decomposition & Verification — structured decomposition of LLM / VLM output for verifiable reasoning traces and formal-verification guarantees. May incorporate the URML RFC (embodied-agents#33) from external contributors. Complements the Formal Recipe Verification initiative on the EMOS side.

  • Collaborative Multi-Agent Discourse — multiple robots share mission goals, exchange semantic memories, and coordinate complex multi-agent tasks through new communication protocols.