Exa-LLM-Draw!

A magic whiteboard with a resident multimodal LLM agent — voice + typing + drawing, all at once — that co-brainstorms with you and presents polished animations on demand.

Status: Pre-implementation. The architecture is frozen; the implementation checklist is the contract. See DOCS/ARCHITECTURE.md and CHECKLIST.md.

License: Proprietary — All Rights Reserved. Derivative of Excalidraw (MIT). See LICENSE and NOTICE.

---

What this is

Exa-LLM-Draw! is not a chatbot glued to the side of a canvas. The LLM is an equal participant in the drawing — it observes the scene, listens to the room, reads what you're typing, and mutates the canvas with the same primitives the human toolbar uses. The result: a whiteboard where the AI is a real collaborator instead of a sidecar.

The product runs in two explicit modes:

Brainstorm mode

Low-latency, collaborative, intimate. The agent is a peer. Voice + typing + drawing flow into a single multimodal context. Gemma 4 (multimodal) runs on-device via LiteRT where hardware permits, with a light Qwen 3.5 fallback for edge devices and an OpenAI-compatible HTTP adapter for everything else. Target first-token latency: ≤ 400 ms on capable hardware. The agent watches the scene, contributes suggestions, and edits the canvas directly when the user asks.

Presentation mode

The agent generates polished, animated reveals from a prompt. It scripts a sequence of scene mutations — add a box here, draw an arrow there, highlight that region — and the AnimationScheduler plays them back step by step along a visible rail. The audience sees a clean, choreographed build; the presenter narrates on top.

Multimodal input

Voice, typing, and drawing are all inputs at the same time. The MultimodalContextComposer stitches live transcript, audio frames, text buffer, and debounced scene deltas into a single MultimodalInput before every turn. The goal: nurture and cultivate the energetic enthusiasm that is itself central to a productive session — never force a context switch.

---

Platform targets

Platform	Form factor	Shell	Status
Windows x64	Desktop	Tauri 2	Primary (50% of target usage)
Android	Tablet/phone	Tauri 2 Android	Primary (50% of target usage)
Linux	Desktop	Tauri 2 (AppImage + .deb)	Secondary
Web	Browser	Vite SPA	Secondary
PWA	Installable web app	Same bundle as Web	Secondary

No Apple targets. All five targets ship from a single Vite frontend bundle; the Tauri shell swaps the runtime host without changing the UI code.

---

Architecture in one paragraph

Every scene edit — whether from a human toolbar action, an LLM tool call, or a scheduled animation step — flows through the same typed discriminated union Mutation, dispatched by a single MutationDispatcher, applied by a pure reducer (applyMutation), and broadcast to subscribers (Excalidraw core, collab bridge, stream meter). The agent layer (AgentRuntime + ModelRouter

• ToolRegistry + ProviderAdapter) emits Mutations as tool calls. The animation layer (AnimationScheduler) emits Mutations on a timeline. The user toolbar emits Mutations directly. One pipeline, one source of truth for scene state, and every observer gets every change.

Read next:

• DOCS/ARCHITECTURE.md — authoritative design, entity table, all 15 architectural decisions, all 20 open questions with resolutions.
• DOCS/ARCHITECTURE/components.puml — top-level component view.
• DOCS/ARCHITECTURE/agent-runtime.puml — class view of the agent layer.
• DOCS/ARCHITECTURE/brainstorm-sequence.mmd — brainstorm-mode turn.
• DOCS/ARCHITECTURE/presentation-sequence.mmd — presentation-mode animation turn.
• DOCS/ARCHITECTURE/mutation-dataflow.mmd — mutation lifecycle and collab fan-out.
• DOCS/ARCHITECTURE/multimodal-input-flow.mmd — how voice, scene, and text compose.
• DOCS/DEVOPS-DASHBOARD-SPECIALIZATION.md — concept brief: DevOps infrastructure dashboard as a brAIstormer profile.
• DOCS/PRODUCT-FAMILY-VISION.md — strategic vision: the four-product family (brAIstormer, CanvasOps, CanvasFlow, CanvasDocs).

---

Repository layout

excallmdraw/
├── LICENSE                         Proprietary — All Rights Reserved
├── NOTICE                          Third-party enumeration
├── LICENSES/
│   └── excalidraw-mit.txt          Preserved upstream MIT License
├── README.md                       This file
├── CHECKLIST.md                    Deterministic build plan (source of truth)
├── CLAUDE.md                       Agent collaboration notes
├── version.txt                     Single cross-platform version string
│
├── DOCS/
│   ├── ARCHITECTURE.md             Authoritative architecture + entity table
│   └── ARCHITECTURE/               Sibling diagrams (PlantUML + Mermaid)
│
├── exa-llm-draw-app/               The product — Vite + React frontend
│   ├── src/
│   │   ├── brand/                  BrandConfig + A/B variants
│   │   ├── platform/               Platform detection + TauriBridge
│   │   ├── agent/                  AgentRuntime, ContextComposer, BargeIn...
│   │   ├── providers/              LiteRTAdapter, OpenAICompatibleAdapter
│   │   ├── multimodal/             VoiceCapture, AmbientSceneObserver,
│   │   │                            MultimodalContextComposer
│   │   ├── scene/                  Mutation, MutationDispatcher, apply.ts
│   │   ├── animation/              AnimationScheduler, PresentationRail
│   │   ├── modes/                  ModeController (brainstorm/presentation)
│   │   ├── settings/               ProviderConfig, SettingsStore
│   │   ├── catalog/                Shape library loader (AWS, Azure, flowchart)
│   │   ├── collab/                 AgentPeerBridge (Firebase-backed)
│   │   └── ui/                     ChatPanel, ModeSwitcher, ProviderPicker...
│   └── public/
│
├── src-tauri/                      Tauri 2 shell (Rust)
│   ├── src/
│   ├── Cargo.toml
│   ├── tauri.conf.json
│   └── icons/
│
├── packages/                       Inherited Excalidraw component library
│   ├── excalidraw/                 The React component (MIT, upstream)
│   ├── common/
│   ├── element/
│   ├── math/
│   └── utils/
│
├── excalidraw-app/                 Upstream reference app (retained for parity)
│
└── .forgejo/                       Self-hosted CI workflows
    └── workflows/

Note: the downloadable shape libraries (AWS architecture icons, Azure icons, flowchart symbols, etc.) are preserved from upstream Excalidraw in their original .excalidrawlib form and remain available through the catalog loader. These are the "library" in the product sense — pre-made shape collections you drop onto the canvas.

---

Development quick start

Prerequisites

• Node.js ≥ 20 (Node 24 LTS preferred)
• Yarn (this repo uses Yarn workspaces)
• Rust stable (for Tauri shell builds — desktop/mobile)
• Android SDK + NDK (for Android builds)
• cargo-xwin (for cross-compiling Windows from Linux CI)

Install

yarn install

Run the product (web / PWA)

yarn workspace exa-llm-draw-app dev

Opens the app at http://localhost:3000. This is the same bundle the Tauri shells load, so UI changes are visible immediately without rebuilding Rust.

Run the product (Tauri desktop — Linux / Windows)

yarn tauri dev

First run compiles the Rust shell (slow). Subsequent runs reuse the compiled binary and hot-reload the frontend.

Build release artifacts

Target	Command	Output
Web bundle	yarn workspace exa-llm-draw-app build	exa-llm-draw-app/dist/
Linux AppImage + .deb	yarn tauri build	src-tauri/target/release/bundle/
Windows .msi (from Linux)	yarn tauri build --target x86_64-pc-windows-msvc	src-tauri/target/.../bundle/msi/
Android .apk	yarn tauri android build	src-tauri/gen/android/app/build/outputs/apk/

All artifacts are reproducible and signed according to the tiered signing policy described in the global rules (debug / staging / production).

Run tests

yarn test:typecheck   # TypeScript type checking across all packages
yarn test:update      # Run all tests with snapshot updates
yarn fix              # Auto-format + lint

---

Choosing an LLM provider

Two provider families ship out of the box. Both are hot-swappable from the in-app Settings panel; no rebuild required.

LiteRT (on-device, preferred)

Runs Gemma 4 (multimodal — voice + text + vision) directly in the browser process via @mediapipe/tasks-genai. Zero network cost, zero data leaves the device, target first-token latency of 400 ms on capable hardware. A smaller Qwen 3.5 bundle is provided for edge devices that can't host Gemma 4.

The CapabilityDetector probes WebGPU availability and estimated RAM at startup and picks the right on-device bundle automatically. If no bundle fits, it falls through to the OpenAI-compatible adapter.

OpenAI-compatible (hosted, universal)

Any endpoint that speaks the OpenAI Chat Completions wire format — OpenAI, Azure OpenAI, Groq, Together, LM Studio, Ollama, vLLM, and hundreds of others — works through a single OpenAICompatibleAdapter. Configure base URL, API key, and model name in the Settings panel. The adapter handles streaming, tool calling, and cancellation uniformly.

There is deliberately no vendor-specific adapter (no "Anthropic adapter", no "OpenAI adapter"). One universal HTTP shape keeps the surface small and avoids lock-in.

---

Contributing

This project is currently closed to external contribution. The source is visible for evaluation and personal reference only; see LICENSE for the exact terms.

If you're collaborating with the Author directly: the workflow is four-phase (MAP → ARCHITECT → PLAN → CODE) and CHECKLIST.md is the single source of truth for implementation. Read it, pick an unchecked task, execute it exactly as written, verify, and tick it off. Do not code work that is not in the checklist — if you find yourself wanting to, stop and update the architecture first.

---

License

Exa-LLM-Draw! itself: Proprietary — All Rights Reserved. See LICENSE.

Upstream Excalidraw (preserved portions): MIT License. See LICENSES/excalidraw-mit.txt and NOTICE.

The dual posture is deliberate. New contributions to this repository are proprietary while development is in flight; the upstream MIT is preserved in full to honor Excalidraw's license. The Author reserves the right to re-license the whole project at a future date — any such change will land as a commit to LICENSE.