​
React Visual Avatar SDK

@vonative/react is the frontend transport and control layer for Vonative realtime voice and avatar sessions. It is not, by itself, the full intelligence layer for a vision-aware avatar assistant. The correct mental model is:

​
What the SDK supports now

The current React SDK supports:
  • joining Vonative realtime sessions
  • reading agent and avatar runtime state
  • toggling camera input
  • toggling screen share
  • uploading image and PDF assets
  • setting visual interaction modes
  • enrolling participants from reference images
  • receiving annotation events for overlays
Current visual APIs:
In practice, a frontend developer can use the SDK to build:
  • avatar call interfaces
  • camera and screen-share controls
  • image and PDF upload flows
  • annotation overlays
  • visual-state aware UI for thinking, observing, speaking, and listening

​
What the SDK does not do by itself

The SDK does not guarantee that the assistant can understand visual input. That depends on the runtime path behind the session. Without matching backend and runtime-agent support, a client can successfully:
  • publish camera
  • publish screen share
  • upload images or PDFs
  • receive annotation and state events
while the assistant still cannot reason over those inputs. This distinction matters:
  • media transport is a frontend capability
  • media understanding is a runtime capability

​
Current capability boundary

Today, @vonative/react should be described as:
React SDK for Vonative realtime voice and avatar sessions with camera, screen-share, visual asset, and annotation support.
It should not be described as:
  • a full computer-vision SDK
  • a standalone multimodal reasoning SDK
  • a complete visual intelligence runtime

​
What works today with the full Vonative stack

When paired with the current Vonative avatar flow:
  • the frontend can publish camera and screen-share tracks
  • the runtime can inject the latest live visual frame into the model context on user turns
  • the assistant can respond to simple questions such as whether it can see the screen or camera
  • image and PDF upload events work as frontend controls, with runtime-side interpretation depending on the specific path implemented
This is enough for:
  • screen-guided support
  • demo and onboarding calls
  • camera-assisted conversations
  • simple presentation and tutoring flows

​
What is still limited

The current stack is not yet a full continuous vision system. Known limits:
  • live visual context is turn-scoped, not a persistent scene model
  • uploaded visual assets and live tracks do not automatically become a unified long-lived memory
  • there is no full pointer grounding or region-level selection pipeline
  • there is no OCR- or document-specialized runtime layer yet
  • provider-native avatar gesture and facial controls are still not exposed

​
Implementation plan

Use this rollout plan to move from frontend visual controls to a true vision-native avatar platform.

​
Phase 1: Stable transport and control surface

Goal: make frontend behavior predictable and developer-safe.
  • keep @vonative/react focused on:
    • media capture
    • upload transport
    • visual state events
    • annotation events
  • keep API and runtime concerns out of the SDK package
  • document capability boundaries clearly in public docs and package README
Exit criteria:
  • developers can integrate camera, screen share, and visual asset upload without guessing runtime behavior
  • SDK docs do not imply unsupported intelligence features

​
Phase 2: Reliable live visual turn grounding

Goal: make the assistant reliably use live visual context during conversation turns.
  • subscribe to the human participant’s camera and screen-share tracks in runtime-agent
  • ignore the avatar participant’s own remote video tracks
  • prefer screen share over camera when both are present
  • attach the latest frame to the current user turn in model context
  • add explicit prompt guidance so the model treats attached frames as visible context
Exit criteria:
  • avatar calls can correctly answer:
    • “Can you see my screen?”
    • “Can you see my camera?”
  • screen share and camera produce materially different grounded answers

​
Phase 3: Uploaded asset understanding

Goal: make uploaded images and PDFs first-class model inputs.
  • normalize uploaded image assets into image context
  • normalize PDFs into image and/or extracted text context, depending on the runtime strategy
  • allow the assistant to refer to uploaded assets in later turns
  • track asset IDs and session-scoped references consistently
Exit criteria:
  • users can upload an image or PDF and ask follow-up questions about it
  • uploaded visual assets are visible in both UI state and model context

​
Phase 4: Continuous observation model

Goal: move from “latest frame only” to sustained scene awareness.
  • add a controlled frame sampling policy
  • persist short-lived visual memory across several turns
  • distinguish current screen share from older uploaded assets
  • expose runtime observability for which visual source the model used
Exit criteria:
  • the assistant can maintain context across several visual turns
  • developers can debug which source informed a response

​
Phase 5: Higher-level visual tooling

Goal: turn raw visual input into product-grade developer primitives.
  • add OCR and document extraction tools
  • add pointer or region selection semantics
  • add presentation-aware helpers
  • add support workflows such as:
    • troubleshoot what is visible on screen
    • summarize slide content
    • inspect UI state
Exit criteria:
  • visual support and tutoring workflows no longer depend only on generic LLM vision behavior
Use wording like:
  • “Supports camera, screen share, visual asset upload, and annotation events”
  • “Pairs with Vonative runtime services for live visual reasoning”
  • “Provides the frontend layer for visual avatar experiences”
Avoid wording like:
  • “Complete vision SDK”
  • “Standalone multimodal runtime”
  • “Built-in computer vision stack”
After this guide, the next useful docs to add are:
  • a dedicated @vonative/react API reference
  • a visual avatar quickstart for React
  • a runtime guide for how visual tracks become model context
  • an asset guide for image and PDF understanding