IoT SDK + AI Agent Example

You can pair the IoT SDK with a VideoSDK AI Agent so a device joins the same room as the agent. The device streams its audio to the agent, and the agent listens, processes the audio, and responds in real time. This gives the device two-way voice interaction.

What You Can Build

Smart Home & Lifestyle

• Adaptive Climate Control: a thermostat that reports the temperature and takes spoken instructions to adjust it.

• Interactive Home Chef: a kitchen device that suggests recipes and guides the user through them by voice.

• Wellness Coach: a wearable that monitors heart rate and gives spoken prompts, such as a breathing exercise when the rate spikes.

Education & Entertainment

• Child-Friendly Storyteller: a toy that reads stories aloud, changes character voices, and answers a child's questions.

• Interactive Tutor: a classroom device that answers students' questions aloud and adjusts its explanations to each student.

Any device that can join a room alongside an AI Agent can support similar two-way voice use cases.

Demo: IoT SDK in Conversation with AI Agent

---

Prerequisites

Before you get started, ensure you have the following:

• Python: Python version >= 3.11, for the AI Agent side
• ESP-IDF: ESP-IDF v5.4.2, v5.4.3, or v5.4.4, and a supported ESP32-S3 board with a microphone and a speaker, such as the ESP32-S3-Korvo-2 v3.1. A send-only board can talk to the agent but cannot play its replies.
• A working device build. Finish the IoT SDK Quick Start first, so you can already flash and join a room.
• Video SDK Developer Account (Not having one, follow Video SDK Dashboard)

---

Step 1: Setup for AI Agent

1

Setup for AI Agent

On the AI Agent side, configure it to join the room as a participant. It listens to the device's audio, processes it, and generates responses in real time, handling speech recognition, reasoning, and synthesis for low-latency two-way interaction between the device and the room.

def make_context() -> JobContext:
    room_options = RoomOptions(
        room_id="YOUR_MEETING_ID",
        name="VideoSDK Cascaded Agent",
        playground=True
    )

    return JobContext(room_options=room_options)

caution

Make sure you keep the same meetingID in the RoomOptions here, which you’ll also use on the IoT device side later.

See the full setup in the AI Agent Quick Start Guide

Step 2: Setup for IoT SDK

2

Setup IoT SDK

Now it’s time to configure the IoT SDK on your ESP device. This step enables the device to capture and publish audio streams into the room, as well as receive and play responses from the AI Agent. With this setup, your device transitions from being a simple endpoint to an active communication node.

// Token and meeting ID come from menuconfig -> "VideoSDK Configuration",
// so no real token has to sit in source.
char *token = CONFIG_VIDEOSDK_TOKEN;

init_config_t init_cfg = {
    .meetingID = CONFIG_VIDEOSDK_MEETING_ID,  // same room as the agent
    .token = token,
    .displayName = "ESP32-Device",
    .participantId = "your participant id",   // "" or NULL => a random id is generated
    .audioCodec = AUDIO_CODEC_OPUS,
    .videoCodec = VIDEO_CODEC_NONE,           // audio-only agent
};

result_t init_result = init(&init_cfg);
printf("Result: %d\n", init_result);

caution

Ensure that the meetingID configured on the IoT device is identical to the meetingID defined in the AI Agent’s RoomOptions.

See the full setup in the IoT SDK Quick Start Guide

Step 3: Connect the IoT Device and AI Agent

3

Connect the IoT Device and AI Agent

Run the two sides together:

1. The AI Agent joins the room first.

if __name__ == "__main__":
    # Start the AI Agent, which will join the room as a participant
    job = WorkerJob(entrypoint=start_session, jobctx=make_context) 
    job.start()

2. The IoT device then joins the same room and starts publishing and subscribing to audio.

  // joining the User through IoT - device once AI - Agent joins the room.
  result_t result_publish = startPublishAudio(); 
  result_t result_subscribe = startSubscribeAudio();
  printf("Result Publish:%d\n", result_publish);
  printf("Result Subscribe:%d\n", result_subscribe); 

Supported hardware

Two-way voice needs a board with a speaker so the device can play the agent's replies. Without one, startSubscribeAudio() returns DEVICE_NOT_SUPPORTED, so the device can send audio to the agent but cannot play its replies. See Supported Microcontrollers.

3. The device and the AI Agent now exchange audio in both directions in real time.