Home IoT Hub Series · 23

[Part 23] AI Butler — The Start of Physical AI: a Smart Home That Talks to an LLM

[Recap of Parts 1–22]: one ESP32 brain, 19 sensor nodes, all the way to a touch TFT. We watch the home's status in real time, control it by touch, and even turn the AC on and off automatically from PIR + temperature.

But all of this ultimately moves only inside "rules written in advance." If the temperature goes above 26℃, turn on the AC; if a door opens, sound the buzzer. It can't step a single inch outside the rules.

What we'll do this time: connect AI to the home. An LLM reads the sensor values, judges on its own, and acts. For example, like this.

Throw it all 20 sensor values and the AI reads the pattern and gives advice. This is something rule-based automation can never do.

Concepts — the hard words and the plain words

Before we look at the code, let's sort out a few terms. I'll explain them in this pattern, so don't panic when an unfamiliar word shows up.

How to get an API key

OpenAI (GPT-3.5/GPT-4)

1. Go to platform.openai.com → sign up / log in 2. Top-right profile → View API keys 3. Create new secret key → type any name → create the key 4. Copy the generated key (starts with sk-proj-... or sk-...). Once you close this window you can't see it again. 5. Left menu Settings → Billing → Add payment method → register a card → set Usage limits 6. Cap it at $5/month and you'll never get hit with a surprise bill. At GPT-3.5-turbo rates, $5 is thousands of calls.

Claude (Anthropic)

console.anthropic.com → sign up → API Keys → Create Key. Just change the endpoint to https://api.anthropic.com/v1/messages.

Gemini (Google)

aistudio.google.com → Get API Key. The free quota is generous.

Ollama (local, free)

Install ollama.com on your home PC → ollama run llama3 → endpoint http://localhost:11434/v1/chat/completions. No API key needed and no internet needed. Completely free.

Three ways to do it

This time it's Method A. One ESP32, no extra gear, connecting directly to the AI API. This is the installment where you feel for yourself that "yes, it's possible, but never actually use it this way."

The big picture of this series: learning AI integration step by step

This is exactly the path toward Physical AI. A chatbot is trapped inside the screen. Physical AI is different. Sensors perceive the real world (eyes, ears, nose), the AI judges (brain), and actuators intervene in reality (hands, feet). The 20 sensors we built are the AI's sense organs, and the IR LED and relays are the AI's hands. This series is the process of connecting those sense organs and hands to an AI brain.

Say "Physical AI" and it's easy to picture only humanoid robots like Tesla Optimus or Hyundai Atlas. But that's a research-lab story worth tens of millions of dollars. The real start of Physical AI is a single ~$2.70 ESP32 board sensing the temperature and a door opening in your home, taking the AI's judgment, and turning on the AC. Perceive the real world, think, intervene — that's the essence of Physical AI, and what you're building right now is exactly that.

This article isn't a tutorial that just says "connect GPT to your ESP32" and ends. It's step-by-step learning. Each Method solves one limitation of the previous stage.

┌─────────────────────────────────────────────────────────────────────┐
│                    AI Butler: a 3-stage evolution                   │
├───────────────┬────────────────────┬────────────────────────────────┤
│               │ Method A (Part 23) │ Method B (Part 24)             │
│ Who calls     │ ESP32 → GPT        │ ESP32 → PC → GPT               │
│ the AI?       │ (direct)           │ (via a middle server)          │
├───────────────┼────────────────────┼────────────────────────────────┤
│ Response      │ Serial monitor     │ TFT + PC screen                │
│ shown where?  │ (only via PC)      │ + Telegram push                │
├───────────────┼────────────────────┼────────────────────────────────┤
│ View from     │ ❌ No              │ ✅ Live on your                │
│ outside?      │                    │   phone via Telegram           │
├───────────────┼────────────────────┼────────────────────────────────┤
│ AI controls   │ ❌ Talk only       │ ✅ Real AC ON/OFF              │
│ directly?     │ (human acts)       │ (Pi acts for you)              │
├───────────────┼────────────────────┼────────────────────────────────┤
│ Security      │ ❌ Key exposed     │ ✅ Key only on PC              │
├───────────────┼────────────────────┼────────────────────────────────┤
│ Extra cost    │ $0                 │ $0 (reuse old PC)              │
└───────────────┴────────────────────┴────────────────────────────────┘

Why learn Method A: to feel in your body "why you must not do it this way." Security holes, the Korean-font problem, the weight of TLS — you have to actually go through these to truly understand why Method B is needed. It's like showing crash-test footage before the written driving exam.

Method A: calling the AI API directly from the ESP32

Why it's dangerous — please read before you start

The ESP32 is a microcontroller. Simply put, it's a tiny computer with one "small brain." Microcontrollers have the following security weaknesses.

1. The API key is stored in flash memory as plain text. — If someone gets hold of your ESP32 board and just runs the command esptool.py read_flash, your API key is exposed wholesale. A single GPT-4 API key can run up hundreds of dollars a month. An attacker cleans it out in 3 minutes.

2. TLS certificate verification is limited. — TLS is the technology that encrypts internet traffic so nobody can eavesdrop in the middle (the padlock icon). The ESP32's WiFiClientSecure does implement TLS, but you have to embed a root CA certificate yourself, and when it expires you have to reflash the firmware. The internet is full of examples that just turn verification off with setInsecure(), which means you won't check whether the other side is really OpenAI's server. Completely defenseless against a man-in-the-middle (MITM) attack.

3. Without OTA updates, changing firmware is a pain. — OTA is "Over The Air," i.e. wireless firmware updates over Wi-Fi. Without it, every time the API endpoint changes or you need to swap keys, you have to plug in a USB cable and re-upload.

What you'll need

The full code

This code has the ESP32 gather 20 sensor values every 5 minutes, send them to the AI API, and print the response to the serial monitor. Put in your API key and it runs.

#include <WiFi.h>
#include <WiFiClientSecure.h>
#include <HTTPClient.h>
// #include <ArduinoJson.h>  // uncomment if you want proper JSON parsing

const char* ssid     = "YOUR_SSID";
const char* password = "YOUR_PASSWORD";

// ⚠⚠⚠ NEVER hardcode a real API key here ⚠⚠⚠
// Use only a temporary key you'll revoke right after testing
const char* OPENAI_API_KEY = "sk-your-temporary-test-key";
const char* OPENAI_HOST    = "api.openai.com";
const int    OPENAI_PORT   = 443;
const char* OPENAI_URL     = "/v1/chat/completions";

// ⚠ WARNING: setInsecure() = TLS verification OFF = vulnerable to MITM
WiFiClientSecure client;

// 20 slots (same as Part 22)
#define SLOT_COUNT 20
String slot[SLOT_COUNT] = {
  "--", "--", "--", "--", "--",
  "--", "--", "--", "--", "--",
  "--", "--", "--", "--", "--",
  "--", "--", "--", "--", "--"
};

const char* label[SLOT_COUNT] = {
  "Living Temp","Living Hum","Outside Temp","AC",
  "Boiler","Stove","Gas Leak","Fire",
  "Front Door","Door Lock","Living Light","Fridge Temp",
  "CO2","Dust PM2.5","Power","Water",
  "Hot Water","Rain","Bath Hum","Pet Bowl"
};

unsigned long lastAI = 0;
#define AI_INTERVAL 300000 // 5 min

// =============================================
// setup
// =============================================
void setup() {
  Serial.begin(115200);

  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  Serial.print("Connecting to WiFi");
  while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); }
  Serial.println("\nConnected! IP: " + WiFi.localIP().toString());

  // ⚠ Security risk: TLS certificate verification disabled
  client.setInsecure();
  Serial.println("⚠ WARNING: TLS verification DISABLED");
  Serial.println("⚠ DO NOT use with real API keys!");
  Serial.println();
}

// =============================================
// loop
// =============================================
void loop() {
  unsigned long now = millis();
  if (now - lastAI < AI_INTERVAL) { delay(1000); return; }
  lastAI = now;

  // Build the prompt from the 20 sensor values
  String prompt = buildPrompt();

  // Call the GPT API
  String response = callGPT(prompt);

  // Print the response
  Serial.println("=== AI RESPONSE ===");
  Serial.println(response);
  Serial.println("====================");

  delay(1000);
}

// =============================================
// Build the prompt
// =============================================
String buildPrompt() {
  String data = "Current smart home sensor data:\n\n";
  for (int i = 0; i < SLOT_COUNT; i++) {
    data += String(i + 1) + ". " + String(label[i]) + ": " + slot[i] + "\n";
  }

  // Model name: gpt-3.5-turbo, gpt-4o-mini, claude-3-haiku, llama3, etc.
  // For non-OpenAI APIs, see "Switching to another model?" below  String prompt = "{\"model\":\"gpt-3.5-turbo\",\"messages\":[{"
    "\"role\":\"system\","
    "\"content\":\"You are an AI home assistant. Analyze the sensor data "
    "and provide a brief English summary: 1) Any warnings? 2) Any anomalies? "
    "3) Recommended actions. Keep it under 3 sentences.\"},"
    "{\"role\":\"user\","
    "\"content\":\"" + data + "\"}],"
    "\"max_tokens\":150}";

  return prompt;
}

// =============================================
// Call the AI API
// =============================================
String callGPT(String body) {
  if (!client.connect(OPENAI_HOST, OPENAI_PORT)) {
    Serial.println("Connection failed!");
    return "ERROR: Connection failed";
  }

  String request = String("POST ") + OPENAI_URL + " HTTP/1.1\r\n"
    "Host: " + OPENAI_HOST + "\r\n"
    "Content-Type: application/json\r\n"
    "Authorization: Bearer " + String(OPENAI_API_KEY) + "\r\n"
    "Content-Length: " + String(body.length()) + "\r\n"
    "Connection: close\r\n"
    "\r\n"
    + body;

  client.print(request);

  // Wait for the response (up to 10 s)
  unsigned long timeout = millis();
  while (!client.available() && millis() - timeout < 10000) {
    delay(50);
  }

  // Skip headers → read only the body
  String resp;
  bool inBody = false;
  while (client.available()) {
    String line = client.readStringUntil('\n');
    if (line == "\r" && !inBody) { inBody = true; continue; }
    if (inBody) resp += line;
  }
  client.stop();

  // Extract content from the JSON
  String content = parseContent(resp);
  return content;
}

// =============================================
// Extract only message.content from the JSON response
// =============================================
String parseContent(String json) {
  int pos = json.indexOf("\"content\":\"");
  if (pos < 0) return json; // Parse failed → return the raw text

  pos += 11; // after "\"content\":\""
  String result;
  while (pos < json.length()) {
    char c = json[pos];
    if (c == '\\' && json[pos + 1] == '"') {
      result += '"'; pos += 2; continue;
    }
    if (c == '\\' && json[pos + 1] == 'n') {
      result += '\n'; pos += 2; continue;
    }
    if (c == '"') break;
    result += c;
    pos++;
  }
  return result;
}

Check that it works

1. Put a test key in OPENAI_API_KEY and upload. (See "How to get an API key" above.) 2. Serial monitor at 115200 bps. 3. Every 5 minutes the ESP32 prints a response like this:

=== AI RESPONSE ===
Indoor temp 24.5C, comfortable. CO2 at 780ppm, slightly
elevated - consider ventilation if still high in 30 min.
All warning sensors at 0 - no alerts.

> **Note: this data does NOT show up in the ChatGPT chat window.** API calls are completely separate from the conversation window at [chat.openai.com](https://chat.openai.com). The API is just a channel for one machine to ask another directly; it leaves no trace in the chats you have with GPT on the web. You can only see your API usage as token counts at [platform.openai.com/usage](https://platform.openai.com/usage).
====================

4. As soon as testing is done, revoke the API key immediately. OpenAI dashboard → API Keys → click the trash icon next to that key → Revoke. Because it's left in the ESP32 flash as plain text.

The real-world problems

The code above works. But it has a few real-world problems.

1. You have to hardcode the API key into the source. — Hardcoding means "writing the password directly into the code." The ESP32 has no elegant option like environment variables. Push it to GitHub and the key is exposed right there in a public repo.

2. WiFiClientSecure is heavy. — The TLS handshake takes 10–20 seconds and RAM usage climbs sharply. Run it at the same time as Part 22's TFT rendering and it can reset from running out of memory.

3. The JSON parsing is flimsy. — JSON parsing means "picking out only the part you need from the long reply the AI sent." The parseContent() above works only if you're lucky. Doing it properly needs the dynamic allocation of the ArduinoJson library, and then the ESP32 heap runs dry fast.

4. When a sensor value is --, the AI talks nonsense. — If an initial value like "Pet Bowl: --" goes in, the LLM may give a weird answer like "looks like there's no dog." You need prompt filtering.

Method A: the verdict

Key lesson: a microcontroller is a device specialized for reading sensors and driving actuators. Storing API keys and doing TLS communication isn't its job. That's a job for a "computer."

Next up

Method B — an old PC as the AI broker. That 10-year-old Windows laptop shoved in a closet, or the MacBook you leave on all the time. Put a single Python Flask server on it and all the ESP32's security problems go away. The API key is stored only on the PC, and the ESP32 just does HTTP: "here's the data, give me a command." 40 lines of Python, 30 lines of ESP32 code. 70 lines total and your AI integration is done.