What does the model do?​

The model monitors a webcam powered by Thingino and served to HomeKit via Scrypted. It produces two primary predictions published to MQTT, which are then exported to HomeKit via Homebridge using mqtt-things:

• Door Open/Closed
• Car Home/Away

Please see the previous post on GarageCam for context on the evolution of this project.

How does it run?​

• Raspberry Pi 5 - set to poll every 30s.
• Raspberry Pi 4 - Currently sidelined due to a build mismatch.

The underlying code maps PyTorch operations to specific CPU sets via ExecuTorch, but this process is currently error-prone. Aligning the right instruction sets with the target architecture requires precise configuration, which led to the RPi 4 hiccups.

2026-02-18 06:01:42,406 [INFO] model_runner.inference: Loading PTE model from /app/model/model.pte
[program.cpp:153] InternalConsistency verification requested but not available
[cpuinfo_utils.cpp:71] Reading file /sys/devices/soc0/image_version
[cpuinfo_utils.cpp:87] Failed to open midr file /sys/devices/soc0/image_version
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu0/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu1/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu2/regs/identification/midr_el1
[cpuinfo_utils.cpp:100] Reading file /sys/devices/system/cpu/cpu3/regs/identification/midr_el1
2026-02-18 06:01:42,408 [INFO] model_runner.inference: PTE model loaded successfully
2026-02-18 06:01:42,408 [INFO] model_runner.mqtt_client: Connecting to MQTT broker mqtt_broker:1883
2026-02-18 06:01:42,412 [INFO] model_runner.mqtt_client: Connected to MQTT broker
2026-02-18 06:01:42,512 [INFO] model_runner.healthcheck: Healthcheck server listening on port 8080
2026-02-18 06:01:42,613 [INFO] model_runner.runner: All components initialized successfully
2026-02-18 06:01:42,613 [INFO] model_runner.runner: Running inference every 10.0s
2026-02-18 06:01:43,281 [INFO] model_runner.camera: ONVIF snapshot URI: http://192.168.000.000/image.jpg
2026-02-18 06:01:44,311 [INFO] model_runner.runner: Debug snapshot saved: /tmp/model_runner_debug/snapshot_20260218_060144.jpg
2026-02-18 06:01:44,326 [INFO] model_runner.state: State change: car_present: UNKNOWN -> OFF (prob=0.198)
2026-02-18 06:01:44,326 [INFO] model_runner.state: State change: door_open: UNKNOWN -> OFF (prob=0.000)
2026-02-18 06:01:44,326 [INFO] model_runner.state: State change: door_closed: UNKNOWN -> ON (prob=0.801)

How does it perform?​

Surprisingly well! The Raspberry Pi 5 handles the inference load with ease, maintaining a consistent polling interval without thermal or resource issues.

How efficient is it?​

• Model Size: Optimized for edge deployment using PTE formats.
• CPU Consumed: Minimal impact on the Pi 5's quad-core processor.
• Memory Consumed: Low footprint, leaving plenty of headroom for other services.

While the resource usage is excellent, I'm still refining the observability stack to better track long-term ML accuracy and model drift.

Does this help avoid Frigate / Scrypted NVR fees?​

Nope, you should still pay them. Eventually, my answer may change, but not without more observability work and scaling this across more cameras.

Where is the code?​

Next Time—it's not presentable to the public (yet) largely as there is still a lot of hard-coded stuff with PII that I want to remove.

ExecuTorch and the CompileVM​

Rather than bloat my main workstation with cross-compilation support, I've added a QEMU/UTM Debian VM for compiling the Python wheels for ExecuTorch. This will be factored out of the main repo as a more generic solution.

• Note: I have a compile container as well, but in principle, I could have run this on an x86 box so it wasn't a generic cross-compile/arch-agnostic pipeline.

Did PyTorch Lightning help?​

Tremendously versus straight PyTorch. At some point, I need to dig into what is going on under the hood.

Did AI Assist with the coding / testing / design?​

Yes. It was used for boilerplate and autocomplete. Sadly, as some of the tech (ExecuTorch, ONNX, TFLite), Daytona, BaseTen, and Modal was not familiar to me, Gemini, Claude (Opus), and Grok helpfully jumped in to help—but the subtle bugs hurt, a lot.

How does this really help?​

I've done nothing special here—but it opens the door to a generic home-ML pipeline of using bigger "Teacher" models to train smaller "Student" models that can run on non-specialized hardware.

Next Steps for GarageCam​

• Model Perf Tracking: Need to add in Arize Phoenix or Sentry for tracking model drift and inference performance.
• Model Architecture: See the next post in the Frontier Research blog where I may reveal my ignorance.
• Model Inference Footprint: Fix the RPi 4 and explore other smaller processors.
• Data Privacy: Add in bounding, labeling, and masking of private data with the local Teacher. Once properly masked, Daytona/Modal/BaseTen/Lightning are excellent for beefy training.
• Differential Privacy: Exploring how much noise can be added to the picture without breaking the classifier.

There are a lot of cool developments that others have been working on, and here is where I'd like to spend any free time I get over the holidays:

Jarvis 2.0: The Path to Private Agency​

The core of "Jarvis 2.0" is moving from a reactive chat interface to a proactive, context-aware agent. My focus for the holidays is to integrate Semantic Router for lightning-fast intent detection and vLLM-SR (vLLM Speech Recognition) for local, low-latency voice commands that stay entirely within my home's firewall.

What makes it "2.0"?

• Proactive Intelligence: Moving beyond basic voice commands to an agent that suggests actions based on observed context and historical patterns.
• Local Vision: Integrating models like Qwen2-VL to understand the visual state of the house via the GarageCam/HomeKit camera network.
• Zero Cloud: Ensuring every byte of audio, video, and text remains on the local M2/RPi mesh, with zero external dependencies for core functionality.

What is hard (or time-consuming) about this?​

• Generating the testing/training data set. I ended up using a VLM for the initial labeling, with manual overrides for edge cases. I started with Moondream, but eventually transitioned to Qwen2-VL, which proved far more robust for object identification. In retrospect, we also should have tried out Molmo instead of LLaVA for its superior zero-shot performance on high-resolution snapshots.
• As I'm trying to run this with limited resources on a Raspberry Pi 4 (4GB RAM), that means starting the ML models from scratch and aggressive pruning/quantization.
• Trying to leave something that is generalizable enough for other cameras in the house—or just give up and pay the $20/mo for Frigate/Scrypted Cloud. The goal of "CheapML" is to see how far we can get with pure open-source and local hardware.

The Conversion Pipeline​

The current pipeline takes a PyTorch-trained model, converts it to ONNX for optimization, and then finally to TFLite to squeeze it onto the RPi 4's CPU/TPU. Each step requires careful calibration to avoid significant accuracy loss during quantization.

1. What I have:

   • Wyze Cam v3
   • Scrypted
   • Thingino
   • HomeKit
   • RPi 4 that isn't melting

2. What I need:

   • A way to detect when the garage door is open or closed
   • A way to update MQTT with the state of the garage door
   • A way to update MQTT with the state of the car

So far: The initial Python script I "vibe coded" was, to put it mildly, very hacky. It used the ONVIF Snapshot protocol to capture a frame every 60 seconds from the camera. While it technically worked with opencv-python and paho-mqtt for a few days, it was brittle and prone to memory leaks on the RPi 4.

The real lesson here was in the data handling—capturing snapshots is easy, but making a reliable sensor out of them required a more robust architecture. This prototype was eventually rewritten to use a more stable event-driven model, which I'll cover in the follow-up post.

Mini Minds: The Power of Local Collaboration​

The "Mini Minds" concept involves using multiple small language models (SLMs), each specialized for a specific task—like intent detection, light control, or temperature monitoring—rather than one monolithic LLM. This approach reduces latency and compute costs while maintaining high accuracy in a local IoT environment.

How do Federated Learning Frameworks fit in?​

Frameworks like Flower are essential for this "Mini Minds" architecture. They allow multiple local devices to participate in training without sharing raw, sensitive data, enabling a collective intelligence that stays within the home's walls. This interoperability between small, specialized models is the key to a truly responsive and private smart home.

Can we shrink the size of the model using fine-tuning, LoRa and quantization to fit on a Raspberry Pi 4 with reasonable performance?​

The quest to run meaningful AI on a Raspberry Pi 4 led us down a rabbit hole of quantization and LoRA (Low-Rank Adaptation). While we were knee-deep in the research, it turns out we didn't get this running to our own satisfaction. Others in the open-source community beat us to the solution, providing robust implementations that far exceeded our initial prototypes. We'll not complain and just accept their work, as it allows us to focus on the next layer of the stack: private orchestration.

The Pim Particle Effect: Linguistic Dimensions​

The "Pim Particle" concept remains relevant as a way to think about semantic compression. By using actual previous calls to confirm the linguistic dimensions of possible intent, we can effectively shrink the search space required for an agent to take action.

To maintain these local agents, we've developed a self-healing MCP (Model Context Protocol) server workflow:

This ensures that even when running smaller, "shrunken" models, the tools they interact with are always up-to-date and verified.

Background​

Caveat Emptor: This is a public blog of what probably should be a private note.

Machine Specification:​

• CPU: Mac M2 Pro
• RAM: 96GB DDR4
• Storage: 1TB NVMe SSD

Tonight turned into an unexpected journey through the rabbit holes of self-hosted technologies. I decided to venture out of my comfort zone, and boy, was it an enlightening experience! Here's a rundown of gotchas:

Models​

Local Models​

• Llama 3.1 8B/70B: Our primary general-purpose LLM, running via Ollama.
• Mistral NeMo: Excellent for 128k context window tasks.
• nomic-embed-text: Our go-to for local embeddings.

Tools / Components​

Podman Machine: The Immutable Disk Conundrum​

Podman Machine on OSX has a fun caveat - the disks were traditionally not resizable! Circa this post, this behavior seemed by design to ensure stability and simplicity. Update: Newer versions of Podman allow for disk expansion via podman machine set --disk-size.

Gemini API​

While not strictly local, the Gemini API (via Google AI Studio) serves as a benchmark for our local models. We use it with 'Data Sharing' disabled to maintain as much privacy as possible during testing.

Langfuse: AI on My Terms​

Langfuse is a platform for managing and deploying AI models on your own infrastructure. The setup was straightforward, but what fascinated me was the control it offers over model iterations and data privacy. If you're into machine learning but worried about cloud dependencies or data security, Langfuse might just be your knight in shining armor. Be sure to turn off phone home in the self-hosted config.

Valkey: Open-Source Redis for the Win​

Valkey came as a surprise. An open-source alternative to Redis, it provides similar data structures and features. My exploration tonight wasn't just about setting it up (which was a breeze) but also understanding its ecosystem. Valkey offers a community-driven approach to key-value stores, which could be particularly appealing in environments where open-source contributions are paramount.

Minio: S3 Permissions Masterclass​

Now, Minio was where things got tricky yet incredibly educational. It’s an object storage server compatible with Amazon S3, which I installed to get a grip on handling S3-like storage locally. The real learning curve was the crash course in S3 Permissions and policy. Setting up bucket policies, managing access, and ensuring security was like solving a puzzle where every piece affects the other. It was tough but rewarding, and now I have a deeper understanding of S3's permission model, which is invaluable in cloud or self-hosting scenarios.

Clickhouse: Permissions and Peculiarities​

Clickhouse, a column-oriented database management system, known for its speed in processing analytical queries, was next on my list. However, setting permissions proved to be wonky. Clickhouse's default user permissions are quite permissive, which I initially found odd. After a few hours of configuration, I realized this approach simplifies many operations but requires careful setup to secure a production environment. It's an excellent lesson in balancing convenience with security.

Milvus: Vector Search Made Local​

Milvus caught my eye for its capabilities in similarity search and AI applications, particularly with unstructured data. Setting up a local instance was a breeze, and playing with vector space indexing opened up new realms of what's possible with local infrastructure. It's clear why Milvus is gaining traction for real-time analytics and AI-powered searches.

Python Inbox Processing: A Practical Challenge​

Finally, I dabbled with Python inbox processing. This wasn't about setting up something new but optimizing an existing script to handle my email with better efficiency. Parsing emails, extracting information, and organizing it into a database - it was a practical challenge, blending technology with everyday utility.

Conclusion: A Night of Learning and Laughter​

Tonight was not just about tech exploration but also about embracing the learning curve with laughter. Each of these technologies presented unique challenges and solutions, teaching me that in the tech world, the journey itself is often more valuable than the destination.

Whether you're considering going self-hosted, exploring AI, or just curious about container management, I hope my little adventure inspires you to dive in yourself. Remember, tech exploration is not just about mastering tools but about enjoying the ride and learning from each step. Here's to many more nights of tech discoveries!

Do let me know if you try any of these out or if you have any other cool self-hosted technologies to explore!

podman-machine​

For those unfamiliar, Podman is a daemonless container engine for developing, managing, and running OCI Containers on your Linux System.

Apple Secure Enclaves​

Apple's Private Cloud Compute provides a blueprint for how hardware-level security can scale to the cloud without compromising privacy.

Securing H100 GPUs​

NVIDIA's Confidential Computing on H100 GPUs ensures that even in multi-tenant environments, your AI workloads remain isolated.

Confident Security​

Confident Security is another player focusing on the intersection of data privacy and AI safety.