March 3, 2026 • Technical deep dive & quickstart

Meet HA Event Processor, a lightweight, production-ready service that subscribes to Home Assistant MQTT events, stores them locally, and batches them to Google BigQuery for analytics and model training. It’s designed to run on your local machine, inside Docker Compose for development, or in a k3s cluster for production.

Note that the original code and this blog post have been generated with Github Copilot.

Project & Source

The full project is open source on GitHub: https://github.com/petrivirkkula/ha-event-processor

Why this project?

  • Capture Home Assistant events (topics under homeassistant/#) reliably via MQTT
  • Persist events to a local database (SQLite for dev; PostgreSQL supported for production)
  • Batch and sync events to Google BigQuery for analytics, ML training, or backups
  • Production-grade considerations: metrics, health-checks, non-root containers, RBAC manifests

Key Features

  • MQTT client with auto-reconnect and robust parsing.
  • Event processing and validation (entity normalization, attribute handling).
  • Local storage using SQLAlchemy (models in src/ha_event_processor/storage/models.py).
  • GCP BigQuery uploader with exponential backoff/retry logic.
  • Prometheus metrics (exposed on /metrics) and FastAPI endpoints (/health, /stats, /sync).
  • k3s-ready manifests live in the k3s/ folder and include Deployment, Service, ConfigMap, Secrets template, PVC and RBAC.
  • Comprehensive test suite and coverage reporting (pytest + pytest-cov).

Architecture (high level)

Home Assistant -> MQTT Broker -> HA Event Processor
                                   ├─ MQTT client (paho-mqtt)
                                   ├─ Event processor (validation & normalization)
                                   ├─ Local DB (SQLite/Postgres)
                                   └─ GCP Uploader (BigQuery batch sync)

Quickstart (Local)

Clone the repository and run locally for development:

git clone https://github.com/petrivirkkula/ha-event-processor.git
cd ha-event-processor
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# run the app (FastAPI + uvicorn)
python src/main.py
# open health
curl http://localhost:8000/health

Run with Docker Compose

# Start the local stack
docker-compose up --build
# Check API
curl http://localhost:8000/stats
# Stop
docker-compose down

Deploy to k3s

There are Kubernetes manifests in the k3s/ directory. Example:

# Update image reference in k3s/deployment.yaml, then:
kubectl apply -f k3s/
kubectl get pods -w
kubectl logs -f deployment/ha-event-processor

Testing & Coverage

Unit tests and coverage are included. To run tests locally:

pip install -r requirements-test.txt
bash scripts/run-tests.sh
# or run pytest directly
pytest tests/ --cov=src/ha-event-processor --cov-report=html
open htmlcov/index.html

Notable Files

  • src/main.py – FastAPI entrypoint and health/metrics endpoints.
  • src/ha_event_processor/mqtt/client.py – MQTT parsing and subscription logic.
  • src/ha_event_processor/events/processor.py – Event validation & normalization.
  • src/ha_event_processor/storage/database.py – Storage & DB helpers.
  • src/ha_event_processor/gcp/__init__.py – BigQuery uploader.
  • k3s/ – Kubernetes manifests for k3s deployment.
  • tests/ – Unit tests (pytest) and helper fixtures.

Observability & Production Readiness

The service exposes Prometheus metrics at /metrics and provides structured logging. The Dockerfile is multi-stage and the Kubernetes manifests contain liveness/readiness probes, resource requests/limits, non-root security context, and minimal RBAC.

How can you contribute?

  • Open issues or PRs on GitHub: Repository
  • Improve tests and add integration tests
  • Extend BigQuery schema or add additional cloud backends (e.g., Cloud Storage)
  • Help with Docker images and CI/CD workflows

Conclusion

If you need a robust pipeline to capture Home Assistant events and prepare them for analytics or model training, the HA Event Processor provides a practical starting point: local persistence, reliable MQTT handling, and seamless batch sync to BigQuery. Check out the codebase and documentation and try it today:

https://github.com/petrivirkkula/ha-event-processor

Questions, feedback, or PRs welcome — star the repo to show your support!