Cloud Audit in a Box
(CAIB)
Self-hosted cloud and SaaS audit-log detection on Vector, PostgreSQL with TimescaleDB, and Grafana—packaged, measured, and honest about what still needs live verification.
Cloud audit logs should be searchable, detectable, and retained without forcing every event into somebody else’s subscription. CAIB packages the full path—from vendor collection and normalization to detections, dashboards, health signals, retention, backup drills, and an HA failure test—on infrastructure an operator can inspect.
Release status: engineering complete; real-cloud dogfooding next. AWS, GCP, Azure, and Cloudflare modules are implemented and fixture-tested. Most live vendor API verification is still pending. The Apache-2.0 source repository remains private during dogfooding and release cleanup.
⚡ Two-minute evidence path
make up # generate credentials, build, and start
make demo # seed 30 days of events and fire sample detections
make test # lint plus 70 automated tests
make demo needs no cloud credentials. It populates Overview, Alerts, Search, and per-module dashboards with synthetic fixture data so the complete path can be inspected before a production account is connected.
What CAIB answers
| Operator question | CAIB surface |
|---|---|
| Are audit events arriving? | Per-module rates, ingest lag, loader/detector heartbeats, dead-man rules |
| What changed across providers? | Normalized actor, action, target, outcome, source IP, and attributes |
| Is this activity suspicious? | SQL detections, severity, recurrence folding, and optional Slack notifications |
| Can I investigate a vague clue? | Indexed substring search across curated fields plus preserved raw JSON |
| Will old events bury the database? | Seven-day hot tier, compression, ninety-day detail retention, long-lived aggregates |
| Can this survive failure? | Backup/restore drill and a Patroni failover test under continuous writes |
One module, one end-to-end contract
A CAIB module owns more than collection. Each source packages:
- collector or poller configuration;
- normalized field mapping while retaining raw vendor JSON;
- migrations and fixtures;
- a provisioned Grafana dashboard;
- seed detections and a dead-man rule;
- automated tests;
- vendor API notes with an explicit live-verification checklist.
| Module | Delivery | Current state |
|---|---|---|
| AWS CloudTrail | S3 → SQS → Vector | Ingest, dashboard, and seven seed rules; fixture and LocalStack tested |
| GCP Audit Logs | Pub/Sub → Vector | Ingest, dashboard, and five rules; live Pub/Sub verification pending |
| Azure Activity Log | ARM REST poller | Ingest, dashboard, and five rules; live verification pending |
| Cloudflare Audit Logs | API poller | Ingest, dashboard, and five rules; live token verification pending |
| Slack / Zoom / KnowBe4 | API pollers | Detailed module recipes; not presented as shipped integrations |
The stack stays boring on purpose
| Component | Role |
|---|---|
| Vector | Push-source collection and disk buffering |
| PostgreSQL 18 + TimescaleDB | Events, raw evidence, alerts, aggregates, compression, and retention |
| Grafana | Overview, alerts, search, and module dashboards |
| CAIB core services | Migration, landing-to-event loading, detection, notification, health |
| CAIB poller | Shared runtime selected by module for pull-based APIs |
PostgreSQL and Grafana bind to loopback by default. First run generates random credentials. A preflight check refuses example passwords on a non-loopback bind unless the operator deliberately overrides it. Production still needs a reverse proxy, TLS verification, narrow network policy, external monitoring, and a restore plan.
Measured, not described as “fast”
The committed benchmark replays one million representative events through the real loader path:
| Measurement | Result |
|---|---|
| Raw vendor JSON | 1.03 GB |
| Loader throughput with indexed search | 14,700 events/second |
| Hot footprint, including indexes | 2.0 GB |
| Compressed data footprint | 0.22 GB |
| Replay against compressed chunks | 19,300 events/second, zero duplicates |
| Thirty-day substring search | 0.14 s hot / 0.21 s compressed |
The cold footprint is 9.3 times smaller than the full hot footprint; pure data compression is approximately six times. The difference is mostly hot indexes. The trigram search index roughly halves ingest throughput compared with the pre-index run and buys sub-second analyst search. That trade is measured and reproducible with make bench.
Failure work, not just feature work
CAIB’s release-confidence path includes:
make test # 70 tests, including real Vector runs
make bench # ingest, compression, query, and dedup evidence
make ha-failover-test # kill the primary under write load; assert zero loss
make restore-test # backup to scratch restore; verify markers and counts
Review and failure testing already caught problems invisible to the demo path:
- connection loss could leave the detector running but ineffective;
- source dead-man rules could not detect a dead detection engine;
- an ingest-lag dashboard query scanned unbounded retained history;
- first-run credentials and bind defaults were too permissive;
- database TLS behavior was hardcoded instead of configurable.
Those paths now fail loudly, expose heartbeat state, bound dashboard queries, generate secrets, bind locally, and share a configurable PostgreSQL TLS mode.
High availability without the fairy tale
The optional lab overlay runs three TimescaleDB/Patroni nodes, etcd, and HAProxy. Writers reach the primary-only frontend; Grafana can read from replicas or the primary. The automated test kills the leader during continuous writes and checks for zero loss after election.
Promoting that lab overlay requires real distributed-system work: three etcd members, separate database hosts with local storage, redundant frontends or multi-host clients, correct node addresses, TLS verification, narrow access rules, monitoring, and backups. Compose cannot turn one machine into three failure domains by being enthusiastic.
Honest boundaries
CAIB is a strong fit for lean security, platform, and cloud teams that want inspectable audit evidence and detections without a mandatory managed SIEM.
It is not:
- a staffed SOC or incident-response process;
- a replacement for provider-side log controls;
- fully live-verified across every shipped vendor integration yet;
- safe to expose directly without reverse-proxy and access-control work;
- done with dogfooding, alert tuning, or operational-table retention;
- publicly downloadable while the release-candidate repository remains private.
The next milestone is two weeks of real audit data, live verification of vendor assumptions, and a real restore drill. That evidence—not a bigger feature matrix—decides when the public source release is ready.
Read the engineering story
The build note covers the architecture, measured trade-offs, review failures, and the line between a release candidate and production evidence.
Read on Packets & Regrets🚀 Need Help Getting Started?
Self-hosting is free and always will be. But if you'd rather have it deployed, configured, and maintained for you — I can help.