Observability tier
Docker Compose stack: Prometheus, Grafana, Loki, Tempo, Alertmanager, object storage. Promtail on workloads ships logs; node metrics scraped for dashboards and AI context.
Repeatable AWS architecture
Infrastructure as code, configuration management, and containerised observability — deployed into your VPC so architects can audit every hop.
Three layers
AI augments the stack; it does not replace your telemetry systems of record.
System of record
Prometheus · Loki · Tempo · MinIO-compatible storage
Operations
Grafana · Alertmanager · dashboards · alerting
AI assistance
Streamlit · MCP · Ollama · AI Ops webhook
Typical deployment topology
Logical tiers — sized and scaled per engagement. Workload hosts can be existing servers or new EC2 instances.
AI tier
Ollama (GPU or CPU profile), FastAPI MCP server proxying Prom/Loki/Alertmanager APIs, Streamlit multipage UI, optional metrics exporters for model and GPU observability.
Workload tier
Application and database hosts — log agents, optional OpenTelemetry collector forwarding traces to Tempo. Linux config collector for History door snapshots.
Provisioning
Terraform for EC2, networking, DNS, security groups. Ansible for Docker, Compose stacks, host baseline, scrape targets, and config collection playbooks.
MCP — the evidence bridge
Streamlit and the AI Ops webhook do not scrape observability APIs directly from arbitrary endpoints. A small MCP service exposes health-checked proxies for Prometheus instant and range queries, Loki log range, and Alertmanager alerts — keeping URLs, timeouts, and credentials centralised.
Why this matters for architects
- Single controlled integration point between AI components and TSDB/log store
- Evidence bundles are explicit and loggable — easier governance review
- Swap models or UI without rewiring every observability client
- Natural extension path toward fuller MCP tool-calling as the product matures
Config change pipeline (History)
Ansible invokes a JSON collector on Linux hosts. A ingest step normalises snapshots, computes diffs between dates, and stores rows in SQLite. Streamlit queries that store — no LLM hallucination on “what changed” when the row exists.
POC vs production
The reference implementation is deliberately honest about POC scope: tighten TLS, authentication for Grafana/MCP/Streamlit, secret management, HA, and model operations before internet-facing or multi-tenant use. We help teams map that gap as part of implementation and advisory engagements.