Making an AIplatform observable
Shipping into a live AI teaching platform used by CBSE and ICSE schools — 200+ pull requests reviewed and merged in two-week Scrum sprints.
The platform was already serving schools, but nobody could see inside it. Latency regressions and error spikes surfaced days later, usually because a school told us. There was no way to answer 'is it slow right now?' or 'which school is our AI spend going to?' without reading logs by hand.
It was a live product with paying schools on it, so nothing could be rebuilt from scratch and nothing could be taken offline to instrument. Every change had to land behind senior review, in sprint, on a system that stayed up the whole time.
What I built
- Step 01
- Step 02
- Step 03
- Step 04
- 01
The first production observability stack
Prometheus for metrics, 7 Grafana dashboards for the views people actually ask for, and Loki so log search stopped meaning SSH. 19 Slack alert rules turned the dashboards into something that pages you instead of something you remember to check.
- 02
Per-call cost tracing
Every AI call now carries attribution, so spend is readable per school rather than as one monthly invoice. That turned an opaque cost line into a number the business can act on.
- 03
An asynchronous textbook pipeline
Textbook processing was rebuilt as an object-oriented background pipeline on Celery and Redis, with S3 pre-slicing so a request carries a chapter instead of a whole book. Retry-backoff and concurrency caps ended the failures under load.
- 04
One authorization path
77 API permission checks were scattered across the codebase. They now run through a single deny-by-default chain — school, then org, then plan — evaluated as layered rules over PostgreSQL row-level security, with fail-closed guards that block AI calls outright when the governance layer is down.