Measure first, then architect.
The tooling matters less than the order of operations. Nothing gets rebuilt before there's a number that says whether the rebuild worked.
The measurement layer
Every engagement ends with this in your CI, owned by your team: a golden set that gates releases, traces you can read when an answer goes wrong, and guardrails on the actions the system is allowed to take.
Eval: the release gate
A golden set built from your real questions. Faithfulness, context recall, tool accuracy and latency run on every pull request; a regression blocks the deploy rather than reaching a customer.
Trace: why it answered that
Every step recorded: the plan, the retrieved spans, the tool calls, the tokens. When a support ticket says the answer was wrong, you can read the run instead of guessing.
Guardrails: what it may do
Policy checks on inputs and outputs, hard limits on consequential tool calls, and explicit escalation paths to a human for the cases that should never be automated.
Tools I reach for
languages & services
- python
- go
- typescript
- fastapi
- aws bedrock
- lambda · ecs
- terraform
retrieval & orchestration
- langgraph
- pgvector
- qdrant
- opensearch
- rerankers
- mcp
evaluation & observability
- ragas
- opentelemetry
- langfuse
- mlflow
- pytest in ci
data platform
- postgres
- dbt
- airflow
- snowflake
- data contracts
i'll work in whatever your team already runs. this is preference, not a requirement.
How I decide things
Baseline before architecture
A number first, even a crude one. Rebuilding an agent without a baseline means nobody can tell whether it helped.
Cheapest thing that could work
Better chunking and a reranker beat an agent framework more often than people expect. Agents come in when the question genuinely needs planning.
Operable by your team
If your engineers can't debug it after I leave, it isn't done. That constraint shapes the design, not just the handover.