The AI Engineering Loop
Building with LLMs is an iterative engineering process. Because outputs are probabilistic, teams need a loop for seeing what happened, finding failure modes, testing changes, and deciding what to ship.
The loop is a working model, not a strict waterfall. Teams move through it repeatedly, and different parts of the loop become more important as a product matures.
The steps
1. Tracing
Tracing captures the full path of a request so you can inspect prompts, retrieved context, tool calls, outputs, latency, and cost in one place. Read Tracing for a breakdown of what a useful trace looks like and why traces become the foundation for everything else.
2. Monitoring
Monitoring turns raw traces into ongoing visibility by tracking trends and surfacing the cases that deserve attention. Read Monitoring to understand how teams watch quality, cost, latency, and production failures over time.
3. Datasets
Datasets turn real scenarios into repeatable test cases so you can check whether a change helps across more than a handful of examples. Read Datasets for how to structure dataset items and when it makes sense to split or grow a dataset.
4. Experiments
Experiments let you change one variable at a time and compare outputs against a stable baseline instead of relying on intuition alone. Read Experiments to see how to isolate variables, compare variants, and learn what actually improved.
5. Evaluation
Evaluation is how you decide whether results are good enough to ship, using manual review, code-based checks, or LLM judges depending on the task. Read Evaluate for how teams score outputs and turn qualitative judgments into a repeatable process.
What the loop helps you balance
Across the loop, teams are usually balancing three things at once: output quality, latency, and cost. The point is not to optimize one number in isolation, but to make tradeoffs explicit and grounded in evidence from your own application.
Where the docs fit
This page gives you the map, and Academy explains the concepts behind each step. When you want Langfuse-specific implementation details, move into the docs and guides.