Synopsis "LLM Evaluation & AI Quality Engineering"
Stop shipping LLM, RAG and agent features on gut feel. This is the hands-on AI testing and LLM evaluation playbook for QA engineers, SDETs and AI engineers who must prove a release is ready. Most teams "evaluate" an LLM application by eyeballing a few outputs or trusting one dashboard score. Then a prompt tweak, a model upgrade or a retrieval change quietly breaks production. LLM Evaluation & AI Quality Engineering shows you a better way: AI quality is an engineering lifecycle, not a single evaluation score. You will learn to Define, Measure, Evaluate, Diagnose, Improve, Regression Test, Release, Monitor and Learn, and to treat every measurement as evidence, not truth. What you will build and learn A vendor-neutral Python evaluation package you own, backed by a free public companion repository with 119 automated tests Quality objectives, golden datasets, error analysis and failure taxonomies Deterministic code evaluators, metrics and rubrics, human annotation with Cohen's kappa and Krippendorff's alpha LLM-as-a-judge evaluators you have actually validated, with TPR/TNR meta-evaluation Statistics for non-deterministic systems: Wilson confidence intervals, bootstrap, McNemar paired comparison, sample size, pass@k and pass^k RAG evaluation: recall@k, precision@k, MRR, nDCG, groundedness, citation validity and retrieval-vs-generation attribution AI agent evaluation: sandboxed outcome checks, trajectory and tool-use evaluation, reliability Safety and red teaming: prompt injection, canaries, refusal and over-refusal, mapped to the OWASP Top 10 for LLM Applications 2025 Evaluation-driven development, regression suites for model migration, and cost, latency and reliability trade-offs AI quality gates in CI/CD with GitHub Actions, and production observability with OpenTelemetry GenAI semantic conventions, drift detection and triage Who this book is for QA engineers, SDETs and test leads moving into AI quality; AI/ML engineers who own LLM, RAG or agent features; and technical leads who sign off AI releases. You need intermediate Python, basic pytest and Git. Why this book is different Written by a software QA professional, it turns evaluation results into release decisions you can defend. DeepEval, Ragas, Inspect AI, Arize Phoenix, MLflow, Langfuse, LangSmith and Promptfoo are compared fairly, but you are never locked into a vendor. The final part shows how to run an AI quality engineering programme: RACI, release sign-off, a maturity model, and a practical mapping to NIST AI RMF and ISO/IEC 42001. 23 chapters, 5 parts, 21 figures, 24 tables and 10 appendices. Part of the author's AI Quality Engineering series. Pair it with AI Testing with Python for foundations and RAG in Production for building the systems you will evaluate.
