Professional Validation Library

Never Lose a Message · Kai London

Kafka Resilience Engineering: Validate Replication, Consumer Recovery and Data-Loss Boundaries

Kafka says the message was acknowledged. Can you prove it was never lost?

A Kafka cluster that looks healthy can still lose acknowledged messages or leave consumers in the wrong place after a failure. This book lets engineers and reliability teams test replication, consumer recovery and data-loss boundaries on purpose, and record what the platform really guarantees.

Part I is a practitioner guide covering failure modes in streaming platforms, a reference cluster and lab set-up, replication and acknowledgement settings, broker and controller failures, network partitions, and producer delivery guarantees. Part II is a field toolkit of labs, case studies and annotated templates, including a full reference lab, failure injection drills from broker kill to partition heal, and automated continuous validation. Part III offers reference and practice material with workbooks, templates, tabletop exercises and a glossary. Part IV is an evidence casebook of worked cases, and Appendix A provides checklists.

Later chapters cover consumer groups and offset recovery, exactly-once semantics and transactions, cross-cluster replication and disaster recovery, measuring data-loss boundaries, and runbooks and game days, ending with a worked case on a zero-loss payments stream.

This independent practitioner reference is written for platform engineers, site reliability engineers, data architects, resilience and risk managers, and assurance teams.

Every chapter is built for working professionals: a real headline, a dramatised boardroom scene, decision tables, worked examples with expected results, the evidence to keep, a checklist, the board question and a self-test. The book closes with a capstone assessment, consolidated checklists, model answers, a standards crosswalk, a certification domain map for CISSP, CISM, CRISC and CISA, and a full index.

Professor Kai London