Stateful containers are a symptom of complexity (and architectural insanity)

Pragmatism or self-induced legacy?

I was listening to DevOps and Docker Talk's early October episode titled Move K8s Stateful Pods Between Nodes^ and it reminded me of some pet peeves. None of them are better exemplified than:

"95% of the customers I deal with don't do Kubernetes the right way" - Phil Andrews (cast.ai)

To me, putting stateful workloads in containers is madness. It’s another example of a recurring pattern I’ve seen throughout my career: the tendency of enterprise IT professionals to reinvent the wheel. Their inner engineer can't resist.

Containers are best suited to ephemeral, immutable, stateless workloads. This was elegantly summarised by my former colleague Tony Iams:

Programmable

Resilient

Immutable

Modular

Elastic

Declarative

The podcast covers many of the challenges of stateful containers, including.

• Networking and connection state management: maintaining TCP/IP connections, session state, and preserving the pod’s IP address during migration is a major technical hurdle. Kubernetes deployments often use different CNIs (e.g., Calico, Cilium, AWS VPC). Reestablishing/replaying traffic flows (especially for long-running jobs and web sockets) demands precise ordering and coordination.​

• Storage and data replication: replicating memory and disk state isn't easy (and was a big part of the magic of VM live migration. Migrating persistent volumes involves API-driven detach/reattach steps that add lag/delay. Local ephemeral storage or NVMe disks may require lengthy background replication. Managing database logs, mirroring, and file access safely during migration remains a difficult engineering problem—one that was solved by modern hypervisors 20 years ago.​

• Workload complexity and statefulness: many clusters are a mix of poorly architected or fragile workloads, making bin packing and node reduction challenging without risk of interruption.​

Stateful containers are often more like pets than cattle

• Resource/time constraints: AI/ML/tuning jobs often have large memory/state requirements. There are physical limits to the speed of data transfer or replication. Migrating jobs with hundreds of GBs or terabytes of working data within short time frames can be a major challenge.​

• Limits of Kubernetes orchestration: coordination between pod names, API calls, containerd states, and Kubernetes control plane is required so the pod is instantiated in a paused state, assigned the right resources, and then switched over to active. Order of cutover events (pod name uniqueness, IP address transfer, routing table updates, etc.) must be handled precisely—any discrepancy may cause failures.​

Most of these problems were already solved by other technologies.

Use the right tool for the job--even if it's not shiny and new.

Stateful is ideally scale-up and either physical or virtual. Modern apps are typically scale-out and better suited to containers, serverless (and/or AI).

Whether you agree or not, the podcast provides plenty of food for thought...and I fully acknowledge that theory and practicality rarely intersect. For those facing these challenges, it might be a good idea to take a look at what cast.ai has developed.

Footnote: I use Podcast Addict, which is free, so I reckon they deserve a free plug in return. Use of their free tool, and the content of the podcast, are the full sum of my compensation for publishing this article. I have no financial relationship with any of the parties involved,