A container restarts again and again after a release

Troubleshooting, Easy

You are on call for a Kubernetes cluster. At 14:05 a release of the report-worker service (version 2.4.0, 3 pods) finished. Since then the pods restart in a loop. At 14:25 you see: - kubectl get pods shows all 3 pods as CrashLoopBackOff with 9 restarts each. The service was stable for months with 0 restarts. - Each pod lives about 40 seconds and then ends. Customers see their report jobs stay in "queued" status. - kubectl describe pod shows: Last State: Terminated, Reason: OOMKilled, Exit Code: 137 and the container memory limit is 256Mi. - The last log line before every restart is loading customer export into memory (412 MB) with no error after it. - Version 2.3.0 ran at about 180Mi of memory. The cluster nodes have plenty of free memory. Say what you do in the first 15 minutes, in order, with the reason for each step and the commands you use. Then say what the cause is, how you confirm it, and how you stop it coming back.

Examples

Input kubectl get pods -w

Output Each pod goes Running, then Error, then CrashLoopBackOff. The age of each run is about 40 seconds and the restart count is 9.

Why Shows the pattern and how long each run lasts. It does not give the reason.

Input kubectl logs report-worker-xxxx --previous | tail -3

Output The last line is "loading customer export into memory (412 MB)". No error line follows.

Why Shows where the previous run stopped. A missing error line is a fact the answer must explain.

Input Decoy: kubectl top nodes

Output Every node is below 40% memory.

Why The nodes have free memory. This does not say whether one container ran into a limit of its own.

Constraints

You can run kubectl with edit rights on the namespace and you can deploy. You cannot change the nodes.
The release is the only change in the last 7 days. The data store the worker reads is healthy.
Customers are waiting, so protect them first. Do not delete the pods or the namespace.

Your answer must contain

An ordered list of steps, with the reason for each
The commands or dashboards you use
The cause, tied to the facts above, and how you confirm it
The fix and what you do so it does not happen again