The site returns errors after a deploy

Troubleshooting, Medium

Your web service runs on 8 instances behind a load balancer, and a rolling deploy finished at 10:00. At 10:10 the error rate is 25% (it was 0.1%). You are on call. What you know: - The errors are HTTP 500 and only on POST /checkout. Checkout is about 25% of all requests. Other pages read from a replica and are fine. - The errors started at 10:02. - The log line on every instance is: FATAL: remaining connection slots are reserved for non-replication superuser connections. - The new release raised the database pool size from 10 to 30 connections per instance. The database allows 200 connections in total. Other services use about 40 of them. Say what you do in the first 15 minutes, in order, with the reason for each step and the commands or dashboards you use. Then say what caused it, how you confirm it, and how you stop it coming back.

Examples

Input git diff v1.41..v1.42 --stat, or the release notes

Output 14 files changed: config files, checkout code and 3 dependency updates.

Why A release holds many changes. Use the time line and the log line to choose which ones to read first.

Input SELECT usename, count(*) FROM pg_stat_activity GROUP BY usename; on the primary, next to SHOW max_connections;

Output web_app 160, billing 25, reports 15 (total 200). max_connections = 200.

Why Shows who holds the database connections right now, and the limit to compare them with.

Input Decoy: the CPU and memory dashboard for the 8 web instances

Output CPU about 30%, memory normal, no restarts.

Why It shows the instances are not short of CPU or memory. It does not tell you why checkout fails.

Constraints

8 instances. The rolling deploy finished at 10:00 and the errors started at 10:02.
Only POST /checkout fails (HTTP 500). It is about 25% of requests and customers cannot pay.
The database allows 200 connections. Other services use about 40. The new release raised the pool size per instance.
You can roll back the release and read the database. Do not restart the database.

Your answer must contain

An ordered list for the first 15 minutes, with the reason for each step
The commands or dashboards you use
The cause, tied to the facts above, and how you confirm it
How you stop it coming back