Railway's write-up of its October security incident has one line I can't stop chewing on. During cleanup, the team rotated the credentials its services use to write to storage. That was the correct move. Then database backups quietly stopped landing in storage. They didn't resume until the affected services were redeployed.
Plenty of platforms hand a program its secrets once, at startup, usually as environment variables. The program copies the value into memory and never looks again. Change the stored secret and every process that's already running keeps presenting the old one, which storage now rejects.
So a rotation is two jobs. You change the secret, and then everything that uses it has to restart and pick up the new value. The first job gives you a confirmation on screen. The second gives you silence, and so does a broken backup job. The database keeps serving traffic the whole time, looking perfectly fine, and nobody learns otherwise until they need a restore.
Before you call a rotation done
- Restart what reads secrets at launch: In Kubernetes, a container that gets a secret as an environment variable keeps the old value until it restarts.
- Deploy the staged change: On Railway, edits to variables sit as pending changes until you deploy them.
- Check secrets mounted as files: Kubernetes refreshes these on its own, after a delay. Files mounted with
subPathnever refresh. - Watch archiving separately: PostgreSQL keeps running even when it falls behind copying its log files to backup storage.
- Watch the local disk: Log files that fail to copy pile up on the database server. If that disk fills, PostgreSQL shuts down.
- Look for gaps: Restoring to a specific moment needs the full backup plus every log file written after it. One missing file breaks the chain.
- Confirm the backlog cleared: Check that copying restarted and that it caught up on everything that piled up while it was stuck.

