I built a backup tool in a day, and almost every bug was a near-miss

The Starting Point

I had a little internal tool — vanilla PHP, no framework, no composer, no git — that did two things: migrate MySQL databases between servers, and sync S3 buckets with rclone. Two tabs, some JSON files for saved connections, deployed on a 1-core VPS.

What I wanted: scheduled nightly backups. Databases to a backup server and/or dated .sql.gz dumps, files between any combination of local paths and S3 buckets, and an audit trail so I could actually prove a backup happened.

By the end of the session it had grown a lot more than that. Five tabs, ~4,250 new lines, TLS, and two-factor auth. But the thing worth writing down isn't the feature list — it's that four separate bugs would have quietly corrupted or deleted backups, and every one of them looked fine on the happy path.

What Actually Happened

The one that would have been catastrophic

I was writing the shell script that does the dump:

mysqldump --defaults-extra-file=src.cnf "$DB" | gzip -c > dump.sql.gz

Standard stuff. Except in a pipeline, $? is the exit status of the last command. If mysqldump dies halfway — connection drops, table locks, whatever — gzip happily compresses the partial output and exits 0.

So you get a truncated dump, exit code 0, and a big green "success" in your audit trail.

I tested both versions side by side with a fake mysqldump that exits 2 after printing a couple of lines:

### Case 1: mysqldump exits 2 mid-pipe (with pipefail)
!! mysqldump failed
   script exit = 1
   .sql.gz published? 0

### Case 2: the SAME pipeline without pipefail
== reported SUCCESS
   script exit = 0
   .sql.gz published? 1     ← a truncated dump, marked good

set -o pipefail (which means bash, not sh). Plus gzip -t on the output before publishing, and writing to dump.sql.gz.part and only mv-ing it into place once it verifies. A disk that fills mid-write leaves a .part that never gets promoted.

That's the whole reason I'd rather generate a shell script than shell out per-step from PHP — the failure semantics are visible and testable.

Deploying to a server that's ahead of your local copy

The server had been edited directly over months. Local had drifted. I knew this going in, so before every upload I diffed my file against the server's copy.

Twice that caught me about to delete the auth system:

removed: -        $users = APP_USERS;
removed: -        if (isset($users[$username]) && password_verify($password, $users[$username])) {

My local index.php still had the old single-user login. The server had a multi-user map. Uploading "my" version would have locked everyone out of the tool that holds every database credential.

The fix was a habit, not a clever tool: pull the server's copy, apply the edit to that, diff, then upload. After the second near-miss I added a guard to the install step that refuses to deploy an index.php that doesn't contain APP_USERS. 48 predeploy snapshots later, that guard never fired again — but it's there.

"It times out" — a 522 that wasn't DNS

Put the tool behind a real domain via Cloudflare. It timed out.

http:// through Cloudflare: 200. https://: 522 after 19 seconds.

Cloudflare's SSL mode was Full, meaning it connects to the origin over HTTPS on port 443. nginx was listening on 80 only. Nothing on 443, no certificate, so the connection just hung.

Fixing it properly (Let's Encrypt rather than downgrading Cloudflare to Flexible — this app posts database passwords, the CF→origin leg shouldn't be plaintext) ran into two more:

The nginx config already blocked the ACME challenge. There was a sensible location ~ /\. { deny all; } for dotfiles, and /.well-known/acme-challenge/ matches it. Renewals would have failed silently until the cert expired months later. The fix is location ^~ /.well-known/acme-challenge/ — the ^~ modifier out-ranks regex locations.

http2 on; isn't valid on nginx 1.24. That directive landed in 1.25.1. On 1.24 it's the listen flag: listen 443 ssl http2;. Caught by nginx -t before reload, which is the only reason it wasn't an outage.

I also deliberately did not add an http→https redirect at the origin — behind a proxy that terminates TLS, that's how you get redirect loops and broken renewals.

The bug that looked like S3 and was actually sessions

Reported as: "adding a new bucket, the test keeps failing, but it works in terminal."

I went looking at rclone config, credential encoding, IAM permissions. Found genuinely interesting stuff — one saved key returned InvalidAccessKeyId, meaning it had been deleted upstream — but none of it was the problem.

Then I asked what the error actually said. "Not authenticated."

That's not an S3 error at all. That's the PHP session check. The session had expired while they were tabbed over to the cloud console copying credentials.

Why so fast? The config file declares SESSION_LIFETIME = 3600 — and nothing in the app ever reads that constant. It's decorative. The real limit was PHP's default session.gc_maxlifetime of 1440 seconds (24 minutes), and the distro's phpsessionclean timer runs at :09 and :39 deleting anything older.

The gotcha in fixing it: the obvious move is php_value[session.gc_maxlifetime] in the FPM pool config. That would have made PHP keep sessions alive while the cleaner still deleted the files, because the cleaner script reads the SAPI's php.ini and conf.d — it never looks at pool files. I only found that by reading /usr/lib/php/sessionclean:

session_config=$(PHP_INI_SCAN_DIR=/etc/php/${version}/${conf_dir}/conf.d/ ... )

So the setting has to go in a conf.d drop-in. Verified through nginx with a temporary probe, because checking via the CLI binary reads a different conf.d and gives a false reading.

While I was there I found why the original error was so unhelpful:

setStatus(statusEl, 'error', truncate(r.error || 'Failed', 80));

rclone puts the useful part at the end of a long line, after a RequestID and HostID. Truncating the front threw away the only bit that explains anything. Now it extracts api error <Code>: <message>:

before:  operation error S3: ListBuckets, htt…
after:   InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.

An unhelpful error hiding a one-word typo

A dump uploaded to S3 failed with:

api error IllegalLocationConstraintException: The unspecified location constraint
is incompatible for the region specific endpoint this request was sent to.

Completely opaque. The clue is the operation name: rclone was calling CreateBucket, not PutObject. When rclone can't see a destination bucket, its default is to create it — and that create request is what failed.

The bucket name was simply wrong. But a backup tool should never create a bucket, so:

before:  IllegalLocationConstraintException: The unspecified location constraint…
after:   NoSuchBucket: The specified bucket does not exist

--s3-no-check-bucket on every write. Same failure, one obvious message.

ContentLength=5565 with Body length 1820

Then every single file in an object-storage→S3 backup started failing:

NOTICE: branding/logo.svg: Not decompressing 'Content-Encoding: gzip' compressed file
ERROR : Failed to copy: ... PutObject ... transport connection broken:
        http: ContentLength=5565 with Body length 1820

Throughput collapsed from 878 KiB/s to 800 B/s with an ETA over a day, because every failing file held one of the four transfer slots through its retries.

The source was R2-compatible storage, which serves some objects with Content-Encoding: gzip. The saved connection recorded provider = "Other" — so rclone applied no provider-specific handling, the body length disagreed with the declared length on upload, and the connection got dropped.

I tested four variants against the actual failing object:

I took the provider fix rather than the flag, because --s3-decompress is global to every remote in the command, whereas the provider is per-remote and simply correct. The provider is now derived from the endpoint — object storage endpoints are unambiguous, and AWS never uses a custom one.

Verified the copies were byte-for-byte identical afterwards with matching SHA-256, because "it stopped erroring" is not the same as "the data is right".

The retention bug that would have hollowed out backups

Snapshot mode: every run copies the whole source into a new timestamped folder. Which needs pruning, or storage grows forever.

The obvious implementation is rclone's --min-age. It would have been quietly destructive. --min-age compares object modification times, and a copy preserves the source's modtime. So a file last edited three years ago, sitting inside last night's snapshot, looks ancient. You'd end up with a folder that exists, appears current, and has been gutted from the inside.

Pruning by folder name instead — the only thing that reliably records when a snapshot was taken.

And then the near-miss inside the fix. My first pattern was:

[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]*) ;;

That trailing * matches any folder that merely starts with a date. My test listing included 2026-09-15_backup_notes and it was marked for deletion. Tightened to exact shapes only:

PURGE   2026-09-01_020000
keep    2026-09-20_020000
SKIP    _replaced                 (not a snapshot name)
SKIP    important-do-not-delete   (not a snapshot name)
SKIP    2026-09-15_backup_notes   (not a snapshot name)

Two "bugs" that were my test being wrong

A stale-response race that was real. I made one connection respond in 2.5s and another in 100ms, then switched between them fast. The slow reply landed last and overwrote the newer one — the dropdown showed connection A's buckets while connection B was selected. Fixed with a sequence token that drops superseded replies. That one was genuine, and I'd never have found it without deliberately making a request slow.

Two that weren't. A save looked like it silently failed — turned out my local PHP uses bcrypt cost 12 (~1s per hash) while the server uses cost 10 (~52ms). Hashing ten recovery codes took 10 seconds locally and 0.5s on the server; my test read the result too early. And a valid TOTP code was rejected at login — which was the replay guard working correctly, because my test was fast enough to reuse the same 30-second window as the enrolment code.

Both cost me time. Both were worth confirming rather than "fixing".

The CPU question

Asked whether more CPU would help. Checked rather than guessed:

            %user  %system  %idle      network
21:30        4.03     0.86   93.02     746 kB/s in
21:40        1.16     1.33   95.83     ← during the failing transfer run

4% CPU. The workload is 778 files averaging ~160 KB — dominated by per-request latency, not computation. More cores don't make a round trip faster. Raised rclone's --transfers from 4 to 16 instead. That costs nothing: rclone only allocates chunk buffers for multipart uploads above a 200 MB cutoff, so small-file streams are nearly free.

Separately, a CPU spike they'd noticed turned out to be the distro's own apt-daily-upgrade consuming 1min 46s of CPU after a reboot made the timers overdue. Nothing to do with the app.

Ending on something that wasn't mine

Last deploy of the session, final health check: HTTP 000. Exit code 6 — DNS resolution failure.

The subdomain returned NXDOMAIN from the authoritative nameservers, while the parent domain resolved fine. It had worked 24 minutes earlier. I never touch DNS.

Worth calling out because the instinct when something breaks right after your deploy is to assume it's yours. Two commands (dig against the authoritative NS, and curl to the origin on localhost returning 200) separated "my change broke it" from "the record is gone".

What Worked

The shape that made all of this testable: PHP generates a shell script, spawns it detached, and reaps it via a sentinel.

$inner = sprintf(
    '%s %s >> %s 2>&1; echo "__EXIT__ $?" >> %s',
    escapeshellarg(BACKUP_BASH_BIN), escapeshellarg($scriptPath),
    escapeshellarg($logPath), escapeshellarg($logPath)
);
$cmd = ($setsid ?: 'nohup ') . 'sh -c ' . escapeshellarg($inner) . ' > /dev/null 2>&1 & echo $!';

Three properties that paid off repeatedly:

1. A run survives everything. Detached from the PHP-FPM process group, so worker recycling, request timeouts and closing the browser can't kill a dump mid-write.
2. The sentinel is how you know it finished. You can't waitpid an orphaned process from PHP. __EXIT__ <code> in the log plus a kill -0 liveness check covers both "done" and "died without reporting" — the latter gets recorded as failed rather than sitting on "running" forever.
3. The script is a plain file you can read and run by hand. Every "is this actually safe?" question became a shell test instead of an argument.

The dump pipeline, with every guard:

set -o pipefail
trap 'rm -f "$DUMP.part"' EXIT          # no half-dumps left behind

mysqldump --defaults-extra-file=src.cnf ... | gzip -c > "$DUMP.part" || FAIL "mysqldump failed"
[ -s "$DUMP.part" ]   || FAIL "Dump file is empty"
gzip -t "$DUMP.part"  || FAIL "Dump failed its integrity check — truncated (disk full?)"
mv "$DUMP.part" "$DUMP"

Credentials go in a --defaults-extra-file (0600, deleted when the run ends) rather than on argv, where they'd be world-readable in ps.

Pruning runs last, and the script exits on first failure — so a failed backup can never delete a good one. That ordering is free and it's the difference between "last night failed" and "last night failed and took the archive with it".

Two-factor went in at the end: TOTP in ~40 lines of HMAC, verified against all five RFC 6238 test vectors before wiring up any UI. The QR is rendered in the browser from the otpauth:// URI — never by a third-party QR service, which would mean handing your shared secret to someone else's server. Recovery codes are bcrypt-hashed and single-use, and there's a CLI escape hatch, because locking yourself out of the thing holding every credential is not a recoverable mistake.

What I Learned

• set -o pipefail is not optional when a pipeline writes a backup. Without it, producer | gzip > file reports gzip's status. The default is to silently publish corruption. This is the single highest-value line in the whole project.
• Test the failure mode explicitly. Every real bug this session was found by making something fail on purpose: a dump that dies mid-pipe, a truncated gzip, a slow API response, a folder that isn't a snapshot. The happy path passed first time in every case.
• Prune by name, never by modification time. Copies preserve source mtimes, so age-based deletion inside a snapshot deletes the oldest content, not the oldest backup.
• A guard's pattern deserves its own test. A trailing * in a shell case turned "delete snapshots" into "delete anything starting with a date".
• When the server is ahead of local, diff before every upload. Not occasionally — every time. It caught the same class of mistake twice in one session.
• Read the first line of the error, not the loudest one. The gzip transfer failure had a NOTICE about Content-Encoding before a wall of red. The 522 was answered by noticing http:// worked. The "S3 test failing" was Not authenticated.
• Truncating an error message is a bug, not a UI choice. Cutting to 80 characters removed the only informative part of every AWS error.
• Provider strings aren't cosmetic. provider = "Other" versus Cloudflare was the difference between every transfer failing and everything working.
• Local-vs-production differences will make working code look broken. bcrypt cost 12 vs 10, setsid not existing on macOS, /var/www not existing. I wasted time on all three before checking the environment rather than the code.
• Measure before buying hardware. "Will more CPU help?" — 4% utilisation says no. The bottleneck was per-request latency; raising concurrency was free.
• Silent skips are worse than errors. The concurrency limit refuses a third simultaneous run before creating an audit row — so it leaves no trace in the UI and doesn't retry. Found it while answering a question, not while testing.

Next time: I'd write the failure tests first. Every single one of these was found by asking "what if this goes wrong" after the feature already worked — which means on a worse day I'd have shipped four data-destroying bugs that all passed their happy-path tests.

And I'd put the thing in git before writing 4,000 lines of it.

Tech Notes

Stack: PHP 8.3 (no framework, no composer), vanilla JS + Tailwind via CDN, nginx, rclone 1.74, MySQL, cron, 1-core VPS.

New files: ~4,250 lines across a backup engine, CLI scheduler, S3 browser, streaming download endpoint, folder picker, connection manager, and TOTP module.

Key flags and settings worth remembering:

set -o pipefail                  # or your backup lies to you
gzip -t file.gz                  # catches truncation a size check misses
rclone --s3-no-check-bucket      # never let a backup create its own bucket
rclone --transfers 16            # latency-bound small-file jobs; CPU was 4%
mysqldump --defaults-extra-file  # keeps the password out of `ps`
setsid sh -c '...' & echo $!     # detach from PHP-FPM, capture the PID

nginx / infra gotchas:

location ^~ /.well-known/acme-challenge/ { ... }  # ^~ beats the dotfile deny
listen 443 ssl http2;                             # `http2 on;` needs nginx 1.25.1+

; must be in conf.d, NOT the FPM pool — the session cleaner never reads pool files
session.gc_maxlifetime = 3600

Useful commands:

dig @<authoritative-ns> sub.example.com A   # NXDOMAIN vs misconfigured
sar -u                                       # historical CPU, answers "was it busy?"
systemd-analyze / journalctl -u <svc>        # "Consumed 1min 46s CPU" told me who spiked it
nginx -t                                     # caught an invalid directive pre-reload
php -l file.php                              # ran before every single deploy

Verification habits that earned their keep: RFC test vectors for the crypto, SHA-256 comparison after transfers, byte counts on downloads, and a deploy guard that greps for a required string and refuses to install without it.