
Every engineer knows the rule: never rewrite a working system. For three years we followed it. The HUB365 sync engine moved millions of records a day, clients were happy, and every incident had a patch. Then the patches started patching each other.
The original engine was designed in 2020 for ten clients and a single region. By 2025 it served two hundred clients across three regions, and every assumption baked into its core — one writer, ordered delivery, small payloads — had quietly become false.
The moment the math flipped
We tracked it honestly for two quarters: 38% of engineering time was going into keeping sync healthy. Not building — keeping alive. When maintenance costs more than a rebuild amortized over two years, the "risky" option is staying put.
So we wrote the smallest possible spec. No new features. Same external behavior, byte for byte, verified by replaying six months of production traffic against both engines in parallel.
A rewrite is only reckless when you change what the system does. We changed nothing — except everything underneath.
What we would do differently
Start the shadow traffic replay on day one, not month three. The replay harness found nine behavioral differences we would never have caught in review — including two bugs in the old engine that clients had silently built workarounds for.
We kept both bugs. That was the hardest and most correct decision of the project: compatibility means compatible, not correct.
The new engine shipped in March. Sync incidents are down 94%, and for the first time since 2023, the on-call channel went a full month without a page.



