172,629 writes, 85 days, and a rate that misled 3 times

In short. Cadence mirrors callback scheduling from a CRM onto a team calendar. There is no webhook on the source, so every cycle reads the whole working set again and reasserts it. Across 85 days and 11,821 cycles the trace stream caught the success rate misleading 3 separate times. Read at cycle grain the mirror looks near perfect, 12 bad runs out of 11,821. Read at record grain, which is what somebody sees when they open their calendar, first pass accuracy is 97.664%, and the mirror converges to 99.9987% within 30 minutes because every pass reasserts. The daily rate then falls 1.35 points while reliability never moves, because throughput dropped 42% underneath a roughly fixed failure count. And the failures turn out to be host loss rather than rejected writes, which is what makes stateless reassertion worth its apparent waste. None of the 3 came from watching the system. All 3 came from having written down what it did.

Cadence mirrors callback scheduling out of a CRM and onto the team's calendar. There is no webhook on the source, so every cycle reads the working set again and reasserts the full state of every record, once every 10 minutes.

It has now been running for 85 days. 11,821 completed cycles. The trace stream logged all of it, which means for once there is enough evidence to ask a harder question than did it work.

The harder question is what the success rate is actually telling you, because over this window it told me 3 different things and 2 of them were wrong.

First, what it is measured against

Not 100%. The comparison is the process that ran before it.

A customer service rep read the callback out of the CRM and typed it onto the calendar by hand, against a scheduling surface not built for it. Every entry was a chance to mistype a time, miss one under load, or skip it entirely, and a missed entry is a customer who does not get called. There was no retry underneath any of that. A miss stayed missed until somebody noticed.

Every number below is the residual left after an automated pass plus recovery, on volume no manual process was absorbing. Read it against that.

2 success rates, same window, both correct

4,033 record failures produced only 12 failed cycles. A cycle can finish cleanly while records inside it fail, so at cycle grain the mirror looks close to perfect.

Record grain is the one that matters, because record grain is what a person sees when they open their calendar. Nobody has ever been let down by a cycle. They get let down by a callback that is not there.

Any monitoring that reports the run rather than the row will tell you this system is fine roughly 336 times for every time it is not.

Where it actually ends up

97.664% is where a record lands the first time. It is not where the mirror ends up, because a failed record is retried on the next pass whether or not anything noticed it failed.

97.664%

First pass accuracy, measured. 168,596 of 172,629 attempts landed first time

97.664%

Recovery accuracy, modelled. Share of a pass's 4,033 failures cleared by the next pass, within 10 minutes

99.9987%

Overall accuracy, modelled. After 3 passes, within 30 minutes, leaving a residual of 0.00128%

Only the first of those 3 is a count. The other 2 carry that measured rate across passes, which is a derivation from 85 days of real data rather than a forecast of anything. The single assumption inside it is named near the end. A derived figure printed as a measurement is the 1 error here that would actually matter.

The line was falling. Reliability was flat.

This is the one I would have got wrong without the trace stream.

Plot daily record success across the window and it declines, 1.35 points from the first 20 days to the last 20. That looks like a system degrading, and the honest first reaction is to go find what broke.

Nothing broke.

0.3380 to 0.3368

Failures per cycle, first 20 days against last 20. Flat to within half a percent

17.90 to 10.41

Records per cycle over the same stretch. Throughput fell by 42%

The failure count per cycle did not move. The denominator did. A roughly fixed number of failures divided into a shrinking volume prints a falling percentage, and the percentage is the only thing anybody was looking at.

That is a general trap, not a Cadence one. A rate is a ratio, and a ratio moves when either half moves. If you only ever chart the rate you cannot tell which half did, and the 2 have opposite meanings. One is your system getting worse. The other is your business getting quieter.

The fix is not a smarter metric. It is charting the numerator underneath the rate, on its own axis, so the 2 can disagree in public.

The failures were not rejections

The second correction the trace stream forced.

I had assumed the 4,033 failures were writes the destination refused, which is the normal shape of this, a rate limit or a validation error. They were not. The failures cluster tightly in windows where the host machine was not running, and twice in this window it was powered off by accident.

So they are not writes that were attempted and refused. They are writes that never happened, during a stretch when nothing was polling at all.

That distinction changes what the design is for. Because every cycle reads the working set again and reasserts full state rather than draining a queue, the mirror does not need to know it was ever off. It reads the world on the next pass and the divergence closes on its own. There is no backlog to replay and nothing in flight to lose.

Stateless reassertion reads as wasteful right up until the power cuts.

It also sharpens the caveat further down rather than softening it. Clustered failures resolve together when the host comes back, so treating them as independent biases the derived figure conservative rather than optimistic.

Recovery load does not follow volume

81 full days in the window. Throughput across them ranges by a factor of 3.32, from 841 records in a day to 2,794. Over the same days the recovery load ranges by a factor of 1.22, from 46 records to 56.

Correlation between the 2 is 0.103, so throughput explains about 1% of the variance in recovery load.

That is a diagnostic, not a curiosity. If the mirror were bumping a write ceiling, recovery load would scale with volume, because more writes into a fixed budget means more refusals. It does not scale. It tracks cycle count instead, at a steady 0.34 failures per cycle regardless of how much work each cycle carried.

Which says the write ceiling has not been reached, and the 10 minute interval has room to come down.

Every time in this system is the interval, not the work

A typical cycle finishes in 7.42 seconds and uses 1.24% of its 10 minute window. The worst cycle ever recorded took 130.7 seconds and still left 78% of the window unused.

Nothing in the pipeline is anywhere near a time limit. So detection lag, repair time, and convergence time are not properties of the code. All 3 are consequences of 1 configuration value, the loop interval, and all 3 shorten together if it does.

Duration tracks throughput as well, so a shorter interval is also a cheaper cycle. Median cycle time already fell from 8.94 seconds to 5.86 as daily volume halved.

What is measured and what is modelled

First pass accuracy is a direct count. Recovery and overall convergence are derived from it, because there is no retry queue to measure. Every cycle reasserts the whole working set, so a failed record is retried 10 minutes later whether or not anything recorded that it failed.

Nothing here is forecast. The failure rate is 4,033 out of 172,629 attempts across 85 days, measured. The derivation carries that rate across 3 passes, and its only assumption is that successive failures on the same record are independent. The dominant failure mode, host loss, does not satisfy that. Aggregate behaviour still supports it, each record in the working set absorbed roughly 283 failures across the window and the mirror never degraded, which permanent failures could not produce. Settling it per record would take the raw trace stream rather than the rollups.

I would rather publish that sentence than a cleaner number without it.

What the trace stream was actually for

None of the 4 corrections above came from watching the system. They came from having written down what it did, every cycle, for 85 days, in a form something could query later.

The instrument that measures the work is part of the work. Build it second and you get a number. Build it first and you get the argument about what the number means, which is the part worth having.

What transfers

The denominator trap is not a Cadence problem. Every rate is a ratio, and a ratio moves when either half moves, so a falling percentage is either your system getting worse or your business getting quieter, and the chart alone cannot tell you which. Anywhere a dashboard reports a rate without the count underneath it, that ambiguity is live and nobody can see it.

Stateless reassertion transfers wherever a mirror has no webhook on its source. A queue has to know what it missed. A pass that reasserts full state does not, so an outage costs a window of divergence rather than a backlog and a replay. It reads as wasteful in every design review until the first time the host goes down.

The grain question applies to any monitoring at all. If the thing you alert on is the run rather than the row, you will be told the system is healthy in exactly the situation where a person is being let down by it.

The architecture, the 2 layer dedupe and the 3 log streams that produced this trace, is on the work page.