How a sales floor alerting system was reasoned into existence, wrong turns included.
In short. A sales floor flagged high cost agents at 2 standard deviations above the cohort mean. The rule failed 4 ways. The underlying process is geometric rather than normal, so "2σ" overstated rarity by an order of magnitude. The threshold loosened through the day as dispersion collapsed. Outliers inflated the standard deviation that defined them, so clusters escaped detection. And agents with 0 sales had no cost per acquisition at all, making the purest waste cases structurally invisible. It was replaced with the quantity the ratio had discarded, the binomial tail probability of an agent's conversions under a leave-one-out peer rate, with the cutoff derived from what each kind of error actually costs.
Why this is written as a log
The final rule is 4 lines, and the 4 lines are not the interesting part. As a piece of statistics it is unremarkable, an exact binomial test against a peer benchmark, the kind of thing you could write down in a minute.
What took the time was establishing that this was the right test rather than one of the 5 reasonable alternatives considered first. Each of those was defeated by something specific about how this operation works, a billing convention, an attribution rule the CRM was already conserving upstream, an algebraic invariance that reassigned blame for a mistimed alert, a 10 lead gap in an actual roster. Those facts are not general. They were not available to any procedure that hadn't looked at this floor.
So the corrections are left in, in the order they happened. They are the part that did the work. The 4 lines are what survived them.
Before stage 0, the 2 systems this one ate
Herald is not the first instrument pointed at this floor. 2 things came before it, and both sit on the work page as retired.
The first was a live sales feed. Every close pushed out of the CRM into the team's messenger the moment it landed. It worked, and the parts that usually break did not, polling was idempotent, the dedupe pruned itself, a restart never reblasted the day. What it could not do was choose. A close is a close, so it announced all of them, and a floor that hears everything eventually hears nothing.
The second was a cost tracker, agent profitability watched live, with a watch list naming whoever was running hot. That is the instrument this post opens on. The σ-rule in stage 0 was its rule.
Herald is those 2 folded into 1, plus the thing neither of them had. The feed knew what happened. The tracker knew what it cost. Neither had any way to decide whether the room needed to be told. Everything below is that decision being worked out, and the reason it takes 5 discarded models to get there is that "worth interrupting the floor for" is a far harder claim to earn than "this happened".
Stage 0, the starting instrument, and why it was wrong
The system began where most do, flag an agent when their cost per acquisition (CPA) exceeds the cohort mean by 2 standard deviations. This is the reflexive answer, and it survived exactly as long as it took to examine the distribution it assumed.
The σ-rule presumes approximate normality. The generating process here, leads consumed until a conversion lands, is a geometric waiting time. The first empirical check settled it, cohort mean $124.62, standard deviation $128.70, a coefficient of variation (CV) of 1.03. A CV of unity is the signature of an exponential or geometric process, not a Gaussian one. Under that distribution, the mean plus 2 sigma point is not a 1 in 44 event as normal theory claims, it is closer to 1 in 20. The label overstated rarity by more than an order of magnitude.
This was the first domain fact doing work, the shape of the process was knowable from the CV, and the CV came from the actual numbers, not from an assumption.
Stage 1, four structural failures, found by interrogation
Once the distributional mismatch was visible, the σ-rule was pressed on every axis. 4 independent failure modes surfaced, 3 of them specific to fixed cost lead economics.
- Distributional misspecification. Covered above, normal theory tail probabilities are wrong when CV ≈ 1.
- Threshold drift. The coefficient of variation was not stable across the day. It fell from 1.03 mid morning to 0.83 by early afternoon, because early in the day nearly every agent sits at 0 or 1 sale, maximally dispersed, and later they compress toward the middle as second and third sales land. The consequence, the same "2σ" label denotes materially weaker evidence as the day advances. Measured at 2 points, the probability of a flagged result arising by chance more than doubled, from roughly 12% to 29%, for the identical rule. It was not the same test at 2 times of day.
- Mutual masking. Because an outlier inflates the standard deviation that defines the outlier threshold, extreme agents raise their own bar. Constructed against a realistic cohort, 3 identical high cost agents drove the 2σ threshold above their own value, all 3 escaped detection. The rule was quietest precisely when the floor was worst, 1 bad agent gets caught, a bad day produces silence.
- Survivorship exclusion. Cost per acquisition is undefined at 0 sales, the denominator is zero. An agent who has consumed leads and produced nothing has no CPA, cannot exceed any dollar threshold, and is absent from both the mean and the standard deviation. The purest case of the exact waste the system exists to detect is the 1 case the metric structurally cannot represent.
The fourth is the one worth dwelling on, because it is the kind of observation that only comes from staring at the actual metric definition. It is not a subtle statistical point. It is a domain point wearing statistical clothing, the worst performers are invisible because you cannot divide by their zero.
Stage 2, the correction sequence (the part that proves authorship)
What follows is the actual order in which hypotheses were raised and discarded. Each was a reasonable model. Each was killed by a specific fact about how the operation bills and credits. This sequence is the core of the document.
Wrong turn A, "transfers inflate cost by double billing"
The floor allows an agent to receive a lead and pass it to a colleague for enrollment. The initial hypothesis, each touch bills, so a transferred lead costs $40 twice, and per agent CPA is corrupted by an attribution artifact, cost landing on one agent, the sale on another.
A full model was built on this. It was wrong. The correction came from the billing rule, a lead is charged once, regardless of how many agents touch it. The error was conflating cost allocation, which ledger a charge is displayed on and which can be more than one, with cost incurrence, which happens exactly once. No general model would have caught this without being told the billing convention. The hypothesis is more natural than the truth. It was discarded only because the operation's actual pricing contradicted it.
Wrong turn B, "per agent attribution is broken and needs fixing"
Following A, the next hypothesis was that credit assignment needed correcting in software, that transferred sales required a split credit scheme to keep per agent CPA honest.
Also wrong, and for an instructive reason, the CRM already conserved the ratio. Its attribution credits a passed off sale to the originating agent as well as the enroller, so credit duplicates in exactly the pattern that keeps the per agent figure internally consistent. The fix that was about to be built already existed upstream. The correction came from checking what the system already did before adding to it, a discipline that is the opposite of how generated solutions accrete.
Refinement C, "credit duplicates, cost does not"
The precise, surviving statement emerged from A and B together, cost is incurred once per lead, credit may attach to more than one agent. This asymmetry is not a bug, it is the actual accounting, and it has a useful consequence. Because credited sales exceed true enrollments by exactly the number of transferred closes, the divergence between the bottom up agent roll up and the top down agency figure measures the transfer rate for free. A latent metric fell out of correctly understanding the books.
Wrong turn D, "the relative threshold fails at the morning peak"
A manager had forwarded an alert about 2 hours after open, and the timing felt wrong. The initial diagnosis blamed the relative (2σ) threshold for firing too early against an immature ratio.
Wrong, and the correction was a proof rather than a fact. Cost lags through the day because the numerator (spend) bills instantly while the denominator (sales) completes slowly, so agency CPA runs very high early and decays toward target. But a standardized score is invariant to any shock that moves the whole cohort together. Writing the diurnal effect as either an additive or a multiplicative common factor B(t), the algebra cancels B(t) out of the z-score entirely. The relative threshold was the only construct that already handled the morning, it was the fixed threshold that could not. The instinct that something was mistimed was right, the blame was misassigned, and the derivation reassigned it.
Correction E, "the alert had drifted beyond reach"
At a single snapshot, only 1 agent was flagged and the question arose whether the threshold had inflated past what any agent could physically hit. The hypothesis, the aperture had closed.
Wrong. Inspection of the full roster showed the flagged agent held 24 leads against a next nearest of 14, a 10 lead gap. Nobody else was near the profile the flag required. The system was not blind, the agent was genuinely alone in the tail. The correction was empirical, look at the whole distribution before concluding the instrument is broken.
5 hypotheses, 5 corrections. None of them came from statistics. Every one came from a fact about the operation, how it bills, how it credits, when it peaks, what the roster actually looked like. That is the fingerprint of derivation.
Stage 3, the reframe that dissolved the metric
The accumulated corrections forced a single realization, under fixed unit cost, cost per acquisition carries no information that conversion rate does not.
The algebra is immediate. With every lead priced identically at a constant, an agent's CPA is the constant divided by their conversion rate, a strict, monotone bijection. Ranking by CPA and ranking by conversion rate produce the identical order. The dollar sign is decorative. Any claim that CPA measures "cost efficiency" while conversion rate measures "skill" is vacuous when there is no cost variation to measure.
This is why no repair of the σ-rule could succeed. CPA is a ratio, and a ratio discards the magnitude of its own denominator. 1 sale on 5 leads and 1 sale on 25 leads can be the same CPA, yet they carry vastly different evidence. The metric threw away the single quantity, sample size, that separates a poor performer from an unlucky one. No threshold placed on a quantity can recover information the quantity has already discarded.
The decisive demonstration was temporal. Modeling an agent who converts at half the floor rate, with sales landing as discrete events, the CPA "sawtooth" has a property that indicts the σ-rule completely, its peaks decay as sales accumulate, so after the third sale the metric can no longer physically reach the threshold, no matter how badly the agent performs thereafter. The σ-rule fires twice, early, during the noisiest window, when the evidence for action is weakest, and then goes permanently silent exactly as the evidence becomes conclusive. The incumbent statistic was not merely blind, it was anti-correlated with the evidence.
Stage 4, the replacement, and why each piece is forced
The metric was replaced with the exact quantity the ratio had discarded, the tail probability of the observed conversions under the peer rate.
r = (agency.sales - agent.sales) / (agency.calls_billable - agent.calls_billable)
flag when binom.cdf(agent.sales, agent.calls_billable, r) < 0.10
Every element is forced by a prior correction, not chosen for elegance.
The exact binomial, not a normal approximation, forced by Stage 0. The process is non-normal and samples are small, only the exact test is valid at any n and near the 0 or 1 boundaries.
Leave-one-out peer rate, forced by the masking failure (Stage 1, mode 3). If the subject is included in the rate they are measured against, a high volume agent depresses their own benchmark and partially self exonerates. Excluding them is the same bias reduction discipline that makes a median more honest than a mean under outliers. On real numbers, this exclusion inverted which of 2 agents was correctly flagged, the pooled rate named one, the leave-one-out rate named the other, and the leave-one-out answer agreed with the evidence ordering.
Diurnal invariance, inherited not added, because the peer rate is computed from the concurrent cohort, any floor wide depression of conversion enters the expectation and cancels from the comparison. The morning problem is handled without any time indexing, for the same algebraic reason the z-score was invariant in Stage 2.
Self gating on small samples, an agent with 5 leads and no sales scores far from any threshold, it takes 12 leads at 0 sales before the test can fire. The rule refuses to flag thin samples as a matter of arithmetic, not policy. This retired an entire proposed mechanism, a 2 hour silence window after open, as redundant. The clock gate was a proxy for "enough evidence," the test measures the evidence directly, so the proxy was unnecessary.
Basis consistency, because credit duplicates while cost does not (Refinement C), the sale count and the lead count sit on different bases, an enroller can acquire a credited sale with no corresponding billable lead. The denominator was redefined as leads the agent worked, billed or not, so both counts share a basis and the peer rate stays honest. This is a correction only visible to someone who understood the credit and cost asymmetry, which was itself the product of 2 earlier wrong turns.
Stage 5, the threshold is a decision, not a statistic
The final element is the one most clearly beyond a generated answer, because it crosses from statistics into a second discipline.
The cutoff, the 0.10, was not chosen by convention. It was derived as a break even point in statistical decision theory. The 2 possible errors are not symmetric, a false alarm sent to a competent agent damages confidence, is slow to recover, and may itself depress the conversion rate being measured, a missed case costs a bounded, recoverable amount of lead spend. Where the errors are that lopsided, the acting threshold cannot be "the rule fired," it must be "the evidence suffices to justify the more expensive error."
Formally, the break even belief is
p* = C_miss / (C_falsealarm + C_miss)
so the significance level is a statement about the cost ratio between the 2 mistakes. Choosing 0.10 rather than 0.20 is equivalent to declaring a false alarm 9 times costlier than a miss, an explicit, defensible claim about intervention cost, not an inherited default. The system was then extended to 2 thresholds bracketing a region of human discretion, and a stopping rule that overrides the daily alert cap when the tail probability, which is monotone for an agent taking leads without closing, falls through a lower urgency line that no plausible cost could justify holding past.
None of that structure comes from statistics. It comes from deciding, deliberately, which error the operation could afford to make.
What the trail demonstrates
The whole thing turned on 5 facts, and none of them came from statistics.
- A lead bills once, not per touch. (killed wrong turn A)
- The CRM already conserves the ratio through duplicated credit. (killed wrong turn B)
- Cost is incurred once, credit may duplicate. (produced a free transfer rate metric)
- Cost peaks 2 hours after open because the denominator lags. (reassigned the diurnal blame by proof)
- The flagged agent held a 10 lead gap over the field. (refuted the drift hypothesis empirically)
Read as a whole, this is not the derivation of a formula. It is a record of 5 reasonable models being defeated by specifics, how this operation bills, how it credits, when it peaks, what the roster actually looked like. The final rule is 4 lines. Everything above them is the reason those 4, and not the others. The wrong turns are not failures preceding the work. They are the work.
Appendix note for adaptation. The final production system consists of a shared engine (leave-one-out peer rate and binomial tail probability), a coaching layer that retains cost per acquisition purely as a financial magnitude, a 2 threshold decision rule with a human discretion band, an override stopping rule, and role aware scoping so that pure enrollers, who incur no lead cost, fall outside a lead cost instrument by design rather than by exception. Each of those, too, arrived through correction rather than specification.