ENGINEERING NOTE 19 / 50 · SMS
Compare SMS channels when samples are small
A channel with two successes from two orders has less evidence than one with hundreds of mature orders. Compare uncertainty and traffic mix, not just the displayed percentage.
Published by OPEN RAMBO. The engineering patterns and pseudocode illustrate designs, not a documented product API. Numerical examples are illustrative.
Keep comparable populations
Compare the same service, country and waiting window. Separate trials, cancellations before sending and confirmed delivery attempts where those distinctions are observable. If sending is not observable, state that limitation rather than inventing a denominator.
Display uncertainty
Use a binomial interval or another documented statistical method alongside the observed rate. The method assumes a sampling model; correlated outages and selected traffic can violate it. Treat the interval as uncertainty under assumptions, not a service guarantee.
Define switching criteria
Require a minimum mature sample and an operationally meaningful difference before changing a routing policy. Track waiting time and refund cost as well. A short spike should trigger investigation, while a sustained failure pattern may justify limiting new traffic.
Illustrative implementation
channel_a = {"received": 2, "allocated": 2}
channel_b = {"received": 180, "allocated": 200}
# Display n and uncertainty; do not rank only by raw percentage.Acceptance check
Recompute comparisons after stratifying by service and country. Check whether the original ranking was caused by a different traffic mix.