ENGINEERING NOTE 39 / 50 · CARD
Design deposit alerts that distinguish failure and recovery
An alert should describe an actionable condition and its lifecycle. Repeating a vague warning is less useful than showing last success, lag and the current recovery phase.
Published by OPEN RAMBO. The engineering patterns and pseudocode illustrate designs, not a documented product API. Numerical examples are illustrative.
Separate signals
Track worker heartbeat, successful scan time and backlog independently. A running process does not prove scans succeed. A successful historical batch does not prove the scanner is near the chain head. Each signal needs a clear threshold and owner.
Control reminders
Deduplicate unchanged incidents with a stable incident key and reminder interval. Brief recovery should not erase the last-notified time and cause another immediate email. Material changes can be reported separately without flooding the same incident thread.
Define recovery honestly
Mark transport recovered when successful requests resume, catching up while lag remains, and synchronized only after a defined threshold. Preserve the incident record for review. Closing an alert must not disable the underlying financial checks.
Illustrative implementation
if stale_success: state = "scan_failed"
elif lag > threshold: state = "catching_up"
else: state = "synchronized"Acceptance check
Simulate repeated failure, a brief recovery and renewed failure. Verify bounded notifications and accurate recovery wording.