Don’t call your Durable Object alarm handler directly
Jamie Barton (@notrab) says a `<` written instead of `<=` in a self-rescheduling alarm cost him 8 billion database operations across 191,000 requests in 30 days, and that the lesson is to call setAlarm() rather than alarm().
- Products
- Durable Objects
- Source
- Author experience𝕏
- Author
- Jamie Barton (@notrab)
Key points
- His Durable Object alarm swept carts that expired 7 days earlier. alarm() called _setNextAlarm() when it finished, and _setNextAlarm() called alarm() directly when it found an expired cart. One compared the timestamp against a float and the other against Math.floor of it, so for a one-second window a cart satisfied the scheduler but stayed invisible to the sweep, and the two ping-ponged with no stack unwinding until the object hit its 128 MB limit and was killed.
- He says this cost 8 billion database operations across 191,000 requests in the last 30 days on a cart API that mostly reads a single row, and took two lines to fix.
- His advice: do not call your alarm handler directly; call setAlarm() and let the runtime invoke it, so a scheduling bug shows up as a slow loop in the logs instead of an out-of-memory kill.
Original sources
Published
Original post
Read on 𝕏Jamie Barton (@notrab) says a `<` written instead of `<=` in a self-rescheduling alarm cost him 8 billion database operations across 191,000 requests in 30 days, and that the lesson is to call setAlarm() rather than alarm().
Jamie Barton 🥕
@notrab · Original 𝕏 post
If you have a Durable Object alarm that reschedules itself, go and read it. I wrote `<` where I meant `<=`. It has cost me 8 billion database operations across 191,000 requests in the last 30 days on a cart API that mostly reads a single row 😫 A Durable Object alarm sweeps carts that expired 7 days ago. alarm() calls _setNextAlarm() when it finishes, and _setNextAlarm() calls alarm() when it finds an expired cart. One compared the timestamp against a float, the other against Math.floor of it, so for a one second window a cart could satisfy the scheduler and stay invisible to the sweep. The two ping-ponged with no stack unwinding until the DO hit its 128MB limit and Cloudflare killed it. Two lines to fix. The one worth passing on!!! don't call your alarm handler directly!!! Call setAlarm() and let the runtime invoke it, so a scheduling bug shows up as a slow loop in the logs and never as an out of memory kill. I can't even blame vibe coding for this one...
More to read
Why hard spending caps take time: Ashley Peacock on billing latency
Ashley Peacock (@ashleypeacock), who works on Durable Objects at Cloudflare, explains from experience why near-real-time spending caps are hard to build, says hard billing caps are coming soon, and suggests budget alerts plus issue alerts in the meantime.
Steven Menke: an Agents SDK bug and a reported $1,700 Durable Objects invoice.
Steven Menke (@Galorious_) reports that runaway Durable Objects caused by an Agents SDK bug produced a $1,700 invoice in three days, citing one trillion SQLite reads (a later post says writes). He says his support ticket, even after a maintainer escalated it, received no reply for months.
Introducing on-demand profiling: Cloudflare production examples
Dominik Picheta introduces on-demand CPU and memory profiling for Workers and Durable Objects, sharing Cloudflare examples of removing duplicate work and unnecessary allocation paths.