Generally speaking, it's much better to use alerts instead of hard limit.

Probably for companies and opportunistic individuals with a high risk tolerance. As for me, just give me hard budget caps. It shouldn't be a choice between supporting one of the approaches, when you could let each client pick what they want.

Want an alert? Configure one.

Want a hard limit? Configure one.

(just make the configuration easy and visible)

Heck, have it do the equivalent of send people a DocuSign equivalent PDF to sign acknowledging the risk before enabling it. Would it still stop pissed off people? Probably not. Would it help with the risk of lawsuits, very possibly.

As a customer, the big number is scary and causes panic but for the provider… customers constantly fail to pay bills, providers are constantly writing off bills because it just isn’t worth the cost to chase, if a customer says “hey that usage was a mistake” it’s usually worth it to write it off to save the relationship. If you write off a big bill that wouldn’t have been paid anyway, the customer will perceive you as wonderful and benevolent and be loyal for life when they are ready to spend their money.

With tokens though the actual cost being incurred is much, much higher. If your service is just a wrapper around tokens, and a customer incurs $10k of usage that you paid OpenAI $5k for, it becomes much more difficult to write off.

Google Cloud is one of the few services that actually pursues unpaid bills even on their high margin services.

The caps were very generous but I wasn’t going to spend a minute worrying about what if a malicious actor took over

And its of course not just people yolo'ing with AI. People were quite capable of causing such outages themselves just fine. Distributed, serverless systems are hard.

A better approach to that scenario might be to split into base load and peak load infra, similar to how we do it with the power grid.

Base load could be an actual metal server that you pay x amount of money for and is fully yours, with peak load being handled by autoscaling dynamic stuff.

Both things being on a fixed budget, if that budget would run out, the service would not degrade as in "grind to a halt" but as in "slows down", which is probably not a downtime as part of a communicated SLA.

My point being that cloud and on-demand is useful, but a hybrid approach might in many cases make more sense. Though of course YMMV. I do not know what the requirements of your product are.

I got bit by some undocumented (or at least very poorly documented) hard limits in AWS when our app went viral. It was extremely difficult to just find out what these hard limits actually were. And most importantly, we never explicitly set them, they were just hidden defaults in AWS.

That's very different from having an easy to use, visual dashboard of where and what all your hard limits actually are, and make it it extremely easy to turn them off or on at a moments notice.

It should be the choice of the customer.

Imposing one option: only hard budget cap or only alerts is the wrong design decision.

you can negotiate with the billing department at comparative leisure.

I'm curious. How likely is the billing department to waive off a huge bill as bad debt because an inexperienced builder misconfigured their infra or was hacked?

This is not rocket salad to implement correctly.

Generally speaking, it's much better to use alerts instead of hard limit.

You can't extrapolate your own experience and generalize like that. There is a huge number of people who explicitly want hard caps, period. AWS giving in and finally offering this option after 20 years of people begging them to do that is a sign of that.