The API Rename That Cost $240 and Went Unnoticed for 60 Days

fee = payload.get("fee", 0)

For 60 days, a real trading system reported profit that didn't exist. $239.82 of it, spread across 1,258 settlement records — a few cents at a time, in both directions, never enough in any single record to look wrong.

Nothing crashed. Nothing threw. No alert fired. The daily reports rendered beautifully and were quietly incorrect for two months.

The root cause was one line of textbook defensive Python:

What actually happened

The system consumed settlement data from a vendor API. At some point the vendor renamed the fee field — providers do this more often than you'd think, usually in a minor API revision with a migration note nobody reads.

The moment the field stopped arriving under the expected name, .get() started returning its default. Every fee became zero. Costs vanished from the books, and profit was overstated by exactly the amount of the missing fees.

Over the life of this one integration, it happened three separate times. One rename left the system running blind for four days before anyone noticed. The worst stretch was subtler: two of the bugs partially cancelled each other out, so the aggregate numbers looked "close enough" and passed a weekly review for weeks. It took a manual reconciliation against the vendor's own statements to surface the truth: $239.82 of phantom P&L across 1,258 settlements, accumulated over 60 days.

$240 is a cheap lesson. The same line of code in a system that settles a thousand times more volume is not a cheap lesson.

The bug class: silent defaults on external data

.get(key, default) is the right tool for genuinely optional data that you control. On a required numeric field parsed from an external API, it becomes something else entirely: a contract-violation silencer.

The vendor broke the contract. The default translated that break into a plausible number and handed it to the rest of the system. Numbers are the worst case for this pattern — a missing string tends to fail loudly somewhere downstream, but a zeroed number flows straight into arithmetic and comes out the other side looking like a result.

That gives the bug its most dangerous property: its magnitude scales with data volume while staying invisible in any individual record. Ten settlements with a zeroed fee is a rounding error. Ten thousand is a materially wrong financial statement. Same line of code.

Why AI assistants write this constantly

AI models are defensive by reflex. In training data, a KeyError in a stack trace looks like a bug; payload.get("fee", 0) looks like a handled edge case. So the model reaches for the default every time, and the human reviewing the diff approves it, because defensive code reads as careful code.

The assistant has no way to know that this particular field is load-bearing — that a missing fee isn't an edge case to smooth over but a signal that the integration just broke. It writes the happy path and pads it with defaults.

It gets worse when the failure finally surfaces. Ask an AI assistant to fix a KeyError and the single most common patch you'll get is adding a default — converting a loud failure into this exact silent one.

What the fix looks like

# Fails loudly the day the vendor renames the field
fee = payload["fee"]

# Better: validate the whole boundary
class Settlement(BaseModel):
    settlement_id: str
    gross: Decimal
    fee: Decimal  # required — a missing fee is an incident, not a zero

Required fields from an external system should fail loudly the moment the contract breaks. Direct indexing is the simplest version; a validation layer at the boundary is the durable one:

Why your tests won't catch it

The uncomfortable part: this bug is nearly immune to testing. Your test fixtures mock the vendor payload using the field names the code expects — because the person (or AI) who wrote the test copied them from the same docs, or from the code itself. When the vendor renames the field, your fixtures don't. The code and the tests drift away from reality together, in perfect agreement, both green.

The only things that catch it are boundary validation that fails loudly in production, or reconciliation against an independent source of truth. This team had the second and eventually used it. The first would have caught the break in minutes instead of months.

If you want a quick audit of your own integrations today: grep for .get( with a numeric literal default, and for each hit ask one question — does the vendor's documentation say this field is optional? If the answer is no, that default is not defensive programming. It's a pre-authorized silent failure.

The rule we're building

This incident is why API008 (silent-default-masks-schema-change) is in our pipeline: a checker that flags numeric defaults on fields parsed from external API responses, especially when the value feeds arithmetic. The pattern is mechanically detectable, and the failure mode — silently wrong money — justifies flagging every instance for a human decision.

In the meantime, our shipped schema checks (SCHEMA001 and SCHEMA002, both in the free tier) catch the adjacent class: code whose declared response models and actual payloads disagree.

If AI wrote your integration code, there is a .get(field, 0) somewhere in it right now. Find out which fields it's guarding: pip install stablestack