AI Can't Count Carbs—and That's the Whole Problem

I spent Monday afternoon watching a developer named Tim Street lose his mind.

He'd fed the same photograph of the same meal into the same AI model 27,000 times. He wanted one number: how many carbohydrates are on this plate? The model never gave him the same answer twice.

Not approximately the same. Not close enough for government work. Literally never the same number twice across 27,000 queries.

This is not a diabetes story. This is the entire AI deployment story in a single experiment.

Street runs DiabetTech, a site dedicated to testing glucose monitors and insulin pumps. He's been writing about the promise of AI-assisted carb counting for months. The use case is perfect: people with Type 1 diabetes dose insulin based on carb estimates. Get it wrong by twenty grams and you're either crashing or spiking. An AI that could nail carb counts from photos would be genuinely life-changing.

So he tested it. Twenty-seven thousand times. And what he found is what every startup CTO privately knows but won't say on pitch calls: large language models are probabilistic by design. They sample from a distribution. They don't compute. They don't measure. They guess, eloquently, within a range.

For creative tasks—summarizing articles, drafting emails, brainstorming product names—that's fine. For anything where you need the same input to produce the same output, it's a category error.

The really damning part? Street wasn't even using a no-name model. He tested industry leaders. The stuff investors are pouring billions into. The tech that's supposed to automate radiology and legal discovery and financial auditing.

Here's the thing: Street's experiment isn't an outlier. It's a reproduction of what OpenAI, Anthropic, and Google all document in their own papers. Temperature settings, sampling methods, non-deterministic tokenization—these aren't bugs. They're features. The models are built to be creative, not consistent.

Which raises the uncomfortable question nobody wants to ask during a Series B: what percentage of current AI deployments are solving problems that actually require determinism?

I've watched three fintech startups pitch AI-powered fraud detection in the last month. I've seen healthcare companies demo diagnostic assistants. I've sat through compliance automation decks. Every single one leans hard on accuracy metrics—F1 scores, precision-recall curves, benchmark performance.

None of them mentioned reproducibility.

None of them addressed what happens when the same transaction, run through the same model six months apart, gets flagged once and cleared five times.

The gap between lab benchmarks and production reliability is where the current wave of AI companies will die. Benchmarks measure aggregate performance across thousands of examples. Production systems need per-instance consistency. Those are not the same thing.

Street's carb-counting test is brutal because the stakes are legible. Dose wrong, go to the hospital. But the same dynamic plays out in less visible domains. Contract review tools that flag different clauses on successive reads. Code review assistants that approve a pull request one day and reject it the next. Customer support bots that escalate identical tickets inconsistently.