How we test whether the estimates are honest
When you type "2 slices of toast with peanut butter" into Gentle Plate, it comes back with something like 280 to 380 calories. The obvious question is how do you know that's right?
The honest answer is that I don't know it's right - nobody does, for a slice of toast. What I can check is whether the range is being honest with you: does the real number land inside it more often than not, and does the app widen the range when it's less sure, rather than pretending it knows. This post is about how I check that, in plain words, and what I found along the way.
What "accurate" means for a range
Most calorie apps give you one number. As far as I can tell it comes from a database entry someone typed in years ago, or from an AI model's memory, and either way it looks a lot more certain than it actually is. A single number can only be right or wrong. A range can be honest or dishonest, and I believe that is the more useful question to be asking.
So I decided early on that the app gets judged on two different things:
- For protein and carbs, how close the middle of the range is to the truth. These follow the weight of the food fairly well, and my partner has a real reason to want those numbers close.
- For calories and fat, whether the truth falls inside the range at all. Cooking fat is basically invisible (nobody measures the oil in the pan!) and it goes into both, so the same uncertainty widens both. A wide calorie range is the app being honest, not the app being bad at its job.
That second rule matters more than it sounds. If I had judged the app on how narrow its calorie ranges were, the easy way to "improve" it would have been to make it more confident - which, as you may imagine, is exactly what every one-number tracker has already done.
The test set
To check any of this you need foods where the true answer is actually known. So I built a list of 70 things we actually eat, typed the way a person would type them ("splash of milk", "half the pasta bake", "a greggs sausage roll"), each with the real calories, protein, carbs and fat written next to it. The truth values come from packet labels, from the nutrition tables the chains publish, and from the UK government's food composition tables for anything homemade. Every item has a note saying where its truth came from, and if I couldn't verify a number the item didn't go in.
The list is frozen. I can add to it, but changing an existing truth value needs a note saying why - a test that quietly drifts towards whatever the app already says is worse than no test at all.
Running it means sending all 70 through the estimator exactly as a user would, saving every answer, and scoring them. The scoring is free and I can re-run it as often as I like. The sending costs real money (around 20p for two full runs, so not much) and the AI that reads your text doesn't read it the same way twice. That second point cost me an afternoon.
The afternoon the test measured deleted code
In July I made the biggest change the estimator has had. Before, I was asking the AI for the numbers directly - a bit like asking a well-read friend how many calories are in a pasta bake. They'll give you a confident answer and it will sometimes be right. Now the AI only does the reading: it turns "half the pasta bake" into a list of foods and amounts, and the app looks each one up in its own table and does the sums itself. The AI is really good at understanding what you meant. It is not so good at remembering numbers.
The rule I'd set myself was that the new version had to beat the old one on the test set. First run: worse. So I fixed what the run had shown me, mostly how much a "portion" of things actually weighed, plus one entry that had priced a bowl of red lentil soup as if it were 300g of plain boiled lentils. I rebuilt everything, restarted the app on my laptop and ran the test again. Still worse, and in the same places, which was odd.
Then I noticed something in the second run. One answer said it had used an entry for "porridge, with semi-skimmed milk". I had deleted that entry before the run started, so the app was quoting something that no longer existed.
What had happened was pretty mundane. When I "restarted" the app, the old copy had never actually closed - it was still sitting there answering, and the new copy I'd started was never being used. Every fix I thought I'd tested had never been switched on. It's a bit like tasting the soup, adding salt, tasting again and finding it just as bland, because someone had quietly swapped your spoon back into yesterday's pot.
The fix is dull and I do it every time now: before a single result counts, check that the app answering the test is the one I just built. Coming from an Engineering background I probably should have been doing that from the start!
The number that was luck
Cleaning that up turned up a second problem, which I found more interesting. Once I could trust a run, I re-ran the old version against itself - same code, same foods, nothing changed. The carb error I had been told to beat was 2.8g. The re-run gave 4.5g.
The only thing that had changed was the AI's mood that day. The number the new version "had to beat" was basically one lucky roll of a noisy process, and my test had no way of telling me so. I had built the whole thing to stop judging changes by feel, and then judged its own output by a single dice throw.
So now every comparison runs at least twice on each side. If the difference is smaller than the swing between two identical runs, it's noise and I say so. If a shift shows up again and again, it's real. Judged that way, the July change was a genuine improvement on calorie honesty and a small cost on protein, and that mixed verdict is the one I wrote down rather than the tidier one.
What the numbers say today
Two runs on the current version, from earlier this month, both over the same 70 foods:
- The true calories fell inside the range 78% of the time in one run and 80% in the other.
- The true fat fell inside its range 77% and 79% of the time.
- Protein came within a fifth of the truth 68% and 67% of the time. Carbs did the same 78% and 80% of the time.
I'd rather give you both numbers than an average. Two runs of the same test never quite agree, and the gap between them tells you how much to trust either one. Think of it a bit like a weather forecast - it can't tell you it will rain, but you can check how often "70% chance of rain" actually rains, and that is what's being measured here.
The more useful reading is underneath. The app has two ways of answering. When a food is in its table, it looks the numbers up and does the sums, and the calorie range contains the truth between 85% and 96% of the time depending on the run. When a food isn't in the table, the app has to fall back on the AI's own guess (the well-read friend from earlier) and that drops to somewhere between 58% and 75%. The table currently holds 215 foods.
So the way to make the app more honest is to teach it more foods, each with a real label behind it, rather than to find a cleverer AI. When I went through every miss in September, about 83% were foods the table simply didn't hold yet - Mars bars, Weetabix, a Big Mac, a gin and tonic, chilli con carne, that sort of thing. Only about one in ten was a food the table had and failed to spot in what you'd typed. So the problem is mostly a thin recipe book rather than a bad reader, which is a much easier thing to fix (and is on my never ending backlog).
When the app doesn't know, it should say so
One more thing the runs caught, and it's the reason the guessing is allowed to exist at all.
I added a "lentil soup, homemade" entry to the table from the government tables. Their recipe comes to 121 calories per 100g. The soup we had was closer to 55. The next run showed the app confidently telling us a bowl was 290 to 436 calories when the real figure was about 165 - and before the entry existed, the AI's unaided guess had been fine.
I found that quite a useful lesson: a wrong match dressed up as arithmetic is worse than an honest guess that admits it's a guess. I removed the entry the same night. The table only holds foods where a published value is likely to be close to what's actually on your plate, and for everything else the app falls back to a wider guess.
What this means for you
If you type a packaged food or a chain item the app knows, the range is fairly tight and it's usually right. If you type "my mum's curry", the range is wide because it should be. And when you're planning meals, the numbers are there as a guide to lean on rather than a target to hit, which is the whole point of showing a range in the first place.
Every run I've mentioned is saved with its date and its verdict, including the ones that went wrong. What the estimator sees and what it keeps is on the privacy page.