Share-Alike Was the Whole Point

The three most common brands in the world's largest open database of packaged food are Carrefour, U and Auchan.

I know this because I have just published the thing: 376,048 products, 514,062 barcodes, ninety megabytes of SQLite. Forty-five percent of it is French. There are five times as many French products in it as British ones, and Picard, a French frozen-food chain, has more entries than Coca-Cola, Kellogg's and Danone put together.

None of that is a decision I made. Open Food Facts was started in France in May 2012 by one programmer, and the French contributors never stopped. A global database of packaged food turns out, in practice, to be a very good database of French supermarkets with the rest of the world filling in behind.

Why it is published

Not out of generosity. The licence says I have to.

Open Food Facts is ODbL, which is share-alike. Take the data, change it, hand the result to anyone, and you owe them a machine-readable copy of what you made. My app carries the whole database offline, so handing someone the app is handing them the database. The clock started when the first tester could install it, not at some future launch I get to schedule.

I want to be plain that this is compliance. But the clause is also the reason the data was worth having.

The clause

Somebody stood in a supermarket and photographed the back of a packet of biscuits. Then somebody else did, four million times. That is the whole database. There is no clever acquisition strategy underneath it and no budget, just a very large number of people doing a small boring thing on the understanding that the result stays open.

Share-alike is what makes that understanding enforceable. You can build a business on the data. You can keep your own code shut. What you cannot do is take the database, improve it, and keep the improved version to yourself, which is otherwise the obvious move and happens constantly.

I benefit from the clause and I am bound by it in the same breath. That seems correct.

What came back out

The published database is not a copy of the upstream one. It has been through a pipeline, and the pipeline goes with it.

Names get repaired. A surprising number arrive as Côte instead of Côte, which is UTF-8 that has been through a Windows codepage and back, and which renders literally in a search result. Some names are not names at all but a weight or a barcode that someone pasted into the wrong field, and those get dropped. So do products with no energy value, and products claiming more than 900 kcal per 100 g, which is more than pure fat.

Barcodes get their check digit verified rather than just counted to the right length. About one in twenty-seven upstream codes fails that test. They are typos and internal warehouse numbers, and no scanner will ever produce them, so they sit in the database doing nothing but occasionally matching a search.

Then there is the one I got wrong. I pointed the app at a bottle of Coca-Cola and nothing happened. Tried again, moved the light, wiped the label. Nothing. Typed coca cola into the search box and there it was, first result, exactly the drink in my hand.

The most photographed barcode on the planet, and my scanner had no idea. What had happened is that duplicate listings of the same product get merged onto whichever row is best documented, and for a long time that merge kept the loser's alternate names and quietly binned its barcode. The number printed on the bottle in front of me had lost to another listing of the same drink and gone nowhere. So the product stayed findable by typing and stopped being findable by scanning, which is the half people actually use. One code in four was in that state. They all work now.

What is not in it

The app also carries thirteen national food composition tables, the government ones with laboratory methods attached. None of them are in the release.

Free to use with credit is not free to hand onward. Spain's BEDCA is the sharpest version: non-commercial, attribution, and the data must not be modified. My pipeline modifies every row it touches, and the release is under a licence that permits commercial reuse. Publishing BEDCA under it would be giving away something that was never mine.

So the release is the Open Food Facts half, which is the half the ODbL is about, and the national tables stay with their publishers. Each is credited inside the app under its own licence. This is the same care I took when I scraped BEDCA and published the extraction, and I am no more confident now than I was then.

The other direction

Publishing is the obligation. The part I wanted to build is the return path.

Scan something the app does not know and it can ask Open Food Facts. If they have never heard of it either, you can type what is on the label and send it, and the next person to scan that packet finds it. Both halves are off until you switch them on, separately, and the submission goes under an account belonging to the app so nobody has to register for anything.

On picking something too big

I chose this project because I wanted one I could not hold in my head. A calorie tracker sounds like a weekend. The database under it is not, and the gap between those two facts is the entire lesson.

The evidence that it worked is a list of things I would have called overkill a year ago and now would not give up. A fixture that scores search results against hand-judged answers, because I once retuned the ranking on instinct and made it worse without noticing. A rule that every measured number is written down with the build it came from, because two figures from different builds got compared and the conclusion was nonsense. A file of things already tried and rejected, so they stay rejected.

And continuous integration, which is the one that still makes me laugh. CI exists so that when you break something, a machine tells you before your colleagues find out the hard way. I have no colleagues. Nobody is waiting on my build. It runs on every push anyway.

It is there because of a day I would rather not repeat. I fixed something in the script that builds the food database, and then did not rebuild the food database. The fix sat in the file, perfectly correct and never once executed, while I spent the day looking at search results from a database that did not contain it and reasoning carefully about what they meant. When I finally rebuilt, it turned out the pipeline had been failing since the previous evening as well.

None of that was subtle. It survived a full day because there was nobody to notice it but me, and I was busy being wrong. That is exactly the gap a build server fills, and it turns out you do not need a team to have it.

Where it is

github.com/TerjeRu/lethio-food-db, under ODbL 1.0, with the build script beside it. SQLite if you want the schema, gzipped CSV if you want a spreadsheet.

No DOI this time. The last dataset I put out did not exist anywhere else, which made archiving it worth something. This one is a filtered slice of a database that publishes a full dump of itself every day, and minting a permanent identifier for that would be ceremony.

The app is Lethio Macro Tracker. It is in closed testing on Google Play as I write this, so unless you are one of the twelve people I asked, there is nothing to install yet.

If you use the database, credit Open Food Facts and keep it open. It is not a large ask. It is the one that was made of me.