Fifty-Six People Wanted a CSV That Did Not Exist

I have a DOI now: 10.5281/zenodo.22017001. Permanent, citable, and pointing at a CSV file that mostly records how much potassium is in a chickpea.

I did not set out to contribute to open science. I set out to look up chickpeas.

What I actually wanted

My current project is a calorie/macro tracker. Macro trackers live or die on their food database, and the boring truth of the genre is that most of them are terrible. A search for "rice" returns forty entries called rice uploaded by strangers, three of which are wrong by a factor of ten.

So I've been building the food database out of national food composition tables instead: the official ones, compiled by government food safety agencies, with lab methods and citations attached. France has Ciqual: 3,484 foods, downloadable as a single file, done in an afternoon. Norway has Matvaretabellen, with an actual JSON API. Denmark's Frida is a free download, and Sweden's is CC-BY.

Then there's Spain.

BEDCA

BEDCA is the Base de Datos Española de Composición de Alimentos, run under AESAN, the Spanish food safety agency. It's real, official, and good: 957 foods, with standard deviations, analytical methods, and literature citations attached to individual values, better provenance than several of the tables I'd already used.

There is no download. Not hard to find, not tucked behind a form. There genuinely isn't one. There's a web query interface at bedca.net where you pick a food and it shows you that food, then you pick another. The layout is fixed-width. The search opens its results in a new window that is also a form. The frameset instinct is present even where there are no framesets. The only thing on the site you can actually download is the documentation, as PDFs.

It's easy to be snide about this, and the people who built it did the hard part. Assembling 957 foods with methodology-level provenance is a lot of laboratory work and coordination. Building a good bulk export is trivial by comparison. They did the real work and I just collected it.

The rabbit hole

Before scraping anything, I did what you're supposed to do: I checked whether it was already published somewhere sane.

It looks like it is. Search for BEDCA and one of the first results tells you the database is published on datos.gob.es, the Spanish open data portal, in CSV, XML and dBase, and links straight to it.

That link goes to datos.gob.es/es/solicitud-de-datos/base-de-datos-bedca. Read the URL slowly. Solicitud de datos. It's a request for the data to be published, filed 14 April 2026. Fifty-six people have signed it. AESAN has not responded.

I shut the laptop at that point, genuinely exasperated. Anyone who has had to get something out of a Spanish official portal knows the shape of this. The thing you need exists. The page describing the thing exists. The form for requesting the thing exists, and has a queue, and the queue has people in it. None of that produces the file. The bureaucracy is complete and the output is nothing.

But the number stuck with me. Fifty-six people had gone to the trouble of finding a government petition form to ask for this specific file. Fifty-six people wanted a CSV that does not exist.

The polite scraper

So I wrote one, and not a fast one.

The BEDCA front end talks to its backend in XML, which turns out to be the accessible part of the whole system. Once you know the query shape, you can ask it for a food by id and get back a structured response with every component value, its unit, its standard deviation, its method, and its citation reference.

I set it to one request every 2.5 seconds. That's almost slower than a human clicking, and it made the whole run take about fifty minutes, which was the point. This is a public agency's server, and I would rather be boring than noticed. Every request and its status went into a manifest, so that if anything came back wrong I'd know which food and why, rather than discovering a hole in the data months later.

The result was all 957 foods and 31,166 individual component values, stored verbatim: no unit conversion, no rounding, no "helpful" normalisation. That was deliberate, and it was a legal decision rather than an aesthetic one.

The part where science happens by accident

At this point I had what I needed and could have stopped. The data went into my build pipeline, and Spanish generic foods started resolving properly, which was the entire goal.

But I kept thinking about the fifty-six. I'd guess most of them are students or researchers who don't have the time, or the inclination, to build the tool that gets them this file.

So I pushed it to GitHub. Four CSVs: a wide table with one food per row for people who just want numbers, a long table with the full provenance for people who need to know how a value was measured, a units reference, and an index with the LanguaL codes intact. Plus the manifest, so anyone can see exactly what was fetched and what came back.

Then I tagged a release. GitHub is wired into Zenodo, so Zenodo mints a DOI.

That's the whole mechanism, the entire ceremony: a tag, a webhook, and I'm suddenly the author of a citable research object with a permanent identifier that will outlive the app it was made for.

What a DOI is and isn't

I want to puncture this slightly, because I enjoyed the joke for about half a day before the realistic version caught up with me.

A DOI is not peer review, and it isn't validation. Nobody read this. Zenodo mints DOIs automatically for anything you tag, which is correct behaviour for an archive: the identifier says "this specific version of this thing existed, and here it is," which is genuinely useful, and genuinely not the same as anyone thinking it's good.

At the time of writing, the record has zero downloads. My contribution to open science is, empirically, a contribution to Zenodo's storage bill.

What I do think is real: the provenance survived. If someone uses this, they can see it was fetched in August 2026, from which endpoint, at what rate, and what failed.

The unresolved bit, which I am not going to pretend is resolved

BEDCA's terms of use permit personal, educational and non-commercial use with attribution. The required credit is "AESAN/BEDCA Base de Datos Española de Composición de Alimentos v1.0 (2010)", and the terms also require that the data not be modified, nor its original meaning altered. That last clause is why every value is stored verbatim. Converting sodium from milligrams to grams preserves meaning under any reasonable reading, and I still didn't do it, because "any reasonable reading" is not the same as "their reading."

The non-commercial part I can meet, and I'd rather state it plainly than let it slide past. The tracker is free: no price, no subscription, no ads, no paid tier holding the Spanish foods hostage. Every database I derive from a national table gets published with its attribution intact, this one included. So the use sits inside what the terms allow, and it stays inside only as long as that remains true. If I ever put a price on the app, the permission I'm relying on lapses that day.

Two things I can't answer from the published terms.

The first is whether republishing the extraction is mine to do. Their terms govern use, not supply. And EU database law has a sui generis right protecting substantial extraction from a database, independently of whether the facts inside it are copyrightable, a specifically European problem and not a grey area I get to shrug at. I think I'm on the right side of it. I also think "I think" is doing a lot of work in that sentence.

The second is whether "no modification" survives a format change. I think a CSV of the same numbers isn't modification. They may reasonably disagree.

There's a tension in here that I've mostly made peace with. I make loud claims in my apps and on my sites about local-first software, privacy, and the freedom of data, and I am also boringly compliant with every scrap of data I hold. I run a honeypot server. The people trying to break into it get the maximum privacy GDPR allows when I log their IP addresses. Nobody asked me to do that, and nobody would have known if I hadn't. It just seems like the deal: if I want to be trusted with data, I don't get to be selective about whose.

Publish yours

I went back to the petition and posted a link to the archive in the comments, so the fifty-six might find it without having to build a scraper first. It has been waiting on a moderator ever since.

If you're scraping a public dataset because the portal is bad, you are almost certainly not the first person to want it. Publishing the result costs you a git tag. Publish the manifest with it: the numbers are the least interesting thing you extracted.

The repository: github.com/TerjeRu/bedca-database. Data source: AESAN/BEDCA Base de Datos Española de Composición de Alimentos v1.0 (2010). Errors in the underlying data should go to AESAN at bedca.adm@gmail.com, the address given in the licence terms, not to me. I only moved it.