Nearly 25 years of posting, in 28,153 files

In September I asked Meta for everything I ever posted on Facebook and Instagram, then I asked LinkedIn too. Since then I’ve added my website, my Google documents and my podcast. The downloads all came back in different shapes.

Facebook’s is JSON, a format written for computers rather than people. LinkedIn’s is spreadsheets. Instagram’s came in two parts, and the same post can turn up in both, described two different ways. My website comes out of Textpattern, the software it runs on, as one big export. Google sent a zip file – a folder squashed small for sending – from which I chose the documents to keep. The podcast I had already turned into transcripts on my own computer.

Having deleted the download, I asked Meta for everything again this month, and requested only low-resolution versions of my pictures (I already have the good ones). The download came to just 213 megabytes, about a quarter of the 876 in September, and I worried at first that it wasn’t the whole thing. It was.

I could read any one of these downloads – Facebook, Google, podcast or whatever – on its own. What I couldn’t do was read them together. I couldn’t find everything I’d said about something, wherever I’d said it.

So I asked Claude to write a script – a list of instructions for a computer to carry out again and again, by itself.

The script opens each download, takes out every post, comment, story, document and transcript, and writes each of them into its own small text file – the plainest kind of file there is, which any computer can open. The files are sorted by year, and each one has the same labels at the top: the date, where it came from, what kind of thing it is, how many words.

There are now 28,153 of them.

Where	Items	Words
Instagram	12,987	209,464
Facebook	8,995	145,244
LinkedIn	3,633	136,302
Twitter, 2011	224	3,599
flintoff.org	2,092	1,235,676
Google documents	163	342,687
ADEQUATE podcast	59	155,370
Total	28,153	2,228,342

I thought I had entirely deleted my Twitter account, but its tweets survived because they were cross-posted to LinkedIn at the time.

Still to come: YouTube, which I don’t use much but have had for a long time. I’m interested in the transcripts more than the videos.

Having downloaded all this stuff, I’m keeping it exactly as it arrived – so that if the script gets something wrong, I can alter the instructions and run it again, rebuilding everything from scratch. Some items are left out every time, on purpose: my contacts, my private messages, and other people’s posts that I passed on.

Each new file also contains a space for my own notes – tags, which book it might belong to, where I’ve reused it. The script is written never to touch that part, however many times it runs.

Putting everything together in the same shape turned up a few things. Pieces of writing appear in more than one place 316 times, mostly because I have set my Instagram posts to appear automatically on Facebook too. There’s also a hole when I stopped posting: falling from 2,691 items in 2017, to 189 in 2018 and 141 in 2019, and up again to 2,168 in 2020. Those were the years I mostly stayed off social media.

Searching it

For searching, Claude wrote a second script. I give it phrases rather than single words – “walk on the Heath” rather than “walk” – words in groups, one group for each idea. Some phrases I mark as broad, so they count for less.

The script reads all 28,153 files and lists the ones where the phrases turn up, with the strongest matches first. It gives me the title and the date, and nothing else, so I go and read the real thing.

It ignores the private notes I hide inside some posts. And it lists a post once, even if I put it on three platforms. Oh, and it keeps my books apart, because a 60,000-word manuscript matches nearly everything. (Another thing I have still to do is add my books to the archive in chapters.)

This morning, I was looking into an idea and decided to interrogate the archive. I used certain technical terms to describe the abstract ideas, but nothing came up. Not one hit, in 2.2 million words.

That’s because I don’t write in technical terms. I write about a grandmother, a funeral, a walk on the Heath. So I changed the search terms to look for those kinds of things instead, and got back more than 900 references.

Brilliant! After all, I’d much rather write stories with proper nouns in them than burden anybody with jargon.

Mind you, 900 references will take me a long time to work through, so there’s more filtering to be done. But the method works for anything. A new idea needs a new list of phrases, and the script turns it into a new reading list in under a minute.

(!)

A note about AI

A lot of people – me included – dislike AI for the energy used in data centres. I wanted to know where I stood.

Claude wrote both scripts, over several conversations this week, and tested them. Those conversations did run in data centres. And along the way Claude read samples of my material, to see how each download was laid out and to check results.

But the scripts themselves don’t use AI, and don’t go online. Once written, they are just instructions, sitting on my own computer, which follows them like any other program. Amazingly, both scripts combined amount to just 92 kilobytes. A single photo on my phone is usually several thousand kilobytes.

And searching all 2.2 million words takes my Mac 49 seconds.

Note: the same goes for the work I described in September. I wrote then that I’d asked Claude to read my entire archive. Misleading, because in fact that work was done by scripts too.

The podcast transcripts were made by Whisper, a program that turns speech into text. Whisper is AI too, but it runs on my Mac, not in a data centre.

The 28,153 files take up 118 megabytes. They live on my Mac and on iCloud, as do the (much bigger) downloads they came from.

So AI wrote the tools, and I use them myself. Running a search doesn’t involve AI at all.

***