magnusmanske.de
BaGLAMa2 – Back to Business
BaGLAMa2 has been collecting view data for Wikipedia pages that contain image from a Commons category tree since March 2010. If your organisation “donated” images, audio, or video to Commons, this kind of data gives highly valuable feedback. Look, out images were viewed 20 million times on Wikipedia last month! There are over 1200 category tree “roots” in the database already.
However, this tool has had problems for a long time now, which started with Toolforge moving from grid computing to k8s. The initial version of the tool was written in PHP, and a single grid job would process one category tree and generate a TSV file. Later, I changed that to write all data directly to MySQL, resulting in a 20GB database, which did not scale well. Then I changed it back to storing the main data in files, sqlite3 this time. Meanwhile, the old TSV format and the MySQL data were both kept and supported. However, the k8s switch made the one-script-one-category approach impossible, and so I rewrote the tool in Rust, to run one single job To Rule Them All™. This worked in principle, but database connections and pageview API requests both did not scale well. I then tried a new, more efficient MySQL table approach, which also never took off. Sadly, BaGLAMa2 was stuck in a broken state for years.
Now, I tried a new, radically different approach (while staying in Rust, building on existing code).
First, I (and my useful slop generator) designed a new, bespoke, and highly optimized format to store results. As before, one file would keep the wiki sites, pages, images, and view counts for one category tree, for one month. However, where the dataset for a whole month would use something like 16GB (sqlite3 files, October 2022), the new format uses ~0.5GB in total for a month.
Then, I switched from using the pageview API, to reading the pageview data dump, which is conveniently available on the file system on Toolforge. All site/image/page data gets prepared in advance, which takes ~1h. Then, the compressed pageview dump file is read in its entirety, keeping the relevant view counts, which takes another 15 minutes. In contrast, the previous PHP or Rust processing took several days, and usually left some failed groups for re-run behind.
I have already back-filled July, August, and September of 2026. I have also converted all TSV and sqlite3 files to the new format. This has now saved almost half a TB of disk space on Toolforge. I will also convert the MySQL datasets into that format, to be able to retire the backwards-compatible reading (and writing!) code, and reduce the database size from 20GB to almost nothing. The total disk space for storing all BaGLAMa2 data will then be <35GB, with 0.5GB added every month. At this speed, it will take many decades to again occupy the space it did a few days ago, in the old formats.
Last not least, the new code will run automatically on the 3rd of each month, processing the pageview data from the previous month.
