iiinova — where your digital self keeps evolving

Geocoding anomalies in TongFen timelines

There is a new vignette that walks through how to use the anomaly detection and dissolution of offending boundaries. The method can be applied to existing timeline data or to results of plain TongFen, and it results in slightly coarser geography by dissolving problematic geographic boundaries.

mountainmath.github.io

140

Geocoding errors in aggregate data – Mountain Doodles

An example of this is the 1971-2011 semi-custom tabulation of census profile data on 2016 DA geography where this method was developed in response to clear signals of cross-boundary geocoding errors. More details on methods and rationale are in this post.

doodles.mountainmath.ca

Geocoding errors can become a problem, especially in temporal data where geocoding processes have changed over time. We propose a method to detect and ameliorate geocoding errors in aggregate data.

130

This update also fixes some edge case bugs and adds a new feature: anomaly detection and fix. Temporally linked fine geography (census) data requires consistent geocoding, but most geocodes are subject to significant model drift that sometimes crosses (census) geographies and introduces artefacts.

130

Make Data Based on Different Geographies Comparable

Tongfen v0.3.9 is now on CRAN. This fixes the problem introduced by StatCan blocking programmatic data download starting last week, we now host the correspondence files independently. This restores package functionality at the cost of complicating data provenance.

mountainmath.github.io

Several functions to allow comparisons of data across different geographies, in particular for Canadian census data from different censuses.

1131

Jens von Bergmann @jensvb.bsky.social · 22h

The simple solution that everyone else has implemented is to require a free API key for data access so abusers can be identified and blocked.

000

Jens von Bergmann @jensvb.bsky.social · 04/10/2026

canivt is also affected, but probably not in a critical way that requires re-hosting. Programmatic data download is not that central to this package, and instructing users to work around the StatCan block via a manual download and deposit is annoying but not a deal breaker.

StatCan is blocking programmatic download of older IVT tables · Issue #1 · mountainMath/canivt

StatCan is now blocking programmatic access to some IVT files, this affects pre-2021 downloads. It also affects functions that build the catalogue. Cached and the list shipped with the package stil...

170

Jens von Bergmann @jensvb.bsky.social · 04/10/2026

Yes! Some already do this, e.g. City of Vancouver Open Data portal now serves parquet files. StatCan unfortunately does not, but the cansim, canivt, and canpumf R packages all use parquet or DuckDB under the hood now to increase performance for users after download.

0122

Still trying to assess the blast radius of StatCan putting their entire site behind Cloudflare verification. TongFen is also affected, and in this case it's core functionality. StatCan correspondence files are the most widely used path to harmonize geographies over time.

StatCan disabled programmatic download of correspondence files · Issue #19 · mountainMath/tongfen

The correspondence files now can't be downloaded programmatically any more. This breaks a key functionality of the TongFen package. For example, the 2021 DA correspondence file now requires Cloufla...

2120

It’s not core package functionality, all of that is controlled by me and goes through CensusMapper (StatCan still does not have a useful API for census data), its more custom functionality for cases where users need geographic data or attributes for all of Canada. But it’s still annoying.

My take, the IT team noticed a pickup in bot traffic and did the lazy thing. Enabled by failure of leadership to be clear about the purpose of making the data available for download in the first place. (I waited a bit to see if this was just a stop-gap measure, but it’s been like this all week now.)

140

Good Saturday morning to everyone except StatCan, who decided to break reproducible workflows by disabling programmatic data download for some of their data and put it behind a Cloudflare "human" check. Leaving me to decide whether to remove cancensus package functionality or re-host the data.

StatCan disabled programmatic download on some census data · Issue #228 · mountainMath/cancensus

StatCan has disabled programmatic download on the attributes file and geographic data and possibly other files. These are now behind a CloudFlare verification that requires manual interactive downl...

84710

And they are maximally inaccessible. Canadian codes are so broken.

080

Glad we re-legalized them. Makes one wonder what other ways to provide housing was legal back then and should be re-legalized…

030

I probably read too much into the language in the TCP application, it suggested to me that it was only available for people with affiliation at academic institutions. But maybe that's not the case?

100

Jens von Bergmann @jensvb.bsky.social · 02/10/2026

Maybe I got the licensing on each of these wrong in my scan that I linked above? Some of these I can access, e.g. IPUMS. Some come with a very permissive license. Some CCRI data is locked up under DLI/OCUL and unavailable.

200

Which is why MD is consedered a professional degree not an a ademic graduate degree.

Jens von Bergmann @jensvb.bsky.social · 02/10/2026

This is great, but also quite unfortunate that the data comes with a very restrictive licence. Just the other day I was looking into what it would take to add this data to campumf, but only the 1881 data come with an Open Data license. Maybe another effort to transcribe these can resolve that.

Historical (pre-1931) census microdata: full-count TCP files and public-use samples, import path for canpumf · Issue #25 · mountainMath/canpumf

Census microdata older than 92 years is fully public in Canada, and several transcribed, machine-readable files exist for 1852–1921 with household linkage. None of them are reachable through canpum...

152

Jens von Bergmann @jensvb.bsky.social · 02/10/2026

A type of failure to return to pre-pandemic levels I can get fully behind. Still not liking the trend though, it's high time we turn this around for good. canviz.mountainmath.ca/plot?v=74726

Timeline of sales of fuel for road motor vehicles.

0193

Reposted by Jens von Bergmann

Gavin Schmidt @climateofgavin.bsky.social · 02/10/2026

Not only are the flat earthers wrong, they are getting wronger by the day.

636371

Reposted by Jens von Bergmann

Nathan Lauster @lausterna.bsky.social · 01/10/2026

Just putting it out there that I am a voter who cares about climate change and regular summer wildfires and my own kids' future and stuff and I would also like to see some election pandering - like ANYTHING - thrown in my direction. I mean, the Carbon Tax was doing a lot of emotional work for me.

0283

That’s totally different. Buildings cast shadows, whereas trees provide shade. 😎

1602

Justin Ling @justinling.ca · 01/10/2026

Yeah, and the method used to harmonize census tracts over time is another one of my pet peeves. Why are we still using non-spatial methods for decidedly spatial problems and pretend p-values and R^2 have meaning?

000

Reposted by Jens von Bergmann

I spent the summer trying to understand what makes Mark Carney's government tick. What I found is a C-Suite obsessed with efficiency — which is trying to ignore what's deeply broken. www.thestar.com/opinion/star...

thestar.com

Justin Ling: Who does Mark Carney really listen to? I asked the insiders

It’s become clear that the prime minister would much rather circumvent problems than fix them.

1622792

oh my, I wish I had not seen this...

100

I got so many opinions. The (forced) lack of amenities is probably a non-negligible part why that area is relatively affordable. (Everything is a trade-off.) At least the city did not repeat that mistake in Marine Gateway.

160

If you are going to do that and cc me so I can be in the loop that would be much appreciated. I like the general direction of this and think it’s useful analysis.

000

Education vs unemployment | CensusMapper

On the other hand, I should not complain too loudly. I am certainly guilty of this myself, and fixing this in an interactive JavaScript setting requires building up the necessary machinery there which is a lot of work. Which I have not put in for CensusMapper.

censusmapper.ca

This map shows the relationship between education (share of the population 25-64 years with a Bachelor degree or higher) and the unemployment rate. And it highlights the geographic variation in this r...

Yes, I imagine it’s something like this, but there are many ways to skin that cat. And some are more useful than others. The baseline number of doubled up would-be (in some counterfactual scenario) separate household units is much larger than the population spike and this needs careful treatment.

120

Rental households can’t increase fast than rental units, they are the same thing by definition. I agree most NPR were renters, although we know that NPR household formation rates are significantly below the overall Canadian average. But that’s all part of the question of how “household” was defined.

131

Interesting! Could not find details on how this was actually done though. E.g. “the number of renter households increases faster than the supply of rental units” is obviously bonkers and makes me think their definition of “household” is unorthodox, but there are no details how it’s defined.

141

Don’t use naive regression but use statistical tools designed for ecological inference. E.g. Gary King’s ecological inference estimator or variants of it.

120

Obvious concerns about unmitigated ecological fallacy aside, this is a lot of fun to play with. (Pretty confident much of this would survive if this were handed properly, missed opportunity to put some methodological rigour behind this that would elevate this from toy to actual insights.)

2110

I guess in an environment where everyone gets used to highly discretionary permitting things like bribery and kickbacks just get casually referred to as getting someone to “work magic” and help “navigating” city hall.

He was, for decades, the city’s preeminent “permit expediter,” leaning on the many workers he placed within the city’s building department to work magic for clients navigating San Francisco’s treacherous construction landscape.

092

Alberto Cairo @albertocairo.com · 28/09/2026

There was a major LFS redesign in 1976 and as far as I know the previous version was never made available as PUMF.

110

Reposted by Jens von Bergmann

Lovely and deep visual op-ed by Giorgia Lupi: "A.I. is a different kind of technology, because it can deliver an end result before creative thinking has happened at all" www.nytimes.com/interactive/...

nytimes.com

Opinion | A.I. Shortcuts Are Ruining How We Think (Gift Article)

Anything worth remembering came from a person who did not know where she or he was going until arriving.

25025

The importance of long division is that it’s an algorithm that always works. The first one kids learn at school. A recipe or procedure to follow, that is unforgiving if you mess up a step. A foundation for learning other types of algorithms in math later on.

1211

This reminds me of when the New Math movement argued kids don’t need to learn fractions because of calculators. Fortunately that got reversed, but only after a generation of children failed algebra. If you can’t manipulate fractions with numbers it’s hard to learn how to manipulate it with symbols.

1161

Happy (slightly late) mid-autumn festival!

Don’t worry, upticks don’t sell. Only downticks make the news.

030

2025 fertility rate estimates came out yesterday, posting the obligatory annual graph. Overall Canada TFR remained steady at 1.26. (As a reminder, TFR is a synthetic measure and general cautions apply, especially if you want to write about this in the news media or have arguments about this.)

Total Fertility Rate across Canadian provinces

190

Historical LFS microdata differs somewhat from their modern 2006+ version, it gets stored in a separate local DuckDB. `get_lfs_timeline` harmonizes the two and hands back a single lazy connection to the joint tables for simple and fast cross-database operations. Code runs in a fraction of a second.

library(tidyverse)
library(canpumf)

lfs_tl <- get_lfs_timeline(refresh = "auto")

lfs_unemployment <- lfs_tl |>
filter(LFSSTAT != "Not in labour force") |>
add_lfs_SURVDATE() |>
summarize(Value=sum(FINALWT), .by=c(SURVDATE, GENDER_SEX, LFSSTAT)) |>
mutate(rate=Value/sum(Value),.by=c(SURVDATE, GENDER_SEX)) |>
filter(LFSSTAT == "Unemployed") |>
collect()

ggplot(lfs_unemployment, aes(x = SURVDATE, y = rate, colour = GENDER_SEX)) +
geom_line(alpha = 0.3) +
geom_smooth(method = "loess", span = 0.05, se = FALSE, linewidth = 0.8) +
geom_vline(xintercept = as.Date("2006-01-01"), linetype = "dashed") +
scale_y_continuous(labels = scales::percent) +
labs(title = "Unemployment rate by gender/sex, 1976 onward",
subtitle = "Monthly, not seasonally adjusted; dashed line: LFS_HIST to LFS",
x = NULL, y = "Unemployment rate", colour = NULL,
caption = "StatCan LFS PUMF (1976-2005 via Borealis/ODESI)")

120