iiinova — where your digital self keeps evolving

Replies

Tom McCoy @rtommccoy.bsky.social · 20/05/2025

We call the resulting system a *prior-trained neural network*, because it has been trained to have a particular prior. 9/n

120 2

Tom McCoy @rtommccoy.bsky.social · 20/05/2025

Even though it is a neural network, the prior-trained model can learn formal languages from small numbers of examples - far outperforming a standard neural network, and matching a Bayesian model at a fraction of the computational cost. 10/n

Plots showing results for formal languages. On the left is a line graph which has “number of training examples” as its x-axis and “F-score” as its y-axis. Three models have lines in this plot: a Bayesian model, a standard neural network, and a prior-trained neural network. The Bayesian model and prior-trained neural network perform similarly, while the standard neural network does much worse than both of them.
On the right is a table showing the amount of training time used by each approach. The Bayesian model uses from 1 minute to 7 days of training time. The neural networks (whether standard or prior-trained) use from 10 milliseconds to 2.5 minutes.

130 3

Tom McCoy @rtommccoy.bsky.social · 20/05/2025

Because it has the flexibility of a neural network, the prior-trained model can also learn in a setting that is intractable for the Bayesian model: Learning aspects of English syntax from millions of words of naturalistic text. 11/n

110 1