iiinova — where your digital self keeps evolving

Finally, we closed the loop by showing that the error codes learned by these models can indeed be used to perform song copying in a very simple RL setup. 14/

A diagram of actor-critic reinforcement learning in which the critic calculates a negative reward (error) via a local circuit of the type discussed above.

A grid of spectrograms. Columns are different bird song syllables. Rows are labeled "target," "before," "during," and "after." The final row matches the first row.

That is, we can imagine the loss landscape moving during sensory learning to provide an error (negative reward) surface that can later guide RL. 13/

A learning diagram in which a curved surface (the loss landscape) changes shape and minimum in the first three panels (labeled "Sensory phase"). In the final panel ("Sensorimotor phase"), a dot representing song descends the loss landscape.

We also examine the learning process and find that it proceeds in distinct stages: auditory neurons first sharpen their response profiles, after which the population shifts its minimum response toward the tutor song pattern. 12/

On the left, "Phase I: learn landscape modes" shows a dotted red line like a shallow bowl curve up into a vase-shaped solid red line. On the right, "Phase II: learn memory modes" shows the dotted blue vase shape (left) shift rightward to a point labeled "tutor song."

And the model's predictions are consistent with population data from calcium imaging of CM (by Fabiola Duarte Ortíz and Rich Mooney), an area in the auditory pallium. We quantify this in a number of ways and find that local plasticity in the EI connections is the best match to data. 11/

Two scatterplots showing the responses of CM neurons to correct singing (x axis) versus either perturbed singing (top plot) or singing post-deafening (bottom plot). Both plots have most points along the diagonal, with a few outliers above, indicated by red arrows.

Same conventions as the last plot, but with more points. There are also gray points in addition to the black ones.

After learning, these models indeed learn to produce sparse error codes that represent deviations from the expected tutor song. 10/

Image plots showing activations of a neural population to correct singing, singing under weak perturbation, singing under strong perturbation, and singing post-deafening.

We investigated a variety of network architectures, learning rules, and sites of local plasticity, among them balanced EI networks of rate-based neurons. 9/

Diagram of a neural network with a premotor population projecting to a local population of excitatory neurons reciprocally connected to a local population of inhibitory neurons.

Now, a few more specifics for those interested: We use a simple setup with sparse coding of spectrogram input that creates a primary auditory representation. Premotor signals and these auditory signals converge in secondary auditory areas, where the action will take place. 8/

A diagram of the problem setup: a box labeled sparse coding sends an arrow to one labeled "auditory input," which in turn sends an arrow to a circle labeled "secondary auditory nuclei." Another box labeled "premotor" also sends an arrow to the circle.

Then, in the practice phase, this same circuit, trained to anticipate the tutor song, computes a *difference* between this memory and the bird's own song. This produces an internal error estimate that can guide learning. 7/

Similar to the previous image. Rows of shapes at the top represent the changing structure of the bird's own song. On the row below, a single repeated shape represents the learned structure of the tutor song. Below, line drawings represent responses of two neurons that diminish over the course of learning as the bird's own song replicates the tutor song.

It works like this: during the memorization phase of learning, auditory regions receive premotor signals from area HVC, which will later control the bird's own song. Local plasticity in these regions works to cancel this anticipated input in local circuits, reducing auditory responses. 6/

A diagram showing two rows of two shapes each. The upper row is labeled "Tutor's Song," and the lower row is labeled "Early Tutoring" and "Late Tutoring." The shapes in the first column do not match, but the shapes in the second column are mirror images of each other.

After a few hundred renditions, he's memorized the tutor song, and he'll spend a full month practicing relentlessly (hundreds of thousands of times) to produce the final product. Most importantly, he can do this with *no feedback* from the tutor. 3/

A male zebra finch with his beak turned to a juvenile zebra finch.

First, off, the setup: This little guy is a juvenile zebra finch. At about a month old, he's going to start memorizing the song of an adult male tutor as a pattern for the one and only courtship song of his life. 2/