New in Neuroscience: Why Treats Come After Training
/
Figure 1. Reinforcement Learning.
We’ve all seen it in action. Tell a puppy dog to sit, and if he obeys, he gets a treat. Rinse and repeat this reinforcement training, and you soon have an obedient puppy dog that sits on command. How does this reinforcement work? And why does it need to occur so soon after the desired behavior? A recent study from the Kasai lab in Tokyo reveals the cellular and molecular mechanisms underlying the timing of reinforcement behavior.
In the striatum, a part of the brain involved in motor learning and motivated behaviors, dendrites, neuronal branches that collect input information from other neurons, have an interesting feature: they are speckled with spines (Fig. 2). These little protuberances jut out from the dendrites to form specialized connections called synapses with the neurons providing input. Spine size reflects the strength of the synapse—the bigger the spine, the more receptive the post-synaptic cell is to the pre-synaptic cell’s communication, and vice versa: think of it like the relative intimacy of a relationship with a person—the more you care about him/her, the more intently you listen. Furthermore, spine growth and shrinkage are believed to underlie behavioral adaptations, thus serving as a cellular correlate of behavioral reinforcement—after all, you may indeed be listening, but actions speak louder than words. Notably, standard pre-synapse to post-synapse communication (called neurotransmission) involves the chemicals glutamate and GABA, which determine the content of the conversation, but synapses also use chemicals like dopamine and serotonin to set what I think of as the emotional tone of the conversation. This study investigates how the timing of that emotional signal affects the post-synaptic neuron’s ability to listen to its pre-synaptic conversation partner.
Figure 2. Medium Spiny Neurons and Spines. Left, a medium spiny neuron labeled in green, with many dendrites projecting outwards. Right, a single dendrite with many spines budding outwards.
In a technical tour de force, the authors combined cutting-edge genetic and optical tools with a technique called patch-clamp essentially used to place an electrical stethoscope onto a single neuron of interest (Fig. 3B for more detail). They then generated focal glutamatergic neurotransmission onto their neuron of interest followed within a few seconds or less by dopamine. Initial results showed that this quick succession of two stimuli—glutamate and then dopamine—enlarged spines in a matter of minutes, and that these enlargements persisted for a long time—up to around an hour in most of their recordings (Fig. 3C, E for more detail). Result recap: If your conversation has emotional depth, you may find yourself caring more about that person than you previously did.
Figure 3. Panels taken from figure 1 of Yagishita et al., 2014. B) Experimental setup. Two-photon glutamate uncaging with a 720 nm laser (red). Optogenetically-induced dopamine release with a 457 nm laser (blue). D1R MSNs are patched and filled with Alexa488 (green). C) Each laser induces intended stimulus. E) Spine growth happens on the order of minutes, and persists for a long time (50 minutes). N) Dopamine release (blue bars/arrows) is paired at various time points with glutamatergic stimulation. O) The variable timing of dopamine release generates variable changes in spine size (ΔVH).
However, and crucially, they found that this enlargement only occurs when dopamine follows glutamate by less than two seconds, and NOT when dopamine precedes glutamate. Follow-up studies using drugs to block dopamine’s action confirmed that this spine enlargement was indeed due to dopamine’s actions. Having confirmed dopamine’s precise temporal role in enlarging spines, the authors performed further experiments to find the intracellular pathways connecting dopamine’s action to spine enlargement. Using drugs targeting proteins activated by dopamine, they found that dopamine release has to be preceded by glutamate because proteins activated by dopamine rely on rapidly-declining Ca2+ levels produced by glutamatergic neurotransmission to drive spine enlargement (Fig. 3N, O for more detail). And this is why you give the puppy a treat after he obeys, and not before. What implications might this finding have on education and economic policy? Hmm…
This study combined several cutting-edge techniques to demonstrate a mechanism for the dependence of reinforcement learning on precisely-timed dopamine release. However, despite their elegant mechanistic discoveries, two issues remain. One is that input from upstream neurons is not well-understood. That’s an understatement—it’s actually a subject of considerable controversy and intrigue. Different populations of neurons have been shown to send excitatory (glutamate), inhibitory (GABA) and dopaminergic signals into the striatum—and some neurons have recently been shown to release several of these signals! Given the narrow time window for dopamine’s actions on these spines, it remains to be determined how all of this is temporally coordinated—by the same neurons using multiple different neurotransmitter molecules, or by different inputs acting in syncopated concert?
The other issue is reproducibility. Reproducibility is a huge issue in science—and in fact, it may even be regarded as the foundation of scientific research. But technically-demanding experiments are typically, and understandably, more difficult to reproduce. This specific group has done some incredible, fascinating work before, but others in the field have informally reported trouble replicating their results. Whether this is due simply to the sheer technical difficulty of such studies or to something (tips, tricks and other treats?) the authors aren’t revealing remains to be determined.
