Cerebral Cortex Advance Access originally published online on January 13, 2007
Cerebral Cortex 2007 17(10):2443-2452; doi:10.1093/cercor/bhl152
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Solving the Distal Reward Problem through Linkage of STDP and Dopamine Signaling
The Neurosciences Institute, 10640 John Jay Hopkins Drive, San Diego, CA 92121, USA
Address correspondence to email: Eugene.Izhikevich{at}nsi.edu.
In Pavlovian and instrumental conditioning, reward typically comes seconds after reward-triggering actions, creating an explanatory conundrum known as "distal reward problem": How does the brain know what firing patterns of what neurons are responsible for the reward if 1) the patterns are no longer there when the reward arrives and 2) all neurons and synapses are active during the waiting period to the reward? Here, we show how the conundrum is resolved by a model network of cortical spiking neurons with spike-timing–dependent plasticity (STDP) modulated by dopamine (DA). Although STDP is triggered by nearly coincident firing patterns on a millisecond timescale, slow kinetics of subsequent synaptic plasticity is sensitive to changes in the extracellular DA concentration during the critical period of a few seconds. Random firings during the waiting period to the reward do not affect STDP and hence make the network insensitive to the ongoing activity—the key feature that distinguishes our approach from previous theoretical studies, which implicitly assume that the network be quiet during the waiting period or that the patterns be preserved until the reward arrives. This study emphasizes the importance of precise firing patterns in brain dynamics and suggests how a global diffusive reinforcement signal in the form of extracellular DA can selectively influence the right synapses at the right time.
Key Words: classical conditioning • dopamine • instrumental conditioning • reward • simulation • spike-timing–dependent plasticity (STDP)
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  B. Pleger, F. Blankenburg, C. C. Ruff, J. Driver, and R. J. Dolan Reward Facilitates Tactile Judgments and Modulates Hemodynamic Responses in Human Primary Somatosensory Cortex J. Neurosci., August 13, 2008; 28(33): 8161 - 8168. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 E. M. Izhikevich and G. M. Edelman Large-scale model of mammalian thalamocortical systems PNAS, March 4, 2008; 105(9): 3593 - 3598. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. A. Farries and A. L. Fairhall Reinforcement Learning With Modulated Spike Timing Dependent Synaptic Plasticity J Neurophysiol, December 1, 2007; 98(6): 3648 - 3665. [Abstract] [Full Text] [PDF]
|
|