Learning to predict rewards is a fundamental driver of adaptive behavior. Midbrain dopamine neuro... more Learning to predict rewards is a fundamental driver of adaptive behavior. Midbrain dopamine neurons (DANs) play a key role in such learning by signaling reward prediction errors (RPEs) that teach recipient circuits about expected rewards given current circumstances and actions. However, the algorithm that DANs are thought to provide a substrate for, temporal difference (TD) reinforcement learning (RL), learns the mean of temporally discounted expected future rewards, discarding useful information concerning experienced distributions of reward amounts and delays. Here we present time-magnitude RL (TMRL), a multidimensional variant of distributional reinforcement learning that learns the joint distribution of future rewards over time and magnitude using an efficient code that adapts to environmental statistics. In addition, we discovered signatures of TMRL-like computations in the activity of optogenetically identified DANs in mice during a classical conditioning task. Specifically, w...
We are constantly faced with decisions between alternatives defined by multiple attributes, neces... more We are constantly faced with decisions between alternatives defined by multiple attributes, necessitating an evaluation and integration of different information sources. Time-varying signals in multiple brain areas are implicated in decision-making; but we lack a rigorous biophysical description of how basic circuit properties, such as excitatory-inhibitory (E/I) tone and cascading nonlinearities, shape attribute processing and choice behavior. Furthermore, how such properties govern choice performance under varying levels of environmental uncertainty is unknown. We investigated two-attribute, two-alternative decision-making in a dynamical, cascading nonlinear neural network with three layers: an input layer encoding choice alternative attribute values; an intermediate layer of modules processing separate attributes; and a final layer producing the decision. Depending on intermediate layer E/I tone, the network displays distinct regimes characterized by linear (I), convex (II) or co...
Adaptation facilitates neural representation of a wide range of diverse inputs, including reward ... more Adaptation facilitates neural representation of a wide range of diverse inputs, including reward values. Adaptive value coding typically relies on contextual information either obtained from the environment or retrieved from and maintained in memory. However, it is unknown whether having to retrieve and maintain context information modulates the brain’s capacity for value adaptation. To address this issue, we measured hemodynamic responses of the prefrontal cortex (PFC) in two studies on risky decision-making. In each trial, healthy human subjects chose between a risky and a safe alternative; half of the participants had to remember the risky alternatives, whereas for the other half they were presented visually. The value of safe alternatives varied across trials. PFC responses adapted to contextual risk information, with steeper coding of safe alternative value in lower-risk contexts. Importantly, this adaptation depended on working memory load, such that response functions relatin...
The notion of value is central to theoretical and empirical approaches to decision-making. In psy... more The notion of value is central to theoretical and empirical approaches to decision-making. In psychological and economic choice theory, value functions quantify the relationship between relevant decision information and choice behavior. Evidence for value coding in neural circuits suggests that value information is explicitly represented in brain activity and plays a critical role in the neurobiological choice process. Here, we review a research approach centered on the computations underlying neural value coding. As in sensation and perception, neural information processing in valuation and choice relies on core computational principles including contextual modulation and divisive gain control. The form of these computations reveals details about the nature of decision-related value information and the constraints inherent in computing with biological systems. Understanding value representation at the intermediate level of computation promises insight into decision-making at the level of both the underlying circuit architecture and the resulting choice behavior.
A body of work spanning neuroscience, economics, and psychology indicates that decisionmaking is ... more A body of work spanning neuroscience, economics, and psychology indicates that decisionmaking is context-dependent, which means that the value of an option depends not only on the option in question, but also on the other options in the choice set—or the ‘context’. While context effects have been observed primarily in small-scale laboratory studies with tightly constrained, artificially constructed choice sets, it remains to be determined whether these context effects take hold in real-world choice problems, where choice sets are large and decisions driven by rich histories of direct experience. Here, we investigate whether valuations are context-dependent in real-world choice by analyzing a large restaurant rating dataset (Yelp.com) as well as two independent replication datasets which provide complementary operationalizations of restaurant choice. We find that users make fewer ratings-maximizing choices in choice sets with higher-rated options—a hallmark of context-dependent choic...
This data set contains temporal discount rates estimated from different models of discounting, al... more This data set contains temporal discount rates estimated from different models of discounting, along with model performance metrics. It also contains risk attitude (aversion or tolerance) data and utility model performance scores.The data set is part of an accepted publication entitled 'Risk preferences impose a hidden distortion on measures of choice impulsivity' accepted for publication on PLoS ONE.
We investigated two-attribute, two-alternative decision-making in a hierarchical neural network w... more We investigated two-attribute, two-alternative decision-making in a hierarchical neural network with three layers: an input layer encoding choice alternative attribute values; an intermediate layer of modules processing separate attributes; and a choice layer producing the decision. Depending on intermediate layer excitatory-inhibitory (E/I) tone, the network displays three distinct regimes characterized by linear (I), convex (II) or concave (III) choice indifference curves. In regimes I and II, each option’s attribute information is additively integrated. To maximize reward at low environmental uncertainty, the system should operate in regime I. At high environmental uncertainty, reward maximization is achieved in regime III, with each attribute module selecting a favored alternative, and the ultimate decision based upon comparison between outputs of attribute processing modules. We then use these principles to examine multi-attribute decisions with autism-related deficits in E/I b...
Rational choice theory assumes optimality in decision-making. Violations of a basic axiom of econ... more Rational choice theory assumes optimality in decision-making. Violations of a basic axiom of economic rationality known as “Independence of Irrelevant Alternatives” (IIA) have been demonstrated in both humans and animals and could stem from common neuronal constraints. Here we develop tests for IIA in the nematode Caenorhabditis elegans, an animal with only 302 neurons, using olfactory chemotaxis assays. We find that in most cases C. elegans make rational decisions. However, by probing multiple neuronal architectures using various choice sets, we show that violations of rationality arise when the circuit of olfactory sensory neurons is asymmetric. We further show that genetic manipulations of the asymmetry between the AWC neurons can make the worm irrational. Last, a context-dependent normalization-based model of value coding and gain control explains how particular neuronal constraints on information coding give rise to irrationality. Thus, we demonstrate that bounded rationality c...
Modern life is replete with choices that offer a diverse and often confounding array of options. ... more Modern life is replete with choices that offer a diverse and often confounding array of options. Even simple decisions like purchasing a book or choosing an entree in a restaurant offer a wide array of options, but an extensive array of alternatives exists for far more consequential choices as well: choosing a university, selecting a retirement plan, deciding on
How do outcomes affect future behavior? A study using precise optogenetic stimulation finds that ... more How do outcomes affect future behavior? A study using precise optogenetic stimulation finds that learning from positive reinforcement is mediated by striatal pathways distinct from those that mediate learning from punishment.
To investigate the representation of information in the hippocampus during memory processes, we s... more To investigate the representation of information in the hippocampus during memory processes, we simultaneously monitored the spiking activity of many single neurons in freely behaving rats during spatial locomotor tasks and periods of sleep. The first experiment examined the effect of differential reinforcement on the hippocampal representation of space, as mediated by the spatial receptive fields, or place fields, of hippocampal pyramidal neurons. We show that there is a bias in both place field distribution and population ...
Learning is widely modeled in psychology, neuroscience, and computer science by prediction error-... more Learning is widely modeled in psychology, neuroscience, and computer science by prediction error-guided reinforcement learning (RL) algorithms. While standard RL assumes linear reward functions, reward-related neural activity is a saturating, nonlinear function of reward; however, the computational and behavioral implications of nonlinear RL are unknown. Here, we show that nonlinear RL incorporating the canonical divisive normalization computation introduces an intrinsic and tunable asymmetry in prediction error coding. At the behavioral level, this asymmetry explains empirical variability in risk preferences typically attributed to asymmetric learning rates. At the neural level, diversity in asymmetries provides a computational mechanism for recently proposed theories of distributional RL, allowing the brain to learn the full probability distribution of future rewards. This behavioral and computational flexibility argues for an incorporation of biologically valid value functions in...
SummaryInhibitory interneurons regulate excitability, information flow, and plasticity in neural ... more SummaryInhibitory interneurons regulate excitability, information flow, and plasticity in neural circuits. Inhibitory synapses are also plastic and can be modified by changes in experience or activity, often together with changes to excitatory synapses. However, given the diversity of inhibitory cell types within the cerebral cortex, it is unclear if plasticity is similar for various inhibitory inputs or what the functional significance of inhibitory plasticity might be. Here we examined spike-timing-dependent plasticity of inhibitory synapses from four major subtypes of GABAergic cells onto layer 2/3 pyramidal cells in mouse auditory cortex. The likelihood of inhibitory potentiation varied across cell types, with somatostatin-positive (SST+) interneuron inputs exhibiting the most potentiation on average. A network simulation of perceptual decision-making revealed that plasticity of SST+-like inputs provided robustness from higher input noise levels to maintain decision accuracy. Di...
This data set contains temporal discount rates estimated from different models of discounting, al... more This data set contains temporal discount rates estimated from different models of discounting, along with model performance metrics. It also contains risk attitude (aversion or tolerance) data and utility model performance scores.The data set is part of an accepted publication entitled 'Risk preferences impose a hidden distortion on measures of choice impulsivity' accepted for publication on PLoS ONE.
Oxford Research Encyclopedia of Neuroscience, 2019
A core question in systems and computational neuroscience is how the brain represents information... more A core question in systems and computational neuroscience is how the brain represents information. Identifying principles of information coding in neural circuits is critical to understanding brain organization and function in sensory, motor, and cognitive neuroscience. This provides a conceptual bridge between the underlying biophysical mechanisms and the ultimate behavioral goals of the organism. Central to this framework is the question of computation: what are the relevant representations of input and output, and what algorithms govern the input-output transformation? Remarkably, evidence suggests that certain canonical computations exist across different circuits, brain regions, and species. Such computations are implemented by different biophysical and network mechanisms, indicating that the unifying target of conservation is the algorithmic form of information processing rather than the specific biological implementation. A prime candidate to serve as a canonical computation ...
Learning to predict rewards is a fundamental driver of adaptive behavior. Midbrain dopamine neuro... more Learning to predict rewards is a fundamental driver of adaptive behavior. Midbrain dopamine neurons (DANs) play a key role in such learning by signaling reward prediction errors (RPEs) that teach recipient circuits about expected rewards given current circumstances and actions. However, the algorithm that DANs are thought to provide a substrate for, temporal difference (TD) reinforcement learning (RL), learns the mean of temporally discounted expected future rewards, discarding useful information concerning experienced distributions of reward amounts and delays. Here we present time-magnitude RL (TMRL), a multidimensional variant of distributional reinforcement learning that learns the joint distribution of future rewards over time and magnitude using an efficient code that adapts to environmental statistics. In addition, we discovered signatures of TMRL-like computations in the activity of optogenetically identified DANs in mice during a classical conditioning task. Specifically, w...
We are constantly faced with decisions between alternatives defined by multiple attributes, neces... more We are constantly faced with decisions between alternatives defined by multiple attributes, necessitating an evaluation and integration of different information sources. Time-varying signals in multiple brain areas are implicated in decision-making; but we lack a rigorous biophysical description of how basic circuit properties, such as excitatory-inhibitory (E/I) tone and cascading nonlinearities, shape attribute processing and choice behavior. Furthermore, how such properties govern choice performance under varying levels of environmental uncertainty is unknown. We investigated two-attribute, two-alternative decision-making in a dynamical, cascading nonlinear neural network with three layers: an input layer encoding choice alternative attribute values; an intermediate layer of modules processing separate attributes; and a final layer producing the decision. Depending on intermediate layer E/I tone, the network displays distinct regimes characterized by linear (I), convex (II) or co...
Adaptation facilitates neural representation of a wide range of diverse inputs, including reward ... more Adaptation facilitates neural representation of a wide range of diverse inputs, including reward values. Adaptive value coding typically relies on contextual information either obtained from the environment or retrieved from and maintained in memory. However, it is unknown whether having to retrieve and maintain context information modulates the brain’s capacity for value adaptation. To address this issue, we measured hemodynamic responses of the prefrontal cortex (PFC) in two studies on risky decision-making. In each trial, healthy human subjects chose between a risky and a safe alternative; half of the participants had to remember the risky alternatives, whereas for the other half they were presented visually. The value of safe alternatives varied across trials. PFC responses adapted to contextual risk information, with steeper coding of safe alternative value in lower-risk contexts. Importantly, this adaptation depended on working memory load, such that response functions relatin...
The notion of value is central to theoretical and empirical approaches to decision-making. In psy... more The notion of value is central to theoretical and empirical approaches to decision-making. In psychological and economic choice theory, value functions quantify the relationship between relevant decision information and choice behavior. Evidence for value coding in neural circuits suggests that value information is explicitly represented in brain activity and plays a critical role in the neurobiological choice process. Here, we review a research approach centered on the computations underlying neural value coding. As in sensation and perception, neural information processing in valuation and choice relies on core computational principles including contextual modulation and divisive gain control. The form of these computations reveals details about the nature of decision-related value information and the constraints inherent in computing with biological systems. Understanding value representation at the intermediate level of computation promises insight into decision-making at the level of both the underlying circuit architecture and the resulting choice behavior.
A body of work spanning neuroscience, economics, and psychology indicates that decisionmaking is ... more A body of work spanning neuroscience, economics, and psychology indicates that decisionmaking is context-dependent, which means that the value of an option depends not only on the option in question, but also on the other options in the choice set—or the ‘context’. While context effects have been observed primarily in small-scale laboratory studies with tightly constrained, artificially constructed choice sets, it remains to be determined whether these context effects take hold in real-world choice problems, where choice sets are large and decisions driven by rich histories of direct experience. Here, we investigate whether valuations are context-dependent in real-world choice by analyzing a large restaurant rating dataset (Yelp.com) as well as two independent replication datasets which provide complementary operationalizations of restaurant choice. We find that users make fewer ratings-maximizing choices in choice sets with higher-rated options—a hallmark of context-dependent choic...
This data set contains temporal discount rates estimated from different models of discounting, al... more This data set contains temporal discount rates estimated from different models of discounting, along with model performance metrics. It also contains risk attitude (aversion or tolerance) data and utility model performance scores.The data set is part of an accepted publication entitled 'Risk preferences impose a hidden distortion on measures of choice impulsivity' accepted for publication on PLoS ONE.
We investigated two-attribute, two-alternative decision-making in a hierarchical neural network w... more We investigated two-attribute, two-alternative decision-making in a hierarchical neural network with three layers: an input layer encoding choice alternative attribute values; an intermediate layer of modules processing separate attributes; and a choice layer producing the decision. Depending on intermediate layer excitatory-inhibitory (E/I) tone, the network displays three distinct regimes characterized by linear (I), convex (II) or concave (III) choice indifference curves. In regimes I and II, each option’s attribute information is additively integrated. To maximize reward at low environmental uncertainty, the system should operate in regime I. At high environmental uncertainty, reward maximization is achieved in regime III, with each attribute module selecting a favored alternative, and the ultimate decision based upon comparison between outputs of attribute processing modules. We then use these principles to examine multi-attribute decisions with autism-related deficits in E/I b...
Rational choice theory assumes optimality in decision-making. Violations of a basic axiom of econ... more Rational choice theory assumes optimality in decision-making. Violations of a basic axiom of economic rationality known as “Independence of Irrelevant Alternatives” (IIA) have been demonstrated in both humans and animals and could stem from common neuronal constraints. Here we develop tests for IIA in the nematode Caenorhabditis elegans, an animal with only 302 neurons, using olfactory chemotaxis assays. We find that in most cases C. elegans make rational decisions. However, by probing multiple neuronal architectures using various choice sets, we show that violations of rationality arise when the circuit of olfactory sensory neurons is asymmetric. We further show that genetic manipulations of the asymmetry between the AWC neurons can make the worm irrational. Last, a context-dependent normalization-based model of value coding and gain control explains how particular neuronal constraints on information coding give rise to irrationality. Thus, we demonstrate that bounded rationality c...
Modern life is replete with choices that offer a diverse and often confounding array of options. ... more Modern life is replete with choices that offer a diverse and often confounding array of options. Even simple decisions like purchasing a book or choosing an entree in a restaurant offer a wide array of options, but an extensive array of alternatives exists for far more consequential choices as well: choosing a university, selecting a retirement plan, deciding on
How do outcomes affect future behavior? A study using precise optogenetic stimulation finds that ... more How do outcomes affect future behavior? A study using precise optogenetic stimulation finds that learning from positive reinforcement is mediated by striatal pathways distinct from those that mediate learning from punishment.
To investigate the representation of information in the hippocampus during memory processes, we s... more To investigate the representation of information in the hippocampus during memory processes, we simultaneously monitored the spiking activity of many single neurons in freely behaving rats during spatial locomotor tasks and periods of sleep. The first experiment examined the effect of differential reinforcement on the hippocampal representation of space, as mediated by the spatial receptive fields, or place fields, of hippocampal pyramidal neurons. We show that there is a bias in both place field distribution and population ...
Learning is widely modeled in psychology, neuroscience, and computer science by prediction error-... more Learning is widely modeled in psychology, neuroscience, and computer science by prediction error-guided reinforcement learning (RL) algorithms. While standard RL assumes linear reward functions, reward-related neural activity is a saturating, nonlinear function of reward; however, the computational and behavioral implications of nonlinear RL are unknown. Here, we show that nonlinear RL incorporating the canonical divisive normalization computation introduces an intrinsic and tunable asymmetry in prediction error coding. At the behavioral level, this asymmetry explains empirical variability in risk preferences typically attributed to asymmetric learning rates. At the neural level, diversity in asymmetries provides a computational mechanism for recently proposed theories of distributional RL, allowing the brain to learn the full probability distribution of future rewards. This behavioral and computational flexibility argues for an incorporation of biologically valid value functions in...
SummaryInhibitory interneurons regulate excitability, information flow, and plasticity in neural ... more SummaryInhibitory interneurons regulate excitability, information flow, and plasticity in neural circuits. Inhibitory synapses are also plastic and can be modified by changes in experience or activity, often together with changes to excitatory synapses. However, given the diversity of inhibitory cell types within the cerebral cortex, it is unclear if plasticity is similar for various inhibitory inputs or what the functional significance of inhibitory plasticity might be. Here we examined spike-timing-dependent plasticity of inhibitory synapses from four major subtypes of GABAergic cells onto layer 2/3 pyramidal cells in mouse auditory cortex. The likelihood of inhibitory potentiation varied across cell types, with somatostatin-positive (SST+) interneuron inputs exhibiting the most potentiation on average. A network simulation of perceptual decision-making revealed that plasticity of SST+-like inputs provided robustness from higher input noise levels to maintain decision accuracy. Di...
This data set contains temporal discount rates estimated from different models of discounting, al... more This data set contains temporal discount rates estimated from different models of discounting, along with model performance metrics. It also contains risk attitude (aversion or tolerance) data and utility model performance scores.The data set is part of an accepted publication entitled 'Risk preferences impose a hidden distortion on measures of choice impulsivity' accepted for publication on PLoS ONE.
Oxford Research Encyclopedia of Neuroscience, 2019
A core question in systems and computational neuroscience is how the brain represents information... more A core question in systems and computational neuroscience is how the brain represents information. Identifying principles of information coding in neural circuits is critical to understanding brain organization and function in sensory, motor, and cognitive neuroscience. This provides a conceptual bridge between the underlying biophysical mechanisms and the ultimate behavioral goals of the organism. Central to this framework is the question of computation: what are the relevant representations of input and output, and what algorithms govern the input-output transformation? Remarkably, evidence suggests that certain canonical computations exist across different circuits, brain regions, and species. Such computations are implemented by different biophysical and network mechanisms, indicating that the unifying target of conservation is the algorithmic form of information processing rather than the specific biological implementation. A prime candidate to serve as a canonical computation ...
Uploads
Papers by Kenway Louie