Action selection

Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection Action selection Tony J. Prescott (2008), Scholarpedia, 3(2):2705. Recommend this on Google doi:10.4249/scholarpedia.2705 revision #137678 [link to/cite this article] Prof. Tony J. Prescott, Dept Psychology, Univ of Sheffield, UK Informally speaking, action selection describes the task of choosing “what to do next”. More formally, given an agent with a repertoire of available actions, some knowledge of its internal state, and some sensory information concerning environmental context, the task is to decide what action (or action sequence) to perform in order for that agent to best achieve its goals. Contents 1 Related terms and scope 2 Is action selection a genuine problem? 3 Requirements for effective action selection 4 Candidate action selection mechanisms 5 Evolution may favor specialized action selection mechanisms 6 Possible action selection substrates in the vertebrate brain 6.1 Conflict resolution for clean escape 6.2 Fixed priority mechanisms 6.3 Recurrent reciprocal inhibition 6.4 The basal ganglia 6.5 The medial core of the reticular formation 6.6 The role of the cortex in action selection 6.7 Self-organising processes in action selection 7 Conclusion 8 External links 9 References 10 See also Related terms and scope The problem of selecting between alternative actions has been a focus of research in ethology, psychology, neurobiology, computational neuroscience, artificial intelligence, and robotics. In these literatures the problem is also sometimes described as that of ‘behavioral choice’, ‘motor program selection’, or ‘decision making’. Action selection is also related to the problem of attention. Overt shifts of attention can be thought of as selected actions, whilst covert shifts (that have no behavioral manifestations) may be thought of as the resolution of internal selection conflicts concerning processing resources. The term ‘behavior switching’ is sometimes used as a synonym for action selection but will be used here to describe the sub-problem of managing the transition between two behavioral states in a smooth and timely manner. Research on action selection includes work on the problem of determining optimal actions (see e.g. Houston, McNamara and Steer, 2007; Seth 2007), which may involve learning, and on the problem of ‘architecture’—that is, the task of designing control systems that implement effective action selection. This 1 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection article will address the architecture problem and not the optimality problem, it will also focus on biological questions concerning how action selection competitions are resolved in vertebrate nervous systems; reviews of action selection in invertebrates have been provided by Davis (1979), Kristan and Shaw (1997), Kupfermann and Weiss (2001) and Prescott (2007), and in artefacts such as robots or software agents by Maes (1995) and Bryson (2000). Is action selection a genuine problem? The very notion that actions are ‘selected’ is controversial. One difficulty is that the concept of action selection, at least as traditionally defined, assumes the decomposition of behavior into distinct elements (actions) that can be selected between. Whilst it is possible to design an artefact, such as a robot, to have a repertoire of discrete acts that it can perform, it is not immediately clear that animal or human behavior decomposes cleanly in this way. For instance, what constitutes an action when you reach for, grasp, and then sip from a glass of water? On one view you might be performing a sequence of three separate acts—reach, grasp, sip—on another you are performing a single integrated act of ‘drinking’. This issue is not merely a descriptive one. On the hypothesis that the brain is solving action selection problems in executing such behaviors, a critical question concerns what the ‘units’ of selection might be. In this example, you might select to take a drink and then resolve a further sequence of more specific action selection conflicts: which arm to use? which grasp? when to stop closing my hand and begin lifting? At some point this cascade of action selection tasks should ground out in a sequence of selected sensorimotor primitives that are allowed to control your effector systems (limbs and muscles). Such a view follows a strong tradition of ideas about the hierarchical decomposition of control in sensorimotor neuroscience (see, e.g. Botvinick, 2007). On another view, however, following your decision to take a drink, your sensory and motor apparatus co-ordinate themselves so as to find and grasp the cup and ensure the liquid reaches your mouth without spilling. This account does not invoke selection hierarchies but relies on the phenomena of self-organization in neural systems. In other words, it assumes that your nervous system is configured so as to generate an appropriate sequence of attractor states that produces the required behavior without explicitly representing its elements (or their alternatives) as discrete components. Such a dynamical systems view of sensorimotor control is gaining ground in the neuroscience community as a powerful alternative hypothesis to the classical hierarchical view (see e.g. Kelso, 1995). The roots of this approach include work on motor patterning in invertebrate nervous systems where the small size of the networks involved has allowed detailed study of the functional role of specific neurons (e.g. Mpitsos and Cohen, 1986). From another direction, the study of model cortical circuits, has also demonstrated the importance of attractor dynamics in state switching and attentional control (see 'self-organising processing in action selection' below). However, if the details of movement are controlled via network dynamics, what of your original decision to take a drink? Might it, too, be the result of attractor dynamics in relevant neural networks? In other words, could the organisation of behaviour be attractor dynamics all the way up, and if so what place is left for the classical notion that actions need to be selected? Note that there is an uncontentious sense in which the brain as a whole is a dynamical system and will thus show dynamical properties in its circuits. However, the question considered here is not one of dynamics per se but of modularity. Specifically, whether it is useful to consider the brain as having a decomposition in which some components have a primarily action selection role, possibly as part of some hierarchical cascade. Research in the synthetic sciences (artificial intelligence and robotics) demonstrates that control systems can be constructed that operate either by the principles of explicit hierarchical action 2 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection selection (see, e.g. Bryson, 2000, Botvinick, 2007), or by designing appropriately configured controllers in which appropriate behavior, and behavior switching, emerges globally through system dynamics (see e.g. Seth 2007). In either case, existing systems do not approach animal nervous systems in their sophistication or reliability but do provide proof of principle that both strategies can work. In the biological sciences the question of which style of control best describes the vertebrate brain architecture remains an important empirical issue. The remainder of this article briefly reviews some evidence pertaining to this question. The position to be outlined rests, in fact, on a compromise: that the co-ordination of behavior by vertebrates brains makes use of specialized action selection circuitry that can guarantee cleanly selected patterns of expressed behavior, but also exploits the powerful attractor dynamics of neural circuits in areas such as the neocortex in order to constrain the choice of candidate actions. Before outlining this view the following sub-sections will (i) briefly review some requirements for effective action selection (taking an ‘actions as units’ perspective), and, on the basis of these requirements, (ii) derive criteria for identifying specialized selection mechanisms in animal nervous systems, and (iii) identify some candidate network configurations that could provide effective substrates for the resolution of action selection conflicts. Requirements for effective action selection In order to facilitate effective selection and timely switching between competitors, we can identify a number of useful properties that an action selection mechanism should possess: A basic principle of action selection is that from a set of incompatible competitors only one should be allowed expression at a given time. In selecting a single winner, a heuristic that appears to be exploited in vertebrate decision-making (McFarland, 1989) is to prefer the most strongly supported, or most salient, competitor as indicated by relevant external and internal cues. A competitor with a slight edge over its rivals should see the competition resolved rapidly and decisively in its favor, so providing clean switching. Following resolution of a selection competition the winner should be fully selected (i.e. allowed unrestricted access to the motor apparatus) and the losers prevented from interfering with its performance (absence of distortion). In many circumstances it may also be useful for a winning competitor to remain active at lower input levels than are initially required for it to overcome the competition. This characteristic, termed persistence (McFarland, 1989), can prevent unnecessary switching, or ‘dithering’, between closely matched competitors. Note that conflict resolution between competitors bidding for incompatible uses of a single resource is only part of the wider problem of generating integrated behavior (Prescott, 2007). Different effector systems, such as the muscle groups underlying locomotion and gaze in mammals, constitute more-or-less independent resources, however, it is clearly important that their activities are appropriately co-ordinated. For instance, the gaze system should be frequently oriented in the current direction of travel to ensure a clear route is available. As this example demonstrates, selection mechanisms for individual resources need to be embedded within a control architecture that can deliver appropriate simultaneous and sequential patterns of activity in multiple output systems. Candidate action selection mechanisms 3 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection Because it is a fundamental property of neurons to be selective with regard to patterns of input activity to which they respond, claims that particular brain sub-systems are specifically or preferentially involved in the selection of action, as distinct to other aspects of control, must meet more stringent requirements. For instance, to be considered as a candidate action-selection mechanism, Redgrave, Prescott, and Gurney (1999) suggested that a neural sub-system should exhibit properties that reflect the requirements for effective action selection identified above, namely: (i) the system of interest should have inputs that carry information about both internal (to the body) and external (outside the body) cues relevant to decision-making, (ii) there should be some mechanism that allows calculation of the salience that should be attached each available action, (iii) there should be mechanisms that allow for the resolution of conflicts between competing actions based on their relative salience, (iv) the outputs of the system should be configured so as to allow the expression of winning actions whilst disallowing losers. This section outlines a selection of network architectures that could potentially serve as the conflict resolution mechanism identified in (iii). Consideration of the other properties listed above is deferred until the review of particular candidate brain sub-systems below. A specific form of neural connectivity, which is often associated with action selection, is recurrent reciprocal inhibition (RRI) (see also neural inhibition) whereby two or more units are connected such that each one has an inhibitory link to every other (see figure 1a). Such circuits display a form of positive feedback since increasing the activation of one unit causes increased inhibition on the remaining units thereby reducing their inhibitory effect on the first. RRI therefore provides many desirable selection properties including full selection of the winner, absence of distortion, and clean switching. An RRI network will also naturally exhibit hysteresis, and thus behavioral persistence, such that once one active unit has become selected it becomes harder for an evenly-matched competitor to wrest control. A second candidate network configuration is the feed-forward circuit illustrated in Figure 1b. Here, the salience of each of the selection candidates is represented by activity in the upper row of input units, and the extent to which they are selected by the lower output units. Each unit in the input layer excites its partner Figure 1: Network architectures that can support action selection. Red: output unit in the same “channel” excitation, Blue: inhibition whilst inhibiting those in competing channels. Note, that whilst such a circuit can be counted on to boost the activity of the most salient channel whilst weakening that of those less salient (Gurney, Prescott, & Redgrave, 2001), this contrast enhancement does not imply full selection or the absence of distortion. However, by adding positive feedback connectivity (1c), that allows each output unit to excite its own input, we can reintroduce recurrence to produce a circuit with similar good selection and hysteresis properties to the RRI model. 4 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection All of the network models just described have a significant disadvantage in terms of scaling cost. Namely, to arbitrate between n competitors each requires n(n-1) inhibitory connections, while adding a new competitor requires a further 2n connections. Moreover, the feed-forward system (1b) requires an additional n excitatory connections, and the version with the positive feedback loop (1c) a further n connections. Note, however, there is an alternative configuration of these latter two networks that scales much more efficiently though at the cost of introducing more complex regulatory control. Specifically, removing the n(n-1) direct inhibitory links and introducing a component whose role is to broadcast global inhibition to the output layer (1d) avoids the exponential growth in connectivity costs. To make this system work, however, requires that the surround inhibition is appropriately balanced with the focused excitation to allow effective action selection to take place. A final example in this, by no means exhaustive (or mutually exclusive), catalogue of possible selection circuits is simply to put all the conflict resolution machinery inside a special purpose selection component (1e). This ‘action selector’ then receives input from all of the selection candidates and broadcasts the relevant output of its internal conflict resolution process back to each of these competitors. This circuit has low extrinsic connectivity though, clearly, the intrinsic network that resolves the competitions within the selection component may be complex and have significant bandwidth requirements. Evolution may favor specialized action selection mechanisms The above discussion of candidate selection networks has raised the issue of connectivity costs as this is a major determinant of the size and metabolic efficiency of animal nervous systems. Ringo (1991) has pointed out that geometrical factors place important limits on the degree of network interconnectivity within the brain. In particular, larger brains cannot support the same degree of connectivity as smaller ones—significant increases in brain size must inevitably be accompanied by decreased connectivity between non-neighboring brain areas. Leise (1990) has further argued that a common feature of both vertebrate and invertebrate nervous systems is that they are composed of anatomically and functionally differentiable local compartments which are restricted in size to a maximum of around 1mm diameter. Connectivity between neurons is highest within compartments, and larger nervous systems have more compartments rather than larger individual compartments. One of the constraints that appears to limit compartment size is the greater cost of high-bandwidth communication over long distances in neural tissue. The nature of the action selection problem is such that functional systems in different parts of the brain will often be in competition for the same motor resources. In evolution, then, the requirements of lower connectivity and increased compartmentalization with increased brain size should therefore have favored selection architectures with lower connectional overheads. Such pressures would appear to work against the emergence of large-scale reciprocal inhibition networks, although their presence within local compartments would invoke a less costly overhead. More generally, specialised selection systems, such as figure 1e, that minimise long-range connectivity would appear to be favoured by this constraint. In this context it is worth noting that decisions at higher levels of the 'selection cascade', such as deciding to take a drink, may be less localized (i.e. involve more long-distance communication within the brain), than those at lower levels such as selecting which grasp to use in picking up a cup. Thus, lower level selection decisions may be less effected by this scaling issue, and more able to make use of relatively costly connectivity schemes such as RRI. A second argument in favour of the emergence of specialized selectors is the advantage conferred by modularity itself (Wagner & Altenberg, 1996). Specifically, to the extent that the problem of selection can be distinguished from the perceptual and motor control problems involved in coordinating a given 5 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection activity it should be advantageous to decouple the selection mechanism from other parts of the control circuitry. As separate components each can be improved or modified independently. In contrast, in a circuit that dynamically ‘flips’ in a global way between distinct behavioural states, a change directed at some other aspect of function (say, the fine details of motor control) could impact on the switching behavior of the network with possibly undesirable consequences. Possible action selection substrates in the vertebrate brain The substrate for action selection in a control architecture as complex as the vertebrate nervous system is likely to involve many different mechanisms and structures. The following brief review is by no means exhaustive but considers a few promising candidates. Conflict resolution for clean escape One of the requirements for effective action selection is timely, sometimes very rapid, decision making. Transmission and response times in neural tissue are not negligible so for urgent tasks it is important to ensure that time is not lost resolving conflicts with competing behaviors. Indeed, there is evidence to suggest, that for tasks such as defensive escape, special circuitry may have evolved in the vertebrate nervous system to provide a very fast override of the competition. The giant Mauthner cells (M-cells) found in the brain-stem of most fish and some amphibians provide an example of this function. M-cells are known to be involved in the ‘C-start’ escape maneuver—the primary behavior used by many species of fish to avoid hazards such as predation. Eaton, Hofve, and Fetcho (1995) have argued that the principal role of the M-cell in the brainstem escape circuit may not be to initiate the C-start as much as to suppress competing behaviors. This conclusion is supported by evidence that removal of the M-cells does not disable the C-start and has only a mild effect on the strength or latency of the response. Instead, the fast conduction of the Mauthner giant axon (one of the largest in the vertebrates) may be crucial in ensuring that contradictory signals, that could otherwise result in fatal errors, do not influence motor output mechanisms. Conservation of brain-stem organization across the vertebrate classes suggests that homologous mechanisms may play a similar role in the escape behaviors of other vertebrates. For instance, giant neurons in the caudal pontine reticular nucleus of rats, have been shown to play a central role in the acoustic startle reflex (Lingenhohl and Friauf, 1994). The connectivity of these cells, in addition other properties such as their high firing threshold and broad frequency tuning, suggests that a circuit homologous to the fish brainstem escape system may have survived largely intact in the mammalian brain (Eaton, Lee, and Foreman, 2001). Fixed priority mechanisms Many studies of the role of the vertebrate brain in behavioral integration suggest that the resolution of conflict problems between the different levels of the neuraxis (spinal cord, hindbrain, midbrain, etc.) may be determined by fixed-priority, vertical links. For instance, Prescott, Gurney, and Redgrave (1999) have reviewed evidence that the vertebrate defense system can be viewed as a set of dissociable layers in which higher levels can suppress or modulate the outputs of lower levels. Fixed-priority mechanisms cannot, however, capture the versatility of behavior switching observed between the different behavior systems (defense, feeding, reproduction, etc.) found in adult vertebrates. Since dominance relationships between behavior systems can fluctuate dramatically with changing circumstances more flexible forms of conflict resolution are required than can be determined by this form of hard-wiring. Recurrent reciprocal inhibition 6 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection RRI connectivity has been identified in many different areas of the vertebrate brain (Windhorst, 1996), could play a role in conflict resolution at multiple levels of the nervous system (Gallistel, 1980), and, modulated by top-down biasing, is likely an important characteristic of cortical mechanisms for selective attention (see e.g. Desimone & Duncan, 1995; Deco and Rolls, 2005). However, due to the scaling issue noted above, its role in selecting between distally located brain sub-systems may be limited to conflicts involving only a small number of competitors. A nice example of such a large-scale RRI circuit, which has been found to play a critical role in regulating mammalian sleep-wake behavior, occurs between the ventrolateral preoptic (VLPO) nucleus of the hypothalamus and a group of related monoaminergic brainstem nuclei (Saper, Scammell, & Lu, 2005). VLPO neurons that are primarily active during sleep have direct, mutual inhibitory connections with cells in these monoaminergic nuclei that fire most rapidly during wakefulness; the resulting circuit instantiates a switch capable of generating rapid transitions between arousal states. A further group of neurons in the lateral hypothalamus appears to modulate the stability of this switch which would otherwise be over-sensitive to small perturbations. The basal ganglia The principal components of the basal ganglia include the striatum, globus pallidus and subthalamic nucleus in the base of the vertebrate forebrain, and the substantia nigra in the midbrain. The proposal that this group of inter-linked nuclei are involved in action selection is based on an emerging consensus amongst neuroscientists that their key function is to enable desired actions and to inhibit undesired, potentially competing, actions (see, e.g. Mink, 1996; Redgrave et al., 1999). The basal ganglia appear to fulfill the requirements noted above for a specialized selection device as follows. Neural signals that may represent ‘requests for access’ to the motor system are continuously projected to the striatum, which is the principal basal ganglia input nucleus, from relevant functional sub-systems in both the brainstem and forebrain of the animal. Afferents from a wide range of sensory and motivational systems also arrive at striatal input neurons. This connectivity should allow both extrinsic and intrinsic motivating factors to influencing the strength of rival bids. The level of activity in different populations of striatal neurons (channels) may then provide a neural representation of action salience. The main output centers of the basal ganglia (parts of the substantia nigra and globus pallidus) are tonically active and direct a continuous flow of inhibition at neural centers throughout the brain that either directly or indirectly generate movement. This tonic inhibition appears to place a powerful brake on these movement systems such that the basal ganglia effectively holds a ‘veto’ over all voluntary activity. As shown in figure 2, intrinsic basal ganglia circuitry, together with feedback loops via the thalamus, appears to be suitably configured to resolve the selection competition between multiple active channels. More specifically, this architecture implements a form of the 'feed-forward selection circuit with positive feedback' previously illustrated in figure 1d. A main difference here is that activity in the output layer is inverted, compared to the previous figure, with full selection corresponding the inhibition of specific basal ganglia output neurons and thus the disinhibition of their motor system targets. Within the basal ganglia the subthalamic nucleus (STN) may play the role of the network component provide the global ‘stop’ signal for losing channels—in this case by increasing the activity of basal ganglia output neurons. Modulation of the balance between focused inhibition (of the winner) and diffuse excitation (of losers) appears to be managed by an intrinsic basal ganglia circuit involving the globus pallidus external segment (GPe) that may appropriately scale the output of the subthalamic nucleus relative to the number of active channels (Gurney et al., 2001). It is worth noting, too, that reciprocal inhibition, at the sub-compartment (<1mm) range, is found within several individual basal ganglia nuclei where it may serve to enhance the overall selection properties of this system. 7 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection The medial core of the reticular formation Studies of infants rats in whom the basal ganglia are not yet developed, and in decerebrate animals in which the forebrain and much of the midbrain have been removed, indicate that, below the basal ganglia, there is a brainstem substrate for selection that can provide some basic behavioral switching while the adult architecture is developing or when it is damaged or incapacitated. A likely locus for this mechanism is in the medial core of the brainstem reticular formation (mRF) (Humphries, Gurney, & Prescott, 2007). The mRF appears to fulfill the connectional requirements of Figure 2: Basal ganglia selection circuitry. The double inhibitory connection Striatum-SNr-Motor systems provides focused selection of a winning action (via disinhibition), whilst diffuse excitation of substantia nigra pars reticulata (SNr), provided by the subthalamic nucleus (STN), maintains inhibition on losing channels. Intrinsic circuitry involving the globus pallidus external segment (GPe) provides regulation of global excitation from STN. The circuit Cortex-StriatumSNr-Thalamus-Cortex implements a positive feedback loop to reinforce a selected channel. a centralized selection system in that it receives afferent input from all of the body’s external and internal sensory systems and projects outputs to the cranial nerves that control movement of the face and to spinal neurons that command limb control and locomotion pattern generation. The intrinsic circuitry of the mRF appears to be configured as a set of clusters, that has been analogized with “a stack of poker chips” (Scheibel & Scheibel, 1967). In each cluster there are two main neural populations: the first consists of large projection neurons, having excitatory outputs, that branch to targets in the spinal cord and midbrain as well as to other clusters within the mRF; the second population consists of inter-neurons that project inhibitory outputs entirely within the same cluster. Since this intrinsic architecture does not resemble any of the candidate selection mechanism reviewed earlier, how, then, might the mRF operate as an action selection substrate? Humphries et al. (2007) have proposed that activity in individual clusters may represent sub-actions—component parts of a coherent behavior. Thus, the expression of a behavior would involve the simultaneous activation of clusters representing compatible sub-actions and inhibition of clusters representing incompatible ones. Both the basal ganglia and the reticular formation lie in central positions along the vertebrate main neuraxis and have been described collectively as forming the brain’s ‘centrencephalic core’ and identified by a number of neurobiologists as playing a key role in the integration of behavior (see Prescott et al., 1999 for review). The mRF is a major target of basal ganglia output via a pathway involving the pedunculopontine nucleus, hence it is possible that the relationship between the two systems may reflect a hierarchical decomposition of control whereby patterns of innate behavior organized in the mRF could be selected “in toto” by the basal ganglia. The role of the cortex in action selection The evolution of mammals saw a substantial increase in the role of the forebrain in action specification and control largely supplementing, rather than replacing, the functionality of motor systems lower down the neuraxis. Whilst cortex itself is not new in evolutionary terms (being homologous to areas of dorsal pallium in other jawed vertebrates), it is larger and more differentiated (more cortical areas) in modern 8 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection mammals compared to ancestral reptiles. New functional circuits have also evolved such as the pathways allowing direct cortical control over brainstem spinal cord motorneurons (Butler and Hodos, 2005). The mammalian brain consequently possesses a complex layered control architecture providing multiple levels of sensorimotor competence (Prescott et al., 1999). Both cortical and sub-cortical motor systems form action selection ‘loops’ via the basal ganglia providing the option to choose between brainstem systems that provide a rapid response to immediate contingencies, and cortical systems that provide more sophisticated adjudication between alternatives, taking greater account of context, past experiences, and future opportunities. Experiments with mammals, such as rats and cats, in which the cerebral cortex has been completely removed (see Gallistel, 1980), show that such radical surgery leaves intact the capacity to generate motivated, and integrated behavioral sequences. Decorticate animals lack skilled motor control, are impaired in various learning tasks, and appear more stimulus-bound than control animals, yet they still eat, drink, and groom under appropriate motivational and stimulus conditions, and display many aspects of normal social and sexual behavior. To this extent, then, the cortex is not a critical locus for action selection with respect to these animals’ basic repertoire of species-typical acts. The mammalian prefrontal cortex, which has long been identified with a general role in ‘executive function’, forms similar closed-loop circuits with the basal ganglia to motor cortical areas more directly associated with the control of movement. This finding, suggests that the basal ganglia selection mechanisms have been appropriated for use in more cognitive tasks involving, for example, planning, or the selection and maintenance of working memory representations (see e.g. Hazy, Frank, & O'Reilly, 2007). Areas of prefrontal cortex also act to process reward information, to predict expected outcomes, to monitor performance and catch errors, to evaluate cost/benefit trade-offs, and to analyse context (Krawczyk, 2002; Ridderinkhof, van den Wildenberg, Segalowitz, & Carter, 2004; Rushworth et al., 2005; Schall, Stuphorn, & Brown, 2002). In other words, by working in concert with specialized action selection machinery that appears to be primarily sub-cortical, the cortex provides a much enriched capacity to ensure that the most appropriate actions are selected. Self-organising processes in action selection It was noted earlier that suitably configured neural circuits, within a given brain area or straddling multiple brain sub-systems, could have self-organizing properties that allow them to exhibit selection capabilities without explicitly choosing between alternative actions. For instance, distributed competitive processes in perceptual networks reduce the number of stimuli that are attended and will thus narrow the range of action plans that are active at a given time (see e.g. Desimone & Duncan, 1995, Duncan, Humphreys, & Ward, 1997). Similarly, attractor dynamics within specific cortical areas, and forward and back projections between different cortical domains, will further contribute to the preference for some courses of action over others (see e.g. Deco & Rolls, 2005). Finally, computational models of motor and pre-motor circuitry have shown that neural representations of movement trajectories can evolve without necessarily separating action specification (e.g. computation of the parameters of movement) from action selection (see e.g. Lukashin et al. 1996; Erlhagen & Schoner, 2002; Cisek, 2007). Conclusion Healthy animals generate integrated behavior, composed, from the observer’s view, of sequences of contextually appropriate and well-timed actions. A key task for the brain is to generate the next action, however, the extent to which the neural processes that give rise to each observed behavioral transition do so by explicitly resolving selection competitions is only partially resolved. This article has identified strong evidence for vertebrate neural circuitry that performs such a specific action selection role, 9 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection however, these modular selection mechanisms are embedded in a distributed action generating system with powerful self-organizing properties. Identifying the balance between specialised and more distributed selection processes will be a central task in further understanding how we choose “what to do next”. External links Prescott, T. J., Bryson, J. J., and Seth, A. K. (editors). Theme Issue on Models of Natural Action Selection (http://rstb.royalsocietypublishing.org/content/362/1485.toc) . Philosophical Transactions of the Royal Society. B. 2007, 362(1485). Bryson, J. J. (editor). Special Issue on Mechanisms of Action Selection (http://adb.sagepub.com/content /vol15/issue1/) . Adaptive Behavior, 15(1), 2007. Homepage of the 2005 International Workshop on Modelling Natural Action Selection (MNAS) (http://www.cs.bath.ac.uk/ai/MNAS-2005/) . Author Homepage (http://www.abrg.group.shef.ac.uk/people/tony/) References Botvinick, M. M. (2007). Multilevel structure in behavior and the brain: a model of Fuster's hierarchy (http://dx.doi.org/10.1098/rstb.2007.2056) . Phil. Trans. R. Soc. B., 362(1485), 1615-1626. Bryson, J. J. (2000). Cross-paradigm analysis of autonomous agent architecture (http://dx.doi.org /10.1080/095281300409829) . Journal of Experimental and Theoretical Artificial Intelligence, 12(2), 165-189. Butler, A. B. & Hodos, W. (2005). Comparative Vertebrate (http://eu.wiley.com/WileyCDA/WileyTitle /productCd-0471210056.html) Neuroanatomy. New York: John Wiley & Sons. Cisek, P. (2007). Cortical mechanisms of action selection: The affordance competition hypothesis (http://dx.doi.org/10.1098/rstb.2007.2054) . Phil. Trans. R. Soc. B., 362(1485), 1585-1600. Davis, W. J. (1979). Behavioral Hierarchies (http://dx.doi.org/10.1016/0166-2236(79)90003-1) . Trends In Neurosciences, 2(1), 5-7. Deco, G., & Rolls, E. T. (2005). Attention, short-term (http://dx.doi.org/10.1016 /j.pneurobio.2005.08.004) memory, and action selection: a unifying theory. Progress in Neurobiology, 76(4), 236-256. Desimone, R. & Duncan, J. (1995). Neural mechanisms of selective visual attention (http://dx.doi.org /10.1146/annurev.ne.18.030195.001205) . Annual Review of Neuroscience, 18, 193-222. Duncan, J., Humphreys, G., & Ward, R. (1997). Competitive brain activity in visual attention (http://dx.doi.org/10.1016/S0959-4388(97)80014-1) . Current Opinion in Neurobiology, 7(2), 255-261. Eaton, R. C., Hofve, J. C., & Fetcho, J. R. (1995). Beating the competition - the reliability hypothesis for Mauthner axon size. Brain Behavior and Evolution, 45(4), 183-194. Eaton, R. C., Lee, R. K. K., & Foreman, M. B. (2001). The Mauthner cell and other (http://dx.doi.org /10.1016/S0301-0082(00)00047-2) identified neurons of the brainstem escape network of fish. Progress in Neurobiology, 63, 467-485. Erlhagen, W. & Schoner, G. (2002). Dynamic field theory of movement preparation (http://dx.doi.org /10.1037/0033-295X.109.3.545) . Psychological Review, 109, 545-573. Gurney, K., Prescott, T. J., & Redgrave, P. (2001). A computational model of action selection in the basal 10 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection ganglia (http://dx.doi.org/10.1007/PL00007984) . Biological Cybernetics, 84(6), 401-423. Hazy, T. E., Frank, M. J., & O'Reilly, R. C. (2007). Toward an executive without a homunculus: Computational models of the prefrontal cortex/basal ganglia system (http://dx.doi.org/10.1098 /rstb.2007.2055) . Phil. Trans. R. Soc. B., 362(1485), 1601-1614. Houston, A. I., McNamara, J. M. & Steer, M. D. Do we expect natural selection to produce rational behaviour? (http://dx.doi.org/10.1098/rstb.2007.2057) . Phil. Trans. R. Soc. B., 362(1485), 1531-1544. Humphries, M. H., Gurney, K., & Prescott, T. J. (2007). Is there a brainstem substrate for action selection? (http://dx.doi.org/10.1098/rstb.2007.2057) Phil. Trans. R. Soc. B., 362(1485), 1627-1640. Kelso, J. A. S. (1995). Dynamic Patterns: The Self-organization of Brain and Behaviour. Cambridge, MA: MIT Press. Krawczyk, D. C. (2002). Contributions of the prefrontal cortex to the neural basis of human decision making (http://dx.doi.org/10.1016/S0149-7634(02)00021-0) . Neuroscience and Biobehavioral Reviews, 26(6), 631-664. Kristan, W. B., & Shaw, B. K. (1997). Population coding and behavioral choice (http://dx.doi.org/10.1016 /S0959-4388(97)80142-0) . Current Opinion In Neurobiology, 7(6), 826-831. Kupfermann, I., & Weiss, K. R. (2001). Motor program selections in simple model systems (http://dx.doi.org/10.1016/S0959-4388(01)00267-7) . Current Opinion In Neurobiology, 11(6), 673-677. Leise, E. M. (1990). Modular construction of nervous systems: a basic principle of design for invertebrates and vertebrates (http://dx.doi.org/10.1016/0165-0173(90)90009-D) . Brain Research Reviews, 15, 1-23. Lingenhohl, K. & Friauf, E. (1994). Giant neurons in the rat reticular formation: A sensorimotor interface in the elementary acoustic startle circuit? (http://www.jneurosci.org/cgi/content/abstract/14/3/1176) Journal of Neuroscience, 14. 1176-1194. Lukashin, A. V., Amirikian, B. R., Mozhaev, V. L., Wilcox, G. L., & Georgopoulos, A. P. (1996). Modeling motor cortical operations by an (http://dx.doi.org/10.1007/s004220050237) attractor network of stochastic neurons. Biological Cybernetics, 74(3), 255-261. Maes, P. (1995). Modelling adaptive autonomous agents. In C. G. Langton (Ed.), Artificial Life: An Overview. Cambridge, MA: MIT Press. McFarland, D. (1989). Problems of Animal Behaviour. Harlow, UK: Longman. Mink, J. (1996). The basal ganglia: focused selection and inhibition of competing motor programs (http://dx.doi.org/doi:10.1016/S0301-0082(96)00042-1) . Progress in Neurobiology, 50(4), 381-425. Mpitsos GJ, Cohan CS. (1986) Convergence in a distributed nervous system: parallel processing and self-organization. Journal of Neurobiology, 17(5):517-45. Prescott, T. J. (2007). Forced moves or good tricks in design space? Landmarks in the evolution of neural mechanisms for action selection (http://dx.doi.org/10.1177/1059712306076252) . Adaptive Behavior, 15(1), 9-31. Prescott, T. J., Redgrave, P., & Gurney, K. N. (1999). Layered control architectures in robots and vertebrates (http://dx.doi.org/10.1177/105971239900700105) . Adaptive Behavior, 7(1), 99-127. Redgrave, P., Prescott, T., & Gurney, K. N. (1999). The basal ganglia: A vertebrate solution to the selection problem? Neuroscience, 89, 1009-1023. Ridderinkhof, K. R., van den Wildenberg, W. P., Segalowitz, S. J., & Carter, C. S. (2004). Neurocognitive mechanisms of cognitive control: the role of prefrontal cortex in action selection, response inhibition, performance monitoring, and reward-based learning (http://dx.doi.org/10.1016/j.bandc.2004.09.016) . Brain and Cognition, 56(2), 129-140. Ringo, J. L. (1991). Neuronal interconnection as a function of brain size. Brain, behaviour, and evolution, 11 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection 38, 1-6. Rushworth, M. F., Buckley, M. J., Gough, P. M., Alexander, I. H., Kyriazis, D., McDonald, K. R., et al. (2005). Attentional selection and action selection in the ventral and orbital prefrontal cortex (http://dx.doi.org/10.1523/JNEUROSCI.2765-05.2005) . Journal of Neuroscience, 25(50), 11628-11636. Saper, C. B., Scammell, T. E., & Lu, J. (2005). Hypothalamic regulation of sleep and circadian rhythms (http://dx.doi.org/10.1038/nature04284) . Nature, 437(7063), 1257-1263. Schall, J. D., Stuphorn, V., & Brown, J. W. (2002). Monitoring and control of action by the frontal lobes (http://dx.doi.org/10.1016/S0896-6273(02)00964-9) . Neuron, 36(2), 309-322. Scheibel, M. E., & Scheibel, A. B. (1967). Anatomical basis of attention mechanisms in vertebrate brains. In G. C. Quarton, M. T. & F. O. Schmitt (Eds.), The Neurosciences: A Study Program (pp. 577-602). New York: Rockefeller University Press. Seth, A. (2007). The ecology of action selection: insights from artificial life (http://dx.doi.org/10.1098 /rstb.2007.2052) . Phil. Trans. R. Soc. B., 362(1485), 1545-1558. Wagner, G. P., & Altenberg, L. (1996). Perspective - complex adaptations and the evolution of evolvability. Evolution, 50(3), 967-976. Windhorst, U. (1996). On the role of recurrent inhibitory feedback in motor control (http://dx.doi.org /10.1016/0301-0082(96)00023-8) . Progress In Neurobiology, 49(6), 517-587. Internal references John W. Milnor (2006) Attractor. Scholarpedia, 1(11):1815. doi:10.4249/scholarpedia.1815 (http://dx.doi.org/10.4249/scholarpedia.1815) . Chris Eliasmith (2007) Attractor network. Scholarpedia, 2(10):1380. doi:10.4249/scholarpedia.1380 (http://dx.doi.org/10.4249/scholarpedia.1380) . Peter Redgrave (2007) Basal ganglia. Scholarpedia, 2(6):1825. doi:10.4249/scholarpedia.1825 (http://dx.doi.org/10.4249/scholarpedia.1825) . Valentino Braitenberg (2007) Brain. Scholarpedia, 2(11):2918. doi:10.4249/scholarpedia.2918 (http://dx.doi.org/10.4249/scholarpedia.2918) . James Meiss (2007) Dynamical systems. Scholarpedia, 2(2):1629. doi:10.4249/scholarpedia.1629 (http://dx.doi.org/10.4249/scholarpedia.1629) . Jim Houk (2007) Models of basal ganglia. Scholarpedia, 2(10):1633. doi:10.4249/scholarpedia.1633 (http://dx.doi.org/10.4249/scholarpedia.1633) . Michael A Arbib (2007) Modular models of brain function. Scholarpedia, 2(3):1869. doi:10.4249/scholarpedia.1869 (http://dx.doi.org/10.4249/scholarpedia.1869) . Peter Jonas and Gyorgy Buzsaki (2007) Neural inhibition. Scholarpedia, 2(9):3286. doi:10.4249/scholarpedia.3286 (http://dx.doi.org/10.4249/scholarpedia.3286) . Florentin Woergoetter and Bernd Porr (2008) Reinforcement learning. Scholarpedia, 3(3):1448. doi:10.4249/scholarpedia.1448 (http://dx.doi.org/10.4249/scholarpedia.1448) . Wolfram Schultz (2007) Reward. Scholarpedia, 2(3):1652. doi:10.4249/scholarpedia.1652 (http://dx.doi.org/10.4249/scholarpedia.1652) . Hermann Haken (2008) Self-organization of brain function. Scholarpedia, 3(4):2555. doi:10.4249/scholarpedia.2555 (http://dx.doi.org/10.4249/scholarpedia.2555) . Robert E. Burke (2008) Spinal cord. Scholarpedia, 3(4):1925. doi:10.4249/scholarpedia.1925 12 of 13 05/01/2017, 20:37 Action selection - Scholarpedia http://www.scholarpedia.org/article/Action_selection (http://dx.doi.org/10.4249/scholarpedia.1925) . Philip Holmes and Eric T. Shea-Brown (2006) Stability. Scholarpedia, 1(10):1838. doi:10.4249/scholarpedia.1838 (http://dx.doi.org/10.4249/scholarpedia.1838) . S. Murray Sherman (2006) Thalamus. Scholarpedia, 1(9):1583. doi:10.4249/scholarpedia.1583 (http://dx.doi.org/10.4249/scholarpedia.1583) . See also Attention; Models of Attention; Recurrent Neural Networks; Neural Inhibition; Attractor Network; Modular Models of Brain Function; Self-organization of Brain Function; Coordination Dynamics; Basal Ganglia; Models of Basal Ganglia; Neocortex; Reinforcement Learning. Sponsored by: Eugene M. Izhikevich, Editor-in-Chief of Scholarpedia, the peer-reviewed open-access encyclopedia Reviewed by (http://www.scholarpedia.org/w/index.php?title=Action_selection&oldid=29807) : Anonymous Reviewed by (http://www.scholarpedia.org/w/index.php?title=Action_selection&oldid=29807) : Dr. Paul Cisek, University of Montreal, CANADA Accepted on: 2008-01-09 01:54:55 GMT (http://www.scholarpedia.org /w/index.php?title=Action_selection&oldid=29807) Categories: Computational Neuroscience Navigation and Control Cognitive Science "Action selection" by Tony J. Prescott is licensed under a Creative Commons AttributionNonCommercialShareAlike 3.0 Unported License. Permissions beyond the scope of this license are described in the Terms of Use 13 of 13 05/01/2017, 20:37