Visual search is a ubiquitous and often challenging daily task, exemplified by looking for the ca... more Visual search is a ubiquitous and often challenging daily task, exemplified by looking for the car keys at home or a friend in a crowd. An intriguing property of some classical search tasks is an asymmetry such that finding a target A among distractors B can be easier than finding B among A. To elucidate the mechanisms responsible for asymmetry in visual search, we propose a computational model that takes a target and a search image as inputs and produces a sequence of eye movements until the target is found. The model integrates eccentricity-dependent visual recognition with target-dependent top-down cues. We compared the model against human behavior in six paradigmatic search tasks that show asymmetry in humans. Without prior exposure to the stimuli or task-specific training, the model provides a plausible mechanism for search asymmetry. We hypothesized that the polarity of search asymmetry arises from experience with the natural environment. We tested this hypothesis by training the model on augmented versions of ImageNet where the biases of natural images were either removed or reversed. The polarity of search asymmetry disappeared or was altered depending on the training protocol. This study highlights how classical perceptual properties can emerge in neural network models, without the need for task-specific training, but rather as a consequence of the statistical properties of the developmental diet fed to the model. All source code and data are publicly available at https: //github.com/kreimanlab/VisualSearchAsymmetry.
In visual search tasks, responses to targets on one trial can influence responses on the next tri... more In visual search tasks, responses to targets on one trial can influence responses on the next trial. Most typically, target repetition speeds response while switching to a different target slows response. Such "priming" effects have sometimes been given very significant roles in theories of search (e.g., Theeuwes, Philosophical Transactions of the Royal Society B: Biological Sciences, 368, 1628, 2013). Most work on priming has involved "singleton" or "popout" tasks. In non-popout priming tasks, observers must often perform a task-switching operation because the guiding template for one target (e.g., a red vertical target in a conjunction task) is incompatible with efficient search for the other target (green horizontal, in this example). We examined priming in inefficient search where the priming feature (Color: Experiments 1-3, Shape: Experiments 4-5) was irrelevant to the task of finding a T among Ls. We wished to determine if finding a red T on one trial helped observers to be more efficient if the next T was also red. In all experiments, we found additive priming effects. The reaction time (RT) for the second trial was shorter if the color of the T was repeated. However, there was no interaction with set size. The slope of the RT × Set Size function was not shallower for runs of the same target color, compared to trials where the target color switched. We propose that priming might produce transient guidance of the earliest deployments of attention on the next trial or it might speed decisions about a selected target. Priming does not appear to guide attention over the entire search.
Cognitive Research: Principles and Implications, 2020
When radiologists search for a specific target (e.g., lung cancer), they are also asked to report... more When radiologists search for a specific target (e.g., lung cancer), they are also asked to report any other clinically significant “incidental findings” (e.g., pneumonia). These incidental findings are missed at an undesirably high rate. In an effort to understand and reduce these errors, Wolfe et al. (Cognitive Research: Principles and Implications 2:35, 2017) developed “mixed hybrid search” as a model system for incidental findings. In this task, non-expert observers memorize six targets: half of these targets are specific images (analogous to the suspected diagnosis in the clinical task). The other half are broader, categorically defined targets, like “animals” or “cars” (analogous to the less well-specified incidental findings). In subsequent search through displays for any instances of any of the targets, observers miss about one third of the categorical targets, mimicking the incidental finding problem. In the present paper, we attempted to reduce the number of errors in the m...
This series of experiments investigates how changing the input from a simulated AI can affect the... more This series of experiments investigates how changing the input from a simulated AI can affect the decisions made by human observers in a two-alternative forced choice task (like the decision to recall a woman for further examination in mammography). In this experiment, we investigate how changing prevalence affects human performance when AI is used as a Second Reader.
How does scene structure to guide search process? How well can Os use scene structure information... more How does scene structure to guide search process? How well can Os use scene structure information to guide their search for target when they are given instruction with different degrees of specificity? In the present study, we design a set of stimuli named Vowel World to answer these questions.
This study examines search performance and search strategies in the reading of Digital Breast Tom... more This study examines search performance and search strategies in the reading of Digital Breast Tomosynthesis (DBT) images.
Data collection began: 11/2/2018 Can useful field of view (UFOV) explain human search performance... more Data collection began: 11/2/2018 Can useful field of view (UFOV) explain human search performance? In this experiment, we test whether the size of UFOV changes with the density of the display. For more detailed information about this experiment, see "Pre-Reg_UFOV_Density.pdf" in the Files section below.
Data were collected in September 2017. See "Files" below for an example of the block de... more Data were collected in September 2017. See "Files" below for an example of the block design.
Cognitive Research: Principles and Implications, 2022
Using an online, medical image labeling app, 803 individuals rated images of skin lesions as eith... more Using an online, medical image labeling app, 803 individuals rated images of skin lesions as either "melanoma" (skin cancer) or "nevus" (a skin mole). Each block consisted of 80 images. Blocks could have high (50%) or low (20%) target prevalence and could provide full, accurate feedback or no feedback. As in prior work, with feedback, decision criteria were more conservative at low prevalence than at high prevalence and resulted in more miss errors. Without feedback, this low prevalence effect was reversed (albeit, not significantly). Participants could participate in up to four different conditions a day on each of 6 days. Our main interest was in the effect of Block N on Block N + 1. Low prevalence with feedback made participants more conservative on a subsequent block. High prevalence with feedback made participants more liberal on a subsequent block. Conditions with no feedback had no significant impact on the subsequent block. The delay between Blocks 1 and ...
In this experiment, we investigate what kind of object visual properties can be used to divide la... more In this experiment, we investigate what kind of object visual properties can be used to divide large sets of objects into different subsets for the independent extraction of statistical features, such as mean size. The accuracy of averaging when subset selection is required will be compared with the baseline accuracies when the target subset is presented alone or when the observer averages the whole set.
How do observers search through familiar scenes? A novel panoramic search method is used to study... more How do observers search through familiar scenes? A novel panoramic search method is used to study the interaction of memory and vision in natural search behavior. In panoramic search, observers see part of an unchanging scene larger than their current field of view. A target object can be visible, present in the display but hidden from view, or absent. Visual search efficiency does not change after hundreds of trials through an unchanging scene (Experiment 1). Memory search, in contrast, begins inefficiently but becomes efficient with practice. Given a choice between vision and memory, observers choose vision (Experiments 2 and 3). However, if forced to use their memory on some trials, they learn to use memory on all trials, even when reliable visual information remains available (Experiment 4). The results suggest that observers make a pragmatic choice between vision and memory, with a strong bias toward visual search even for memorized stimuli. The human visual environment is rela...
In Hybrid Foraging tasks, observers search for multiple instances of several types of target. Col... more In Hybrid Foraging tasks, observers search for multiple instances of several types of target. Collecting all the dirty laundry and kitchenware out of a child’s room would be a real-world example. How are such foraging episodes structured? A series of four experiments shows that selection of one item from the display makes it more likely that the next item will be of the same type. This pattern holds if the targets are defined by basic features like color and shape but not if they are defined by their identity (e.g., the letters p & d). Additionally, switching between target types during search is expensive in time, with longer response times between successive selections if the target type changes than if they are the same. Finally, the decision to leave a screen/patch for the next screen in these foraging tasks is imperfectly consistent with the predictions of optimal foraging theory. The results of these hybrid foraging studies cast new light on the ways in which prior selection h...
Radiologists can discriminate between normal and abnormal breast tissue at a glance, suggesting t... more Radiologists can discriminate between normal and abnormal breast tissue at a glance, suggesting that radiologists might be using some “global signal” of abnormality. Our study investigated whether texture descriptions can be used to characterize the global signal of abnormality and whether radiologists use this information during interpretation. Synthetic images were generated using a texture synthesis algorithm trained on texture descriptions extracted from sections of mammograms. Radiologists completed a task that required rating the abnormality of briefly presented tissue sections. When the abnormal tissue had no visible lesion, radiologists seemed to use texture descriptions; performance was similar across real and synthesized tissue sections. However, when the abnormal tissue had a visible lesion, radiologists seemed to rely on additional mechanisms beyond the texture descriptions; performance increased for the real tissue sections. These findings suggest that radiologists can ...
Purpose: Radiologists sometimes fail to report clearly visible, clinically significant findings. ... more Purpose: Radiologists sometimes fail to report clearly visible, clinically significant findings. Eye tracking can provide insight into the causes of such errors. Approach: We tracked eye movements of 17 radiologists, searching for masses in 80 mammograms (60 with masses). Results: Errors were classified using the Kundel et al. (1978) taxonomy: search errors (target never fixated), recognition errors (fixated <500 ms), or decision errors (fixated >500 ms). Error proportions replicated Krupinski (1996): search 25%, recognition 25%, and decision 50%. Interestingly, we found few differences between experts and residents in accuracy or eye movement metrics. Error categorization depends on the definition of the useful field of view (UFOV) around fixation. We explored different UFOV definitions, based on targeting saccades and search saccades. Targeting saccades averaged slightly longer than search saccades. Of most interest, we found that the probability that the eyes would move to the target on the next saccade or even on one of the next three saccades was strikingly low (∼33%, even when the eyes were <2 deg from the target). This makes it clear that observers do not fully process everything within a UFOV. Using a probabilistic UFOV, we find, unsurprisingly, that observers cover more of the image when no target is present than when it is found. Interestingly, we do not find evidence that observers cover too little of the image on trials when they miss the target. Conclusions: These results indicate that many errors in mammography reflect failed deployment of attention; not failure to fixate clinically significant locations.
Visual search is a ubiquitous and often challenging daily task, exemplified by looking for the ca... more Visual search is a ubiquitous and often challenging daily task, exemplified by looking for the car keys at home or a friend in a crowd. An intriguing property of some classical search tasks is an asymmetry such that finding a target A among distractors B can be easier than finding B among A. To elucidate the mechanisms responsible for asymmetry in visual search, we propose a computational model that takes a target and a search image as inputs and produces a sequence of eye movements until the target is found. The model integrates eccentricity-dependent visual recognition with target-dependent top-down cues. We compared the model against human behavior in six paradigmatic search tasks that show asymmetry in humans. Without prior exposure to the stimuli or task-specific training, the model provides a plausible mechanism for search asymmetry. We hypothesized that the polarity of search asymmetry arises from experience with the natural environment. We tested this hypothesis by training the model on augmented versions of ImageNet where the biases of natural images were either removed or reversed. The polarity of search asymmetry disappeared or was altered depending on the training protocol. This study highlights how classical perceptual properties can emerge in neural network models, without the need for task-specific training, but rather as a consequence of the statistical properties of the developmental diet fed to the model. All source code and data are publicly available at https: //github.com/kreimanlab/VisualSearchAsymmetry.
In visual search tasks, responses to targets on one trial can influence responses on the next tri... more In visual search tasks, responses to targets on one trial can influence responses on the next trial. Most typically, target repetition speeds response while switching to a different target slows response. Such "priming" effects have sometimes been given very significant roles in theories of search (e.g., Theeuwes, Philosophical Transactions of the Royal Society B: Biological Sciences, 368, 1628, 2013). Most work on priming has involved "singleton" or "popout" tasks. In non-popout priming tasks, observers must often perform a task-switching operation because the guiding template for one target (e.g., a red vertical target in a conjunction task) is incompatible with efficient search for the other target (green horizontal, in this example). We examined priming in inefficient search where the priming feature (Color: Experiments 1-3, Shape: Experiments 4-5) was irrelevant to the task of finding a T among Ls. We wished to determine if finding a red T on one trial helped observers to be more efficient if the next T was also red. In all experiments, we found additive priming effects. The reaction time (RT) for the second trial was shorter if the color of the T was repeated. However, there was no interaction with set size. The slope of the RT × Set Size function was not shallower for runs of the same target color, compared to trials where the target color switched. We propose that priming might produce transient guidance of the earliest deployments of attention on the next trial or it might speed decisions about a selected target. Priming does not appear to guide attention over the entire search.
Cognitive Research: Principles and Implications, 2020
When radiologists search for a specific target (e.g., lung cancer), they are also asked to report... more When radiologists search for a specific target (e.g., lung cancer), they are also asked to report any other clinically significant “incidental findings” (e.g., pneumonia). These incidental findings are missed at an undesirably high rate. In an effort to understand and reduce these errors, Wolfe et al. (Cognitive Research: Principles and Implications 2:35, 2017) developed “mixed hybrid search” as a model system for incidental findings. In this task, non-expert observers memorize six targets: half of these targets are specific images (analogous to the suspected diagnosis in the clinical task). The other half are broader, categorically defined targets, like “animals” or “cars” (analogous to the less well-specified incidental findings). In subsequent search through displays for any instances of any of the targets, observers miss about one third of the categorical targets, mimicking the incidental finding problem. In the present paper, we attempted to reduce the number of errors in the m...
This series of experiments investigates how changing the input from a simulated AI can affect the... more This series of experiments investigates how changing the input from a simulated AI can affect the decisions made by human observers in a two-alternative forced choice task (like the decision to recall a woman for further examination in mammography). In this experiment, we investigate how changing prevalence affects human performance when AI is used as a Second Reader.
How does scene structure to guide search process? How well can Os use scene structure information... more How does scene structure to guide search process? How well can Os use scene structure information to guide their search for target when they are given instruction with different degrees of specificity? In the present study, we design a set of stimuli named Vowel World to answer these questions.
This study examines search performance and search strategies in the reading of Digital Breast Tom... more This study examines search performance and search strategies in the reading of Digital Breast Tomosynthesis (DBT) images.
Data collection began: 11/2/2018 Can useful field of view (UFOV) explain human search performance... more Data collection began: 11/2/2018 Can useful field of view (UFOV) explain human search performance? In this experiment, we test whether the size of UFOV changes with the density of the display. For more detailed information about this experiment, see "Pre-Reg_UFOV_Density.pdf" in the Files section below.
Data were collected in September 2017. See "Files" below for an example of the block de... more Data were collected in September 2017. See "Files" below for an example of the block design.
Cognitive Research: Principles and Implications, 2022
Using an online, medical image labeling app, 803 individuals rated images of skin lesions as eith... more Using an online, medical image labeling app, 803 individuals rated images of skin lesions as either "melanoma" (skin cancer) or "nevus" (a skin mole). Each block consisted of 80 images. Blocks could have high (50%) or low (20%) target prevalence and could provide full, accurate feedback or no feedback. As in prior work, with feedback, decision criteria were more conservative at low prevalence than at high prevalence and resulted in more miss errors. Without feedback, this low prevalence effect was reversed (albeit, not significantly). Participants could participate in up to four different conditions a day on each of 6 days. Our main interest was in the effect of Block N on Block N + 1. Low prevalence with feedback made participants more conservative on a subsequent block. High prevalence with feedback made participants more liberal on a subsequent block. Conditions with no feedback had no significant impact on the subsequent block. The delay between Blocks 1 and ...
In this experiment, we investigate what kind of object visual properties can be used to divide la... more In this experiment, we investigate what kind of object visual properties can be used to divide large sets of objects into different subsets for the independent extraction of statistical features, such as mean size. The accuracy of averaging when subset selection is required will be compared with the baseline accuracies when the target subset is presented alone or when the observer averages the whole set.
How do observers search through familiar scenes? A novel panoramic search method is used to study... more How do observers search through familiar scenes? A novel panoramic search method is used to study the interaction of memory and vision in natural search behavior. In panoramic search, observers see part of an unchanging scene larger than their current field of view. A target object can be visible, present in the display but hidden from view, or absent. Visual search efficiency does not change after hundreds of trials through an unchanging scene (Experiment 1). Memory search, in contrast, begins inefficiently but becomes efficient with practice. Given a choice between vision and memory, observers choose vision (Experiments 2 and 3). However, if forced to use their memory on some trials, they learn to use memory on all trials, even when reliable visual information remains available (Experiment 4). The results suggest that observers make a pragmatic choice between vision and memory, with a strong bias toward visual search even for memorized stimuli. The human visual environment is rela...
In Hybrid Foraging tasks, observers search for multiple instances of several types of target. Col... more In Hybrid Foraging tasks, observers search for multiple instances of several types of target. Collecting all the dirty laundry and kitchenware out of a child’s room would be a real-world example. How are such foraging episodes structured? A series of four experiments shows that selection of one item from the display makes it more likely that the next item will be of the same type. This pattern holds if the targets are defined by basic features like color and shape but not if they are defined by their identity (e.g., the letters p & d). Additionally, switching between target types during search is expensive in time, with longer response times between successive selections if the target type changes than if they are the same. Finally, the decision to leave a screen/patch for the next screen in these foraging tasks is imperfectly consistent with the predictions of optimal foraging theory. The results of these hybrid foraging studies cast new light on the ways in which prior selection h...
Radiologists can discriminate between normal and abnormal breast tissue at a glance, suggesting t... more Radiologists can discriminate between normal and abnormal breast tissue at a glance, suggesting that radiologists might be using some “global signal” of abnormality. Our study investigated whether texture descriptions can be used to characterize the global signal of abnormality and whether radiologists use this information during interpretation. Synthetic images were generated using a texture synthesis algorithm trained on texture descriptions extracted from sections of mammograms. Radiologists completed a task that required rating the abnormality of briefly presented tissue sections. When the abnormal tissue had no visible lesion, radiologists seemed to use texture descriptions; performance was similar across real and synthesized tissue sections. However, when the abnormal tissue had a visible lesion, radiologists seemed to rely on additional mechanisms beyond the texture descriptions; performance increased for the real tissue sections. These findings suggest that radiologists can ...
Purpose: Radiologists sometimes fail to report clearly visible, clinically significant findings. ... more Purpose: Radiologists sometimes fail to report clearly visible, clinically significant findings. Eye tracking can provide insight into the causes of such errors. Approach: We tracked eye movements of 17 radiologists, searching for masses in 80 mammograms (60 with masses). Results: Errors were classified using the Kundel et al. (1978) taxonomy: search errors (target never fixated), recognition errors (fixated <500 ms), or decision errors (fixated >500 ms). Error proportions replicated Krupinski (1996): search 25%, recognition 25%, and decision 50%. Interestingly, we found few differences between experts and residents in accuracy or eye movement metrics. Error categorization depends on the definition of the useful field of view (UFOV) around fixation. We explored different UFOV definitions, based on targeting saccades and search saccades. Targeting saccades averaged slightly longer than search saccades. Of most interest, we found that the probability that the eyes would move to the target on the next saccade or even on one of the next three saccades was strikingly low (∼33%, even when the eyes were <2 deg from the target). This makes it clear that observers do not fully process everything within a UFOV. Using a probabilistic UFOV, we find, unsurprisingly, that observers cover more of the image when no target is present than when it is found. Interestingly, we do not find evidence that observers cover too little of the image on trials when they miss the target. Conclusions: These results indicate that many errors in mammography reflect failed deployment of attention; not failure to fixate clinically significant locations.
Uploads
Papers by Jeremy Wolfe