The aim of this study was to describe the prevalence of same-sex experience, from deep kissing, t... more The aim of this study was to describe the prevalence of same-sex experience, from deep kissing, through petting to genital contact, same-sex romantic attraction and reported same-sex arousal in a representative sample of young men and women aged 17-18 in Norway. While same-sex experiences were reported as being more prevalent among young women (27.4%) than among young men (6.5%), the gender difference in the case of same-sex genital experiences was smaller (F = 4.6%, M = 2.7%) than in the case of deep kissing (F = 25.7%, M = 3.9%). While same-sex romantic attraction was more prevalent among young men with same-sex experiences than young women, such attraction was confined to a minority of all young men. Results suggest that same-sex experiences among Norwegian youth are not unequivocally tied to same-sex attraction and possible homosexual self-identity. We argue that among young men, same-sex behaviour is more often associated with same-sex orientation whereas young women's same-sex relations seem to be more exploratory.
Clinical text de-identification can potentially overlap with clinical information such as medical... more Clinical text de-identification can potentially overlap with clinical information such as medical problems or treatments, therefore causing this information to be lost. In this study, we focused on the analysis of the overlap between the 2010 i2b2 NLP challenge concept annotations, with the PHI annotations of our best-of-breed clinical text de-identification application. Overall, 0.81% of the annotations overlapped exactly, and 1.78% partly overlapped.
Manually annotating clinical document corpora to generate reference standards for Natural Languag... more Manually annotating clinical document corpora to generate reference standards for Natural Language Processing (NLP) systems or Machine Learning (ML) is a time-consuming and labor-intensive endeavor. Although a variety of open source annotation tools currently exist, there is a clear opportunity to develop new tools and assess functionalities that introduce efficiencies into the process of generating reference standards. These features include: management of document corpora and batch assignment, integration of machine-assisted verification functions, semi-automated curation of annotated information, and support of machine-assisted pre-annotation. The goals of reducing annotator workload and improving the quality of reference standards are important considerations for development of new tools. An infrastructure is also needed that will support large-scale but secure annotation of sensitive clinical data as well as crowdsourcing which has proven successful for a variety of annotation ...
Objective To identify epidemiologically important factors such as infectious disease exposure his... more Objective To identify epidemiologically important factors such as infectious disease exposure history, travel or specific variables from unstructured data using natural language processing (NLP) methods.
In this paper, we present an evaluation of the hybrid best-of-breed automated VHA (Veteran's ... more In this paper, we present an evaluation of the hybrid best-of-breed automated VHA (Veteran's Health Administration) clinical text de-identification system, nicknamed BoB, developed within the VHA Consortium for Healthcare Informatics Research. We also evaluate two available machine learning-based text de-identifications systems: MIST and HIDE. Two different clinical corpora were used for this evaluation: a manually annotated VHA corpus, and the 2006 i2b2 de-identification challenge corpus. These experiments focus on the generalizability and portability of the classification models across different document sources. BoB demonstrated good recall (92.6%), satisfactorily prioritizing patient privacy, and also achieved competitive precision (83.6%) for preserving subsequent document interpretability. MIST and HIDE reached very competitive results, in most cases with high precision (92.6% and 93.6%), although recall was sometimes lower than desired for the most sensitive PHI categories.
Mining the free text of electronic medical records (EMR) using natural language processing (NLP) ... more Mining the free text of electronic medical records (EMR) using natural language processing (NLP) is an effective method of extracting information not always captured in administrative data. We sought to determine if concepts related to homelessness, a non-medical condition, were amenable to extraction from the EMR of Veterans Affairs (VA) medical records. As there were no off-the-shelf products, a lexicon of terms related to homelessness was created. A corpus of free text documents from outpatient encounters was reviewed to create the reference standard for NLP training and testing. V3NLP Framework was used to detect instances of lexical terms and was compared to the reference standard. With a positive predictive value of 77% for extracting relevant concepts, this study demonstrates the feasibility of extracting positively asserted concepts related to homelessness from the free text of medical records.
We tested automated detection of influenza-like illness (ILI) from free-text clinician notes extr... more We tested automated detection of influenza-like illness (ILI) from free-text clinician notes extracted from the VA electronic medical record using a simple negation algorithm coupled with string matching for relevant ILI symptoms mapped to UMLS concepts. Additionally, we documented negation problems encountered and adaptations made to the negation algorithm when applied to the full texts of a diverse array of VA note types and templates.
AMIA Joint Summits on Translational Science proceedings AMIA Summit on Translational Science, 2010
Text classifiers have been used for biosurveillance tasks to identify patients with diseases or c... more Text classifiers have been used for biosurveillance tasks to identify patients with diseases or conditions of interest. When compared to a clinical reference standard of 280 cases of Acute Respiratory Infection (ARI), a text classifier consisting of simple rules and NegEx plus string matching for specific concepts of interest produced 569 (4%) false positive (FP) cases. Using instance level manual annotation we estimate the prevalence of contextual attributes and error types leading to FP cases. Errors were due to (1) Deletion errors from abbreviations, spelling mistakes and missing synonyms (57%); (2) Insertion errors from templated document structures such as check boxes, and lists of signs and symptoms (36%) and; (3) Substitution errors from irrelevant concepts and alternate meanings for the same word (6%). We demonstrate that specific concept attributes contribute to false positive cases. These results will inform modifications and adaptations to improve text classifier performa...
Using a publicly available corpus of clinical texts called MTSamples originally created to train ... more Using a publicly available corpus of clinical texts called MTSamples originally created to train medical coders and transcriptionists we applied de-identification annotation guidelines and schema developed by the Consortium for Healthcare Informatics Research (CHIR). We assess the feasibility of a Best of Breed (BOB) de- identification tool to reduce workload associated with building a de-identification annotation layer on this corpus. We also report prevalence estimates, inter-annotator agreement metrics and F1-measures for annotators and the BOB tool.
The Department of Veterans Affairs (VA) and the Informatics for Integrating Biology and the Bedsi... more The Department of Veterans Affairs (VA) and the Informatics for Integrating Biology and the Bedside (i2b2) team partnered to generate the reference standard for the 2010 i2b2/VA challenge task on concept extraction, assertion classification, and relation classification. The purpose of this paper is to report an in-depth qualitative analysis of the experience and perceptions of human annotators for these tasks. Transcripts of semi-structured interviews were analyzed using qualitative methods to identify key constructs and themes related to these annotation tasks. Interventions were embedded with these tasks using pre-annotation of clinical concepts and a modified annotation workflow. From the human perspective, annotation tasks involve an inherent conflict between bias, accuracy, and efficiency. This analysis deepens understanding of the biases, complexities and impact of variations in the annotation process that may affect annotation task reliability and reference standard validity ...
Journal of the American Medical Informatics Association : JAMIA, 2015
The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art ... more The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of rules and machine learners. We used a subset of the Shared Annotated Resources (ShARe) corpus of annotated clinical text-199 clinical notes for training and 99 for testing (roughly 180 K words in total). We provided the community with the annotated gold standard training documents to build systems to identify and normalize disorder mentions. The systems were tested on a held-out gold standard test set to measure their performance. For Task 1a, the best-performing system achieved an F1 score of 0.75 (0.80...
Information retrieval algorithms based on natural language processing (NLP) of the free text of m... more Information retrieval algorithms based on natural language processing (NLP) of the free text of medical records have been used to find documents of interest from databases. Homelessness is a high priority non-medical diagnosis that is noted in electronic medical records of Veterans in Veterans Affairs (VA) facilities. Using a human-reviewed reference standard corpus of clinical documents of Veterans with evidence of homelessness and those without, an open-source NLP tool (Automated Retrieval Console v2.0, ARC) was trained to classify documents. The best performing model based on document level work-flow performed well on a test set (Precision 94%, Recall 97%, F-Measure 96). Processing of a naïve set of 10,000 randomly selected documents from the VA using this best performing model yielded 463 documents flagged as positive, indicating a 4.7% prevalence of homelessness. Human review noted a precision of 70% for these flags resulting in an adjusted prevalence of homelessness of 3.3% wh...
BackgroundThe electronic medical record (EMR) contains a rich source of information that could be... more BackgroundThe electronic medical record (EMR) contains a rich source of information that could be harnessed for epidemic surveillance. We asked if structured EMR data could be coupled with computerized processing of free-text clinical entries to enhance detection of acute respiratory infections (ARI).MethodologyA manual review of EMR records related to 15,377 outpatient visits uncovered 280 reference cases of ARI. We used
Syndromic surveillance systems that incorporate electronic free-text data have primarily focused ... more Syndromic surveillance systems that incorporate electronic free-text data have primarily focused on extracting concepts of interest from chief complaint text, emergency department visit notes, and nurse triage notes. Due to availability and access, there has been limited work in the area of surveilling the full text of all electronic note documents compared with more specific document sources. This study provides an evaluation of the performance of a text classifier for detection of influenza-like illness (ILI) by document sources that are commonly used for biosurveillance by comparing them to routine visit notes, and a full electronic note corpus approach. Evaluating the performance of an automated text classifier for syndromic surveillance by source document will inform decisions regarding electronic textual data sources for potential use by automated biosurveillance systems. Even when a full electronic medical record is available, commonly available surveillance source documents ...
This paper reports on a shared task involving the assignment of emotions to suicide notes. Two fe... more This paper reports on a shared task involving the assignment of emotions to suicide notes. Two features distinguished this task from previous shared tasks in the biomedical domain. One is that it resulted in the corpus of fully anonymized clinical text and annotated suicide notes. This resource is permanently available and will (we hope) facilitate future research. The other key feature of the task is that it required categorization with respect to a large set of labels. The number of participants was larger than in any previous biomedical challenge task. We describe the data production process and the evaluation measures, and give a preliminary analysis of the results. Many systems performed at levels approaching the inter-coder agreement, suggesting that human-like performance on this task is within the reach of currently available technologies.
ABSTRACT Objective To highlight the importance of templates in extracting surveillance data from ... more ABSTRACT Objective To highlight the importance of templates in extracting surveillance data from the free text of electronic medical records using natural language processing (NLP) techniques. Introduction The main stay of recording patient data is the free text of electronic medical records (EMR). While stating the chief complaint and history of presenting illness in the patients ‘own words’, the rest of the electronic note is written by the provider in their words. Providers often use boiler-plate templates from EMR pull-downs to document information on the patient in the form of checklists, check boxes, yes/no and free text responses to questions. When these templates are used for recording symptoms, demographic information or medical, social or travel history, they represent an important source of surveillance data [1]. There is a dearth of literature on the use of natural language processing in extracting data from templates in the EMR. Methods A corpus of 1000 free text medical notes from the VA integrated electronic medical record (CPRS) was reviewed to identify commonly used templates. Of these, 500 were enriched for the surveillance domain of interest for this project (homelessness). The other 500 were randomly sampled from a large corpus of electronic notes. An NLP algorithm was developed to extract concepts related to our target surveillance domain. A manual review of the notes was performed by three human reviewers to generate a document-level reference standard that classified this set of documents as either demonstrating evidence of homelessness (H) or not (NH). A rule-based NLP algorithm was developed that used a combination of key word searches and negation based on an extensive lexicon of terms developed for this purpose. A random sample of 50 documents each of H and NH documents were reviewed after each iteration of the NLP algorithm to determine the false positive rate of the extracted concepts. Results The corpus consisted of 48% H and 52% NH documents as determined by human review. The NLP algorithm successfully extracted concepts from these documents. The H set had an average of 8 concepts related to homelessness per document (median 8, range 1 to 34). The NH set had an average 2 concepts (median 1, range 1 to 13)”. Thirteen template patterns were identified in this set of documents. The three most common were check boxes with square brackets, Yes/No and free text answer after a question. Several positively and negatively asserted concepts were noted to be in the responses to templated questions such as “Are you currently homeless: Yes or No”; “How many times have you been homeless in the past 3 years: (free text response)”; “Have you ever been in jail? [Y] or [N]”; Are you in need of substance abuse services? Yes or No”. Human review of a random sample of documents at the concept level indicated that the NLP algorithm generated 28% false positives in extracting concepts related to homelessness when templates were ignored among the H documents. When the algorithm was refined to include templates, the false positive rate declined to 22%. For the NH documents, the corresponding false positive rates were 56% and 21%. Conclusions To our knowledge, this is one of the first attempts to address the problem of information extraction from templates or templated sections of the EMR. A key challenge of templates is that they will most likely lead to poor performance of NLP algorithms and cause bottlenecks in processing if they are not considered. Acknowledging the presence of templates and refining NLP algorithms to handle them improves information extraction from free text medical notes, thus creating an opportunity for improved surveillance using the EMR. Algorithms will likely need to be customized to the electronic medical record and the surveillance domain of interest. A more detailed analysis of the templated sections is underway.
2011 44th Hawaii International Conference on System Sciences, 2011
... Architecture Scott L. DuVall VA Salt Lake City Health Care System and University of Utah scot... more ... Architecture Scott L. DuVall VA Salt Lake City Health Care System and University of Utah [email protected] Brett R. South VA Salt Lake City Health Care System and University of Utah [email protected] Leonard ...
The aim of this study was to describe the prevalence of same-sex experience, from deep kissing, t... more The aim of this study was to describe the prevalence of same-sex experience, from deep kissing, through petting to genital contact, same-sex romantic attraction and reported same-sex arousal in a representative sample of young men and women aged 17-18 in Norway. While same-sex experiences were reported as being more prevalent among young women (27.4%) than among young men (6.5%), the gender difference in the case of same-sex genital experiences was smaller (F = 4.6%, M = 2.7%) than in the case of deep kissing (F = 25.7%, M = 3.9%). While same-sex romantic attraction was more prevalent among young men with same-sex experiences than young women, such attraction was confined to a minority of all young men. Results suggest that same-sex experiences among Norwegian youth are not unequivocally tied to same-sex attraction and possible homosexual self-identity. We argue that among young men, same-sex behaviour is more often associated with same-sex orientation whereas young women's same-sex relations seem to be more exploratory.
Clinical text de-identification can potentially overlap with clinical information such as medical... more Clinical text de-identification can potentially overlap with clinical information such as medical problems or treatments, therefore causing this information to be lost. In this study, we focused on the analysis of the overlap between the 2010 i2b2 NLP challenge concept annotations, with the PHI annotations of our best-of-breed clinical text de-identification application. Overall, 0.81% of the annotations overlapped exactly, and 1.78% partly overlapped.
Manually annotating clinical document corpora to generate reference standards for Natural Languag... more Manually annotating clinical document corpora to generate reference standards for Natural Language Processing (NLP) systems or Machine Learning (ML) is a time-consuming and labor-intensive endeavor. Although a variety of open source annotation tools currently exist, there is a clear opportunity to develop new tools and assess functionalities that introduce efficiencies into the process of generating reference standards. These features include: management of document corpora and batch assignment, integration of machine-assisted verification functions, semi-automated curation of annotated information, and support of machine-assisted pre-annotation. The goals of reducing annotator workload and improving the quality of reference standards are important considerations for development of new tools. An infrastructure is also needed that will support large-scale but secure annotation of sensitive clinical data as well as crowdsourcing which has proven successful for a variety of annotation ...
Objective To identify epidemiologically important factors such as infectious disease exposure his... more Objective To identify epidemiologically important factors such as infectious disease exposure history, travel or specific variables from unstructured data using natural language processing (NLP) methods.
In this paper, we present an evaluation of the hybrid best-of-breed automated VHA (Veteran's ... more In this paper, we present an evaluation of the hybrid best-of-breed automated VHA (Veteran's Health Administration) clinical text de-identification system, nicknamed BoB, developed within the VHA Consortium for Healthcare Informatics Research. We also evaluate two available machine learning-based text de-identifications systems: MIST and HIDE. Two different clinical corpora were used for this evaluation: a manually annotated VHA corpus, and the 2006 i2b2 de-identification challenge corpus. These experiments focus on the generalizability and portability of the classification models across different document sources. BoB demonstrated good recall (92.6%), satisfactorily prioritizing patient privacy, and also achieved competitive precision (83.6%) for preserving subsequent document interpretability. MIST and HIDE reached very competitive results, in most cases with high precision (92.6% and 93.6%), although recall was sometimes lower than desired for the most sensitive PHI categories.
Mining the free text of electronic medical records (EMR) using natural language processing (NLP) ... more Mining the free text of electronic medical records (EMR) using natural language processing (NLP) is an effective method of extracting information not always captured in administrative data. We sought to determine if concepts related to homelessness, a non-medical condition, were amenable to extraction from the EMR of Veterans Affairs (VA) medical records. As there were no off-the-shelf products, a lexicon of terms related to homelessness was created. A corpus of free text documents from outpatient encounters was reviewed to create the reference standard for NLP training and testing. V3NLP Framework was used to detect instances of lexical terms and was compared to the reference standard. With a positive predictive value of 77% for extracting relevant concepts, this study demonstrates the feasibility of extracting positively asserted concepts related to homelessness from the free text of medical records.
We tested automated detection of influenza-like illness (ILI) from free-text clinician notes extr... more We tested automated detection of influenza-like illness (ILI) from free-text clinician notes extracted from the VA electronic medical record using a simple negation algorithm coupled with string matching for relevant ILI symptoms mapped to UMLS concepts. Additionally, we documented negation problems encountered and adaptations made to the negation algorithm when applied to the full texts of a diverse array of VA note types and templates.
AMIA Joint Summits on Translational Science proceedings AMIA Summit on Translational Science, 2010
Text classifiers have been used for biosurveillance tasks to identify patients with diseases or c... more Text classifiers have been used for biosurveillance tasks to identify patients with diseases or conditions of interest. When compared to a clinical reference standard of 280 cases of Acute Respiratory Infection (ARI), a text classifier consisting of simple rules and NegEx plus string matching for specific concepts of interest produced 569 (4%) false positive (FP) cases. Using instance level manual annotation we estimate the prevalence of contextual attributes and error types leading to FP cases. Errors were due to (1) Deletion errors from abbreviations, spelling mistakes and missing synonyms (57%); (2) Insertion errors from templated document structures such as check boxes, and lists of signs and symptoms (36%) and; (3) Substitution errors from irrelevant concepts and alternate meanings for the same word (6%). We demonstrate that specific concept attributes contribute to false positive cases. These results will inform modifications and adaptations to improve text classifier performa...
Using a publicly available corpus of clinical texts called MTSamples originally created to train ... more Using a publicly available corpus of clinical texts called MTSamples originally created to train medical coders and transcriptionists we applied de-identification annotation guidelines and schema developed by the Consortium for Healthcare Informatics Research (CHIR). We assess the feasibility of a Best of Breed (BOB) de- identification tool to reduce workload associated with building a de-identification annotation layer on this corpus. We also report prevalence estimates, inter-annotator agreement metrics and F1-measures for annotators and the BOB tool.
The Department of Veterans Affairs (VA) and the Informatics for Integrating Biology and the Bedsi... more The Department of Veterans Affairs (VA) and the Informatics for Integrating Biology and the Bedside (i2b2) team partnered to generate the reference standard for the 2010 i2b2/VA challenge task on concept extraction, assertion classification, and relation classification. The purpose of this paper is to report an in-depth qualitative analysis of the experience and perceptions of human annotators for these tasks. Transcripts of semi-structured interviews were analyzed using qualitative methods to identify key constructs and themes related to these annotation tasks. Interventions were embedded with these tasks using pre-annotation of clinical concepts and a modified annotation workflow. From the human perspective, annotation tasks involve an inherent conflict between bias, accuracy, and efficiency. This analysis deepens understanding of the biases, complexities and impact of variations in the annotation process that may affect annotation task reliability and reference standard validity ...
Journal of the American Medical Informatics Association : JAMIA, 2015
The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art ... more The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of rules and machine learners. We used a subset of the Shared Annotated Resources (ShARe) corpus of annotated clinical text-199 clinical notes for training and 99 for testing (roughly 180 K words in total). We provided the community with the annotated gold standard training documents to build systems to identify and normalize disorder mentions. The systems were tested on a held-out gold standard test set to measure their performance. For Task 1a, the best-performing system achieved an F1 score of 0.75 (0.80...
Information retrieval algorithms based on natural language processing (NLP) of the free text of m... more Information retrieval algorithms based on natural language processing (NLP) of the free text of medical records have been used to find documents of interest from databases. Homelessness is a high priority non-medical diagnosis that is noted in electronic medical records of Veterans in Veterans Affairs (VA) facilities. Using a human-reviewed reference standard corpus of clinical documents of Veterans with evidence of homelessness and those without, an open-source NLP tool (Automated Retrieval Console v2.0, ARC) was trained to classify documents. The best performing model based on document level work-flow performed well on a test set (Precision 94%, Recall 97%, F-Measure 96). Processing of a naïve set of 10,000 randomly selected documents from the VA using this best performing model yielded 463 documents flagged as positive, indicating a 4.7% prevalence of homelessness. Human review noted a precision of 70% for these flags resulting in an adjusted prevalence of homelessness of 3.3% wh...
BackgroundThe electronic medical record (EMR) contains a rich source of information that could be... more BackgroundThe electronic medical record (EMR) contains a rich source of information that could be harnessed for epidemic surveillance. We asked if structured EMR data could be coupled with computerized processing of free-text clinical entries to enhance detection of acute respiratory infections (ARI).MethodologyA manual review of EMR records related to 15,377 outpatient visits uncovered 280 reference cases of ARI. We used
Syndromic surveillance systems that incorporate electronic free-text data have primarily focused ... more Syndromic surveillance systems that incorporate electronic free-text data have primarily focused on extracting concepts of interest from chief complaint text, emergency department visit notes, and nurse triage notes. Due to availability and access, there has been limited work in the area of surveilling the full text of all electronic note documents compared with more specific document sources. This study provides an evaluation of the performance of a text classifier for detection of influenza-like illness (ILI) by document sources that are commonly used for biosurveillance by comparing them to routine visit notes, and a full electronic note corpus approach. Evaluating the performance of an automated text classifier for syndromic surveillance by source document will inform decisions regarding electronic textual data sources for potential use by automated biosurveillance systems. Even when a full electronic medical record is available, commonly available surveillance source documents ...
This paper reports on a shared task involving the assignment of emotions to suicide notes. Two fe... more This paper reports on a shared task involving the assignment of emotions to suicide notes. Two features distinguished this task from previous shared tasks in the biomedical domain. One is that it resulted in the corpus of fully anonymized clinical text and annotated suicide notes. This resource is permanently available and will (we hope) facilitate future research. The other key feature of the task is that it required categorization with respect to a large set of labels. The number of participants was larger than in any previous biomedical challenge task. We describe the data production process and the evaluation measures, and give a preliminary analysis of the results. Many systems performed at levels approaching the inter-coder agreement, suggesting that human-like performance on this task is within the reach of currently available technologies.
ABSTRACT Objective To highlight the importance of templates in extracting surveillance data from ... more ABSTRACT Objective To highlight the importance of templates in extracting surveillance data from the free text of electronic medical records using natural language processing (NLP) techniques. Introduction The main stay of recording patient data is the free text of electronic medical records (EMR). While stating the chief complaint and history of presenting illness in the patients ‘own words’, the rest of the electronic note is written by the provider in their words. Providers often use boiler-plate templates from EMR pull-downs to document information on the patient in the form of checklists, check boxes, yes/no and free text responses to questions. When these templates are used for recording symptoms, demographic information or medical, social or travel history, they represent an important source of surveillance data [1]. There is a dearth of literature on the use of natural language processing in extracting data from templates in the EMR. Methods A corpus of 1000 free text medical notes from the VA integrated electronic medical record (CPRS) was reviewed to identify commonly used templates. Of these, 500 were enriched for the surveillance domain of interest for this project (homelessness). The other 500 were randomly sampled from a large corpus of electronic notes. An NLP algorithm was developed to extract concepts related to our target surveillance domain. A manual review of the notes was performed by three human reviewers to generate a document-level reference standard that classified this set of documents as either demonstrating evidence of homelessness (H) or not (NH). A rule-based NLP algorithm was developed that used a combination of key word searches and negation based on an extensive lexicon of terms developed for this purpose. A random sample of 50 documents each of H and NH documents were reviewed after each iteration of the NLP algorithm to determine the false positive rate of the extracted concepts. Results The corpus consisted of 48% H and 52% NH documents as determined by human review. The NLP algorithm successfully extracted concepts from these documents. The H set had an average of 8 concepts related to homelessness per document (median 8, range 1 to 34). The NH set had an average 2 concepts (median 1, range 1 to 13)”. Thirteen template patterns were identified in this set of documents. The three most common were check boxes with square brackets, Yes/No and free text answer after a question. Several positively and negatively asserted concepts were noted to be in the responses to templated questions such as “Are you currently homeless: Yes or No”; “How many times have you been homeless in the past 3 years: (free text response)”; “Have you ever been in jail? [Y] or [N]”; Are you in need of substance abuse services? Yes or No”. Human review of a random sample of documents at the concept level indicated that the NLP algorithm generated 28% false positives in extracting concepts related to homelessness when templates were ignored among the H documents. When the algorithm was refined to include templates, the false positive rate declined to 22%. For the NH documents, the corresponding false positive rates were 56% and 21%. Conclusions To our knowledge, this is one of the first attempts to address the problem of information extraction from templates or templated sections of the EMR. A key challenge of templates is that they will most likely lead to poor performance of NLP algorithms and cause bottlenecks in processing if they are not considered. Acknowledging the presence of templates and refining NLP algorithms to handle them improves information extraction from free text medical notes, thus creating an opportunity for improved surveillance using the EMR. Algorithms will likely need to be customized to the electronic medical record and the surveillance domain of interest. A more detailed analysis of the templated sections is underway.
2011 44th Hawaii International Conference on System Sciences, 2011
... Architecture Scott L. DuVall VA Salt Lake City Health Care System and University of Utah scot... more ... Architecture Scott L. DuVall VA Salt Lake City Health Care System and University of Utah [email protected] Brett R. South VA Salt Lake City Health Care System and University of Utah [email protected] Leonard ...
Uploads
Papers by Brett South