Dean Sittig

EDUCATION

PhD University of Utah

RESEARCH, TEACHING, or OTHER INTERESTS

Health Informatics, General Medicine, Health (social science), Safety Research

Scopus Publications

Recommendations to Ensure Safety of AI in Real-World Clinical Care
Dean F. Sittig and Hardeep Singh
American Medical Association (AMA)
This Viewpoint provides recommendations for health care organizations (HCOs) and clinicians to facilitate the use of artificial intelligence (AI)–enabled systems, including electronic health records with AI features, in routine clinical care and provides pragmatic guidance for HCOs and clinicians at all stages of AI implementation.


Implementation of Electronic Triggers to Identify Diagnostic Errors in Emergency Departments
Viralkumar Vaghani, Ashish Gupta, Usman Mir, Li Wei, Daniel R. Murphy, Umair Mushtaq, Dean F. Sittig, Andrew J. Zimolzak, and Hardeep Singh
American Medical Association (AMA)
ImportanceMissed diagnosis can lead to preventable patient harm.ObjectiveTo develop and implement a portfolio of electronic triggers (e-triggers) and examine their performance for identifying missed opportunities in diagnosis (MODs) in emergency departments (EDs).Design, Setting, and ParticipantsIn this retrospective medical record review study of ED visits at 1321 Veterans Affairs health care sites, rules-based e-triggers were developed and implemented using a national electronic health record repository. These e-triggers targeted 6 high-risk presentations for MODs in treat-and-release ED visits. A high-risk stroke e-trigger was applied to treat-and-release ED visits from January 1, 2016, to December 31, 2020. A symptom-disease dyad e-trigger was applied to visits from January 1, 2018, to December 31, 2019. High-risk abdominal pain, unexpected ED return, unexpected hospital return, and test result e-triggers were applied to visits from January 1, 2019, to December 31, 2019. At least 100 randomly selected flagged records were reviewed by physician reviewers for each e-trigger. Data were analyzed between January 2024 and April 2024.ExposuresTreat-and-release ED visits involving high-risk stroke, symptom-disease dyads, high-risk abdominal pain, unexpected ED return, unexpected hospital return, and abnormal test results not followed up after initial ED visit.Main Outcomes and MeasuresTrained physician reviewers evaluated the presence/absence of MODs at ED visits and recorded data on patient and clinician characteristics, types of diagnostic process breakdowns, and potential harm from MODs.ResultsThe high-risk stroke e-trigger was applied to 8 792 672 treat-and-release ED visits (4 967 283 unique patients); the symptom-disease dyad e-trigger was applied to 3 692 454 visits (2 070 979 patients); and high-risk abdominal pain, unexpected ED return, unexpected hospital return, and test result e-triggers were applied to 1 845 905 visits (1 032 969 patients), overall identifying 203, 1981, 170, 116 785, 14 879, and 2090 trigger-positive records, respectively. Review of 625 randomly selected patient records (mean [SD] age, 62.5 [15.2] years; 553 [88.5%] male) showed the following MOD counts and positive predictive values (PPVs) within each category: 47 MODs (PPV, 47.0%) for stroke, 31 MODs (PPV, 25.8%) for abdominal pain, 11 MODs (PPV, 11.0%) for ED returns, 23 MODs (PPV, 23.0%) for hospital returns, 18 MODs (PPV, 18.0%) for symptom-disease dyads, and 55 MODs (PPV, 52.4%) for test results. Patients with MODs were slightly older than those without (mean [SD] age, 65.6 [14.5] vs 61.2 [15.3] years; P < .001). Reviewer agreement was favorable (range, 72%-100%). In 108 of 130 MODs (83.1%; excluding MODs related to the test result e-trigger), the most common diagnostic process breakdown involved the patient-clinician encounter. In 185 total MODs, 20 patients experienced severe harm (10.8%), and 54 patients experienced moderate harm (29.2%).Conclusions and RelevanceIn this retrospective medical record review study, rules-based e-triggers were useful for post hoc detection of MODs in ED visits. Interventions to target ED work system factors are urgently needed to support patient-clinician encounters and minimize harm from diagnostic errors.


Regulation of artificial intelligence in healthcare: Clinical Laboratory Improvement Amendments (CLIA) as a model
Brian R Jackson, Mark P Sendak, Anthony Solomonides, Suresh Balu, and Dean F Sittig
Oxford University Press (OUP)
Abstract Objectives To assess the potential to adapt an existing technology regulatory model, namely the Clinical Laboratory Improvement Amendments (CLIA), for clinical artificial intelligence (AI). Materials and Methods We identify overlap in the quality management requirements for laboratory testing and clinical AI. Results We propose modifications to the CLIA model that could make it suitable for oversight of clinical AI. Discussion In national discussions of clinical AI, there has been surprisingly little consideration of this longstanding model for local technology oversight. While CLIA was specifically designed for laboratory testing, most of its principles are applicable to other technologies in patient care. Conclusion A CLIA-like approach to regulating clinical AI would be complementary to the more centralized schemes currently under consideration, and it would ensure institutional and professional accountability for the longitudinal quality management of clinical AI.


Maximizing the Ability of Health IT and AI to Improve Patient Safety




Toward a responsible future: recommendations for AI-enabled clinical decision support
Steven Labkoff, Bilikis Oladimeji, Joseph Kannry, Anthony Solomonides, Russell Leftwich, Eileen Koski, Amanda L Joseph, Monica Lopez-Gonzalez, Lee A Fleisher, Kimberly Nolen,et al.
Oxford University Press (OUP)
Abstract Background Integrating artificial intelligence (AI) in healthcare settings has the potential to benefit clinical decision-making. Addressing challenges such as ensuring trustworthiness, mitigating bias, and maintaining safety is paramount. The lack of established methodologies for pre- and post-deployment evaluation of AI tools regarding crucial attributes such as transparency, performance monitoring, and adverse event reporting makes this situation challenging. Objectives This paper aims to make practical suggestions for creating methods, rules, and guidelines to ensure that the development, testing, supervision, and use of AI in clinical decision support (CDS) systems are done well and safely for patients. Materials and Methods In May 2023, the Division of Clinical Informatics at Beth Israel Deaconess Medical Center and the American Medical Informatics Association co-sponsored a working group on AI in healthcare. In August 2023, there were 4 webinars on AI topics and a 2-day workshop in September 2023 for consensus-building. The event included over 200 industry stakeholders, including clinicians, software developers, academics, ethicists, attorneys, government policy experts, scientists, and patients. The goal was to identify challenges associated with the trusted use of AI-enabled CDS in medical practice. Key issues were identified, and solutions were proposed through qualitative analysis and a 4-month iterative consensus process. Results Our work culminated in several key recommendations: (1) building safe and trustworthy systems; (2) developing validation, verification, and certification processes for AI-CDS systems; (3) providing a means of safety monitoring and reporting at the national level; and (4) ensuring that appropriate documentation and end-user training are provided. Discussion AI-enabled Clinical Decision Support (AI-CDS) systems promise to revolutionize healthcare decision-making, necessitating a comprehensive framework for their development, implementation, and regulation that emphasizes trustworthiness, transparency, and safety. This framework encompasses various aspects including model training, explainability, validation, certification, monitoring, and continuous evaluation, while also addressing challenges such as data privacy, fairness, and the need for regulatory oversight to ensure responsible integration of AI into clinical workflow. Conclusions Achieving responsible AI-CDS systems requires a collective effort from many healthcare stakeholders. This involves implementing robust safety, monitoring, and transparency measures while fostering innovation. Future steps include testing and piloting proposed trust mechanisms, such as safety reporting protocols, and establishing best practice guidelines.


Active Learning Pipeline to Identify Candidate Terms for a CDSS Ontology
Xia Jing, Rohan Goli, Keerthana Komatineni, Shailesh Alluri, Nina Hubig, Hua Min, Yang Gong, Dean F. Sittig, Paul Biondich, David Robinson,et al.
IOS Press
Ontology is essential for achieving health information and information technology application interoperability in the biomedical fields and beyond. Traditionally, ontology construction is carried out manually by human domain experts (HDE). Here, we explore an active learning approach to automatically identify candidate terms from publications, with manual verification later as a part of a deep learning model training and learning process. We introduce the overall architecture of the active learning pipeline and present some preliminary results. This work is a critical and complementary component in addition to manually building the ontology, especially during the long-term maintenance stage.


Patient-centered clinical decision support challenges and opportunities identified from workflow execution models
Dean F Sittig, Aziz Boxwala, Adam Wright, Courtney Zott, Nicole A Gauthreaux, James Swiger, Edwin A Lomotan, and Prashila Dullabh
Oxford University Press (OUP)
Abstract Objective To use workflow execution models to highlight new considerations for patient-centered clinical decision support policies (PC CDS), processes, procedures, technology, and expertise required to support new workflows. Methods To generate and refine models, we used (1) targeted literature reviews; (2) key informant interviews with 6 external PC CDS experts; (3) model refinement based on authors’ experience; and (4) validation of the models by a 26-member steering committee. Results and Discussion We identified 7 major issues that provide significant challenges and opportunities for healthcare systems, researchers, administrators, and health IT and app developers. Overcoming these challenges presents opportunities for new or modified policies, processes, procedures, technology, and expertise to: (1) Ensure patient-generated health data (PGHD), including patient-reported outcomes (PROs), are documented, reviewed, and managed by appropriately trained clinicians, between visits and after regular working hours. (2) Educate patients to use connected medical devices and handle technical issues. (3) Facilitate collection and incorporation of PGHD, PROs, patient preferences, and social determinants of health into existing electronic health records. (4) Troubleshoot erroneous data received from devices. (5) Develop dashboards to display longitudinal patient-reported data. (6) Provide reimbursement to support new models of care. (7) Support patient engagement with remote devices. Conclusion Several new policies, processes, technologies, and expertise are required to ensure safe and effective implementation and use of PC CDS. As we gain more experience implementing and working with PC CDS, we should be able to begin realizing the long-term positive impact on patient health that the patient-centered movement in healthcare promises.


Implementation of a health information technology safety classification system in the Veterans Health Administration’s Informatics Patient Safety Office
Danielle Kato, Joe Lucas, and Dean F Sittig
Oxford University Press (OUP)
Abstract Objective Implement the 5-type health information technology (HIT) patient safety concern classification system for HIT patient safety issues reported to the Veterans Health Administration’s Informatics Patient Safety Office. Materials and methods A team of informatics safety analysts retrospectively classified 1 year of HIT patient safety issues by type of HIT patient safety concern using consensus discussions. The processes established during retrospective classification were then applied to incoming HIT safety issues moving forward. Results Of 140 issues retrospectively reviewed, 124 met the classification criteria. The majority were HIT failures (eg, software defects) (33.1%) or configuration and implementation problems (29.8%). Unmet user needs and external system interactions accounted for 20.2% and 10.5%, respectively. Absence of HIT safety features accounted for 2.4% of issues, and 4% did not have enough information to classify. Conclusion The 5-type HIT safety concern classification framework generated actionable categories helping organizations effectively respond to HIT patient safety risks.


Leveraging explainable artificial intelligence to optimize clinical decision support
Siru Liu, Allison B McCoy, Josh F Peterson, Thomas A Lasko, Dean F Sittig, Scott D Nelson, Jennifer Andrews, Lorraine Patterson, Cheryl M Cobb, David Mulherin,et al.
Oxford University Press (OUP)
Abstract Objective To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches. Methods We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert’s historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful. Results The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues. Conclusion We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.


A guide to mitigating audit log-related risk in medical professional liability cases
Dean F. Sittig and Adam Wright
Wiley
AbstractFollowing the American Recovery and Reinvestment Act in 2009, use of electronic health records (EHRs) has become ubiquitous. Accordingly, one should expect most medical professional liability cases to involve review of patient records produced from EHRs. When questions arise regarding who was involved in care of a patient, what they knew and when, or the meaning, completeness, integrity, validity, timeliness, confidentiality, accuracy, or legitimacy of data, or ways that the EHR's user interface or automated clinical decision support tools may have contributed to the alleged events, one often turns to the EHR and its audit log. This manuscript discusses lines of defense incorporated into the design, development, implementation, and use of EHRs to ensure their integrity and the types of EHR transaction logs (e.g., audit log) that exist. Using these logs can help one answer questions that often arise in medical malpractice cases. Finally, there are “best practices” surrounding EHR audit logs that health care organizations should implement. When used appropriately, EHRs and their audit logs provide another source of information to help hospital risk managers, legal counsel, and EHR expert witnesses to investigate adverse incidents and, if needed, prosecute or defend clinicians and/or health care organizations involved in the patient's care.


Five Strategies for a Safer EHR Modernization Journey
Dean F. Sittig, Edward E. Yackel, and Hardeep Singh
Springer Science and Business Media LLC



A lifecycle framework illustrates eight stages necessary for realizing the benefits of patient-centered clinical decision support
Dean F Sittig, Aziz Boxwala, Adam Wright, Courtney Zott, Priyanka Desai, Rina Dhopeshwarkar, James Swiger, Edwin A Lomotan, Angela Dobes, and Prashila Dullabh
Oxford University Press (OUP)
AbstractThe design, development, implementation, use, and evaluation of high-quality, patient-centered clinical decision support (PC CDS) is necessary if we are to achieve the quintuple aim in healthcare. We developed a PC CDS lifecycle framework to promote a common understanding and language for communication among researchers, patients, clinicians, and policymakers. The framework puts the patient, and/or their caregiver at the center and illustrates how they are involved in all the following stages: Computable Clinical Knowledge, Patient-specific Inference, Information Delivery, Clinical Decision, Patient Behaviors, Health Outcomes, Aggregate Data, and patient-centered outcomes research (PCOR) Evidence. Using this idealized framework reminds key stakeholders that developing, deploying, and evaluating PC-CDS is a complex, sociotechnical challenge that requires consideration of all 8 stages. In addition, we need to ensure that patients, their caregivers, and the clinicians caring for them are explicitly involved at each stage to help us achieve the quintuple aim.


Developing electronic clinical quality measures to assess the cancer diagnostic process
Daniel R Murphy, Andrew J Zimolzak, Divvy K Upadhyay, Li Wei, Preeti Jolly, Alexis Offner, Dean F Sittig, Saritha Korukonda, Riyaa Murugaesh Rekha, and Hardeep Singh
Oxford University Press (OUP)
Abstract Objective Measures of diagnostic performance in cancer are underdeveloped. Electronic clinical quality measures (eCQMs) to assess quality of cancer diagnosis could help quantify and improve diagnostic performance. Materials and Methods We developed 2 eCQMs to assess diagnostic evaluation of red-flag clinical findings for colorectal (CRC; based on abnormal stool-based cancer screening tests or labs suggestive of iron deficiency anemia) and lung (abnormal chest imaging) cancer. The 2 eCQMs quantified rates of red-flag follow-up in CRC and lung cancer using electronic health record data repositories at 2 large healthcare systems. Each measure used clinical data to identify abnormal results, evidence of appropriate follow-up, and exclusions that signified follow-up was unnecessary. Clinicians reviewed 100 positive and 20 negative randomly selected records for each eCQM at each site to validate accuracy and categorized missed opportunities related to system, provider, or patient factors. Results We implemented the CRC eCQM at both sites, while the lung cancer eCQM was only implemented at the VA due to lack of structured data indicating level of cancer suspicion on most chest imaging results at Geisinger. For the CRC eCQM, the rate of appropriate follow-up was 36.0% (26 746/74 314 patients) in the VA after removing clinical exclusions and 41.1% at Geisinger (1009/2461 patients; P < .001). Similarly, the rate of appropriate evaluation for lung cancer in the VA was 61.5% (25 166/40 924 patients). Reviewers most frequently attributed missed opportunities at both sites to provider factors (84 of 157). Conclusions We implemented 2 eCQMs to evaluate the diagnostic process in cancer at 2 large health systems. Health care organizations can use these eCQMs to monitor diagnostic performance related to cancer.


Using AI-generated suggestions from ChatGPT to optimize clinical decision support
Siru Liu, Aileen P Wright, Barron L Patterson, Jonathan P Wanderer, Robert W Turer, Scott D Nelson, Allison B McCoy, Dean F Sittig, and Adam Wright
Oxford University Press (OUP)
Abstract Objective To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. Results Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. Conclusion AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.


Visualization of Patient-Generated Health Data: A Scoping Review of Dashboard Designs
Edna Shenvi, Aziz Boxwala, Dean Sittig, Courtney Zott, Edwin Lomotan, James Swiger, and Prashila Dullabh
Georg Thieme Verlag KG
Abstract Background Patient-centered clinical decision support (PC CDS) aims to assist with tailoring decisions to an individual patient's needs. Patient-generated health data (PGHD), including physiologic measurements captured frequently by automated devices, provide important information for PC CDS. The volume and availability of such PGHD is increasing, but how PGHD should be presented to clinicians to best aid decision-making is unclear. Objectives Identify best practices in visualizations of physiologic PGHD, for designing a software application as a PC CDS tool. Methods We performed a scoping review of studies of PGHD dashboards that involved clinician users in design or evaluations. We included only studies that used physiologic PGHD from single patients for usage in decision-making. Results We screened 468 titles and abstracts, 63 full-text papers, and identified 15 articles to include in our review. Some research primarily sought user input on PGHD presentation; other studies garnered feedback only as a side effort for other objectives (e.g., integration with electronic health records [EHRs]). Development efforts were often in the domains of chronic diseases and collected a mix of physiologic parameters (e.g., blood pressure and heart rate) and activity data. Users' preferences were for data to be presented with statistical summaries and clinical interpretations, alongside other non-PGHD data. Recurrent themes indicated that users desire longitudinal data display, aggregation of multiple data types on the same screen, actionability, and customization. Speed, simplicity, and availability of data for other purposes (e.g., documentation) were key to dashboard adoption. Evaluations were favorable for visualizations using common graphing or table formats, although best practices for implementation have not yet been established. Conclusion Although the literature identified common themes on data display, measures, and usability, more research is needed as PGHD usage grows. Ensuring that care is tailored to individual needs will be important in future development of clinical decision support.


A multi-site randomized trial of a clinical decision support intervention to improve problem list completeness
Adam Wright, Richard Schreiber, David W Bates, Skye Aaron, Angela Ai, Raja Arul Cholan, Akshay Desai, Miguel Divo, David A Dorr, Thu-Trang Hickman,et al.
Oxford University Press (OUP)
Abstract Objective To improve problem list documentation and care quality. Materials and methods We developed algorithms to infer clinical problems a patient has that are not recorded on the coded problem list using structured data in the electronic health record (EHR) for 12 clinically significant heart, lung, and blood diseases. We also developed a clinical decision support (CDS) intervention which suggests adding missing problems to the problem list. We evaluated the intervention at 4 diverse healthcare systems using 3 different EHRs in a randomized trial using 3 predetermined outcome measures: alert acceptance, problem addition, and National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) clinical quality measures. Results There were 288 832 opportunities to add a problem in the intervention arm and the problem was added 63 777 times (acceptance rate 22.1%). The intervention arm had 4.6 times as many problems added as the control arm. There were no significant differences in any of the clinical quality measures. Discussion The CDS intervention was highly effective at improving problem list completeness. However, the improvement in problem list utilization was not associated with improvement in the quality measures. The lack of effect on quality measures suggests that problem list documentation is not directly associated with improvements in quality measured by National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) quality measures. However, improved problem list accuracy has other benefits, including clinical care, patient comprehension of health conditions, accurate CDS and population health, and for research. Conclusion An EHR-embedded CDS intervention was effective at improving problem list completeness but was not associated with improvement in quality measures.


Changes in Availability of and Prices for Shoppable Services at US News and World Report Honor Roll Hospitals: a Longitudinal Cross-Sectional Study
Peter Cram, Elliot Cram, Joseph Antos, Dean F. Sittig, Ajay Anand, and Yue Li
Springer Science and Business Media LLC



Computer clinical decision support that automates personalized clinical care: A challenging but needed healthcare delivery strategy
Alan H Morris, Christopher Horvat, Brian Stagg, David W Grainger, Michael Lanspa, James Orme, Terry P Clemmer, Lindell K Weaver, Frank O Thomas, Colin K Grissom,et al.
Oxford University Press (OUP)
AbstractHow to deliver best care in various clinical settings remains a vexing problem. All pertinent healthcare-related questions have not, cannot, and will not be addressable with costly time- and resource-consuming controlled clinical trials. At present, evidence-based guidelines can address only a small fraction of the types of care that clinicians deliver. Furthermore, underserved areas rarely can access state-of-the-art evidence-based guidelines in real-time, and often lack the wherewithal to implement advanced guidelines. Care providers in such settings frequently do not have sufficient training to undertake advanced guideline implementation. Nevertheless, in advanced modern healthcare delivery environments, use of eActions (validated clinical decision support systems) could help overcome the cognitive limitations of overburdened clinicians. Widespread use of eActions will require surmounting current healthcare technical and cultural barriers and installing clinical evidence/data curation systems. The authors expect that increased numbers of evidence-based guidelines will result from future comparative effectiveness clinical research carried out during routine healthcare delivery within learning healthcare systems.


i-CLIMATE: a "clinical climate informatics" action framework to reduce environmental pollution from healthcare
Dean F Sittig, Jodi D Sherman, Matthew J Eckelman, Andrew Draper, and Hardeep Singh
Oxford University Press (OUP)
Abstract Addressing environmental pollution and climate change is one of the biggest sociotechnical challenges of our time. While information technology has led to improvements in healthcare, it has also contributed to increased energy usage, destructive natural resource extraction, piles of e-waste, and increased greenhouse gases. We introduce a framework “Information technology-enabled Clinical cLimate InforMAtics acTions for the Environment” (i-CLIMATE) to illustrate how clinical informatics can help reduce healthcare’s environmental pollution and climate-related impacts using 5 actionable components: (1) create a circular economy for health IT, (2) reduce energy consumption through smarter use of health IT, (3) support more environmentally friendly decision-making by clinicians and health administrators, (4) mobilize healthcare workforce environmental stewardship through informatics, and (5) Inform policies and regulations for change. We define Clinical Climate Informatics as a field that applies data, information, and knowledge management principles to operationalize components of the i-CLIMATE Framework.


A Systematic Approach to Configuring MetaMap for Optimal Performance
Xia Jing, Akash Indani, Nina Hubig, Hua Min, Yang Gong, James J. Cimino, Dean F. Sittig, Lior Rennert, David Robinson, Paul Biondich,et al.
Georg Thieme Verlag KG
Abstract Background MetaMap is a valuable tool for processing biomedical texts to identify concepts. Although MetaMap is highly configurative, configuration decisions are not straightforward. Objective To develop a systematic, data-driven methodology for configuring MetaMap for optimal performance. Methods MetaMap, the word2vec model, and the phrase model were used to build a pipeline. For unsupervised training, the phrase and word2vec models used abstracts related to clinical decision support as input. During testing, MetaMap was configured with the default option, one behavior option, and two behavior options. For each configuration, cosine and soft cosine similarity scores between identified entities and gold-standard terms were computed for 40 annotated abstracts (422 sentences). The similarity scores were used to calculate and compare the overall percentages of exact matches, similar matches, and missing gold-standard terms among the abstracts for each configuration. The results were manually spot-checked. The precision, recall, and F-measure (β =1) were calculated. Results The percentages of exact matches and missing gold-standard terms were 0.6–0.79 and 0.09–0.3 for one behavior option, and 0.56–0.8 and 0.09–0.3 for two behavior options, respectively. The percentages of exact matches and missing terms for soft cosine similarity scores exceeded those for cosine similarity scores. The average precision, recall, and F-measure were 0.59, 0.82, and 0.68 for exact matches, and 1.00, 0.53, and 0.69 for missing terms, respectively. Conclusion We demonstrated a systematic approach that provides objective and accurate evidence guiding MetaMap configurations for optimizing performance. Combining objective evidence and the current practice of using principles, experience, and intuitions outperforms a single strategy in MetaMap configurations. Our methodology, reference codes, measurements, results, and workflow are valuable references for optimizing and configuring MetaMap.


Clinical decision support malfunctions related to medication routes: A case series
Adam Wright, Scott Nelson, David Rubins, Richard Schreiber, and Dean F Sittig
Oxford University Press (OUP)
Abstract Objective To identify common medication route-related causes of clinical decision support (CDS) malfunctions and best practices for avoiding them. Materials and Methods Case series of medication route-related CDS malfunctions from diverse healthcare provider organizations. Results Nine cases were identified and described, including both false-positive and false-negative alert scenarios. A common cause was the inclusion of nonsystemically available medication routes in value sets (eg, eye drops, ear drops, or topical preparations) when only systemically available routes were appropriate. Discussion These value set errors are common, occur across healthcare provider organizations and electronic health record (EHR) systems, affect many different types of medications, and can impact the accuracy of CDS interventions. New knowledge management tools and processes for auditing existing value sets and supporting the creation of new value sets can mitigate many of these issues. Furthermore, value set issues can adversely affect other aspects of the EHR, such as quality reporting and population health management. Conclusion Value set issues related to medication routes are widespread and can lead to CDS malfunctions. Organizations should make appropriate investments in knowledge management tools and strategies, such as those outlined in our recommendations.


Identifying a Clinical Informatics or Electronic Health Record Expert Witness for Medical Professional Liability Cases
Dean F. Sittig and Adam Wright
Georg Thieme Verlag KG
Abstract Background The health care field is experiencing widespread electronic health record (EHR) adoption. New medical professional liability (i.e., malpractice) cases will likely involve the review of data extracted from EHRs as well as EHR workflows, audit logs, and even the potential role of the EHR in causing harm. Objectives Reviewing printed versions of a patient's EHRs can be difficult due to differences in printed versus on-screen presentations, redundancies, and the way printouts are often grouped by document or information type rather than chronologically. Simply recreating an accurate timeline often requires experts with training and experience in designing, developing, using, and reviewing EHRs and audit logs. Additional expertise is required if questions arise about data's meaning, completeness, accuracy, and timeliness or ways that the EHR's user interface or automated clinical decision support tools may have contributed to alleged events. Such experts often come from the sociotechnical field of clinical informatics that studies the design, development, implementation, use, and evaluation of information and communications technology, specifically, EHRs. Identifying well-qualified EHR experts to aid a legal team is challenging. Methods Based on literature review and experience reviewing cases, we identified seven criteria to help in this assessment. Results The criteria are education in clinical informatics; clinical informatics knowledge; experience with EHR design, development, implementation, and use; communication skills; academic publications on clinical informatics; clinical informatics certification; and membership in informatics-related professional organizations. Conclusion While none of these criteria are essential, understanding the breadth and depth of an individual's qualifications in each of these areas can help identify a high-quality, clinical informatics expert witness.


Translating electronic health record-based patient safety algorithms from research to clinical practice at multiple sites
Andrew J Zimolzak, Hardeep Singh, Daniel R Murphy, Li Wei, Sahar A Memon, Divvy K Upadhyay, Saritha Korukonda, Lisa Zubkoff, and Dean F Sittig
BMJ
IntroductionResearchers are increasingly developing algorithms that impact patient care, but algorithms must also be implemented in practice to improve quality and safety.ObjectiveWe worked with clinical operations personnel at two US health systems to implement algorithms to proactively identify patients without timely follow-up of abnormal test results that warrant diagnostic evaluation for colorectal or lung cancer. We summarise the steps involved and lessons learned.MethodsTwelve sites were involved across two health systems. Implementation involved extensive software documentation, frequent communication with sites and local validation of results. Additionally, we used automated edits of existing code to adapt it to sites’ local contexts.ResultsAll sites successfully implemented the algorithms. Automated edits saved sites significant work in direct code modification. Documentation and communication of changes further aided sites in implementation.ConclusionPatient safety algorithms developed in research projects were implemented at multiple sites to monitor for missed diagnostic opportunities. Automated algorithm translation procedures can produce more consistent results across sites.


Challenges and opportunities for advancing patient-centered clinical decision support: Findings from a horizon scan
Prashila Dullabh, Shana F Sandberg, Krysta Heaney-Huls, Lauren S Hovey, David F Lobach, Aziz Boxwala, Priyanka J Desai, Elise Berliner, Chris Dymek, Michael I Harrison,et al.
Oxford University Press (OUP)
AbstractObjectiveWe conducted a horizon scan to (1) identify challenges in patient-centered clinical decision support (PC CDS) and (2) identify future directions for PC CDS.Materials and MethodsWe engaged a technical expert panel, conducted a scoping literature review, and interviewed key informants. We qualitatively analyzed literature and interview transcripts, mapping findings to the 4 phases for translating evidence into PC CDS interventions (Prioritizing, Authoring, Implementing, and Measuring) and to external factors.ResultsWe identified 12 challenges for PC CDS development. Lack of patient input was identified as a critical challenge. The key informants noted that patient input is critical to prioritizing topics for PC CDS and to ensuring that CDS aligns with patients’ routine behaviors. Lack of patient-centered terminology standards was viewed as a challenge in authoring PC CDS. We found a dearth of CDS studies that measured clinical outcomes, creating significant gaps in our understanding of PC CDS’ impact. Across all phases of CDS development, there is a lack of patient and provider trust and limited attention to patients’ and providers’ concerns.DiscussionThese challenges suggest opportunities for advancing PC CDS. There are opportunities to develop industry-wide practices and standards to increase transparency, standardize terminologies, and incorporate patient input. There is also opportunity to engage patients throughout the PC CDS research process to ensure that outcome measures are relevant to their needs.ConclusionAddressing these challenges and embracing these opportunities will help realize the promise of PC CDS—placing patients at the center of the healthcare system.


The Technology Landscape of Patient-Centered Clinical Decision Support-Where Are We and What Is Needed?
Prashila Dullabh, Krysta Heaney-Huls, Lauren Hovey, Shana Sandberg, David F. Lobach, Aziz Boxwala, Priyanka Desai, and Dean F. Sittig
IOS Press
Patient Centered Outcomes Research (PCOR) and health care delivery system transformation require investments in development of tools and techniques for rapid dissemination of clinical and operational best practices. This paper explores the current technology landscape for patient-centered clinical decision support (PC CDS) and what is needed to make it more shareable, standards-based, and publicly available with the goal of improving patient care and clinical outcomes. The landscape assessment used three sources of information: (1) a 22-member technical expert panel; (2) a literature review of peer-reviewed and grey literature; and (3) key informant interviews with PC CDS stakeholders. We identified ten salient technical considerations that span all phases of PC CDS development; our findings suggest there has been significant progress in the development and implementation of PC CDS but challenges remain.

RECENT SCHOLAR PUBLICATIONS

Recommendations to Ensure Safety of AI in Real-World Clinical Care
DF Sittig, H Singh
JAMA 333 (6), 457-458 2025



Regulation of artificial intelligence in healthcare: Clinical Laboratory Improvement Amendments (CLIA) as a model
BR Jackson, MP Sendak, A Solomonides, S Balu, DF Sittig
Journal of the American Medical Informatics Association 32 (2), 404-407 2025



Implementation of electronic triggers to identify diagnostic errors in emergency departments
V Vaghani, A Gupta, U Mir, L Wei, DR Murphy, U Mushtaq, DF Sittig, ...
JAMA Internal Medicine 185 (2), 143-151 2025



Vaccination Schedules Recommended by the Centers for Disease Control and Prevention: From Human-Readable to Machine-Processable
X Jing, H Min, Y Gong, M Ernst, A Weaver, C Crozier, D Robinson, ...
2025



Maximizing the Ability of Health IT and AI to Improve Patient Safety
H Singh, DF Sittig, DC Classen
JAMA Internal Medicine 185 (1), 10-12 2025



Recommendations to Ensure Safety of AI in Real-World Clinical Care
DF Sittig, H Singh
Journal of the American Medical Association 2024



Keyphrase Identification Using Minimal Labeled Data with Hierarchical Contexts and Transfer Learning
R Goli, K Komatineni, S Alluri, N Hubig, H Min, Y Gong, DF Sittig, ...
medRxiv, 2023.01. 26.23285060 2024



Toward a responsible future: recommendations for AI-enabled clinical decision support
S Labkoff, B Oladimeji, J Kannry, A Solomonides, R Leftwich, E Koski, ...
Journal of the American Medical Informatics Association 31 (11), 2730-2739 2024



Maximizing the Ability of Health IT and AI to Improve Patient Safety
H Singh, DF Sittig, DC Classen
JAMA Intern Med 2024



Reusable Generic Clinical Decision Support System Module for Immunization Recommendations in Resource-Constraint Settings
S Orlioglu, AS Boobalan, K Abanyie, RD Boyce, H Min, Y Gong, DF Sittig, ...
medRxiv 2024



PAIGE Chatbot For Patient-Clinician Communication
C Zott, DF Sittig, N Gauthreaux, A Wright, F FAMIA, E Russo, PMPL Zahn, ...
2024



Quartz App to Support Medication Adherence
N Gauthreaux, C Zott, A Boxwala, DF Sittig, PM Dullabh
2024



Patient-centered clinical decision support challenges and opportunities identified from workflow execution models
DP Sittig DF, Boxwala A, Wright A, Zott C, Gauthreaux NA, Swiger J, Lomotan EA
J Am Med Inform Assoc. 31(8):1682-1692. doi: 10.1093/jamia/ocae138. 31 (8 2024



Implementation of a health information technology safety classification system in the Veterans Health Administration’s Informatics Patient Safety Office
D Kato, J Lucas, DF Sittig
Journal of the American Medical Informatics Association 31 (7), 1588-1595 2024



Leveraging explainable artificial intelligence to optimize clinical decision support
S Liu, AB McCoy, JF Peterson, TA Lasko, DF Sittig, SD Nelson, J Andrews, ...
Journal of the American Medical Informatics Association 31 (4), 968-974 2024



Active Learning Pipeline to Identify Candidate Terms for a CDSS Ontology
X Jing, R Goli, K Komatineni, S Alluri, N Hubig, H Min, Y Gong, DF Sittig, ...
Digital Health and Informatics Innovations for Sustainable Health Care 2024



Five Strategies for a Safer EHR Modernization Journey
DF Sittig, EE Yackel, H Singh
Journal of General Internal Medicine 38 (Suppl 4), 940-942 2023



Visualization of patient-generated health data: a scoping review of dashboard designs
E Shenvi, A Boxwala, D Sittig, C Zott, E Lomotan, J Swiger, P Dullabh
Applied Clinical Informatics 14 (05), 913-922 2023



A guide to mitigating audit log‐related risk in medical professional liability cases
DF Sittig, A Wright
Journal of Healthcare Risk Management 43 (2), 37-47 2023



A lifecycle framework illustrates eight stages necessary for realizing the benefits of patient-centered clinical decision support
DF Sittig, A Boxwala, A Wright, C Zott, P Desai, R Dhopeshwarkar, ...
Journal of the American Medical Informatics Association 30 (9), 1583-1589 2023

MOST CITED SCHOLAR PUBLICATIONS

Randomized clinical trial of pressure-controlled inverse ratio ventilation and extracorporeal CO2 removal for adult respiratory distress syndrome.
AH Morris, CJ Wallace, RL Menlove, TP Clemmer, JF Orme Jr, LK Weaver, ...
American journal of respiratory and critical care medicine 149 (2), 295-305 1994
Citations: 1274



Types of unintended consequences related to computerized provider order entry
EM Campbell, DF Sittig, JS Ash, KP Guappone, RH Dykstra
Journal of the American Medical Informatics Association 13 (5), 547-556 2006
Citations: 1104



A new socio-technical model for studying health information technology in complex adaptive healthcare systems
DF Sittig, H Singh
Cognitive informatics for biomedicine, 59-80 2015
Citations: 1017



Grand challenges in clinical decision support
DF Sittig, A Wright, JA Osheroff, B Middleton, JM Teich, JS Ash, ...
Journal of biomedical informatics 41 (2), 387-392 2008
Citations: 751



Improving Outcomes: A Practical Guide to Clinical Decision Support Implementation
JA Osheroff, EA Pifer, DF Sittig, RA Jenders, JM Teich
Chicago: HIMSS 2005
Citations: 685



The extent and importance of unintended consequences related to computerized provider order entry
JS Ash, DF Sittig, EG Poon, K Guappone, E Campbell, RH Dykstra
Journal of the American Medical Informatics Association 14 (4), 415-423 2007
Citations: 675



Communication breakdown in the outpatient referral process
TK Gandhi, DF Sittig, M Franklin, AJ Sussman, DG Fairchild, DW Bates
Journal of general internal medicine 15, 626-631 2000
Citations: 497



Computer-based physician order entry: the state of the art
DF Sittig, WW Stead
Journal of the American Medical Informatics Association 1 (2), 108-123 1994
Citations: 422



Some unintended consequences of clinical decision support systems
JS Ash, DF Sittig, EM Campbell, KP Guappone, RH Dykstra
Amia annual Symposium proceedings 2007, 26 2007
Citations: 397



The emerging science of very early detection of disease outbreaks
MM Wagner, FC Tsui, JU Espino, VM Dato, DF Sittig, RA Caruana, ...
Journal of Public Health Management and Practice 7 (6), 51-59 2001
Citations: 392



Categorizing the unintended sociotechnical consequences of computerized provider order entry
JS Ash, DF Sittig, RH Dykstra, K Guappone, JD Carpenter
Int J Med Inform 76, 21-27 2007
Citations: 364



Using AI-generated suggestions from ChatGPT to optimize clinical decision support
S Liu, AP Wright, BL Patterson, JP Wanderer, RW Turer, SD Nelson, ...
Journal of the American Medical Informatics Association 30 (7), 1237-1245 2023
Citations: 333



Clinical decision support: a 25 year retrospective and a 25 year vision
B Middleton, DF Sittig, A Wright
Yearbook of medical informatics 25 (S 01), S103-S116 2016
Citations: 302



Electronic health records and national patient-safety goals
DF Sittig, H Singh
New England Journal of Medicine 367 (19), 1854-1860 2012
Citations: 283



Defining and measuring diagnostic uncertainty in medicine: a systematic review
V Bhise, SS Rajan, DF Sittig, RO Morgan, P Chaudhary, H Singh
Journal of general internal medicine 33, 103-115 2018
Citations: 276



The unintended consequences of computerized provider order entry: findings from a mixed methods exploration
JS Ash, DF Sittig, R Dykstra, E Campbell, K Guappone
International journal of medical informatics 78, S69-S76 2009
Citations: 273



Defining health information technology–related errors: new developments since to err is human
DF Sittig, H Singh
Archives of internal medicine 171 (14), 1281-1284 2011
Citations: 261



Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential?
H Singh, EJ Thomas, S Mani, D Sittig, H Arora, D Espadas, MM Khan, ...
Archives of internal medicine 169 (17), 1578-1586 2009
Citations: 258



An analysis of electronic health record-related patient safety concerns
DW Meeks, MW Smith, L Taylor, DF Sittig, JM Scott, H Singh
Journal of the American Medical Informatics Association 21 (6), 1053-1059 2014
Citations: 254



Information overload and missed test results in electronic health record–based settings
H Singh, C Spitzmueller, NJ Petersen, MK Sawhney, DF Sittig
JAMA internal medicine 173 (8), 702-704 2013
Citations: 245

CONSULTANCY

Expert witness for Electronic Health Record-related issues