Dean Sittig
@sbmi.uth.edu
Professor Clinical and Health Informatics
University of Texas Health Science Center at Houston
EDUCATION
PhD University of Utah
RESEARCH, TEACHING, or OTHER INTERESTS
Health Informatics, General Medicine, Health (social science), Safety Research
Scopus Publications
Scholar Citations
Scholar h-index
Scholar i10-index
Scopus Publications
Xia Jing, Aneesa Weaver, Phyllis MacGilvray, Timothy Law, David Robinson, Dean F. Sittig, Christian Nøhr, Arild Faxvaag, Paul Biondich, and Adam Wright
IOS Press
We conducted an online survey and structured interviews with a convenience sample of primary care providers in the USA to identify requirements related to the clinical decision support systems (CDSS) in primary care settings.
Xia Jing, Hua Min, Yang Gong, Mytchell A. Ernst, Aneesa Weaver, Chloe Crozier, David Robinson, Dean F. Sittig, Paul G. Biondich, Samuil Orlioglu,et al.
MDPI AG
Background: Reusable, machine-processable clinical decision support system (CDSS) rules have not been widely achieved in the medical informatics field. This study introduces the process, results, challenges faced, and lessons learned while converting the United States of America Centers for Disease Control and Prevention (CDC)-recommended immunization schedules (2022) to machine-processable CDSS rules. Methods: We converted the vaccination schedules into tabular, charts, MS Excel, and clinical quality language (CQL) formats. The CQL format can be automatically converted to a machine-processable format using existing tools. Therefore, it was regarded as a machine-processable format. The results were reviewed, verified, and tested. Results: We have developed 465 rules for 19 vaccines in 13 categories, and we have shared the rules via GitHub to make them publicly available. We used cross-review and cross-checking to validate the CDSS rules in tabular and chart formats. The CQL files were tested for syntax and logic with hypothetical patient HL7 FHIR resources. Our rules can be reused and shared by the health IT industry, CDSS developers, medical informatics educators, or clinical care institutions. The unique contributions of our work are twofold: (1) we created ontology-based, machine-processable, and reusable immunization recommendation rules, and (2) we created and shared multiple formats of immunization recommendation rules publicly which can be a valuable resource for medical and medical informatics communities. Conclusions: These CDSS rules can be important contributions to informatics communities, reducing redundant efforts, which is particularly significant in resource-limited settings. Despite the maturity and concise presentation of the CDC recommendations, careful attention and multiple layers of verification and review are necessary to ensure accurate conversion. The publicly shared CDSS rules can also be used for health and biomedical informatics education and training purposes.
Dean F Sittig, Trisha Flanagan, Patricia Sengstack, Rosann T Cholankeril, Sara Ehsan, Amanda Heidemann, Daniel R Murphy, Hojjat Salmasian, Jason S Adelman, and Hardeep Singh
Oxford University Press (OUP)
Abstract The Safety Assurance Factors for Electronic Health Record (EHR) Resilience (SAFER) Guides provide recommendations to healthcare organizations for conducting proactive self-assessments of the safety and effectiveness of their EHR implementation and use. Originally released in 2014, they were last updated in 2016. In 2022, the Centers for Medicare and Medicaid Services required their annual attestation by US hospitals. Objectives This case study describes how SAFER Guide recommendations were updated to align with current evidence and clinical practice. Materials and Methods Over nine months, a multidisciplinary team updated SAFER Guides through literature reviews, iterative feedback, and online meetings. Results We reduced the number of recommended practices across all Guides by 40% and consolidated 9 Guides into 8 to maximize ease of use, feasibility, and utility. We provide a 4-level evidence grading hierarchy for each recommendation and a new 5-point rating scale to self-assess implementation status of the recommendation. We included 429 citations of which 289 (67%) were published since the 2016 revision. Discussion SAFER Guides were revised to offer EHR best practices, adaptable to unique organizational needs, with interactive content available at: https://www.healthit.gov/topic/safety/safer-guides. Conclusion Revisions ensure that the 2025 SAFER Guides represent the best available current evidence for EHR developers and healthcare organizations.
Dean F. Sittig and Hardeep Singh
American Medical Association (AMA)
This Viewpoint provides recommendations for health care organizations (HCOs) and clinicians to facilitate the use of artificial intelligence (AI)–enabled systems, including electronic health records with AI features, in routine clinical care and provides pragmatic guidance for HCOs and clinicians at all stages of AI implementation.
Viralkumar Vaghani, Ashish Gupta, Usman Mir, Li Wei, Daniel R. Murphy, Umair Mushtaq, Dean F. Sittig, Andrew J. Zimolzak, and Hardeep Singh
American Medical Association (AMA)
ImportanceMissed diagnosis can lead to preventable patient harm.ObjectiveTo develop and implement a portfolio of electronic triggers (e-triggers) and examine their performance for identifying missed opportunities in diagnosis (MODs) in emergency departments (EDs).Design, Setting, and ParticipantsIn this retrospective medical record review study of ED visits at 1321 Veterans Affairs health care sites, rules-based e-triggers were developed and implemented using a national electronic health record repository. These e-triggers targeted 6 high-risk presentations for MODs in treat-and-release ED visits. A high-risk stroke e-trigger was applied to treat-and-release ED visits from January 1, 2016, to December 31, 2020. A symptom-disease dyad e-trigger was applied to visits from January 1, 2018, to December 31, 2019. High-risk abdominal pain, unexpected ED return, unexpected hospital return, and test result e-triggers were applied to visits from January 1, 2019, to December 31, 2019. At least 100 randomly selected flagged records were reviewed by physician reviewers for each e-trigger. Data were analyzed between January 2024 and April 2024.ExposuresTreat-and-release ED visits involving high-risk stroke, symptom-disease dyads, high-risk abdominal pain, unexpected ED return, unexpected hospital return, and abnormal test results not followed up after initial ED visit.Main Outcomes and MeasuresTrained physician reviewers evaluated the presence/absence of MODs at ED visits and recorded data on patient and clinician characteristics, types of diagnostic process breakdowns, and potential harm from MODs.ResultsThe high-risk stroke e-trigger was applied to 8 792 672 treat-and-release ED visits (4 967 283 unique patients); the symptom-disease dyad e-trigger was applied to 3 692 454 visits (2 070 979 patients); and high-risk abdominal pain, unexpected ED return, unexpected hospital return, and test result e-triggers were applied to 1 845 905 visits (1 032 969 patients), overall identifying 203, 1981, 170, 116 785, 14 879, and 2090 trigger-positive records, respectively. Review of 625 randomly selected patient records (mean [SD] age, 62.5 [15.2] years; 553 [88.5%] male) showed the following MOD counts and positive predictive values (PPVs) within each category: 47 MODs (PPV, 47.0%) for stroke, 31 MODs (PPV, 25.8%) for abdominal pain, 11 MODs (PPV, 11.0%) for ED returns, 23 MODs (PPV, 23.0%) for hospital returns, 18 MODs (PPV, 18.0%) for symptom-disease dyads, and 55 MODs (PPV, 52.4%) for test results. Patients with MODs were slightly older than those without (mean [SD] age, 65.6 [14.5] vs 61.2 [15.3] years; P < .001). Reviewer agreement was favorable (range, 72%-100%). In 108 of 130 MODs (83.1%; excluding MODs related to the test result e-trigger), the most common diagnostic process breakdown involved the patient-clinician encounter. In 185 total MODs, 20 patients experienced severe harm (10.8%), and 54 patients experienced moderate harm (29.2%).Conclusions and RelevanceIn this retrospective medical record review study, rules-based e-triggers were useful for post hoc detection of MODs in ED visits. Interventions to target ED work system factors are urgently needed to support patient-clinician encounters and minimize harm from diagnostic errors.
Brian R Jackson, Mark P Sendak, Anthony Solomonides, Suresh Balu, and Dean F Sittig
Oxford University Press (OUP)
Abstract Objectives To assess the potential to adapt an existing technology regulatory model, namely the Clinical Laboratory Improvement Amendments (CLIA), for clinical artificial intelligence (AI). Materials and Methods We identify overlap in the quality management requirements for laboratory testing and clinical AI. Results We propose modifications to the CLIA model that could make it suitable for oversight of clinical AI. Discussion In national discussions of clinical AI, there has been surprisingly little consideration of this longstanding model for local technology oversight. While CLIA was specifically designed for laboratory testing, most of its principles are applicable to other technologies in patient care. Conclusion A CLIA-like approach to regulating clinical AI would be complementary to the more centralized schemes currently under consideration, and it would ensure institutional and professional accountability for the longitudinal quality management of clinical AI.
Prashila Dullabh, Courtney Zott, Nicole Gauthreaux, James Swiger, Edwin Lomotan, and Dean F Sittig
JMIR Publications Inc.
Background Patient-centered clinical decision support (PC CDS) exists on a continuum that reflects the degree to which its knowledge base, data, delivery, and use focus on patient needs and experiences. A new focus on value-based, whole-person care has resulted in broader development of PC CDS technologies, yet there is limited information on how to measure their performance and effectiveness. To address these gaps, there is a need for more measurement guidance to assess PC CDS interventions. Objective This paper presents a new framework that incorporates patient-centered principles into traditional health IT and clinical decision support (CDS) evaluation frameworks to create a unified guide to PC CDS performance measurement. Methods We conducted a targeted literature review of 147 sources on health IT, CDS, and PC CDS measurement and evaluation to develop the framework. Sources were reviewed if they included the sociotechnical components relevant to PC CDS, covered the full IT life cycle of PC CDS, and addressed measurement considerations at different user and system levels. We then validated and refined the measurement framework through key informant interviews with 6 experts in measurement, CDS, and clinical informatics. Throughout the framework development, we gathered feedback from a 7-member expert committee on the methods, findings, and the framework’s relevance and application. Results The PC CDS performance measurement framework includes 6 domains: safe, timely, effective, efficient, equitable, and patient centered. The 6 domains contain 34 subdomains that can be selected to assess performance, depending on the type of PC CDS intervention or the specific research focus. In addition, there are 4 levels of aggregation at which subdomains can be measured (individual, population, organization, or IT system) that account for the multilevel impact of PC CDS. We provide examples of measures and approaches to patient centeredness for each subdomain, followed by 2 illustrative use cases demonstrating the framework application. Conclusions This framework can be used by researchers, health system leaders, informaticians, and patients to understand the full breadth of performance and impact of PC CDS technology. The framework is significant in that it (1) covers the entire PC CDS life cycle, (2) has a direct focus on the patient, (3) covers measurement at different levels, (4) encompasses 6 independent but related domains, and (5) requires additional research and development to fully characterize all domains and subdomains. As the field of PC CDS matures, researchers and evaluators can build upon the framework to assess which components of PC CDS technologies work; whether PC CDS technologies are being used as anticipated; and whether the intended outcomes of delivering evidence-based, patient-centered care are being achieved.
Steven Labkoff, Bilikis Oladimeji, Joseph Kannry, Anthony Solomonides, Russell Leftwich, Eileen Koski, Amanda L Joseph, Monica Lopez-Gonzalez, Lee A Fleisher, Kimberly Nolen,et al.
Oxford University Press (OUP)
Abstract Background Integrating artificial intelligence (AI) in healthcare settings has the potential to benefit clinical decision-making. Addressing challenges such as ensuring trustworthiness, mitigating bias, and maintaining safety is paramount. The lack of established methodologies for pre- and post-deployment evaluation of AI tools regarding crucial attributes such as transparency, performance monitoring, and adverse event reporting makes this situation challenging. Objectives This paper aims to make practical suggestions for creating methods, rules, and guidelines to ensure that the development, testing, supervision, and use of AI in clinical decision support (CDS) systems are done well and safely for patients. Materials and Methods In May 2023, the Division of Clinical Informatics at Beth Israel Deaconess Medical Center and the American Medical Informatics Association co-sponsored a working group on AI in healthcare. In August 2023, there were 4 webinars on AI topics and a 2-day workshop in September 2023 for consensus-building. The event included over 200 industry stakeholders, including clinicians, software developers, academics, ethicists, attorneys, government policy experts, scientists, and patients. The goal was to identify challenges associated with the trusted use of AI-enabled CDS in medical practice. Key issues were identified, and solutions were proposed through qualitative analysis and a 4-month iterative consensus process. Results Our work culminated in several key recommendations: (1) building safe and trustworthy systems; (2) developing validation, verification, and certification processes for AI-CDS systems; (3) providing a means of safety monitoring and reporting at the national level; and (4) ensuring that appropriate documentation and end-user training are provided. Discussion AI-enabled Clinical Decision Support (AI-CDS) systems promise to revolutionize healthcare decision-making, necessitating a comprehensive framework for their development, implementation, and regulation that emphasizes trustworthiness, transparency, and safety. This framework encompasses various aspects including model training, explainability, validation, certification, monitoring, and continuous evaluation, while also addressing challenges such as data privacy, fairness, and the need for regulatory oversight to ensure responsible integration of AI into clinical workflow. Conclusions Achieving responsible AI-CDS systems requires a collective effort from many healthcare stakeholders. This involves implementing robust safety, monitoring, and transparency measures while fostering innovation. Future steps include testing and piloting proposed trust mechanisms, such as safety reporting protocols, and establishing best practice guidelines.
Xia Jing, Rohan Goli, Keerthana Komatineni, Shailesh Alluri, Nina Hubig, Hua Min, Yang Gong, Dean F. Sittig, Paul Biondich, David Robinson,et al.
IOS Press
Ontology is essential for achieving health information and information technology application interoperability in the biomedical fields and beyond. Traditionally, ontology construction is carried out manually by human domain experts (HDE). Here, we explore an active learning approach to automatically identify candidate terms from publications, with manual verification later as a part of a deep learning model training and learning process. We introduce the overall architecture of the active learning pipeline and present some preliminary results. This work is a critical and complementary component in addition to manually building the ontology, especially during the long-term maintenance stage.
Dean F Sittig, Aziz Boxwala, Adam Wright, Courtney Zott, Nicole A Gauthreaux, James Swiger, Edwin A Lomotan, and Prashila Dullabh
Oxford University Press (OUP)
Abstract Objective To use workflow execution models to highlight new considerations for patient-centered clinical decision support policies (PC CDS), processes, procedures, technology, and expertise required to support new workflows. Methods To generate and refine models, we used (1) targeted literature reviews; (2) key informant interviews with 6 external PC CDS experts; (3) model refinement based on authors’ experience; and (4) validation of the models by a 26-member steering committee. Results and Discussion We identified 7 major issues that provide significant challenges and opportunities for healthcare systems, researchers, administrators, and health IT and app developers. Overcoming these challenges presents opportunities for new or modified policies, processes, procedures, technology, and expertise to: (1) Ensure patient-generated health data (PGHD), including patient-reported outcomes (PROs), are documented, reviewed, and managed by appropriately trained clinicians, between visits and after regular working hours. (2) Educate patients to use connected medical devices and handle technical issues. (3) Facilitate collection and incorporation of PGHD, PROs, patient preferences, and social determinants of health into existing electronic health records. (4) Troubleshoot erroneous data received from devices. (5) Develop dashboards to display longitudinal patient-reported data. (6) Provide reimbursement to support new models of care. (7) Support patient engagement with remote devices. Conclusion Several new policies, processes, technologies, and expertise are required to ensure safe and effective implementation and use of PC CDS. As we gain more experience implementing and working with PC CDS, we should be able to begin realizing the long-term positive impact on patient health that the patient-centered movement in healthcare promises.
Danielle Kato, Joe Lucas, and Dean F Sittig
Oxford University Press (OUP)
Abstract Objective Implement the 5-type health information technology (HIT) patient safety concern classification system for HIT patient safety issues reported to the Veterans Health Administration’s Informatics Patient Safety Office. Materials and methods A team of informatics safety analysts retrospectively classified 1 year of HIT patient safety issues by type of HIT patient safety concern using consensus discussions. The processes established during retrospective classification were then applied to incoming HIT safety issues moving forward. Results Of 140 issues retrospectively reviewed, 124 met the classification criteria. The majority were HIT failures (eg, software defects) (33.1%) or configuration and implementation problems (29.8%). Unmet user needs and external system interactions accounted for 20.2% and 10.5%, respectively. Absence of HIT safety features accounted for 2.4% of issues, and 4% did not have enough information to classify. Conclusion The 5-type HIT safety concern classification framework generated actionable categories helping organizations effectively respond to HIT patient safety risks.
Siru Liu, Allison B McCoy, Josh F Peterson, Thomas A Lasko, Dean F Sittig, Scott D Nelson, Jennifer Andrews, Lorraine Patterson, Cheryl M Cobb, David Mulherin,et al.
Oxford University Press (OUP)
Abstract Objective To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches. Methods We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert’s historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful. Results The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues. Conclusion We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.
Dean F. Sittig and Adam Wright
Wiley
AbstractFollowing the American Recovery and Reinvestment Act in 2009, use of electronic health records (EHRs) has become ubiquitous. Accordingly, one should expect most medical professional liability cases to involve review of patient records produced from EHRs. When questions arise regarding who was involved in care of a patient, what they knew and when, or the meaning, completeness, integrity, validity, timeliness, confidentiality, accuracy, or legitimacy of data, or ways that the EHR's user interface or automated clinical decision support tools may have contributed to the alleged events, one often turns to the EHR and its audit log. This manuscript discusses lines of defense incorporated into the design, development, implementation, and use of EHRs to ensure their integrity and the types of EHR transaction logs (e.g., audit log) that exist. Using these logs can help one answer questions that often arise in medical malpractice cases. Finally, there are “best practices” surrounding EHR audit logs that health care organizations should implement. When used appropriately, EHRs and their audit logs provide another source of information to help hospital risk managers, legal counsel, and EHR expert witnesses to investigate adverse incidents and, if needed, prosecute or defend clinicians and/or health care organizations involved in the patient's care.
Dean F. Sittig, Edward E. Yackel, and Hardeep Singh
Springer Science and Business Media LLC
Dean F Sittig, Aziz Boxwala, Adam Wright, Courtney Zott, Priyanka Desai, Rina Dhopeshwarkar, James Swiger, Edwin A Lomotan, Angela Dobes, and Prashila Dullabh
Oxford University Press (OUP)
AbstractThe design, development, implementation, use, and evaluation of high-quality, patient-centered clinical decision support (PC CDS) is necessary if we are to achieve the quintuple aim in healthcare. We developed a PC CDS lifecycle framework to promote a common understanding and language for communication among researchers, patients, clinicians, and policymakers. The framework puts the patient, and/or their caregiver at the center and illustrates how they are involved in all the following stages: Computable Clinical Knowledge, Patient-specific Inference, Information Delivery, Clinical Decision, Patient Behaviors, Health Outcomes, Aggregate Data, and patient-centered outcomes research (PCOR) Evidence. Using this idealized framework reminds key stakeholders that developing, deploying, and evaluating PC-CDS is a complex, sociotechnical challenge that requires consideration of all 8 stages. In addition, we need to ensure that patients, their caregivers, and the clinicians caring for them are explicitly involved at each stage to help us achieve the quintuple aim.
Daniel R Murphy, Andrew J Zimolzak, Divvy K Upadhyay, Li Wei, Preeti Jolly, Alexis Offner, Dean F Sittig, Saritha Korukonda, Riyaa Murugaesh Rekha, and Hardeep Singh
Oxford University Press (OUP)
Abstract Objective Measures of diagnostic performance in cancer are underdeveloped. Electronic clinical quality measures (eCQMs) to assess quality of cancer diagnosis could help quantify and improve diagnostic performance. Materials and Methods We developed 2 eCQMs to assess diagnostic evaluation of red-flag clinical findings for colorectal (CRC; based on abnormal stool-based cancer screening tests or labs suggestive of iron deficiency anemia) and lung (abnormal chest imaging) cancer. The 2 eCQMs quantified rates of red-flag follow-up in CRC and lung cancer using electronic health record data repositories at 2 large healthcare systems. Each measure used clinical data to identify abnormal results, evidence of appropriate follow-up, and exclusions that signified follow-up was unnecessary. Clinicians reviewed 100 positive and 20 negative randomly selected records for each eCQM at each site to validate accuracy and categorized missed opportunities related to system, provider, or patient factors. Results We implemented the CRC eCQM at both sites, while the lung cancer eCQM was only implemented at the VA due to lack of structured data indicating level of cancer suspicion on most chest imaging results at Geisinger. For the CRC eCQM, the rate of appropriate follow-up was 36.0% (26 746/74 314 patients) in the VA after removing clinical exclusions and 41.1% at Geisinger (1009/2461 patients; P < .001). Similarly, the rate of appropriate evaluation for lung cancer in the VA was 61.5% (25 166/40 924 patients). Reviewers most frequently attributed missed opportunities at both sites to provider factors (84 of 157). Conclusions We implemented 2 eCQMs to evaluate the diagnostic process in cancer at 2 large health systems. Health care organizations can use these eCQMs to monitor diagnostic performance related to cancer.
Siru Liu, Aileen P Wright, Barron L Patterson, Jonathan P Wanderer, Robert W Turer, Scott D Nelson, Allison B McCoy, Dean F Sittig, and Adam Wright
Oxford University Press (OUP)
Abstract Objective To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. Results Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. Conclusion AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.
Edna Shenvi, Aziz Boxwala, Dean Sittig, Courtney Zott, Edwin Lomotan, James Swiger, and Prashila Dullabh
Georg Thieme Verlag KG
Abstract Background Patient-centered clinical decision support (PC CDS) aims to assist with tailoring decisions to an individual patient's needs. Patient-generated health data (PGHD), including physiologic measurements captured frequently by automated devices, provide important information for PC CDS. The volume and availability of such PGHD is increasing, but how PGHD should be presented to clinicians to best aid decision-making is unclear. Objectives Identify best practices in visualizations of physiologic PGHD, for designing a software application as a PC CDS tool. Methods We performed a scoping review of studies of PGHD dashboards that involved clinician users in design or evaluations. We included only studies that used physiologic PGHD from single patients for usage in decision-making. Results We screened 468 titles and abstracts, 63 full-text papers, and identified 15 articles to include in our review. Some research primarily sought user input on PGHD presentation; other studies garnered feedback only as a side effort for other objectives (e.g., integration with electronic health records [EHRs]). Development efforts were often in the domains of chronic diseases and collected a mix of physiologic parameters (e.g., blood pressure and heart rate) and activity data. Users' preferences were for data to be presented with statistical summaries and clinical interpretations, alongside other non-PGHD data. Recurrent themes indicated that users desire longitudinal data display, aggregation of multiple data types on the same screen, actionability, and customization. Speed, simplicity, and availability of data for other purposes (e.g., documentation) were key to dashboard adoption. Evaluations were favorable for visualizations using common graphing or table formats, although best practices for implementation have not yet been established. Conclusion Although the literature identified common themes on data display, measures, and usability, more research is needed as PGHD usage grows. Ensuring that care is tailored to individual needs will be important in future development of clinical decision support.
Adam Wright, Richard Schreiber, David W Bates, Skye Aaron, Angela Ai, Raja Arul Cholan, Akshay Desai, Miguel Divo, David A Dorr, Thu-Trang Hickman,et al.
Oxford University Press (OUP)
Abstract Objective To improve problem list documentation and care quality. Materials and methods We developed algorithms to infer clinical problems a patient has that are not recorded on the coded problem list using structured data in the electronic health record (EHR) for 12 clinically significant heart, lung, and blood diseases. We also developed a clinical decision support (CDS) intervention which suggests adding missing problems to the problem list. We evaluated the intervention at 4 diverse healthcare systems using 3 different EHRs in a randomized trial using 3 predetermined outcome measures: alert acceptance, problem addition, and National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) clinical quality measures. Results There were 288 832 opportunities to add a problem in the intervention arm and the problem was added 63 777 times (acceptance rate 22.1%). The intervention arm had 4.6 times as many problems added as the control arm. There were no significant differences in any of the clinical quality measures. Discussion The CDS intervention was highly effective at improving problem list completeness. However, the improvement in problem list utilization was not associated with improvement in the quality measures. The lack of effect on quality measures suggests that problem list documentation is not directly associated with improvements in quality measured by National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) quality measures. However, improved problem list accuracy has other benefits, including clinical care, patient comprehension of health conditions, accurate CDS and population health, and for research. Conclusion An EHR-embedded CDS intervention was effective at improving problem list completeness but was not associated with improvement in quality measures.
Peter Cram, Elliot Cram, Joseph Antos, Dean F. Sittig, Ajay Anand, and Yue Li
Springer Science and Business Media LLC
Alan H Morris, Christopher Horvat, Brian Stagg, David W Grainger, Michael Lanspa, James Orme, Terry P Clemmer, Lindell K Weaver, Frank O Thomas, Colin K Grissom,et al.
Oxford University Press (OUP)
AbstractHow to deliver best care in various clinical settings remains a vexing problem. All pertinent healthcare-related questions have not, cannot, and will not be addressable with costly time- and resource-consuming controlled clinical trials. At present, evidence-based guidelines can address only a small fraction of the types of care that clinicians deliver. Furthermore, underserved areas rarely can access state-of-the-art evidence-based guidelines in real-time, and often lack the wherewithal to implement advanced guidelines. Care providers in such settings frequently do not have sufficient training to undertake advanced guideline implementation. Nevertheless, in advanced modern healthcare delivery environments, use of eActions (validated clinical decision support systems) could help overcome the cognitive limitations of overburdened clinicians. Widespread use of eActions will require surmounting current healthcare technical and cultural barriers and installing clinical evidence/data curation systems. The authors expect that increased numbers of evidence-based guidelines will result from future comparative effectiveness clinical research carried out during routine healthcare delivery within learning healthcare systems.
Dean F Sittig, Jodi D Sherman, Matthew J Eckelman, Andrew Draper, and Hardeep Singh
Oxford University Press (OUP)
Abstract Addressing environmental pollution and climate change is one of the biggest sociotechnical challenges of our time. While information technology has led to improvements in healthcare, it has also contributed to increased energy usage, destructive natural resource extraction, piles of e-waste, and increased greenhouse gases. We introduce a framework “Information technology-enabled Clinical cLimate InforMAtics acTions for the Environment” (i-CLIMATE) to illustrate how clinical informatics can help reduce healthcare’s environmental pollution and climate-related impacts using 5 actionable components: (1) create a circular economy for health IT, (2) reduce energy consumption through smarter use of health IT, (3) support more environmentally friendly decision-making by clinicians and health administrators, (4) mobilize healthcare workforce environmental stewardship through informatics, and (5) Inform policies and regulations for change. We define Clinical Climate Informatics as a field that applies data, information, and knowledge management principles to operationalize components of the i-CLIMATE Framework.
Xia Jing, Akash Indani, Nina Hubig, Hua Min, Yang Gong, James J. Cimino, Dean F. Sittig, Lior Rennert, David Robinson, Paul Biondich,et al.
Georg Thieme Verlag KG
Abstract Background MetaMap is a valuable tool for processing biomedical texts to identify concepts. Although MetaMap is highly configurative, configuration decisions are not straightforward. Objective To develop a systematic, data-driven methodology for configuring MetaMap for optimal performance. Methods MetaMap, the word2vec model, and the phrase model were used to build a pipeline. For unsupervised training, the phrase and word2vec models used abstracts related to clinical decision support as input. During testing, MetaMap was configured with the default option, one behavior option, and two behavior options. For each configuration, cosine and soft cosine similarity scores between identified entities and gold-standard terms were computed for 40 annotated abstracts (422 sentences). The similarity scores were used to calculate and compare the overall percentages of exact matches, similar matches, and missing gold-standard terms among the abstracts for each configuration. The results were manually spot-checked. The precision, recall, and F-measure (β =1) were calculated. Results The percentages of exact matches and missing gold-standard terms were 0.6–0.79 and 0.09–0.3 for one behavior option, and 0.56–0.8 and 0.09–0.3 for two behavior options, respectively. The percentages of exact matches and missing terms for soft cosine similarity scores exceeded those for cosine similarity scores. The average precision, recall, and F-measure were 0.59, 0.82, and 0.68 for exact matches, and 1.00, 0.53, and 0.69 for missing terms, respectively. Conclusion We demonstrated a systematic approach that provides objective and accurate evidence guiding MetaMap configurations for optimizing performance. Combining objective evidence and the current practice of using principles, experience, and intuitions outperforms a single strategy in MetaMap configurations. Our methodology, reference codes, measurements, results, and workflow are valuable references for optimizing and configuring MetaMap.
Adam Wright, Scott Nelson, David Rubins, Richard Schreiber, and Dean F Sittig
Oxford University Press (OUP)
Abstract Objective To identify common medication route-related causes of clinical decision support (CDS) malfunctions and best practices for avoiding them. Materials and Methods Case series of medication route-related CDS malfunctions from diverse healthcare provider organizations. Results Nine cases were identified and described, including both false-positive and false-negative alert scenarios. A common cause was the inclusion of nonsystemically available medication routes in value sets (eg, eye drops, ear drops, or topical preparations) when only systemically available routes were appropriate. Discussion These value set errors are common, occur across healthcare provider organizations and electronic health record (EHR) systems, affect many different types of medications, and can impact the accuracy of CDS interventions. New knowledge management tools and processes for auditing existing value sets and supporting the creation of new value sets can mitigate many of these issues. Furthermore, value set issues can adversely affect other aspects of the EHR, such as quality reporting and population health management. Conclusion Value set issues related to medication routes are widespread and can lead to CDS malfunctions. Organizations should make appropriate investments in knowledge management tools and strategies, such as those outlined in our recommendations.
RECENT SCHOLAR PUBLICATIONS
MOST CITED SCHOLAR PUBLICATIONS
CONSULTANCY
Expert witness for Electronic Health Record-related issues