Kddm

Chapter 4 Knowledge Discovery, Data Mining, and Practice-Based Evidence

Mollie R. Cummins

Ginette A. Pepper

Susan D. Horn

The next step to comparative effectiveness research is to conduct more prospective large-scale observational cohort studies with the rigor described here for knowledge discovery and data mining (KDDM) and practice-based evidence (PBE) studies.

Objectives

At the completion of this chapter the reader will be prepared to:

1.Define the goals and processes employed in knowledge discovery and data mining (KDDM) and practice-based evidence (PBE) designs

2.Analyze the strengths and weaknesses of observational designs in general and of KDDM and PBE specifically

3.Identify the roles and activities of the informatics specialist in KDDM and PBE in healthcare environments

Key Terms

Comparative effectiveness research, 69

Confusion matrix, 62

Data mining, 61

Knowledge discovery and data mining (KDDM), 56

Machine learning, 56

Natural language processing (NLP), 58

Practice-based evidence (PBE), 56

Preprocessing, 56

Abstract

The advent of the electronic health record (EHR) and other large electronic datasets has revolutionized efficient access to comprehensive data across large numbers of patients and the concomitant capacity to detect subtle patterns in these data even with missing or less than optimal data quality. This chapter introduces two approaches to knowledge building from clinical data: (1) knowledge discovery and data mining (KDDM) and (2) practice-based evidence (PBE). The use of machine learning methods in retrospective analysis of routinely collected clinical data characterizes KDDM. KDDM enables us to efficiently and effectively analyze large amounts of data and develop clinical knowledge models for decision support. PBE integrates health information technology (health IT) products with cohort identification, prospective data collection, and extensive front-line clinician and patient input for comparative effectiveness research. PBE can uncover best practices and combinations of treatments for specific types of patients while achieving many of the presumed advantages of randomized controlled trials (RCTs).

Introduction

Leaders need to foster a shared learning culture for improving healthcare. This extends beyond the local department or institution to a value for creating generalizable knowledge to improve care worldwide. Sound, rigorous methods are needed by researchers and health professionals to create this knowledge and address practical questions about risks, benefits, and costs of interventions as they occur in actual clinical practice. Typical questions are as follows:

•Are treatments used in daily practice associated with intended outcomes?

•Can we predict adverse events in time to prevent or ameliorate them?

•What treatments work best for which patients?

•With limited financial resources, what are the best interventions to use for specific types of patients?

•What types of individuals are at risk for certain conditions?

Answers to these questions can help clinicians, patients, researchers, healthcare administrators, and policy-makers to learn from and improve real-world, everyday clinical practice. Two important emerging approaches to knowledge building from clinical data are KDDM and PBE, both of which are described in this chapter.

Research Designs for Knowledge Discovery

The gold standard research design for answering questions about the efficacy of treatments is an experimental design, often referred to as a randomized controlled trial (RCT). An RCT requires random assignment of patients to treatment condition as well as other design features, such as tightly controlled inclusion criteria, to assure as much as posible that the only difference between the experimental and control groups is the treatment (or placebo) that each group receives. The strength of the RCT is the degree of confidence in causal inferences, in other words, that the therapeutic intervention caused the clinical effects (or lack of effects). Drawbacks of the RCT include the time and expense required to conduct a comparison of a small number of treatment options and the limited generalizability of the results to patients, settings, intervention procedures, and measures that differ from the specific conditions in the study condition. Further, RCTs have little value in generating unique hypotheses and possibilities.

Observational research designs can also yield valuable information to characterize disease risk and generate hypotheses about potentially effective treatments. In addition, observational research is essential to determine the effectiveness of treatments or how well treatments work in actual practice. In observational studies the investigator merely records what occurs under naturalistic conditions, such as which individual gets what therapy and what outcomes result or which variables are associated with what outcomes. Of course, with observational studies the patients who receive different treatments generally differ on many other variables (selection bias) since treatments were determined by clinician judgment rather than random assignment and selection. For example, one therapy may be prescribed for sicker patients under natural conditions or may not be accessible to uninsured patients. Since diagnostic approaches vary in clinical practice, patients with the same diagnosis may have considerable differences in the actual condition.

Observational studies can be either prospective (data are generated after the study commences) or retrospective (data were generated before the study). Chart review has traditonally been the most common approach to retrospective observational research. However, chart review previously required tedious and time-consuming data extraction and the requisite data may be missing, inconsistent, or of poor quality. Prospective studies have the advantage that the measurements can be standardized, but recording both research data and clinical data constitutes a documentation burden for clinicians that cannot be accommodated in typical clinical settings unless the research and clinical data elements are combined to become the standard for documentation.

EHRs and Knowledge Discovery

The advent of the EHR and other large electronic datasets has revolutionized observational studies by increasing the potential for efficient access to comprehensive data, reflecting large numbers of patients and the capacity to detect subtle patterns in the data, even with missing or less than optimal data quality. With very large samples available from EHRs at relatively low cost, it is often possible to compensate with statistical controls for the lack of randomization in the practice setting. With electronic data, standardized data collection is facilitated and data validity can be enhanced, minimizing the documentation burden by using clinical data for research purposes.

Increased adoption of EHRs and other health information systems has resulted in vast amounts of structured and textual data. Stored on servers in a data warehouse (a large data repository integrating data across clinical, administrative, and other systems), the data may be a partial or complete copy of all data collected in the course of care provision. The data can include billing information, physician and nursing notes, laboratory results, radiology images, and numerous other diverse types of data. In some settings data describing individual patients and their characteristics, health issues, treatments, and outcomes has accumulated for years, forming longitudinal records. The clinical record can also be linked to repositories of genetic or familial data.1–3 These data constitute an incredible resource that is underused for scientific research in biomedicine and nursing.

The potential of using these data stores for the advancement of scientific knowledge and patient care is widely acknowledged. However, the lack of availability of tools and technology to adequately manage the data deluge has proven to be an Achilles' heel. Very large data resources, typically on the terabyte scale or larger, require highly specialized approaches to storage, management, extraction, and analysis. Moreover, the data may not be useful. Data quality can be poor and require substantial additional processing prior to use.

Clinical concepts are typically represented in the EHR in a way that supports healthcare delivery but not necessarily research. For example, pain might be qualitatively described in a patient's note and EHR as “mild” or “better.” This may meet the immediate need for documentation and care but it does not allow the researcher to measure differences in pain over time and across patients, as would measurement using a pain scale. Clinical concepts may not be adequately measured or represented in a way that enables scientific analysis. Data quality affects the feasibility of secondary analysis.

TABLE 4-1 Characteristics of Knowledge Discovery and Data Mining (KDDM) and Practice-Based Evidence (PBE)

Characteristic KDDM PBE

Description Application of machine learning and statistical methods for pattern discovery

Participatory research approach requiring documentation of predefined process and outcome data and analysis

Goal Develop models to predict future events or infer missing information

Determine the effectiveness of multiple interventions on multiple outcomes in actual practice environment

Design classification Observational (descriptive) Observational (descriptive)

Temporal aspects Retrospective Prospective

Typical sample size 1000-1,000,000 or more, depending on project and available data

800-2000+

Knowledge Building Using Health IT

Two observational approaches to knowledge building from health IT can be employed for research and clinical performance improvement. One approach is based on machine learning applied to retrospective analysis of routinely collected clinical data and a second approach is based on increasing integration of health IT with cohort identification, front-line knowledge, and prospective data collection for research and clinical care.

Knowledge discovery and data mining (KDDM), the first approach, uses pattern discovery in large amounts of clinical and biomedical data and entails the use of software tools that facilitate the extraction, sampling, and large-scale cleaning and preprocessing of data. KDDM also makes use of specialized analytic methods, characteristically machine learning methods, to identify patterns in a semiautomated fashion. This level of analysis far exceeds the types of descriptive summaries typically presented by dashboard applications, such as a clinical summary for a patient. Instead, KDDM is used to build tools that support clinical decision making, generate hypotheses for scientific evaluation, and identify links between genotype and phenotype. KDDM can also be used to “patch” weaknesses in clinical data that pose a barrier to research. For example, if poor data quality is a barrier to automatic identification of patients with type II diabetes from diagnostic codes, a machine learning approach could be used to more completely and accurately identify the patients on the basis of text documents and laboratory and medication data.

Practice-based evidence (PBE) is an example of the second approach. PBE studies are observational cohort studies that attempt to mitigate the weaknesses traditionally associated with observational designs. This is accomplished by exhaustive attention to determining patient characteristics that may confound conclusions about the effectiveness of an intervention.

For example, observational studies might indicate that aerobic exercise is superior to nonaerobic exercise in preventing falls. But if the prescribers tend to order nonaerobic exercise for those who are more debilitated, severity of illness is a confounder and should be controlled in the analysis. PBE studies use large samples and diverse sources of patients to improve sample representativeness, power, and external validity. Generally there are 800 or more subjects, which is considerably more than in a typical RCT but far less than in a KDDM study. PBE uses approaches similar to community-based participatory research by including front-line clinicians and patients in the design, execution, and analysis of studies, as well as their data elements, to improve relevance to real-world practice. Finally, PBE uses detailed standardized structured documentation of interventions, which is ideally incorporated into the standard electronic documentation.

This method requires training and quality control checks for reliability of the measures of the actual process of care. Statistical analysis involves determining bivariate and multivariate correlations among patient characteristics, intervention process steps, and outcomes. PBE can uncover best practices and combinations of treatments for specific types of patients while achieving many of the presumed advantages of RCTs, especially the presumed advantage that RCTs control for patient differences through randomization. Front-line clinicians treating the study patients lead the study design and analyses of the data prospectively based on clinical expertise, rather than relying on machines to detect patterns as in KDDM. The characteristics of KDDM and PBE are summarized in Table 4-1. Both techniques are detailed in the following sections.

Knowledge Discovery and Data Mining

KDDM is a process in which machine learning and statistical methods are applied to analyze large amounts of data. Frequently, the goal of analysis is to develop models that predict future events or infer missing information based on available data. Methods of KDDM are preferred for this type of endeavor because they are effective for analyzing very large repositories of clinical data and for analyzing complex, nonlinear relationships. Models developed on the basis of routinely collected clinical data are advantageous for several reasons:

1.KDDM models access and leverage the valuable information contained in large repositories of clinical data

2.Models can be developed from very large sample sizes or entire populations

3.Models based on routinely collected data can be implemented in computerized systems to support decision making for individual patients

4.Models induced directly from data using machine learning methods often perform better than models manually developed by human experts

For example, Walton and colleagues developed a model that forecasts an impending respiratory syncytial virus (RSV) outbreak.4 RSV is a virus that causes bronchiolitis in children, and severe cases warrant hospitalization. RSV outbreaks cause dramatic increases in census at pediatric hospitals, so advance warning of an impending RSV outbreak would allow pediatric hospitals to plan staffing and supplies. Some evidence indicates that weather is related to outbreaks of RSV and RSV outbreaks are known to follow a biennial pattern, information that may be useful for predicting outbreaks in advance. Given these circumstances the authors built a model using historical data that predicts RSV outbreaks up to 3 weeks in advance. These types of models can be especially effective in designing clinical decision support (CDS) systems. CDS systems are computer applications that assist healthcare providers in making clinical decisions about patients and are explained in detail in Chapter 10. The design of individual CDS systems varies and can be as simple as an alert that warns about potential drug–drug interaction.5 Every CDS system is based on some underlying algorithm or rules and on existing or entered patient data. These rules must be specified in machine-readable code that is compatible with patient data stored in an EHR or other applications. Historically, clinical practice guidelines have not been expressed as a set of adequately explicit rules and could not be executed by a machine. See, for example, Lyng and Pederson and Isern and Moreno for a detailed discussion of this issue.6,7 While a human being can reason on the basis of conditions such as “moderate improvement” or “diminished level of consciousness,” a machine cannot. CDS models must consist of rules, conditions, and dependencies described in terms of machine-interpretable relationships and specific data values. Moreover, the algorithms and rules must be executable over the data as they are coded in the information system.

For example, gender may be included in a set of rules. If the rule is based on a gender variable coded with the values male, female, or unknown, it will not work in a system where gender is coded as 0, 1, 2, 3, or null, where 0 = male, 1 = female, 2 = transgender, 3 = unknown, and null = missing. While relatively simple changes could adapt the rule set for use in a system with different coding of gender, other variables pose a greater challenge. Some necessary variables may not exist as coded data in an information system or may be represented in a variety of ways that cannot be resolved as easily as gender can be.

In recent years there has been a substantial effort to develop computer-interpretable guidelines—guidelines that are expressed as an adequately explicit set of rules—with some success.8 KDDM is also advantageous in this situation because it develops only machine-executable algorithms or rules, based on native data. Every model could potentially be used in a CDS system. Moreover, in situations where there is insufficient evidence to fully specify rules, the rules can be induced from a large sample of real-life examples using KDDM.

Retrieving a Dataset for Analysis

The process of KDDM, depicted in Figure 4-1, encompasses multiple steps and actions. Data must first be extracted from the clinical data warehouse. To review, data warehouses are a complex, vast collection of databases and it is usually necessary to join a number of tables to construct a usable dataset for KDDM purposes. To accomplish this investigators must collaborate closely with informatics specialists to develop effective queries, queries that select the clinical data relevant to the specific KDDM project with a sufficient but not overwhelming sample size.

To request the appropriate data, investigators and clinicians first need to understand how the concepts of interest are represented (coded) in the health IT product. In many health IT products, for example, laboratory tests are coded according to the standard Logical Observation Identifier Names and Codes (LOINC) terminology.9 To ensure that the extracted dataset contains urinalysis data, for example, it will be necessary to first determine how and where a urinalysis is coded. For example, in the case of Veterans Health Administration (VHA) data, this may entail identification of the LOINC codes used to represent urinalysis results. For less precise concepts, such as mental health diagnoses, multiple codes may be relevant. Some data may not be structured and may be captured only in text documents such as discharge summaries. Information extraction from these documents can be accomplished and represents an active area of research and development.10

Queries written in a specialized programming language (Structured Query Language or SQL) are used to retrieve data from the data warehouse according to a researcher's specifications. Currently, investigators and healthcare organization IT personnel collaborate to develop effective queries. The code used to execute the query is saved as a file and can be reused in the future or repeatedly reused on a scheduled basis. In some cases healthcare organizations opt to support ongoing investigator data needs by creating separate repositories of aggregated, processed clinical data that relate to a particular clinical domain. In the VHA, investigators in infectious disease have developed procedures to aggregate a specialized set of nationwide patient data related to methicillin-resistant Staphylococcus aureus (MRSA).11 These specialized repositories of data can be more readily analyzed on an ongoing basis to support quality improvement, health services research, and clinical research.

The amount of data retrieved from clinical data warehouses can be enormous, especially when data originate from multiple sites. Investigators will want to define a sampling plan that limits the number of selected records, according to the needs of the study or project. For KDDM, it may not be possible to import the data fully into analytic software as a single flat file. Fortunately, many statistical and machine learning software packages can be used to analyze data contained within an SQL database. For example, SAS Enterprise Miner can be used to analyze data within an SQL database using Open Database Connectivity (ODBC).12 Clinicians or investigators who are new to KDDM should plan to collaborate with statistical and informatics personnel to plan an optimal approach.

Preprocessing Clinical Data

EHRs include both coded (structured) data and text data that must be cleaned and processed prior to analysis. EHRs collect and store data according to a coding system consisting of one or more terminologies (Box 4-1). While standard terminologies exist, many systems make use of a local terminology, a distinct set of variables, and a distinct coding system for those variables that are not necessarily shared across systems. Different sites, clinics, or hospitals within a healthcare organization could use different terminologies, coding data in different ways. Within a single site, changes in information systems and terminologies over time can also result in variations in data coding. When data are aggregated across time and across sites, the variations in terminology result in a dataset that

C:\Users\chanda\AppData\Local\Temp\978-0-323-10095-3_0015.jpg

FIG 4-1 Steps of the knowledge discovery and data mining (KDDM) process.

represents similar concepts in multiple ways. For example, one large healthcare organization recognized that within its clinical data the relatively simple concepts of “yes” and “no” were represented using 30 unique coding schemes.13 Unlike data collected using a prospective approach, clinical data often require extensive cleaning and preprocessing. Thus preprocessing constitutes the majority of effort in the clinical KDDM process shown in Figure 4-1.

Box 4-1 Tools for Processing Clinical Data

The U.S. National Library of Medicine maintains a repository of the many available informatics tools, called the Online Registry of Biomedical Informatics Tools (ORBIT) Project, at http://orbit.nlm.nih.gov/. However, given the complexities of processing clinical data with these tools, investigators should consider collaboration or consultation with an informatics specialist versed in these techniques.

Preprocessing Text Data

In clinical records, the richest and most descriptive data are often unstructured, captured only in the text notes entered by clinicians. Text data can be analyzed in a large number of clinical records using a specialized approach known as natural language processing (NLP) or, more specifically, information extraction.14 Methods of information extraction identify pieces of meaningful information in sequences of text, pieces of information that represent concepts and can be coded as such for further analysis. Machine interpretation of text written in the form of natural language is not straightforward because natural language is rife with spelling errors, acronyms, and abbreviations, among other issues.14 Consequently, information extraction is usually a computationally expensive, multistep process in which text data are passed through a pipeline of sequential NLP procedures. These procedures deal with common NLP challenges such as word disambiguation and negation and may involve the use of machine learning methods. However, each pipeline may differ according to the NLP task at hand.15 Unstructured Information Management Architecture (UIMA) (http://uima.apache.org) is one example of an NLP pipeline framework. Information extraction for clinical text is an active area of research and development. However, information extraction tools are not commonly used outside of informatics research settings and text data are included infrequently in KDDM projects. This technique is another example of an area where researchers or clinicians need to partner with informaticists.

Preprocessing Coded (Structured) Data

In a set of consistently coded clinical data, the data should be analyzed using descriptive statistics and visualization with respect to the following:

•Distribution: Normally distributed data are most amenable to modeling. If the data distribution is skewed, the data can be transformed using a function of the original data or using nonparametric statistical methods.

•Frequency: The frequency of specific values for categorical variables may reveal a need for additional preprocessing. It is not uncommon for identical concepts to be represented using multiple values. Also, some values are so rare that their exclusion from analysis should be considered.

•Missingness: Missingness can be meaningful. For example, a missing hemoglobin A1c (HgA1c) laboratory test may indicate that a patient does not have diabetes. In that case, a binary variable indicating whether or not HgA1c values are missing can be added to the dataset. In other circumstances, the values are simply missing at random. If values are missing at random, they can be replaced using a number of statistical imputation approaches.

•Sparsity: Sparse data are data for which binary values are mostly zero. Categorical variables with a large number of possible values contribute to sparsity. For example, a field called “primary diagnosis” has a set of possible values equal to the number of diagnoses found in the ICD-9 coding system. Upon 1 of n encoding, the number of possible values becomes the number of new columns added to the dataset. Some diagnoses will be more common than others. For uncommon diagnoses the value of “primary diagnosis” will almost always equal zero. The value of “1” will be found in only a small percentage of records.

•Outliers: Outliers, data points that fall far outside the distribution of data, should be considered for elimination or further analysis prior to modeling.

•Identifiers: Codes or other values that uniquely identify patients should be excluded from the modeling process.

•Erroneous data: Absurd, impossible data values are routinely found in clinical data. These can be treated as randomly missing values and replaced.

Descriptive analysis is facilitated by many software packages. For example, Figure 4-2 depicts a screenshot from Weka, a freely available data mining software package.16 In this software, when a variable from the dataset is selected, basic descriptive statistics and a graph of the frequency distribution are displayed. A variety of filters can then be applied to address issues with the data.

The considerations in preprocessing the data at this stage are numerous and readers are referred to an excellent text by Dorian Pyle, Data Preparation for Data Mining.17 Preprocessing is always best accomplished through a joint effort by the analyst and one or more domain experts, such as clinicians who are familiar with the concepts the data represent. The domain experts can lend valuable insight to the analyst, who must develop an optimal representation of each variable. Review of the data at this point may reveal conceptual gaps, the absence of data, or the lack of quality data that represent important concepts. For example, age and functional status (e.g., activities of daily living) might be important data to include in a project related to the prediction of patient falls in the hospital. By mapping concepts to variables, or vice versa, teams can communicate about gaps and weaknesses in the data.

Sampling and Partitioning