Tools for immunization guideline knowledge maintenance. I. Automated generation of the logic "kernel" for immunization forecasting

Evaluation of a clinical decision support algorithm for patient-specific childhood immunization

OBJECTIVE

To evaluate the effectiveness of a clinical decision support system (CDSS) implementing standard childhood immunization guidelines, using real-world patient data from the Regenstrief Medical Record System (RMRS).

METHODS

Study subjects were age 6-years or younger in 2008 and had visited the pediatric clinic on the campus of Wishard Memorial Hospital. Immunization records were retrieved from the RMRS for 135 randomly selected pediatric patients. We compared vaccine recommendations from the CDSS for both eligible and recommended timelines, based on the child's date of birth and vaccine history, to recommendations from registered nurses who routinely selected vaccines for administration in a busy inner city hospital, using the same date of birth and vaccine history. Aggregated and stratified agreement and Kappa statistics were reported. The reasons for disagreement between suggestions from the CDSS and nurses were also identified.

RESULTS

For the 135 children, a total of 1215 vaccination suggestions were generated by nurses and were compared to the recommendations of the CDSS. The overall agreement rates were 81.3% and 90.6% for the eligible and recommended timelines, respectively. The overall Kappa values were 0.63 for the eligible timeline and 0.80 for the recommended timeline. Common reasons for disagreement between the CDSS and nurses were: (1) missed vaccination opportunities by nurses, (2) nurses sometimes suggested a vaccination before the minimal age and minimal waiting interval, (3) nurses usually did not validate patient immunization history, and (4) nurses sometimes gave an extra vaccine dose.

CONCLUSION

Our childhood immunization CDSS can assist providers in delivering accurate childhood vaccinations.

Maintaining and incrementally revalidating a computer-based clinical guideline: a case study

The paper explores the issues involved in maintaining the logic within a complex computer-based clinical guideline, using as a case study IMM/Serve, an operational guideline whose domain is childhood immunization. For a period of more than a year and a half, we have maintained a log of (1) the national changes to the immunization recommendations, (2) the local customizations of IMM/Serve's logic, and (3) certain logic problems that arose in the process of accommodating these changes and customizations. We describe the nature of these changes, customizations, and problems. We also discuss how different types of domain knowledge might assist in the automated process of validating successive versions of the logic. The paper's goal is to use the immunization domain to provide specific examples of the issues and problems that arise in maintaining a computer-based clinical guideline.

Tools for immunization guideline knowledge maintenance. II. Automated Web-based generation of user-customized test cases

IMM/Test is a prototype software tool built to generate test cases that can be used to help test and verify the internal logic of an immunization forecasting program. A forecasting program takes as input a child's immunization history and produces recommendations as to which vaccinations are due and which should be scheduled next. IMM/Test was developed to test a specific immunization forecasting program, IMM/Serve. In addition, IMM/Test has been incorporated into a broader Web-based tool, IMM/Web, which allows the user (e.g., a member of an immunization registry staff) to customize the parameters used for immunization forecasting (e.g., the minimum ages for each dose and the minimum wait intervals between doses) to reflect local practice. IMM/Web then generates a customized set of test cases that may be used to test the user's immunization forecasting program. The user may also request that the test cases be automatically passed to IMM/Serve to analyze using the newly defined parameters. The paper describes the internal design of IMM/Test and IMM/Web and discusses certain lessons learned in the implementation of the two programs.