Importance of aligning the implementation of new payment models for innovative pharmaceuticals in European countries

INTRODUCTION

The uptake of complex technologies and platforms has resulted in several challenges in the pricing and reimbursement of innovative pharmaceuticals. To address these challenges, plenty of concepts have already been described in the scientific literature about innovative value judgment or payment models, which are either (1) remaining theoretical; or (2) applied only in pilots with limited impact on patient access; or (3) applied so heterogeneously in many different countries that it prevents the health care industry from meeting expectations of HTA bodies and health care payers in the evidence requirements or offerings in different jurisdictions.

AREAS COVERED

This paper provides perspectives on how to reduce the heterogeneity of pharmaceutical payment models across European countries in five areas, including 1) extended evaluation frameworks, 2) performance-based risk-sharing agreements, 3) pooled procurement for low volume or urgent technologies, 4) alternative access schemes, and 5) delayed payment models for technologies with high upfront costs.

EXPERT OPINION

Whilst pricing and reimbursement decisions will remain a competence of EU member states, there is a need for alignment of European pharmaceutical payment model components in critical areas with the ultimate objective of improving the equitable access of European patients to increasingly complex pharmaceutical technologies.

Global Perceptions on Social Accountability and Outcomes: A Survey of Medical Schools

Phenomenon: Social accountability has become a universal component in medical education. However, medical schools have little guidance for operationalizing and applying this concept in practice. This study explored institutional practices and administrative perceptions of social accountability in medical education. Approach: An online survey was distributed to a purposeful sample of English-speaking undergraduate medical school deans and program directors/leads from 245 institutions in 14 countries. The survey comprised of 38-items related to program mission statements, admission processes, curricular content, and educational outcomes. Survey items were developed using previous literature and categorized using a context-input-process-products (CIPP) evaluation model. Exploratory Factor Analysis (EFA) was used to assess the inter-relationship among survey items. Reliability and internal consistency of items were evaluated using McDonald's Omega. Findings: Results from 81 medical schools in 14 countries collected between February and June 2020 are presented. Institutional commonalities of social accountability were observed. However, our findings suggest programs focus predominately on educational inputs and processes, and not necessarily on outcomes. Findings from our EFA demonstrated excellent internal consistency and reliability. Four-factors were extracted: (1) selection and recruitment; (2) institutional mandates; (3) institutional activities; and (4) community awareness, accounting for 71% of the variance. McDonald's Omega reliability estimates for subscales ranged from 0.80-0.87. Insights: This study identified common practices of social accountability. While many medical schools expressed an institutional commitment to social accountability, their effects on the community remain unknown and not evaluated. Overall, this paper offers programs and educators a psychometrically supported tool to aid in the operationalization and reliability of evaluating social accountability.

Adapting Clinical Guidelines for the Digital Age: Summary of a Holistic and Multidisciplinary Approach

The Centers for Disease Control and Prevention's Adapting Clinical Guidelines for the Digital Age initiative aims to redesign and improve guideline development, implementation, and standardization. Historically, aspects of guideline development and implementation have been siloed. This leads to long lag times for guidelines to reach patient care, unnecessary redundancy, and potential for misinterpretation, leading to inconsistencies in how the recommendations are applied. A multidisciplinary, multiorganizational holistic approach brought together experts in guideline development, informatics, communication, implementation, and evaluation to understand and identify problems in guideline development and implementation, define an ideal state with no constraints, and then design a future state that advances the process close to the ideal state. The Adapting Clinical Guidelines for the Digital Age workgroups each worked on one focus area and included experts from the other areas to help analyze the current state and develop holistic solutions for the future state. Each workgroup produced interrelated standards, processes, and tools that can be used across the continuum of guideline development and implementation.

An Evaluation Framework for a Novel Process to Codevelop Written and Computable Guidelines

Clinical practice guidelines (CPGs) support individual and population health by translating new, evidence-based knowledge into recommendations for health practice. CPGs can be provided as computable, machine-readable guidelines that support the translation of recommendations into shareable, interoperable clinical decision support and other digital tools (eg, quality measures, case reports, care plans). Interdisciplinary collaboration among guideline developers and health information technology experts can facilitate the translation of written guidelines into computable ones. The benefits of interdisciplinary work include a focus on the needs of end-users who apply guidelines in practice through clinic decision support systems as part of the Centers for Disease Control and Prevention's (CDC's) Adapting Clinical Guidelines for the Digital Age (ACG) initiative, a group of interdisciplinary experts proposed a process to facilitate the codevelopment of written and computable CPGs, referred to as the "integrated process (IP)."1 This paper presents a framework for evaluating the IP based on a combination of vetted evaluation models and expert opinions. This framework combines 3 types of evaluations: process, product, and outcomes. These evaluations assess the value of interdisciplinary expert collaboration in carrying out the IP, the quality, usefulness, timeliness, and acceptance of the guideline, and the guideline's health impact, respectively. A case study is presented that illustrates application of the framework.

Assessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework

Artificial intelligence (AI) software in radiology is becoming increasingly prevalent and performance is improving rapidly with new applications for given use cases being developed continuously, oftentimes with development and validation occurring in parallel. Several guidelines have provided reporting standards for publications of AI-based research in medicine and radiology. Yet, there is an unmet need for recommendations on the assessment of AI software before adoption and after commercialization. As the radiology AI ecosystem continues to grow and mature, a formalization of system assessment and evaluation is paramount to ensure patient safety, relevance and support to clinical workflows, and optimal allocation of limited AI development and validation resources before broader implementation into clinical practice. To fulfil these needs, we provide a glossary for AI software types, use cases and roles within the clinical workflow; list healthcare needs, key performance indicators and required information about software prior to assessment; and lay out examples of software performance metrics per software category. This conceptual framework is intended to streamline communication with the AI software industry and provide healthcare decision makers and radiologists with tools to assess the potential use of these software. The proposed software evaluation framework lays the foundation for a radiologist-led prospective validation network of radiology AI software. Learning Points: The rapid expansion of AI applications in radiology requires standardization of AI software specification, classification, and evaluation. The Canadian Association of Radiologists' AI Tech & Apps Working Group Proposes an AI Specification document format and supports the implementation of a clinical expert evaluation process for Radiology AI software.

The USMLE® STEP 1 Pass or Fail Era of the Vascular Surgery Residency Application Process: Implications for Structural Bias and Recommendations

MOTIVES BEHIND THE SHIFT

USMLE® STEP 1 score reporting has been changed to a binary pass/fail format since January 26, 2022. The motives behind this change were (1) the questionable validity of using USMLE® STEP 1 as a screening tool during the candidate selection process and (2) the negative impact of using standardized examination scores as an initial gatekeeping threshold for the underrepresented in medicine (URiM) candidates applying to GME programs, given their generally lower mean standardized exams scores compared to non-URiM students. The USMLE® administrators justified this change as a tactic to enhance the overall educational experience for all students and to increase the representation of URiM groups. Moreover, they advised the program directors (PDs) to give more attention to other important qualities and components such as the applicant's personality traits, leadership roles and other extracurricular accomplishments, as part of a holistic evaluation strategy. IMPACT OF USMLE® STEP 1 PASS/FAIL: At this early stage, it is unclear how this change will impact Vascular Surgery Integrated residency (VSIR) programs. Several questions are outstanding, most importantly, how VSIR PDs will evaluate applicants absent the variable which heretofore was the primary screening tool. Our previously published survey showed that VSIR PDs will move their attention to other measures such as USMLE® STEP 2CK and letters of recommendation during the VSIR selection process. Furthermore, more emphasis on subjective measures such as the applicant's medical school rank and extracurricular student activities is expected. Given the expected higher weight of USMLE® STEP 2CK in the selection process than ever, many anticipate that medical students will dedicate more of their limited time to its preparation at the expense of both clinical and non-clinical activities. Potentially leaving less time to explore specialty pathways and to determine whether VS is the appropriate career for them.

A FRAMEWORK FOR CANDIDATE EVALUATION

The critical juncture in the VSIR candidate evaluation paradigm presents an opportunity to thoughtfully transform the process via current (Standardized Letter of Recommendation, USMLE® STEP 2CK, and clinical research) and future (Emotional Intelligence, Structure Interview and Personality Assessment) measures which constitute a framework to follow in the USMLE® STEP 1 pass/fail era.

A Framework for Evaluating the Implementation of Biosimilar Drugs

Background

The introduction of biosimilar drugs has significant effects on health care systems, and a variety of approaches are required to support acceptance, adoption, and use of these drugs. Literature exists on the enablers of, and barriers to, biosimilar implementation, but frameworks that support the evaluation of biosimilar implementation strategies are currently lacking.

Objective

To develop an evaluation framework for assessing the effects of biosimilar implementation strategies on patients, clinicians, and publicly funded drug programs.

Methods

The scope of the evaluation was determined by a pan-Canadian working group through the creation of a logic model of activities and expected outcomes associated with biosimilar implementation. Each component of the logic model was considered under the RE-AIM framework, which led to a set of evaluation questions and indicators. Feedback to inform the final framework was sought from stakeholders through focus group sessions and written responses.

Results

An evaluation framework was created that articulates evaluation questions and indicators across 5 priority areas: stakeholder engagement, patient experience, patient outcomes, clinician experience, and system sustainability and affordability. Stakeholder feedback was obtained through 9 focus group sessions with a total of 87 participants. Feedback was used to refine the framework on the basis of stakeholder priorities and feasibility.

Conclusions

Through extensive stakeholder consultation, an evaluation framework was developed to measure and monitor the effects of biosimilar implementation on the 5 identified priority areas, as well as to inform future biosimilar implementations. This framework can be used as a starting point for evaluating the implementation of biosimilars across health care systems.

Role of innovation in pharmaceutical regulation: A proposal for principles to evaluate EU General Pharmaceutical Legislation from the innovator perspective

Because the EU General Pharmaceutical Legislation is under review, the EFPIA Innovation Board developed evaluation principles for the policy proposals and key considerations on how the regulatory framework can support innovation while ensuring only safe, efficacious and quality medicines are authorized. The evaluation principles are anchored on actions to promote: agile adoption of new methodologies with soft law tools; continued emphasis on regulatory science to inform policies; a cost/benefit assessment of the new regulation to ensure they have an overall positive impact; and mitigation of any negative externalities or unintended effects for any type of innovation or products. The evaluation principles are intended to guide the impact assessment of the pharmaceutical legislation in the EU but the principles can be applied globally.

How does the structure of data impact cell-cell similarity? Evaluating how structural properties influence the performance of proximity metrics in single cell RNA-seq data

Accurately identifying cell-populations is paramount to the quality of downstream analyses and overall interpretations of single-cell RNA-seq (scRNA-seq) datasets but remains a challenge. The quality of single-cell clustering depends on the proximity metric used to generate cell-to-cell distances. Accordingly, proximity metrics have been benchmarked for scRNA-seq clustering, typically with results averaged across datasets to identify a highest performing metric. However, the 'best-performing' metric varies between studies, with the performance differing significantly between datasets. This suggests that the unique structural properties of an scRNA-seq dataset, specific to the biological system under study, have a substantial impact on proximity metric performance. Previous benchmarking studies have omitted to factor the structural properties into their evaluations. To address this gap, we developed a framework for the in-depth evaluation of the performance of 17 proximity metrics with respect to core structural properties of scRNA-seq data, including sparsity, dimensionality, cell-population distribution and rarity. We find that clustering performance can be improved substantially by the selection of an appropriate proximity metric and neighbourhood size for the structural properties of a dataset, in addition to performing suitable pre-processing and dimensionality reduction. Furthermore, popular metrics such as Euclidean and Manhattan distance performed poorly in comparison to several lessor applied metrics, suggesting that the default metric for many scRNA-seq methods should be re-evaluated. Our findings highlight the critical nature of tailoring scRNA-seq analyses pipelines to the dataset under study and provide practical guidance for researchers looking to optimize cell-similarity search for the structural properties of their own data.

Implementation of Interventions and Policies on Opioids and Awareness of Opioid-Related Harms in Canada: A Multistage Mixed Methods Descriptive Study

In Canada, interventions and policies have been implemented to minimize the risk of opioid-related harms. This mixed methods study aimed at describing trends over time in implementation, as well as in awareness and health outcomes. For implementation, we conducted a scoping review to identify opioids interventions and policies implemented in Canada between 1 January 2016 and 15 November 2019. Awareness was measured through a descriptive analysis of opioid-related harm cases reported by consumers and health care professionals (HCPs) to the national spontaneous reporting system and of social media coverage, while health outcome consisted of opioid-related deaths recorded in the coroner's reports database of the province of Quebec, Canada. Trends over time in implementation of interventions were compared to trends in awareness and opioid-related deaths, without implying causality. There were 413 national or provincial interventions on opioids implemented over the study period, with a four-fold increase in 2016. The most common (31.5%) was harm reduction strategies, such as naloxone distribution. The reporting rate of opioid-related harms ranged between 0.1 and 0.2 per 100,000 persons with no observed time trend. Compared to 2015, the number of social media posts increased in 2016 by 35.4% (Reddit), 329.0% (Facebook), and 381.5% (Twitter). Between 2016 and 2019, there was a slight decrease in the number of opioid-related deaths recorded in the coroner's database. Overall, the increase in the number of policies did not see a parallel increase in spontaneous reports of opioid-related harms as an indicator of consumer or HCP awareness. Conversely, the dramatic increase in social media coverage was consistent with heightened public awareness. Although no inferences of causality were made in this study, the decrease in opioid-related deaths observed in the recent years may indicate a potential effectiveness of interventions and policies.

Development of an Advance Care Planning Policy within an Evidenced-Based Evaluation Framework

Background: As the population is aging and medical advancements enable people to live longer, advance care planning (ACP) becomes increasingly important in guiding future care decisions; however, they are often incomplete or absent from the patient chart. This study describes the development and implementation of an ACP policy in a post-acute care and long-term care setting using a systematic implementation framework. Methods: A process evaluation that parallels the Replicating Effective Programs (REP) framework was used to understand stakeholder experiences with ACP and identify gaps in practice. Physicians, multidisciplinary staff, patients, and substitute decision makers engaged in focus groups and interviews, and completed surveys. A retrospective chart review determined Plan for Life Sustaining Treatment (PLST) form completion rates. Results: Stakeholder feedback identified barriers and facilitators to ACP including a need for staff training, user-friendly resources, and standardization of ACP practice. The PLST form was developed and embedded in the electronic medical record, and had a 92% and an 87% PLST completion rate on 2 pilot units. Conclusion: The study showed the usefulness of the REP model in guiding the evaluation as an effective tool to enhance implementation practices and inform ACP policy development that can be replicated in other organizations.

Participatory Systems Modelling for Youth Mental Health: An Evaluation Study Applying a Comprehensive Multi-Scale Framework

The youth mental health sector is persistently challenged by issues such as service fragmentation and inefficient resource allocation. Systems modelling and simulation, particularly utilizing participatory approaches, is offering promise in supporting evidence-informed decision making with limited resources by testing alternative strategies in safe virtual environments before implementing them in the real world. However, improved evaluation efforts are needed to understand the critical elements involved in and to improve methods for implementing participatory modelling for youth mental health system and service delivery. An evaluation protocol is described to evaluate the feasibility, value, impact, and sustainability of participatory systems modelling in delivering advanced decision support capabilities for youth mental health. This study applies a comprehensive multi-scale evaluation framework, drawing on participatory action research principles as well as formative, summative, process, and outcome evaluation techniques. Novel data collection procedures are presented, including online surveys that incorporate gamification to enable social network analysis and patient journey mapping. The evaluation approach also explores the experiences of diverse stakeholders, including young people with lived (or living) experience of mental illness. Social and technical opportunities will be uncovered, as well as challenges implementing these interdisciplinary methods in complex settings to improve youth mental health policy, planning, and outcomes. This study protocol can also be adapted for broader international applications, disciplines, and contexts.

A Protocol for a Comprehensive Monitoring and Evaluation Framework With a Compendium of Tools to Assess Quality of Project ECHO (Extension for Community Healthcare Outcomes) Implementation Using Mixed Methods, Developmental Evaluation Design

Introduction: The United States Centers for Disease Control and Prevention (CDC), through U.S. President's Emergency Plan for AIDS Relief (PEPFAR), supports a third of all people receiving HIV care globally. CDC works with local partners to improve methods to find, treat, and prevent HIV and tuberculosis. However, a shortage of trained medical professionals has impeded efforts to control the HIV epidemic in Sub-Saharan Africa and Asia. The Project Extension for Community Healthcare Outcomes (ECHO^TM) model expands capacity to manage complex diseases, share knowledge, disseminate best practices, and build communities of practice. This manuscript describes a practical protocol for an evaluation framework and toolkit to assess ECHO implementation.

Methods and Analysis: This mixed methods, developmental evaluation design uses an appreciative inquiry approach, and includes a survey, focus group discussion, semi-structured key informant interviews, and readiness assessments. In addition, ECHO session content will be objectively reviewed for accuracy, content validity, delivery, appropriateness, and consistency with current guidelines. Finally, we offer a mechanism to triangulate data sources to assess acceptability and feasibility of the evaluation framework and compendium of monitoring and evaluation tools.

Expected impact of the study on public health: This protocol offers a unique approach to engage diverse group of stakeholders using an appreciative inquiry process to co-create a comprehensive evaluation framework and a compendium of assessment tools. This evaluation framework utilizes mixed methods (quantitative and qualitative data collection tools), was pilot tested in Tanzania, and has the potential for contextualized use in other countries who plan to evaluate their Project ECHO implementation.

Evaluation Framework for Successful Artificial Intelligence-Enabled Clinical Decision Support Systems: Mixed Methods Study

Background

Clinical decision support systems are designed to utilize medical data, knowledge, and analysis engines and to generate patient-specific assessments or recommendations to health professionals in order to assist decision making. Artificial intelligence–enabled clinical decision support systems aid the decision-making process through an intelligent component. Well-defined evaluation methods are essential to ensure the seamless integration and contribution of these systems to clinical practice.

Objective

The purpose of this study was to develop and validate a measurement instrument and test the interrelationships of evaluation variables for an artificial intelligence–enabled clinical decision support system evaluation framework.

Methods

An artificial intelligence–enabled clinical decision support system evaluation framework consisting of 6 variables was developed. A Delphi process was conducted to develop the measurement instrument items. Cognitive interviews and pretesting were performed to refine the questions. Web-based survey response data were analyzed to remove irrelevant questions from the measurement instrument, to test dimensional structure, and to assess reliability and validity. The interrelationships of relevant variables were tested and verified using path analysis, and a 28-item measurement instrument was developed. Measurement instrument survey responses were collected from 156 respondents.

Results

The Cronbach α of the measurement instrument was 0.963, and its content validity was 0.943. Values of average variance extracted ranged from 0.582 to 0.756, and values of the heterotrait-monotrait ratio ranged from 0.376 to 0.896. The final model had a good fit (χ₂₆²=36.984; P=.08; comparative fit index 0.991; goodness-of-fit index 0.957; root mean square error of approximation 0.052; standardized root mean square residual 0.028). Variables in the final model accounted for 89% of the variance in the user acceptance dimension.

Conclusions

User acceptance is the central dimension of artificial intelligence–enabled clinical decision support system success. Acceptance was directly influenced by perceived ease of use, information quality, service quality, and perceived benefit. Acceptance was also indirectly influenced by system quality and information quality through perceived ease of use. User acceptance and perceived benefit were interrelated.

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA

Multimodal Learning Analytics (MMLA) researchers are progressively employing machine learning (ML) techniques to develop predictive models to improve learning and teaching practices. These predictive models are often evaluated for their generalizability using methods from the ML domain, which do not take into account MMLA’s educational nature. Furthermore, there is a lack of systematization in model evaluation in MMLA, which is also reflected in the heterogeneous reporting of the evaluation results. To overcome these issues, this paper proposes an evaluation framework to assess and report the generalizability of ML models in MMLA (EFAR-MMLA). To illustrate the usefulness of EFAR-MMLA, we present a case study with two datasets, each with audio and log data collected from a classroom during a collaborative learning session. In this case study, regression models are developed for collaboration quality and its sub-dimensions, and their generalizability is evaluated and reported. The framework helped us to systematically detect and report that the models achieved better performance when evaluated using hold-out or cross-validation but quickly degraded when evaluated across different student groups and learning contexts. The framework helps to open up a “wicked problem” in MMLA research that remains fuzzy (i.e., the generalizability of ML models), which is critical to both accumulating knowledge in the research community and demonstrating the practical relevance of these techniques.

Scoping review: Development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers

Objective

The study sought to review the different assessment items that have been used within existing health app evaluation frameworks aimed at individual, clinician, or organizational users, and to analyze the scoring and evaluation methods used in these frameworks.

Materials and Methods

We searched multiple bibliographic databases and conducted backward searches of reference lists, using search terms that were synonyms of “health apps,” “evaluation,” and “frameworks.” The review covered publications from 2011 to April 2020. Studies on health app evaluation frameworks and studies that elaborated on the scaling and scoring mechanisms applied in such frameworks were included.

Results

Ten common domains were identified across general health app evaluation frameworks. A list of 430 assessment criteria was compiled across 97 identified studies. The most frequently used scaling mechanism was a 5-point Likert scale. Most studies have adopted summary statistics to generate the total scoring of each app, and the most popular approach taken was the calculation of mean or average scores. Other frameworks did not use any scaling or scoring mechanism and adopted criteria-based, pictorial, or descriptive approaches, or “threshold” filter.

Discussion

There is wide variance in the approaches to evaluating health apps within published frameworks, and this variance leads to ongoing uncertainty in how to evaluate health apps.

Conclusions

A new evaluation framework is needed that can integrate the full range of evaluative criteria within one structure, and provide summative guidance on health app rating, to support individual app users, clinicians, and health organizations in choosing or recommending the best health app.

Resource Management Techniques for Cloud/Fog and Edge Computing: An Evaluation Framework and Classification

Processing IoT applications directly in the cloud may not be the most efficient solution for each IoT scenario, especially for time-sensitive applications. A promising alternative is to use fog and edge computing, which address the issue of managing the large data bandwidth needed by end devices. These paradigms impose to process the large amounts of generated data close to the data sources rather than in the cloud. One of the considerations of cloud-based IoT environments is resource management, which typically revolves around resource allocation, workload balance, resource provisioning, task scheduling, and QoS to achieve performance improvements. In this paper, we review resource management techniques that can be applied for cloud, fog, and edge computing. The goal of this review is to provide an evaluation framework of metrics for resource management algorithms aiming at the cloud/fog and edge environments. To this end, we first address research challenges on resource management techniques in that domain. Consequently, we classify current research contributions to support in conducting an evaluation framework. One of the main contributions is an overview and analysis of research papers addressing resource management techniques. Concluding, this review highlights opportunities of using resource management techniques within the cloud/fog/edge paradigm. This practice is still at early development and barriers need to be overcome.

Expanded Neonatal Bloodspot Screening Programmes: An Evaluation Framework to Discuss New Conditions With Stakeholders

Neonatal bloodspot screening (NBS) programmes that screen for rare but serious conditions are expanding worldwide. Fast developments for testing and treatment put pressure on implementation processes. In 2015 the Netherlands embarked on an NBS expansion from 17 to 31 conditions. An evaluation framework was developed based on international NBS frameworks to gain insight in test properties, clinical findings, follow-up and implementation. A stakeholder process took place with implications for the planning of the expanded NBS panel. The evaluation framework progressed into a go/no go framework to start national screening, and is currently explored as basis for continuous evaluation of the NBS panel. The framework and stakeholder process may serve as an example for other programmes.

Evaluating Research Centers in Minority Institutions: Framework, Metrics, Best Practices, and Challenges

The NIH-funded Research Centers in Minority Institutions (RCMI) program is currently funding 18 academic institutions to strengthen the research environment and contribution to health disparities research. The purpose of this multiphase mixed-methods study was to establish a uniform evaluation framework for demonstrating the collective success of this research consortium. Methods included discussions of aims and logic models at the RCMI Evaluators' Workshop, a literature review to inform an evaluation conceptual framework, and a case study survey to obtain evaluation-related information and metrics. Ten RCMIs participated in the workshop and 14 submitted responses to the survey. The resultant RCMI Evaluation Conceptual Model presents a practical ongoing approach to document RCMIs' impacts on health disparities. Survey results identified 37 common metrics under four primary categories. Evaluation challenges were issues related to limited human resources, data collection, decision-making, defining metrics, cost-sharing, and revenue-generation. There is a need for further collaborative efforts across RCMI sites to engage program leadership and community stakeholders in addressing the identified evaluation challenges and measurement. Program leadership should be engaged to apply the Evaluation Conceptual Framework and common metrics to allow for valid inter-institutional comparisons and consortium-wide evaluations. Stakeholders could ensure evaluation metrics are used to facilitate community impacts.

"Co-production Compass" (COCO): An Analytical Framework for Monitoring Patient Preferences in Co-production of Healthcare Services in Mental Health Settings

Background: Engaging patients in raising their voices to advocate for their priorities being taken into account is today acknowledged as essential to improve research and decision-making in healthcare. However, literature is scarce regarding an evaluation framework to monitor the extent to which this approach is successful, in particular in mental health, where the application of patient-reported outcome measures (PROMs) is particularly difficult. In this study, we describe the process of development and first implementation of a new assessment framework-"Co-production Compass" (COCO) framework-for monitoring patient preference collection in co-production of healthcare services within the scope of a national-based project (namely, Recovery.Net) in the mental health field. Method: We conducted (1) a narrative scan of relevant scientific literature on patient engagement in service co-production and (2) qualitative analysis of five subsequent workshops involving-in total-144 expert stakeholders (i.e., expert patients, doctors, nurses, psychologists, healthcare managers…). Data analysis involved three phases: identifying the themes, developing a framework, and confirming the framework. We coded and organized the data and abstracted, illustrated, described, and explored the emergent themes using thematic analysis. At the same time, content analysis was conducted to retrieve concepts and insights from relevant literature about health services co-production to integrate and extend the emergent conceptual framework. The framework was finally reviewed by the research partners belonging to the study project and preliminarily implemented. Results: According to the results of both the literature scan and the participatory workshops, the COCO evaluation framework for monitoring patient preference collection when coproducing medical pathways was drafted. The framework comprised of three organizing themes, corresponding to the three code clusters, which emerged from both the stakeholders' workshop data and relevant scientific literature: "the need for shared and practice-oriented evaluation standards"; "the quest for a multi-dominion approach"; "the need for a multi-stakeholder evaluation". These themes were interconnected and formed a conceptual framework to measure the phenomenon of meaningful patient involvement in healthcare co-production. This framework was endorsed by the research partners of the project and preliminarily applied in a mental health setting. Conclusion: The COCO framework provides guidance on aspects of co-production in healthcare to address for meaningful patient involvement in giving their inputs for more effective service and drug development processes. It could be particularly useful when monitoring patient-researcher partnership initiatives.

A comparative feasibility study of two parent education and training programmes for autism spectrum disorder in a low-resource South African setting

LAY ABSTRACT

After a diagnosis of autism, it is an important first step to give families information about autism and skills to help them support their child. These interventions are called Parent Education and Training programmes. Little is known about these programmes or if they can make a difference to families, particularly in countries with few autism resources. In this study, we compared two Parent Education and Training programmes in South Africa. EarlyBird/EarlyBird Plus was developed in the United Kingdom, and Autism Cares in South Africa. We wanted to know if parents found the programmes useful, and if any changes were required. We collected information from parents through questionnaires and interviews before and after the group. We also asked a panel of experts (including parents) to compare the programmes. In total, 18 parents attended the EarlyBird/EarlyBird Plus group and 11 attended the Autism Cares group. Parents found both programmes helpful and made suggestions for improvements. Parents showed less stress, more knowledge of autism, and saw improvements in their children. The expert panel rated EarlyBird/EarlyBird Plus as better because it was more supportive of parents and was seen as easier to run in the country. Our study showed that Parent Education and Training programmes are important, but that researchers must study not only the outcomes but also the implementation needs of these programmes.

Health system decision-makers at the helm of implementation research: development of a framework to evaluate the processes and effectiveness of embedded approaches

Background

Embedded approaches to implementation research (IR), whereby health system decision-makers participate actively in the research process, are gaining traction as effective approaches to optimise the delivery of health programmes and policies. However, the evidence base on the processes and effectiveness of such collaborative research remains inchoate. Standardised approaches to evaluate these initiatives are needed to identify core elements of ‘embeddedness’, unveil the underlying pathways of change, and assess contribution to evidence uptake in decision-making and overall outcomes of effect. The framework presented in this paper responds to this need, designed to guide the systematic evaluation of embedded IR.

Methods

This evaluation framework for embedded IR approaches is based on the experience of a joint initiative by the Pan American Health Organization/Alliance for Health Policy and Systems Research, which has supported 19 IR grants in 10 Latin American and Caribbean countries from 2014 to 2017. The conceptualisation of this framework drew on various sources of information, including empirical evidence and conceptual insights from the literature, interviews with content experts, and a prospective evaluation of the 2016 cohort that included semi-structured key informant interviews, document analysis, and a research team survey to examine key aspects of embedded research.

Results

We developed a widely applicable conceptual framework to guide the evaluation of embedded IR in various contexts. Focused on uncovering how this collaborative research approach influences programme improvement, it outlines expected processes and intermediate outcomes. It also highlights constructs with which to assess ‘embeddedness’ as well as critical contextual factors. The framework is intended to provide a structure by which to systematically examine such embedded research initiatives, proposing three key stages of evidence-informed decision-making – co-production of evidence, engagement with research, and enactment of programme changes.

Conclusion

Rigorous evaluation of embedded IR is needed to build the evidence on its processes and effectiveness in influencing decision-making. The evaluation framework presented here addresses this gap with consideration of the complexity of such efforts. Its applicability to similar initiatives is bolstered by virtue of being founded on real-world experience; its potential to contribute to a nuanced understanding of embedded IR is significant.

How Health Promoters Can Assess Capacity Building Processes in Setting-Based Approaches-Development and Testing of a Monitoring Instrument

Background: Health promoters often use stakeholder groups to jointly plan and implement local interventions. Stakeholder groups should take over responsibility to later run the health promotion program independently. Monitoring this process of capacity building can help health promoters improve the quality of the process. Instruments for the systematic assessment of capacity building among stakeholder groups are scarce. The goal of this study was to develop, and pilot test a generic assessment instrument for setting-based capacity building. Methods: We drafted a semi-standardized monitoring instrument to be used in stakeholder groups in various settings. This “EVA-protocol” (short for evaluation protocol) was based on capacity building domains e.g., leadership, resource mobilization. It was pilot implemented in a research network on increasing an active lifestyle in various settings. The respective health promoters documented 78 meetings of 15 different stakeholder groups. We performed feedback interviews and member checking among the facilitating health promoters, asking for comprehensibility, length, usability and perceived benefits of the instrument. Findings: Data collected in the “EVA-protocol” helped the facilitating health promoters understand the development of competences and capacities in the stakeholder groups and identify factors that favor or hinder the capacity building process. The instrument was rated as user friendly, but it was remarked that it is best filled out by two persons and reflected upon by those to offer the greatest benefit. Not all projects could afford this procedure due to lack of time/staff resources. Conclusions: The drafted instrument can serve as quality management tool for health promoters who facilitate participatory stakeholder groups in different settings and intend to build capacities for sustainable health promotion structures.

Strategic Priorities for Research on Antibiotic Alternatives in Animal Agriculture-Results From an Expert Workshop

The emergence, spread, and expansion of antibiotic resistance and increasing restrictions on the use of antibiotics in animal agriculture have created a need for efficacious alternatives that remains unmet. Prioritizing research needs in the development of alternatives is key to ensuring that scarce research resources are dedicated to the most promising approaches. However, frameworks to enable a consistent, systematic, and transparent evaluation of antibiotic alternative candidates are lacking. Here, we present such an evaluation framework.

A conceptual framework for an oral cancer awareness campaign in Northern Germany - Challenges in campaign development and assessment

Initial impetus for action: Oral cancer is still an underestimated disease in terms of incidence as well as mortality rates; it requires urgent prevention and early detection. At present, there is no best-practice systematic approach to raising awareness and informing the public about about this type of cancer in Germany. This article describes a framework that covers the significant stages of conceptual development and campaign design to promote oral cancer awareness in Germany. Solution: The challenges of the development, as well as evaluation of an oral cancer awareness campaign are shared in this article. Four key stages of the campaign are defined: (1) mass media, (2) target groups, (3) health care professionals, and (4) epidemiology. For each section, the following levels of assessment are proposed: (a) campaign development (formative assessment), (b) controlling and optimising campaign implementation (process assessment) and (c) measuring outcomes (summative assessment). Outcome: A process-oriented assessment concept for each of the four campaign sections was developed and merged to form a matrix, which includes each of the above sections regarding the prevention and early detection of oral cancer, as well as the three stages of campaign assessment. Future implications and learning points: The conceptual framework demonstrated that systematic planning and evaluation of different components helped to describe and evaluate an oral cancer campaign: For future campaigns, the use of a matrix covering different campaign targets as well as the entire campaign process, is recommended as a basis for campaign design and evaluation.

Examining the Multi-Scalar Unevenness of High-Quality Healthcare Resources Distribution in China

Healthcare disparity is, to a large extent, ascribable to the uneven distribution of high-quality healthcare resources, which remains insufficiently examined, largely due to data unavailability. To overcome this barrier, we synthesized multiple sources of data, employed integrated methods and made a comprehensive analysis of government administrative structures and the socio-economic environment to build probably the most inclusive dataset of Chinese 3-A hospitals thus far. Calibrated on a sample of 379 hospitals rated by a reputable organization, we developed a realistic and viable evaluation framework for assessing hospital quality in China. We then calculated performance scores for 1246 3-A hospitals, which were aggregated and further analyzed at multiple scales (cities, provinces, regions, and economic zones) using general entropy indexes. This research shows that the fragmented governance and incoordination of "kuai" and "tiao" is rooted deeply in China's legacy of centrally-planned systems, and has had a far-reaching yet partially contradictory influence over the contemporary distribution and performance of healthcare resources. Additionally, the unevenness in the distribution of healthcare resources is related closely to a city's administrative rank and power. This study thus suggests that the policy design of healthcare systems should be coordinated with external socio-economic transformation in a sustainable manner.

Adopting a logical framework model to help achieve a balanced and healthy vaccine R&D portfolio

Vaccines are currently the 5th biggest therapy area with global sales for prophylactic and therapeutic vaccines to be ~ $30B, which is expected to increase to $45B by 2024. Immunization is globally recognized as one of the best investments to improve health, with impact lasting beyond saving 2-3M lives every year with benefits accrued over a lifetime.

Enterprise value of any R&D company is a cumulative sum of its projects and proprietary technologies. Hence organizations need to continuously evaluate their portfolios to review the health of projects as changes in external environment may impact project viability. Simultaneously, addition of any new project in a company’s portfolio is a significant investment and needs to be evaluated using an objective multi-parametric framework. In this pursuit, Hilleman Labs, an equal joint venture by MSD and Wellcome Trust, has created a logical framework to evaluate potential vaccine candidates before they are added to the portfolio.

An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process

Background

The number of e-mental health apps is increasing rapidly. Studies have shown that the use of some apps is beneficial, whereas others are ineffective or do not meet users’ privacy expectations. Individuals and organizations that curate, recommend, host, use, or pay for apps have an interest in categorizing apps according to the consensus criteria of usability and effectiveness. Others have previously published recommendations for assessing health-related apps; however, the extent to which these recommendations can be generalized across different population groups (eg, culture, gender, and language) remains unclear. This study describes an attempt by Canadian stakeholders to develop an e-mental health assessment framework that responds to the unique needs of people living in Canada in an evidence-based manner.

Objective

The objective of our study was to achieve consensus from a broad group of Canadian stakeholders on guiding principles and criteria for a framework to assess e-mental health apps in Canada.

Methods

We developed an initial set of guiding principles and criteria from a rapid review and environmental scan of pre-existing app assessment frameworks. The initial list was refined through a two-round modified Delphi process. Participants (N=25) included app developers and users, health care providers, mental health advocates, people with lived experience of a mental health problem or mental illness, policy makers, and researchers. Consensus on each guideline or criterion was defined a priori as at least 70% agreement. The first round of voting was conducted electronically. Prior to Round 2 voting, in-person presentations from experts and a persona empathy mapping process were used to explore the perspectives of diverse stakeholders.

Results

Of all respondents, 68% (17/25) in Round 1 and 100% (13/13) in Round 2 agreed that a framework for evaluating health apps is needed to help Canadian consumers identify high-quality apps. Consensus was reached on 9 guiding principles: evidence based, gender responsive, culturally appropriate, user centered, risk based, internationally aligned, enabling innovation, transparent and fair, and based on ethical norms. In addition, 15 informative and evaluative criteria were defined to assess the effectiveness, functionality, clinical applicability, interoperability, usability, transparency regarding security and privacy, security or privacy standards, supported platforms, targeted users, developers’ transparency, funding transparency, price, user desirability, user inclusion, and meaningful inclusion of a diverse range of communities.

Conclusions

Canadian mental health stakeholders reached the consensus on a framework of 9 guiding principles and 15 criteria important in assessing e-mental health apps. What differentiates the Canadian framework from other scales is explicit attention to user inclusion at all stages of the development, gender responsiveness, and cultural appropriateness. Furthermore, an empathy mapping process markedly influenced the development of the framework. This framework may be used to inform future mental health policies and programs.

RE-AIM in Clinical, Community, and Corporate Settings: Perspectives, Strategies, and Recommendations to Enhance Public Health Impact

The RE-AIM Framework is a planning and evaluation model that has been used in a variety of settings to address various programmatic, environmental, and policy innovations for improving population health. In addition to the broad application and diverse use of the framework, there are lessons learned and recommendations for the future use of the framework across clinical, community, and corporate settings. The purposes of this article are to: (A) provide a brief overview of the RE-AIM Framework and its pragmatic use for planning and evaluation; (B) offer recommendations to facilitate the application of RE-AIM in clinical, community, and corporate settings; and (C) share perspectives and lessons learned about employing RE-AIM dimensions in the planning, implementation, and evaluation phases within these different settings. In this article, we demonstrate how the RE-AIM concepts and elements within each dimension can be applied by researchers and practitioners in diverse settings, among diverse populations and for diverse health topics.

A Systems Approach to Evaluate One Health Initiatives

Challenges calling for integrated approaches to health, such as the One Health (OH) approach, typically arise from the intertwined spheres of humans, animals, and ecosystems constituting their environment. Initiatives addressing such wicked problems commonly consist of complex structures and dynamics. As a result of the EU COST Action (TD 1404) "Network for Evaluation of One Health" (NEOH), we propose an evaluation framework anchored in systems theory to address the intrinsic complexity of OH initiatives and regard them as subsystems of the context within which they operate. Typically, they intend to influence a system with a view to improve human, animal, and environmental health. The NEOH evaluation framework consists of four overarching elements, namely: (1) the definition of the initiative and its context, (2) the description of the theory of change with an assessment of expected and unexpected outcomes, (3) the process evaluation of operational and supporting infrastructures (the "OH-ness"), and (4) an assessment of the association(s) between the process evaluation and the outcomes produced. It relies on a mixed methods approach by combining a descriptive and qualitative assessment with a semi-quantitative scoring for the evaluation of the degree and structural balance of "OH-ness" (summarised in an OH-index and OH-ratio, respectively) and conventional metrics for different outcomes in a multi-criteria-decision-analysis. Here, we focus on the methodology for Elements (1) and (3) including ready-to-use Microsoft Excel spreadsheets for the assessment of the "OH-ness". We also provide an overview of Element (2), and refer to the NEOH handbook for further details, also regarding Element (4) (http://neoh.onehealthglobal.net). The presented approach helps researchers, practitioners, and evaluators to conceptualise and conduct evaluations of integrated approaches to health and facilitates comparison and learning across different OH activities thereby facilitating decisions on resource allocation. The application of the framework has been described in eight case studies in the same Frontiers research topic and provides first data on OH-index and OH-ratio, which is an important step towards their validation and the creation of a dataset for future benchmarking, and to demonstrate under which circumstances OH initiatives provide added value compared to disciplinary or conventional health initiatives.

Comparison of Feature Learning Methods for Human Activity Recognition Using Wearable Sensors

Getting a good feature representation of data is paramount for Human Activity Recognition (HAR) using wearable sensors. An increasing number of feature learning approaches-in particular deep-learning based-have been proposed to extract an effective feature representation by analyzing large amounts of data. However, getting an objective interpretation of their performances faces two problems: the lack of a baseline evaluation setup, which makes a strict comparison between them impossible, and the insufficiency of implementation details, which can hinder their use. In this paper, we attempt to address both issues: we firstly propose an evaluation framework allowing a rigorous comparison of features extracted by different methods, and use it to carry out extensive experiments with state-of-the-art feature learning approaches. We then provide all the codes and implementation details to make both the reproduction of the results reported in this paper and the re-use of our framework easier for other researchers. Our studies carried out on the OPPORTUNITY and UniMiB-SHAR datasets highlight the effectiveness of hybrid deep-learning architectures involving convolutional and Long-Short-Term-Memory (LSTM) to obtain features characterising both short- and long-term time dependencies in the data.

Critical evaluation of international health programs: Reframing global health and evaluation

Striking changes in the funding and implementation of international health programs in recent decades have stimulated debate about the role of communities in deciding which health programs to implement. An important yet neglected piece of that discussion is the need to change norms in program evaluation so that analysis of community ownership, beyond various degrees of "participation," is seen as central to strong evaluation practices. This article challenges mainstream evaluation practices and proposes a framework of Critical Evaluation with 3 levels: upstream evaluation assessing the "who" and "how" of programming decisions; midstream evaluation focusing on the "who" and "how" of selecting program objectives; and downstream evaluation, the focus of current mainstream evaluation, which assesses whether the program achieved its stated objectives. A vital tenet of our framework is that a community possesses the right to determine the path of its health development. A prerequisite of success, regardless of technical outcomes, is that programs must address communities' high priority concerns. Current participatory methods still seldom practice community ownership of program selection because they are vulnerable to funding agencies' predetermined priorities. In addition to critiquing evaluation practices and proposing an alternative framework, we acknowledge likely challenges and propose directions for future research.

Towards an IMU Evaluation Framework for Human Body Tracking

Existing full-body tracking systems, which use Inertial Measurement Units (IMUs) as sensing unit, require expert knowledge for setup and data collection. Thus, the daily application for human body tracking is difficult. In particular, in the field of active and assisted living (AAL), tracking human movements would enable novel insights not only into the quantity but also into the quality of human movement, for example by monitoring functional training. While the current market offers a wide range of products with vastly different properties, literature lacks guidelines for choosing IMUs for body tracking applications. Therefore, this paper introduces developments towards an IMU evaluation framework for human body tracking which compares IMUs against five requirement areas that consider device features and data quality. The data quality is assessed by conducting a static and a dynamic error analysis. In a first application to four IMUs of different component consumption, the IMU evaluation framework convinced as promising tool for IMU selection.

Disability Policy Evaluation: Combining Logic Models and Systems Thinking

Policy evaluation focuses on the assessment of policy-related personal, family, and societal changes or benefits that follow as a result of the interventions, services, and supports provided to those persons to whom the policy is directed. This article describes a systematic approach to policy evaluation based on an evaluation framework and an evaluation process that combine the use of logic models and systems thinking. The article also includes an example of how the framework and process have recently been used in policy development and evaluation in Flanders (Belgium), as well as four policy evaluation guidelines based on relevant published literature.

A Blueprint to Evaluate One Health

One Health (OH) positions health professionals as agents for change and provides a platform to manage determinants of health that are often not comprehensively captured in medicine or public health alone. However, due to the organization of societies and disciplines, and the sectoral allocation of resources, the development of transdisciplinary approaches requires effort and perseverance. Therefore, there is a need to provide evidence on the added value of OH for governments, researchers, funding bodies, and stakeholders. This paper outlines a conceptual framework of what OH approaches can encompass and the added values they can provide. The framework was developed during a workshop conducted by the “Network for Evaluation of One Health,” an Action funded by the European Cooperation in Science and Technology. By systematically describing the various aspects of OH, we provide the basis for measuring and monitoring the integration of disciplines, sectors, and stakeholders in health initiatives. The framework identifies the social, economic, and environmental drivers leading to integrated approaches to health and illustrates how these evoke characteristic OH operations, i.e., thinking, planning, and working, and require supporting infrastructures to allow learning, sharing, and systemic organization. It also describes the OH outcomes (i.e., sustainability, health and welfare, interspecies equity and stewardship, effectiveness, and efficiency), which are not possible to obtain through sectoral approaches alone, and their alignment with aspects of sustainable development based on society, environment, and economy.

Enabler for Interdisciplinary eHealthcare: A Qualitative Study

The complex relations between Health Technologies and clinical practices have been the focus of intensive research in recent years. This research represents a shift towards a holistic view where evaluation of health technologies is linked to organisational practices. In this paper, we address the gaps in existing literature regarding the holistic evaluation of e-health in clinical practice. We report the results from a qualitative study conducted to gain insight into e-health in practice within an interdisciplinary healthcare domain. Findings from this qualitative study, provides the foundation for the creation of a generic measurement model that allows for the comparative analysis of health technologies and assist in the decision-making of its stakeholders.

The Dynamic Integrated Evaluation Model (DIEM): Achieving Sustainability in Organizational Intervention through a Participatory Evaluation Approach

Recently, there have been calls to develop ways of using a participatory approach when conducting interventions, including evaluating the process and context to improve and adapt the intervention as it evolves over time. The need to integrate interventions into daily organizational practices, thereby increasing the likelihood of successful implementation and sustainable changes, has also been highlighted. We propose an evaluation model-the Dynamic Integrated Evaluation Model (DIEM)-that takes this into consideration. In the model, evaluation is fitted into a co-created iterative intervention process, in which the intervention activities can be continuously adapted based on collected data. By explicitly integrating process and context factors, DIEM also considers the dynamic sustainability of the intervention over time. It emphasizes the practical value of these evaluations for organizations, as well as the importance of their rigorousness for research purposes. Copyright © 2016 John Wiley & Sons, Ltd.

A framework for evaluating and designing citizen science programs for natural resources monitoring

We present a framework of resource characteristics critical to the design and assessment of citizen science programs that monitor natural resources. To develop the framework we reviewed 52 citizen science programs that monitored a wide range of resources and provided insights into what resource characteristics are most conducive to developing citizen science programs and how resource characteristics may constrain the use or growth of these programs. We focused on 4 types of resource characteristics: biophysical and geographical, management and monitoring, public awareness and knowledge, and social and cultural characteristics. We applied the framework to 2 programs, the Tucson (U.S.A.) Bird Count and the Maui (U.S.A.) Great Whale Count. We found that resource characteristics such as accessibility, diverse institutional involvement in resource management, and social or cultural importance of the resource affected program endurance and success. However, the relative influence of each characteristic was in turn affected by goals of the citizen science programs. Although the goals of public engagement and education sometimes complimented the goal of collecting reliable data, in many cases trade-offs must be made between these 2 goals. Program goals and priorities ultimately dictate the design of citizen science programs, but for a program to endure and successfully meet its goals, program managers must consider the diverse ways that the nature of the resource being monitored influences public participation in monitoring.

Framework for Evaluating the Impact of Advanced Practice Nursing Roles

PURPOSE

To address the gap in evidence-based information required to support the development of advanced practice nursing (APN) roles in Switzerland, stakeholders identified the need for guidance to generate strategic evaluation data. This article describes an evaluation framework developed to inform decisions about the effective utilization of APN roles across the country.

APPROACH

A participatory approach was used by an international group of stakeholders. Published literature and an evidenced-based framework for introducing APN roles were analyzed and applied to define the purpose, target audiences, and essential elements of the evaluation framework. Through subsequent meetings and review by an expert panel, the framework was developed and refined.

FINDINGS

A framework to evaluate different types of APN roles as they evolve to meet dynamic population health, practice setting, and health system needs was created. It includes a matrix of key concepts to guide evaluations across three stages of APN role development: introduction, implementation, and long-term sustainability. For each stage, evaluation objectives and questions examining APN role structures, processes, and outcomes from different perspectives (e.g., patients, providers, managers, policy-makers) were identified.

CONCLUSIONS

A practical, robust framework based on well-established evaluation concepts and current understanding of APN roles can be used to conduct systematic evaluations.

CLINICAL RELEVANCE

The evaluation framework is sufficiently generic to allow application in developed countries globally, both for evaluation as well as research purposes.

Analysis of higher education policy frameworks for open and distance education in Pakistan

BACKGROUND

The constant rise in demand for higher education has become the biggest challenge for educational planners. This high demand has paved a way for distance education across the globe.

OBJECTIVES

This article innovatively analyzes the policy documentation of a major distance education initiative in Pakistan for validity that will identify the utility of policy linkages.

RESEARCH DESIGN

The study adopted a qualitative research design that consisted of two steps. In the first step, a content analysis of distance learning policy framework was made. For this purpose, two documents were accessed titled "Framework for Launching Distance Learning Programs in HEIs of Pakistan" and "Guideline on Quality of Distance Education for External Students at the HEIs of Pakistan." In the second step, the policy guidelines mentioned in these two documents were evaluated at two levels. At the first level, the overall policy documents were assessed against a criterion proposed by Cheung, Mirzaei, and Leeder. At the second level, the proposed program of distance learning was assessed against a criterion set by Gellman-Danley and Fetzner and Berge.

CONCLUSION

The distance education program initiative in Pakistan is of promising nature which needs to be assessed regularly. This study has made an initial attempt to assess the policy document against a criterion identified from literature. The analysis shows that the current policy documents do offer some strengths at this initial level, however, they cannot be considered a comprehensive policy guide. The inclusion or correction of missing or vague areas identified in this study would make this policy guideline document a treasured tool for Higher Education Commission (HEC). For distance education policy makers, this distance education policy framework model recognizes several fundamental areas with which they should be concerned. The findings of this study in the light of two different policy framework measures highlight certain opportunities that can help strengthening the distance education policies. The criteria and findings are useful for the reviewers of policy proposals to identify the gaps where policy documents can be improved to bring the desired outcomes.

Uniform presentation of process evaluation results facilitates the evaluation of complex interventions: development of a graph

RATIONALE, AIMS AND OBJECTIVES

Process evaluation is a highly essential element for the increasing number of studies regarding multi-component interventions. Yet, researchers are challenged to collect and present appropriate process outcomes in such way that it is easy and valuable to be used by other researchers and policy makers in interpreting and comparing intervention effects because of the absence of standards for conducting and publishing process evaluation. This article describes the development of a method to concisely summarize the results of process evaluations of complex multi-component interventions.

METHODS

Development of a graph with the aim to facilitate the reporting of process evaluation's results, based on a narrative review of the literature for process measures used in complex interventions for elderly people.

RESULTS

Seventeen articles of process evaluations alongside effect studies of complex interventions were reviewed. From these articles, it was found that process evaluations should address whether the intervention (1) was implemented successfully; (2) was evaluated properly; and (3) can be continued in the future. A flow chart based on the essential components of an adequate process evaluation was developed. A simplified but highly informative figure reporting a summary of the results of the process evaluation is proposed and its use is explained by administering the figure to two studies including a process and effect evaluation of a complex intervention.

CONCLUSION

A graphical approach - which includes the core results of process evaluation and can be used directly in reporting effectiveness studies - will help researchers and policy makers to interpret and compare effects of complex multi-component interventions.

"Real-world" practical evaluation strategies: a review of telehealth evaluation

Background

Currently, the increasing interest in telehealth and significant technological breakthroughs of the past decade create favorable conditions for the widespread adoption of telehealth services. Therefore, expectations are high that telehealth can help alleviate prevailing challenges in health care delivery. However, in order to translate current research to policy and facilitate adoption by patients and health care providers, there is need for compelling evidence of the effectiveness of telehealth interventions. Such evidence is gathered from rigorously designed research studies, which may not always be practical in many real-world settings.

Objective

Our aim was to summarize current telehealth evaluation strategies and challenges and to outline practical approaches to conduct evaluation in real-world settings using one of our previously reported telehealth initiatives, the Diabetes Connect program, as a case study.

Methods

We reviewed commonly used current evaluation frameworks and strategies, as well as best practices based on successful evaluative efforts to date to address commonly encountered challenges in telehealth evaluation. These challenges in telehealth evaluation and commonly used frameworks are described relevant to the evaluation of Diabetes Connect, a 12-month Web-based blood glucose monitoring program.

Results

Designers of telehealth evaluation frameworks must give careful consideration to the elements of planning, implementation, and impact assessment of interventions. Evaluating performance at each of these phases is critical to the overall success of an intervention. Although impact assessment occurs at the end of a program, our review shows that it should begin at the point of problem definition. Critical to the success of an evaluative strategy is early planning that involves all stakeholders to identify the overall goals of the program and key measures of success at each phase of the program life cycle. This strategy should enable selection of an appropriate evaluation strategy and measures to aid in the ongoing development and implementation of telehealth and provide better evidence of program impact.

Conclusions

We recommend a pragmatic, multi-method, multi-phase approach to telehealth evaluation that is flexible and can be adapted to the characteristics and challenges unique to each telehealth program.

Development of key indicators of hospital resilience: a modified Delphi study

OBJECTIVES

Hospital resilience is an emerging concept, which can be defined as 'a hospital's ability to resist, absorb, and respond to the shock of disasters while maintaining its critical health care functions, and then recover to its original state or adapt to a new one'. Our aim was to develop a comprehensive framework of key indicators of hospital resilience.

METHODS

A panel of 33 Chinese experts was invited to participate in a three-round, modified Delphi study to develop a set of potential measures previously derived from a literature review. In the first round, these potential measures were modified to cover the comprehensive domains of hospital resilience. The importance of proposed measures was scored by experts on a five-point Likert scale. Subsequently, the experts reconsidered their voting in light of the previous aggregated results. Agreement on measures was defined as at least 70% of the responders agreeing or strongly agreeing to the inclusion of a measure.

RESULTS

A large proportion of preliminary measures (89.5%) were identified as having good potential for assessing hospital resilience. These measures were categorized into eight domains, 17 subdomains, and 43 indicators. The highest rated indicators (mean score) were: equipment for on-site rescue (4.7), plan initiation (4.6), equipment for referral of patients with complex care needs (4.5), the plan execution (4.4), medication management strategies (4.4), emergency medical treatment conditions (4.4), disaster committee (4.4), stock types and quantities for essential medications (4.4), surge capacity of emergency beds (4.4), and mass-casualty triage protocols (4.4).

CONCLUSIONS

This framework identifies a comprehensive set of indicators of hospital resilience. It can be used for hospital assessment, as well as informing priority practices to address future disasters better.

A systematic framework for functional connectivity measures

Various methods have been proposed to characterize the functional connectivity between nodes in a network measured with different modalities (electrophysiology, functional magnetic resonance imaging etc.). Since different measures of functional connectivity yield different results for the same dataset, it is important to assess when and how they can be used. In this work, we provide a systematic framework for evaluating the performance of a large range of functional connectivity measures-based upon a comprehensive portfolio of models generating measurable responses. Specifically, we benchmarked 42 methods using 10,000 simulated datasets from 5 different types of generative models with different connectivity structures. Since all functional connectivity methods require the setting of some parameters (window size and number, model order etc.), we first optimized these parameters using performance criteria based upon (threshold free) ROC analysis. We then evaluated the performance of the methods on data simulated with different types of models. Finally, we assessed the performance of the methods against different levels of signal-to-noise ratios and network configurations. A MATLAB toolbox is provided to perform such analyses using other methods and simulated datasets.

Evaluating the impact of equity focused health impact assessment on health service planning: three case studies

BACKGROUND

Health impact assessment has been identified internationally as a mechanism to ensure potential health impacts and health equity impacts of proposals are considered before implementation. This paper looks at the impact of three equity focused health impact assessments (EFHIAs) of health service plans on subsequent decision-making and implementation, and then utilises these findings to test and refine an existing conceptual framework for evaluating the impact and effectiveness of health impact assessments for use in relation to EFHIAs.

METHODS

Case study analysis of three EFHIAs conducted on health sector plans in New South Wales, Australia. Data was drawn from 14 semi-structured interviews and the analysis of seven related documents (draft plans and EFHIA reports).

RESULTS

The case studies showed that the EFHIAs all had some impact on the decision-making about the plans and their implementation, most clearly in relation to participants' understandings of equity and in the development of options for modifying service plans to ensure this was addressed. The timing of the EFHIA and individual responses to the EFHIA process and its recommendations were identified as critical factors influencing the impact of the EFHIAs. Several modifications to the conceptual framework are identified, principally adding factors to recognise the role individuals play in influencing the impact and effectiveness of EFHIAs.

CONCLUSION

EFHIA has the potential to improve the consideration of health equity in health service planning processes, though a number of contextual and individual factors affect this. Current approaches can be strengthened by taking into account personal and organisational responses to the EFHIA process.

An evaluation framework for defining the contributions of telestration in surgical telementoring

BACKGROUND

An increasing quantity of research in the domain of telemedicine show a growing popularity and acceptance of care over distance systems among both clinicians and patients. We focus on telementoring solutions, developed for providing remote guidance to less experienced surgeons. Telestration is often regarded as an extra functionality of some telementoring systems. However, we advocate that telestration must be viewed as a core feature of telementoring due to its advantages.

OBJECTIVE

To analyze and define concepts, parameters, and measurement procedures to evaluate the impact of using telestration while telementoring.

METHODS

A systematic review of research dealing with telestration during remote guidance sessions was performed by querying three major online research databases (MEDLINE, Association of Computing Machinery, and Institute of Electrical and Electronics Engineers) using a predefined set of keywords ("laparoscopy", "annotate", "telestrate", "telestration", "annotation", "minimally invasive", and "MIS").

RESULTS

The keyword-based search identified 117 papers. Following the guidelines for performing a systematic review, only 8 publications were considered relevant for the final study. Moreover, a gap in research defining the impacts of telestration during telementoring was identified. To fill this niche, a framework for analyzing, reporting, and measuring the impacts of telestration was proposed.

CONCLUSIONS

The presented framework lays the basics for the structured analysis and reporting of telestration applied to telementoring systems. It is the first step toward building an evidence knowledge base documenting the advantages of live video content annotation and supporting the presented connections between the concepts.

Evaluating Biosurveillance System Components using Multi-Criteria Decision Analysis

Objective

The use of Multi-Criteria Decision Analysis (MCDA) has traditionally been limited to the field of operations research, however many of the tools and methods developed for MCDA can also be applied to biosurveillance. Our project demonstrates the utility of MCDA for this purpose by applying it to the evaluation of data streams for use in an integrated, global biosurveillance system.

Introduction

The evaluation of biosurveillance system components is a complex, multi-objective decision that requires consideration of a variety of factors. Multi-Criteria Decision Analysis provides a methodology to assist in the objective analysis of these types of evaluation by creating a mathematical model that can simulate decisions. This model can utilize many types of data, both quantitative and qualitative, that can accurately describe components. The decision-maker can use this model to determine which of the system components best accomplish the goals being evaluated. Before MCDA can be utilized effectively, an evaluation framework needs to be developed. We built a robust framework that identified unique metrics, surveillance goals, and priorities for metrics. Using this framework, we were able to use MCDA to assist in the evaluation of data streams and to determine which types would be of most use within a global biosurveillance system.

Methods

MCDA was implemented using the Logical Decisions® software. The construction of the evaluation framework was carried out in several steps: identification and definition of data streams, metrics and surveillance goals, and the determination of the relative importance of each metric to the respective surveillance goal being evaluated. Sixteen data streams types were defined and identified for evaluation from a survey we conducted that collected over 200 surveillance products. A subject matter expert (SME) panel was assembled to help identify the biosurveillance goals and metrics in which to evaluate the data streams. To assign values for the metrics, we referenced properties of data streams used in currently operational systems.

Results

Our survey identified sixteen different classes of data streams: Ambulance Records, Clinic/Health Care Provider Records, ED/Hospital Records, Employment/School Records, Established Databases, Financial Records, Help Lines, Internet Search Queries, Laboraotry Records, News Aggregators, Official Reports, Police/Fire Department Records, Personal Communication, Prediction Markets, Sales, and Social Media.

Four biosurveillance goals were identified: Early Warning of Health Threats, Early Detection of Health Events, Situational Awareness, and Consequence Management.

Eleven metrics were identified: Accessibility, Cost, Credibility, Flexibility, Integrability, Geographic/Population Coverage, Granularity, Specificity of Detection, Sustainability, Time to Indication, and Timeliness.

Using the framework, it was possible to use MCDA to rank the utility of each data stream for each goal.

Conclusions

The results suggest that a “one size fits all” approach does not work and that there is no ideal data stream that is most useful for each goal. Data streams that scored more highly for speed tended to rank more highly when the biosurveillance goal is early warning or early detection, whereas data streams that scored more highly for data credibility and geographic/population coverage ranked highly when the goal was situational awareness or consequence management. However, there are several data streams that rank consistently within the top 5 for each goal: Internet Search Queries, News Aggregators, Clinic/Health Care Provider records, ED/Hospital Records, and Laboratory Records and may be considered useful for integrated, global biosurveillance for infectious disease.

A user-centred evaluation framework for the Sealife semantic web browsers

BACKGROUND

Semantically-enriched browsing has enhanced the browsing experience by providing contextualized dynamically generated Web content, and quicker access to searched-for information. However, adoption of Semantic Web technologies is limited and user perception from the non-IT domain sceptical. Furthermore, little attention has been given to evaluating semantic browsers with real users to demonstrate the enhancements and obtain valuable feedback. The Sealife project investigates semantic browsing and its application to the life science domain. Sealife's main objective is to develop the notion of context-based information integration by extending three existing Semantic Web browsers (SWBs) to link the existing Web to the eScience infrastructure.

METHODS

This paper describes a user-centred evaluation framework that was developed to evaluate the Sealife SWBs that elicited feedback on users' perceptions on ease of use and information findability. Three sources of data: i) web server logs; ii) user questionnaires; and iii) semi-structured interviews were analysed and comparisons made between each browser and a control system.

RESULTS

It was found that the evaluation framework used successfully elicited users' perceptions of the three distinct SWBs. The results indicate that the browser with the most mature and polished interface was rated higher for usability, and semantic links were used by the users of all three browsers.

CONCLUSION

Confirmation or contradiction of our original hypotheses with relation to SWBs is detailed along with observations of implementation issues.