Clinical information modeling processes for semantic interoperability of electronic health records: systematic review and inductive analysis

Semantic Interoperability of Electronic Health Records: Systematic Review of Alternative Approaches for Enhancing Patient Information Availability

Background

Semantic interoperability facilitates the exchange of and access to health data that are being documented in electronic health records (EHRs) with various semantic features. The main goals of semantic interoperability development entail patient data availability and use in diverse EHRs without a loss of meaning. Internationally, current initiatives aim to enhance semantic development of EHR data and, consequently, the availability of patient data. Interoperability between health information systems is among the core goals of the European Health Data Space regulation proposal and the World Health Organization's Global Strategy on Digital Health 2020-2025.

Objective

To achieve integrated health data ecosystems, stakeholders need to overcome challenges of implementing semantic interoperability elements. To research the available scientific evidence on semantic interoperability development, we defined the following research questions: What are the key elements of and approaches for building semantic interoperability integrated in EHRs? What kinds of goals are driving the development? and What kinds of clinical benefits are perceived following this development?

Methods

Our research questions focused on key aspects and approaches for semantic interoperability and on possible clinical and semantic benefits of these choices in the context of EHRs. Therefore, we performed a systematic literature review in PubMed by defining our study framework based on previous research.

Results

Our analysis consisted of 14 studies where data models, ontologies, terminologies, classifications, and standards were applied for building interoperability. All articles reported clinical benefits of the selected approach to enhancing semantic interoperability. We identified 3 main categories: increasing the availability of data for clinicians (n=6, 43%), increasing the quality of care (n=4, 29%), and enhancing clinical data use and reuse for varied purposes (n=4, 29%). Regarding semantic development goals, data harmonization and developing semantic interoperability between different EHRs was the largest category (n=8, 57%). Enhancing health data quality through standardization (n=5, 36%) and developing EHR-integrated tools based on interoperable data (n=1, 7%) were the other identified categories. The results were closely coupled with the need to build usable and computable data out of heterogeneous medical information that is accessible through various EHRs and databases (eg, registers).

Conclusions

When heading toward semantic harmonization of clinical data, more experiences and analyses are needed to assess how applicable the chosen solutions are for semantic interoperability of health care data. Instead of promoting a single approach, semantic interoperability should be assessed through several levels of semantic requirements A dual model or multimodel approach is possibly usable to address different semantic interoperability issues during development. The objectives of semantic interoperability are to be achieved in diffuse and disconnected clinical care environments. Therefore, approaches for enhancing clinical data availability should be well prepared, thought out, and justified to meet economically sustainable and long-term outcomes.

Key interoperability Factors for patient portals and Electronic health Records: A scoping review

AIM

To identify the key requirements and challenges to interoperability between patient portals and electronic health records (EHRs).

INTRODUCTION

Patient portals provide patients with access to their health information directly from EHRs within hospitals, primary care centres and general practices (GPs). Patient portals offer many benefits to patients including improved communication with healthcare providers and care coordination. However, many challenges exist with the integration and automatic and secure sharing of information between EHRs and patient portals. It is critical that countries learn from international experiences to successfully develop interoperable national patient portals.

METHODS

A scoping review methodology was undertaken. A search strategy using index terms and keywords was applied across four key databases, an additional grey literature search was also run. The identified studies were screened by two reviewers to determine eligibility against defined inclusion criteria. Data were abstracted from the eligible studies and reviewed to identify the key requirements and challenges to interoperability of patient portals with EHRs.

RESULTS

After screening 3,462 studies, 34 were included across 11 countries. Of the 29 unique patient portals studied, few offered patients access to their entire healthcare record across multiple sites and a number of different functionalities were available. Key interoperability requirements and challenges identified were: Data Sharing Incentives & Supports; Heterogenous Organisations & Information Systems; Data Storage & Management; Available Information & Functionalities; Data Formats & Standards; Identification of Individuals; User Access, Control & Consent; and Security & Privacy.

CONCLUSION

Seamless exchange of health information across patient portals and EHRs required organisational and individual factors, as well as technical considerations. Interorganisational collaboration and engagement of key stakeholders to determine standards and guidelines for consent and sharing of information, as well as technical standards and security measures were recommended.

Can OpenEHR, ISO 13606, and HL7 FHIR Work Together? An Agnostic Approach for the Selection and Application of Electronic Health Record Standards to the Next-Generation Health Data Spaces

In order to maximize the value of electronic health records (EHRs) for both health care and secondary use, it is necessary for the data to be interoperable and reusable without loss of the original meaning and context, in accordance with the findable, accessible, interoperable, and reusable (FAIR) principles. To achieve this, it is essential for health data platforms to incorporate standards that facilitate addressing needs such as formal modeling of clinical knowledge (health domain concepts) as well as the harmonized persistence, query, and exchange of data across different information systems and organizations. However, the selection of these specifications has not been consistent across the different health data initiatives, often applying standards to address needs for which they were not originally designed. This issue is essential in the current scenario of implementing the European Health Data Space, which advocates harmonization, interoperability, and reuse of data without regulating the specific standards to be applied for this purpose. Therefore, this viewpoint aims to establish a coherent, agnostic, and homogeneous framework for the use of the most impactful EHR standards in the new-generation health data spaces: OpenEHR, International Organization for Standardization (ISO) 13606, and Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR). Thus, a panel of EHR standards experts has discussed several critical points to reach a consensus that will serve decision-making teams in health data platform projects who may not be experts in these EHR standards. It was concluded that these specifications possess different capabilities related to modeling, flexibility, and implementation resources. Because of this, in the design of future data platforms, these standards must be applied based on the specific needs they were designed for, being likewise fully compatible with their combined functional and technical implementation.

Toward an Interdisciplinary Approach to Constructing Care Delivery Pathways From Electronic Health Care Databases to Support Integrated Care in Chronic Conditions: Systematic Review of Quantification and Visualization Methods

BACKGROUND

Electronic health care databases are increasingly used for informing clinical decision-making. In long-term care, linking and accessing information on health care delivered by different providers could improve coordination and health outcomes. Several methods for quantifying and visualizing this information into data-driven care delivery pathways (CDPs) have been proposed. To be integrated effectively and sustainably into routine care, these methods need to meet a range of prerequisites covering 3 broad domains: clinical, technological, and behavioral. Although advances have been made, development to date lacks a comprehensive interdisciplinary approach. As the field expands, it would benefit from developing common standards of development and reporting that integrate clinical, technological, and behavioral aspects.

OBJECTIVE

We aimed to describe the content and development of long-term CDP quantification and visualization methods and to propose recommendations for future work.

METHODS

We conducted a systematic review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) recommendations. We searched peer-reviewed publications in English and reported the CDP methods by using the following data in the included studies: long-term care data and extracted data on clinical information and aims, technological development and characteristics, and user behaviors. The data are summarized in tables and presented narratively.

RESULTS

Of the 2921 records identified, 14 studies were included, of which 13 (93%) were descriptive reports and 1 (7%) was a validation study. Clinical aims focused primarily on treatment decision-making (n=6, 43%) and care coordination (n=7, 50%). Technological development followed a similar process from scope definition to tool validation, with various levels of detail in reporting. User behaviors (n=3, 21%) referred to accessing CDPs, planning care, adjusting treatment, or supporting adherence.

CONCLUSIONS

The use of electronic health care databases for quantifying and visualizing CDPs in long-term care is an emerging field. Detailed and standardized reporting of clinical and technological aspects is needed. Early consideration of how CDPs would be used, validated, and implemented in clinical practice would likely facilitate further development and adoption.

TRIAL REGISTRATION

PROSPERO CRD42019140494; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=140494.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.1136/bmjopen-2019-033573.

Electronic Health Record Optimization for Artificial Intelligence

Laboratory clinical decision support (CDS) typically relies on data from the electronic health record (EHR). The implementation of a sustainable, effective laboratory CDS program requires a commitment to standardization and harmonization of key EHR data elements that are the foundation of laboratory CDS. The direct use of artificial intelligence algorithms in CDS programs will be limited unless key elements of the EHR are structured. The identification, curation, maintenance, and preprocessing steps necessary to implement robust laboratory-based algorithms must account for the heterogeneity of data present in a typical EHR.

Using Existing Clinical Information Models for Dutch Quality Registries to Reuse Data and Follow COUMT Paradigm

BACKGROUND

 Reuse of health care data for various purposes, such as the care process, for quality measurement, research, and finance, will become increasingly important in the future; therefore, "Collect Once Use Many Times" (COUMT). Clinical information models (CIMs) can be used for content standardization. Data collection for national quality registries (NQRs) often requires manual data entry or batch processing. Preferably, NQRs collect required data by extracting data recorded during the health care process and stored in the electronic health record.

OBJECTIVES

 The first objective of this study was to analyze the level of coverage of data elements in NQRs with developed Dutch CIMs (DCIMs). The second objective was to analyze the most predominant DCIMs, both in terms of the coverage of data elements as well as in their prevalence across existing NQRs.

METHODS

 For the first objective, a mapping method was used which consisted of six steps, ranging from a description of the clinical pathway to a detailed mapping of data elements. For the second objective, the total number of data elements that matched with a specific DCIM was counted and divided by the total number of evaluated data elements.

RESULTS

 An average of 83.0% (standard deviation: 11.8%) of data elements in studied NQRs could be mapped to existing DCIMs . In total, 5 out of 100 DCIMs were needed to map 48.6% of the data elements.

CONCLUSION

 This study substantiates the potential of using existing DCIMs for data collection in Dutch NQRs and gives direction to further implementation of DCIMs. The developed method is applicable to other domains. For NQRs, implementation should start with the five DCIMs that are most prevalently used in the NQRs. Furthermore, a national agreement on the leading principle of COUMT for the use and implementation for DCIMs and (inter)national code lists is needed.

Successful integration of EN/ISO 13606 standardized extracts from a patient mobile app into an electronic health record: description of a methodology (Preprint)

Background

There is an increasing need to integrate patient-generated health data (PGHD) into health information systems (HISs). The use of health information standards based on the dual model allows the achievement of semantic interoperability among systems. Although there is evidence in the use of the Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources (SMART on FHIR) framework for standardized communication between mobile apps and electronic health records (EHRs), the use of European Norm/International Organization for Standardization (EN/ISO) 13606 has not been explored yet, despite some advantages over FHIR in terms of modeling and formalization of clinical knowledge, as well as flexibility in the creation of new concepts.

Objective

This study aims to design and implement a methodology based on the dual-model paradigm to communicate clinical information between a patient mobile app (Xemio Research) and an institutional ontology-based clinical repository (OntoCR) without loss of meaning.

Methods

This paper is framed within Artificial intelligence Supporting CAncer Patients across Europe (ASCAPE), a project that aims to use artificial intelligence (AI)/machine learning (ML) mechanisms to support cancer patients’ health status and quality of life (QoL). First, the variables “side effect” and “daily steps” were defined and represented with EN/ISO 13606 archetypes. Next, ontologies that model archetyped concepts and map them to the standard were created and uploaded to OntoCR, where they were ready to receive instantiated patient data. Xemio Research used a conversion module in the ASCAPE Local Edge to transform data entered into the app to create EN/ISO 13606 extracts, which were sent to an Application Programming Interface (API) in OntoCR that maps each element in the normalized XML files to its corresponding location in the ontology. This way, instantiated data of patients are stored in the clinical repository.

Results

Between December 22, 2020, and April 4, 2022, 1100 extracts of 47 patients were successfully communicated (234/1100, 21.3%, extracts of side effects and 866/1100, 78.7%, extracts of daily activity). Furthermore, the creation of EN/ISO 13606–standardized archetypes allows the reuse of clinical information regarding daily activity and side effects, while with the creation of ontologies, we extended the knowledge representation of our clinical repository.

Conclusions

Health information interoperability is one of the requirements for continuity of health care. The dual model allows the separation of knowledge and information in HISs. EN/ISO 13606 was chosen for this project because of the operational mechanisms it offers for data exchange, as well as its flexibility for modeling knowledge and creating new concepts. To the best of our knowledge, this is the first experience reported in the literature of effective communication of EN/ISO 13606 EHR extracts between a patient mobile app and an institutional clinical repository using a scalable standard-agnostic methodology that can be applied to other projects, data sources, and institutions.

MedicalForms: Integrated Management of Semantics for Electronic Health Record Systems and Research Platforms

(1) Background: Clinical information modeling tools are software instruments designed to support the definition of semantic structures able to be implemented in health information systems. Based on the analysis of existing tools, this research developed a tool that proposes new approaches to promoting clinician involvement and supporting information modeling processes through mechanisms that ensure governance, information consistency and consensus building. (2) Method: This research developed the MedicalForms system, which is based on the requirements identified in both a Delphi study about tool requirements and the ISO/TS 13972 specifications. (3) Results: This system allows the management of projects, information structures and implementable forms related to clinical documentation. Users can easily define clinical documents in collaboration with the rest of the professionals in their team by being able to reuse previously defined forms, terminologies and information structures. The system is able to export the defined forms as interoperable specifications or as several implementable form formats compatible with multiple open source EHR systems and research platforms. End user perception of this tool was evaluated through the Technology Acceptance Questionnaire with satisfactory results. Finally, the system was applied to develop 12 research registries and 2 clinical trial research forms, 3 mobile applications and 1 decision support system.

How Can a Clinical Data Modelling Tool Be Used to Represent Data Items of Relevance to Paediatric Clinical Trials? Learning from the Conect4children (c4c) Consortium

Data dictionaries for clinical trials are often created manually, with data structures and controlled vocabularies specific for a trial or family of trials within a sponsor’s portfolio. Microsoft Excel is commonly used to capture the representation of data dictionary items but has limited functionality for this purpose. The conect4children (c4c) network is piloting the Direcht clinical data modelling tool to model their Cross Cutting Paediatric Data Dictionary (CCPDD) in a more formalised way. The first pilot had the key objective of testing whether a clinical data modelling tool could be used to represent data items from the CCPDD. The key objective of the second pilot is to establish whether a small team with little or no experience of clinical data modelling can use Direcht to expand the CCPDD. Clinical modelling is the process of structuring clinical data so it can be understood by computer systems and humans. The model contains all of the elements that are needed to define the data item. Results from the pilots show that Direcht creates a structured environment to build data items into models that fit into the larger CCPDD. Models can be represented as an HTML document, mind map, or exported in various formats for import into a computer system. Challenges identified over the course of both pilots are being addressed with c4c partners and external stakeholders.

A State-of-the Art Review of SNOMED CT Terminology Binding and Recommendations for Practice and Research

BACKGROUND

Unambiguous sharing of data requires information models and terminology in combination, but there is a lack of knowledge as to how they should be combined, leading to impaired interoperability.

OBJECTIVES

To facilitate creation of guidelines for SNOMED CT terminology binding we have performed a literature review to find existing recommendations and expose knowledge gaps. The primary audience is practitioners and researchers working with terminology binding.

METHODS

PubMed, Scopus, and Web of Science were searched for papers containing "terminology binding," "subset," "map," "information model" or "implement" and the term "SNOMED."

RESULTS

The search yielded 616 unique papers published from 2004 to 2020, from which 55 papers were selected and analyzed inductively. Topics described in the papers include problems related to input material, SNOMED CT, information models, and lack of appropriate tools as well as recommendations regarding competence.

CONCLUSION

Recommendations are given for practitioners and researchers. Many of the stated problems can be solved by better co-operation between domain experts and informaticians and better knowledge of SNOMED CT. Settings where these competences either work together or where staff with knowledge of both act as brokers are well equipped for terminology binding. Tooling is not thoroughly researched and might be a possible way to facilitate terminology binding.

Harmonization of the ICHOM Quality Measures to Enable Health Outcomes Measurement in Multimorbid Patients

Objectives: To update the sets of patient-centric outcomes measures (“standard-sets”) developed by the not-for-profit organization ICHOM to become more readily applicable in patients with multimorbidity and to facilitate their implementation in health information systems. To that end we set out to (i) harmonize measures previously defined separately for different conditions, (ii) create clinical information models from the measures, and (iii) restructure the annotation to make the sets machine-readable.

Materials and Methods: First, we harmonized the semantic meaning of individual measures across all the 28 standard-sets published to date, in a harmonized measure repository. Second, measures corresponding to four conditions (Breast cancer, Cataracts, Inflammatory bowel disease and Heart failure) were expressed as logical models and mapped to reference terminologies in a pilot study.

Results: The harmonization of semantic meaning resulted in a consolidation of measures used across the standard-sets by 15%, from 3,178 to 2,712. These were all converted into a machine-readable format. 61% of the measures in the 4 pilot sets were bound to existing concepts in either SNOMED CT or LOINC.

Discussion: The harmonization of ICHOM measures across conditions is expected to increase the applicability of ICHOM standard-sets to multi-morbid patients, as well as facilitate their implementation in health information systems.

Conclusion: Harmonizing the ICHOM measures and making them machine-readable is expected to expedite the global adoption of systematic and interoperable outcomes measurement. In turn, we hope that the improved transparency on health outcomes that follows will let health systems across the globe learn from each other to the ultimate benefit of patients.

Verifying the Feasibility of Implementing Semantic Interoperability in Different Countries Based on the OpenEHR Approach: Comparative Study of Acute Coronary Syndrome Registries

Background

The semantic interoperability of health care information has been a critical challenge in medical informatics and has influenced the integration, sharing, analysis, and use of medical big data. International standard organizations have developed standards, approaches, and models to improve and implement semantic interoperability. The openEHR approach—one of the standout semantic interoperability approaches—has been implemented worldwide to improve semantic interoperability based on reused archetypes.

Objective

This study aimed to verify the feasibility of implementing semantic interoperability in different countries by comparing the openEHR-based information models of 2 acute coronary syndrome (ACS) registries from China and New Zealand.

Methods

A semantic archetype comparison method was proposed to determine the semantics reuse degree of reused archetypes in 2 ACS-related clinical registries from 2 countries. This method involved (1) determining the scope of reused archetypes; (2) identifying corresponding data items within corresponding archetypes; (3) comparing the semantics of corresponding data items; and (4) calculating the number of mappings in corresponding data items and analyzing results.

Results

Among the related archetypes in the two ACS-related, openEHR-based clinical registries from China and New Zealand, there were 8 pairs of reusable archetypes, which included 89 pairs of corresponding data items and 120 noncorresponding data items. Of the 89 corresponding data item pairs, 87 pairs (98%) were mappable and therefore supported semantic interoperability, and 71 pairs (80%) were labeled as “direct mapping” data items. Of the 120 noncorresponding data items, 114 (95%) data items were generated via archetype evolution, and 6 (5%) data items were generated via archetype localization.

Conclusions

The results of the semantic comparison between the two ACS-related clinical registries prove the feasibility of establishing the semantic interoperability of health care data from different countries based on the openEHR approach. Archetype reuse provides data on the degree to which semantic interoperability exists when using the openEHR approach. Although the openEHR community has effectively promoted archetype reuse and semantic interoperability by providing archetype modeling methods, tools, model repositories, and archetype design patterns, the uncontrolled evolution of archetypes and inconsistent localization have resulted in major challenges for achieving higher levels of semantic interoperability.

Expressiveness of an International Semantic Standard for Wound Care: Mapping a Standardized Item Set for Leg Ulcers to the Systematized Nomenclature of Medicine-Clinical Terms

Background

Chronic health conditions are on the rise and are putting high economic pressure on health systems, as they require well-coordinated prevention and treatment. Among chronic conditions, chronic wounds such as cardiovascular leg ulcers have a high prevalence. Their treatment is highly interdisciplinary and regularly spans multiple care settings and organizations; this places particularly high demands on interoperable information exchange that can be achieved using international semantic standards, such as Systematized Nomenclature of Medicine–Clinical Terms (SNOMED CT).

Objective

This study aims to investigate the expressiveness of SNOMED CT in the domain of wound care, and thereby its clinical usefulness and the potential need for extensions.

Methods

A clinically consented and profession-independent wound care item set, the German National Consensus for the Documentation of Leg Wounds (NKDUC), was mapped onto the precoordinated concepts of the international reference terminology SNOMED CT. Before the mapping took place, the NKDUC was transformed into an information model that served to systematically identify relevant items. The mapping process was carried out in accordance with the ISO/TR 12300 formalism. As a result, the reliability, equivalence, and coverage rate were determined for all NKDUC items and sections.

Results

The developed information model revealed 268 items to be mapped. Conducted by 3 health care professionals, the mapping resulted in moderate reliability (κ=0.512). Regarding the two best equivalence categories (symmetrical equivalence of meaning), the coverage rate of SNOMED CT was 67.2% (180/268) overall and 64.3% (108/168) specifically for wounds. The sections general medical condition (55/66, 83%), wound assessment (18/24, 75%), and wound status (37/57, 65%), showed higher coverage rates compared with the sections therapy (45/73, 62%), wound diagnostics (8/14, 57%), and patient demographics (17/34, 50%).

Conclusions

The results yielded acceptable reliability values for the mapping procedure. The overall coverage rate shows that two-thirds of the items could be mapped symmetrically, which is a substantial portion of the source item set. Some wound care sections, such as general medical conditions and wound assessment, were covered better than other sections (wound status, diagnostics, and therapy). These deficiencies can be mitigated either by postcoordination or by the inclusion of new concepts in SNOMED CT. This study contributes to pushing interoperability in the domain of wound care, thereby responding to the high demand for information exchange in this field. Overall, this study adds another puzzle piece to the general knowledge about SNOMED CT in terms of its clinical usefulness and its need for further extensions.

Technological progress in electronic health record system optimization: Systematic review of systematic literature reviews

BACKGROUND

The recent, rapid development of digital technologies offers new possibilities for more efficient implementation of electronic health record (EHR) and personal health record (PHR) systems. A growing volume of healthcare data has been the hallmark of this digital transformation. The large healthcare datasets' complexity and their dynamic nature pose various challenges related to processing, analysis, storage, security, privacy, data exchange, and usability.

MATERIALS AND METHODS

We performed a systematic review of systematic reviews to assess technological progress in EHR and PHR systems. We searched MEDLINE, Cochrane, Web of Science, and Scopus for systematic literature reviews on technological advancements that support EHR and PHR systems published between January 1, 2010, and October 06, 2020.

RESULTS

The searches resulted in a total of 2,448 hits. Of these, we finally selected 23 systematic reviews. Most of the included papers dealt with information extraction tools and natural language processing technology (n = 10), followed by studies that assessed the use of blockchain technology in healthcare (n = 8). Other areas of digital technology research included EHR and PHR systems in austere settings (n = 1), de-identification methods (n = 1), visualization techniques (n = 1), communication tools within EHR and PHR systems (n = 1), and methodologies for defining Clinical Information Models that promoted EHRs and PHRs interoperability (n = 1).

CONCLUSIONS

Technological advancements can improve the efficiency in the implementation of EHR and PHR systems in numerous ways. Natural language processing techniques, either rule-based, machine-learning, or deep learning-based, can extract information from clinical narratives and other unstructured data locked in EHRs and PHRs, allowing secondary research (i.e., phenotyping). Moreover, EHRs and PHRs are expected to be the primary beneficiaries of the blockchain technology implementation on Health Information Systems. Governance regulations, lack of trust, poor scalability, security, privacy, low performance, and high cost remain the most critical challenges for implementing these technologies.

COVID-19 and the Digitalisation of Cardiovascular Training and Education-A Review of Guiding Themes for Equitable and Effective Post-graduate Telelearning

The coronavirus disease-2019 (COVID-19) pandemic has had an unprecedented impact leading to novel adaptations in post-graduate medical education for cardiovascular and general internal medicine. Whilst the results of initial community COVID-19 vaccination are awaited, continuation of multimodality teaching and training that incorporates telelearning will have enduring benefit to post-graduate education and will place educational establishments in good stead to nimbly respond in future pandemic-related public health emergencies. With the rise in innovative virtual learning solutions, medical educators will have to leverage technology to develop electronic educational materials and virtual courses that facilitate adult learning. Technology-enabled virtual learning is thus a timely progression of hybrid classroom initiatives that are already adopted to varying degrees, with a need for faculty to serve as subject matter experts, to host and moderate online discussions, and to provide feedback and overall mentorship. As an extension from existing efforts, simulation-based teaching (SBT) and learning and the use of mixed reality technology should also form a greater core in the cardiovascular medicine curriculum. We highlight five foundational themes for building a successful e-learning model in cardiovascular and general post-graduate medical training: (1) digital solutions and associated infrastructure; (2) equity in access; (3) participant engagement; (4) diversity and inclusion; and (5) patient confidentiality and governance framework. With digitalisation impacting our everyday lives and now how we teach and train in medicine, these five guiding principles provide a cognitive scaffold for careful consideration of the required ecosystem in which cardiovascular and general post-graduate medical education can effectively operate. With due consideration of various e-learning options and associated infrastructure needs; and adoption of strategies for participant engagement under sound and just governance, virtual training in medicine can be effective, inclusive and equitable through the COVID-19 era and beyond.

Using NLP in openEHR archetypes retrieval to promote interoperability: a feasibility study in China

BACKGROUND

With the development and application of medical information system, semantic interoperability is essential for accurate and advanced health-related computing and electronic health record (EHR) information sharing. The openEHR approach can improve semantic interoperability. One key improvement of openEHR is that it allows for the use of existing archetypes. The crucial problem is how to improve the precision and resolve ambiguity in the archetype retrieval.

METHOD

Based on the query expansion technology and Word2Vec model in Nature Language Processing (NLP), we propose to find synonyms as substitutes for original search terms in archetype retrieval. Test sets in different medical professional level are used to verify the feasibility.

RESULT

Applying the approach to each original search term (n = 120) in test sets, a total of 69,348 substitutes were constructed. Precision at 5 (P@5) was improved by 0.767, on average. For the best result, the P@5 was up to 0.975.

CONCLUSIONS

We introduce a novel approach that using NLP technology and corpus to find synonyms as substitutes for original search terms. Compared to simply mapping the element contained in openEHR to an external dictionary, this approach could greatly improve precision and resolve ambiguity in retrieval tasks. This is helpful to promote the application of openEHR and advance EHR information sharing.

Linking clinotypes to phenotypes and genotypes from laboratory test results in comprehensive physical exams

BACKGROUND

In this work, we aimed to demonstrate how to utilize the lab test results and other clinical information to support precision medicine research and clinical decisions on complex diseases, with the support of electronic medical record facilities. We defined "clinotypes" as clinical information that could be observed and measured objectively using biomedical instruments. From well-known 'omic' problem definitions, we defined problems using clinotype information, including stratifying patients-identifying interested sub cohorts for future studies, mining significant associations between clinotypes and specific phenotypes-diseases, and discovering potential linkages between clinotype and genomic information. We solved these problems by integrating public omic databases and applying advanced machine learning and visual analytic techniques on two-year health exam records from a large population of healthy southern Chinese individuals (size n = 91,354). When developing the solution, we carefully addressed the missing information, imbalance and non-uniformed data annotation issues.

RESULTS

We organized the techniques and solutions to address the problems and issues above into CPA framework (Clinotype Prediction and Association-finding). At the data preprocessing step, we handled the missing value issue with predicted accuracy of 0.760. We curated 12,635 clinotype-gene associations. We found 147 Associations between 147 chronic diseases-phenotype and clinotypes, which improved the disease predictive performance to AUC (average) of 0.967. We mined 182 significant clinotype-clinotype associations among 69 clinotypes.

CONCLUSIONS

Our results showed strong potential connectivity between the omics information and the clinical lab test information. The results further emphasized the needs to utilize and integrate the clinical information, especially the lab test results, in future PheWas and omic studies. Furthermore, it showed that the clinotype information could initiate an alternative research direction and serve as an independent field of data to support the well-known 'phenome' and 'genome' researches.

A refined methodology for validation of information models derived from flowsheet data and applied to a genitourinary case

Use of electronic health record data is expanding to support quality improvement and research; however, this requires standardization of the data and validation within and across organizations. Information models (IMs) are created to standardize data elements into a logical organization that includes data elements, definitions, data types, values, and relationships. To be generalizable, these models need to be validated across organizations. The purpose of this case report is to describe a refined methodology for validation of flowsheet IMs and apply the revised process to a genitourinary IM created in one organization. The refined IM process, adding evidence and input from experts, produced a clinically relevant and evidence-based model of genitourinary care. The refined IM process provides a foundation for optimizing electronic health records with comparable nurse sensitive data that can add to common data models for continuity of care and ongoing use for quality improvement and research.

Quantification and visualisation methods of data-driven chronic care delivery pathways: protocol for a systematic review and content analysis

INTRODUCTION

Chronic conditions require long periods of care and often involve repeated interactions with multiple healthcare providers. Faced with increasing illness burden and costs, healthcare systems are currently working towards integrated care to streamline these interactions and improve efficiency. To support this, one promising resource is the information on routine care delivery stored in various electronic healthcare databases (EHD). In chronic conditions, care delivery pathways (CDPs) can be constructed by linking multiple data sources and extracting time-stamped healthcare utilisation events and other medical data related to individual or groups of patients over specific time periods; CDPs may provide insights into current practice and ways of improving it. Several methods have been proposed in recent years to quantify and visualise CDPs. We present the protocol for a systematic review aiming to describe the content and development of CDP methods, to derive common recommendations for CDP construction.

METHODS AND ANALYSIS

This protocol followed the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols. A literature search will be performed in PubMed (MEDLINE), Scopus, IEEE, CINAHL and EMBASE, without date restrictions, to review published papers reporting data-driven chronic CDPs quantification and visualisation methods. We will describe them using several characteristics relevant for EHD use in long-term care, grouped into three domains: (1) clinical (what clinical information does the method use and how was it considered relevant?), (2) data science (what are the method's development and implementation characteristics?) and (3) behavioural (which behaviours and interactions does the method aim to promote among users and how?). Data extraction will be performed via deductive content analysis using previously defined characteristics and accompanied by an inductive analysis to identify and code additional relevant features. Results will be presented in descriptive format and used to compare current CDPs and generate recommendations for future CDP development initiatives.

ETHICS AND DISSEMINATION

Database searches will be initiated in May 2019. The review is expected to be completed by February 2020. Ethical approval is not required for this review. Results will be disseminated in peer-reviewed journals and conference presentations.

PROSPERO REGISTRATION NUMBER

CRD42019140494.

ToxRefDB version 2.0: Improved utility for predictive and retrospective toxicology analyses

The Toxicity Reference Database (ToxRefDB) structures information from over 5000 in vivo toxicity studies, conducted largely to guidelines or specifications from the US Environmental Protection Agency and the National Toxicology Program, into a public resource for training and validation of predictive models. Herein, ToxRefDB version 2.0 (ToxRefDBv2) development is described. Endpoints were annotated (e.g. required, not required) according to guidelines for subacute, subchronic, chronic, developmental, and multigenerational reproductive designs, distinguishing negative responses from untested. Quantitative data were extracted, and dose-response modeling for nearly 28,000 datasets from nearly 400 endpoints using Benchmark Dose (BMD) Modeling Software were generated and stored. Implementation of controlled vocabulary improved data quality; standardization to guideline requirements and cross-referencing with United Medical Language System (UMLS) connects ToxRefDBv2 observations to vocabularies linked to UMLS, including PubMed medical subject headings. ToxRefDBv2 allows for increased connections to other resources and has greatly enhanced quantitative and qualitative utility for predictive toxicology.

Discovering Clinical Information Models Online to Promote Interoperability of Electronic Health Records: A Feasibility Study of OpenEHR

Background

Clinical information models (CIMs) enabling semantic interoperability are crucial for electronic health record (EHR) data use and reuse. Dual model methodology, which distinguishes the CIMs from the technical domain, could help enable the interoperability of EHRs at the knowledge level. How to help clinicians and domain experts discover CIMs from an open repository online to represent EHR data in a standard manner becomes important.

Objective

This study aimed to develop a retrieval method to identify CIMs online to represent EHR data.

Methods

We proposed a graphical retrieval method and validated its feasibility using an online CIM repository: openEHR Clinical Knowledge Manager (CKM). First, we represented CIMs (archetypes) using an extended Bayesian network. Then, an inference process was run in the network to discover relevant archetypes. In the evaluation, we defined three retrieval tasks (medication, laboratory test, and diagnosis) and compared our method with three typical retrieval methods (BM25F, simple Bayesian network, and CKM), using mean average precision (MAP), average precision (AP), and precision at 10 (P@10) as evaluation metrics.

Results

We downloaded all available archetypes from the CKM. Then, the graphical model was applied to represent the archetypes as a four-level clinical resources network. The network consisted of 5513 nodes, including 3982 data element nodes, 504 concept nodes, 504 duplicated concept nodes, and 523 archetype nodes, as well as 9867 edges. The results showed that our method achieved the best MAP (MAP=0.32), and the AP was almost equal across different retrieval tasks (AP=0.35, 0.31, and 0.30, respectively). In the diagnosis retrieval task, our method could successfully identify the models covering “diagnostic reports,” “problem list,” “patients background,” “clinical decision,” etc, as well as models that other retrieval methods could not find, such as “problems and diagnoses.”

Conclusions

The graphical retrieval method we propose is an effective approach to meet the uncertainty of finding CIMs. Our method can help clinicians and domain experts identify CIMs to represent EHR data in a standard manner, enabling EHR data to be exchangeable and interoperable.

Big data hurdles in precision medicine and precision public health

BACKGROUND

Nowadays, trendy research in biomedical sciences juxtaposes the term 'precision' to medicine and public health with companion words like big data, data science, and deep learning. Technological advancements permit the collection and merging of large heterogeneous datasets from different sources, from genome sequences to social media posts or from electronic health records to wearables. Additionally, complex algorithms supported by high-performance computing allow one to transform these large datasets into knowledge. Despite such progress, many barriers still exist against achieving precision medicine and precision public health interventions for the benefit of the individual and the population.

MAIN BODY

The present work focuses on analyzing both the technical and societal hurdles related to the development of prediction models of health risks, diagnoses and outcomes from integrated biomedical databases. Methodological challenges that need to be addressed include improving semantics of study designs: medical record data are inherently biased, and even the most advanced deep learning's denoising autoencoders cannot overcome the bias if not handled a priori by design. Societal challenges to face include evaluation of ethically actionable risk factors at the individual and population level; for instance, usage of gender, race, or ethnicity as risk modifiers, not as biological variables, could be replaced by modifiable environmental proxies such as lifestyle and dietary habits, household income, or access to educational resources.

CONCLUSIONS

Data science for precision medicine and public health warrants an informatics-oriented formalization of the study design and interoperability throughout all levels of the knowledge inference process, from the research semantics, to model development, and ultimately to implementation.

Health Data for Research Through a Nationwide Privacy-Proof System in Belgium: Design and Implementation

Background

Health data collected during routine care have important potential for reuse for other purposes, especially as part of a learning health system to advance the quality of care. Many sources of bias have been identified through the lifecycle of health data that could compromise the scientific integrity of these data. New data protection legislation requires research facilities to improve safety measures and, thus, ensure privacy.

Objective

This study aims to address the question on how health data can be transferred from various sources and using multiple systems to a centralized platform, called Healthdata.be, while ensuring the accuracy, validity, safety, and privacy. In addition, the study demonstrates how these processes can be used in various research designs relevant for learning health systems.

Methods

The Healthdata.be platform urges uniformity of the data registration at the primary source through the use of detailed clinical models. Data retrieval and transfer are organized through end-to-end encrypted electronic health channels, and data are encoded using token keys. In addition, patient identifiers are pseudonymized so that health data from the same patient collected across various sources can still be linked without compromising the deidentification.

Results

The Healthdata.be platform currently collects data for >150 clinical registries in Belgium. We demonstrated how the data collection for the Belgian primary care morbidity register INTEGO is organized and how the Healthdata.be platform can be used for a cluster randomized trial.

Conclusions

Collecting health data in various sources and linking these data to a single patient is a promising feature that can potentially address important concerns on the validity and quality of health data. Safe methods of data transfer without compromising privacy are capable of transporting these data from the primary data provider or clinician to a research facility. More research is required to demonstrate that these methods improve the quality of data collection, allowing researchers to rely on electronic health records as a valid source for scientific data.

Semantic interoperability challenges to process large amount of data perspectives in forensic and legal medicine

This article is a position paper dealing with semantic interoperability challenges. It addresses the Variety and Veracity dimensions when integrating, sharing and reusing large amount of heterogeneous data for data analysis and decision making applications in the healthcare domain. Many issues are raised by the necessity to conform Big Data to interoperability standards. We discuss how semantics can contribute to the improvement of information sharing and address the problem of data mediation with domain ontologies. We then introduce the main steps for building domain ontologies as they could be implemented in the context of Forensic and Legal medicine. We conclude with a particular emphasis on the current limitations in standardisation and the importance of knowledge formalization.

The Rise of Big Data in Oncology

OBJECTIVES

To describe big data and data science in the context of oncology nursing care.

DATA SOURCES

Peer-reviewed and lay publications.

CONCLUSION

The rapid expansion of real-world evidence from sources such as the electronic health record, genomic sequencing, administrative claims and other data sources has outstripped the ability of clinicians and researchers to manually review and analyze it. To promote high-quality, high-value cancer care, big data platforms must be constructed from standardized data sources to support extraction of meaningful, comparable insights.

IMPLICATIONS FOR NURSING PRACTICE

Nurses must advocate for the use of standardized vocabularies and common data elements that represent terms and concepts that are meaningful to patient care.

Archetype modeling methodology

Clinical Information Models (CIMs) expressed as archetypes play an essential role in the design and development of current Electronic Health Record (EHR) information structures. Although there exist many experiences about using archetypes in the literature, a comprehensive and formal methodology for archetype modeling does not exist. Having a modeling methodology is essential to develop quality archetypes, in order to guide the development of EHR systems and to allow the semantic interoperability of health data. In this work, an archetype modeling methodology is proposed. This paper describes its phases, the inputs and outputs of each phase, and the involved participants and tools. It also includes the description of the possible strategies to organize the modeling process. The proposed methodology is inspired by existing best practices of CIMs, software and ontology development. The methodology has been applied and evaluated in regional and national EHR projects. The application of the methodology provided useful feedback and improvements, and confirmed its advantages. The conclusion of this work is that having a formal methodology for archetype development facilitates the definition and adoption of interoperable archetypes, improves their quality, and facilitates their reuse among different information systems and EHR projects. Moreover, the proposed methodology can be also a reference for CIMs development using any other formalism.

Combining multivariate statistics and the think-aloud protocol to assess Human-Computer Interaction barriers in symptom checkers

Symptom checkers are software tools that allow users to submit a set of symptoms and receive advice related to them in the form of a diagnosis list, health information or triage. The heterogeneity of their potential users and the number of different components in their user interfaces can make testing with end-users unaffordable. We designed and executed a two-phase method to test the respiratory diseases module of the symptom checker Erdusyk. Phase I consisted of an online test with a large sample of users (n=53). In Phase I, users evaluated the system remotely and completed a questionnaire based on the Technology Acceptance Model. Principal Component Analysis was used to correlate each section of the interface with the questionnaire responses, thus identifying which areas of the user interface presented significant contributions to the technology acceptance. In the second phase, the think-aloud procedure was executed with a small number of samples (n=15), focusing on the areas with significant contributions to analyze the reasons for such contributions. Our method was used effectively to optimize the testing of symptom checker user interfaces. The method allowed kept the cost of testing at reasonable levels by restricting the use of the think-aloud procedure while still assuring a high amount of coverage. The main barriers detected in Erdusyk were related to problems understanding time repetition patterns, the selection of levels in scales to record intensities, navigation, the quantification of some symptom attributes, and the characteristics of the symptoms.

A Fast Healthcare Interoperability Resources (FHIR) layer implemented over i2b2

BACKGROUND

Standards and technical specifications have been developed to define how the information contained in Electronic Health Records (EHRs) should be structured, semantically described, and communicated. Current trends rely on differentiating the representation of data instances from the definition of clinical information models. The dual model approach, which combines a reference model (RM) and a clinical information model (CIM), sets in practice this software design pattern. The most recent initiative, proposed by HL7, is called Fast Health Interoperability Resources (FHIR). The aim of our study was to investigate the feasibility of applying the FHIR standard to modeling and exposing EHR data of the Georges Pompidou European Hospital (HEGP) integrating biology and the bedside (i2b2) clinical data warehouse (CDW).

RESULTS

We implemented a FHIR server over i2b2 to expose EHR data in relation with five FHIR resources: DiagnosisReport, MedicationOrder, Patient, Encounter, and Medication. The architecture of the server combines a Data Access Object design pattern and FHIR resource providers, implemented using the Java HAPI FHIR API. Two types of queries were tested: query type #1 requests the server to display DiagnosticReport resources, for which the diagnosis code is equal to a given ICD-10 code. A total of 80 DiagnosticReport resources, corresponding to 36 patients, were displayed. Query type #2, requests the server to display MedicationOrder, for which the FHIR Medication identification code is equal to a given code expressed in a French coding system. A total of 503 MedicationOrder resources, corresponding to 290 patients, were displayed. Results were validated by manually comparing the results of each request to the results displayed by an ad-hoc SQL query.

CONCLUSION

We showed the feasibility of implementing a Java layer over the i2b2 database model to expose data of the CDW as a set of FHIR resources. An important part of this work was the structural and semantic mapping between the i2b2 model and the FHIR RM. To accomplish this, developers must manually browse the specifications of the FHIR standard. Our source code is freely available and can be adapted for use in other i2b2 sites.

Cross border semantic interoperability for learning health systems: The EHR4CR semantic resources and services

With the development of platforms enabling the integration and use of phenome, genome, and exposome data in the context of international research, data management challenges are increasing, and scalable solutions for cross border and cross domain semantic interoperability need to be developed. Reusing routinely collected clinical data, especially, requires computable portable phenotype algorithms running across different electronic health record (EHR) products and healthcare systems. We propose a framework for describing and comparing mediation platforms enabling cross border phenotype identification within federated EHRs. This framework was used to describe the experience gained during the EHR4CR project and the evaluation of the platform developed for accessing semantically equivalent data elements across 11 European participating EHR systems from 5 countries. Developers of semantic interoperability platforms are beginning to address a core set of requirements in order to reach the goal of developing cross border semantic integration of data.

An information model for computable cancer phenotypes

BACKGROUND

Standards, methods, and tools supporting the integration of clinical data and genomic information are an area of significant need and rapid growth in biomedical informatics. Integration of cancer clinical data and cancer genomic information poses unique challenges, because of the high volume and complexity of clinical data, as well as the heterogeneity and instability of cancer genome data when compared with germline data. Current information models of clinical and genomic data are not sufficiently expressive to represent individual observations and to aggregate those observations into longitudinal summaries over the course of cancer care. These models are acutely needed to support the development of systems and tools for generating the so called clinical "deep phenotype" of individual cancer patients, a process which remains almost entirely manual in cancer research and precision medicine.

METHODS

Reviews of existing ontologies and interviews with cancer researchers were used to inform iterative development of a cancer phenotype information model. We translated a subset of the Fast Healthcare Interoperability Resources (FHIR) models into the OWL 2 Description Logic (DL) representation, and added extensions as needed for modeling cancer phenotypes with terms derived from the NCI Thesaurus. Models were validated with domain experts and evaluated against competency questions.

RESULTS

The DeepPhe Information model represents cancer phenotype data at increasing levels of abstraction from mention level in clinical documents to summaries of key events and findings. We describe the model using breast cancer as an example, depicting methods to represent phenotypic features of cancers, tumors, treatment regimens, and specific biologic behaviors that span the entire course of a patient's disease.

CONCLUSIONS

We present a multi-scale information model for representing individual document mentions, document level classifications, episodes along a disease course, and phenotype summarization, linking individual observations to high-level summaries in support of subsequent integration and analysis.

Too many, too few, or too unsafe? Impact of inappropriate prescribing on mortality, and hospitalization in a cohort of community-dwelling oldest old

AIMS

Little is known about the impact of inappropriate prescribing (IP) in community-dwelling adults, aged 80 years and older. The prevalence at baseline (November 2008September 2009) and impact of IP (misuse and underuse) after 18 months on mortality and hospitalization in a cohort of community-dwelling adults, aged 80 years and older (n = 503) was studied.

METHODS

Screening Tool of Older People's Prescriptions (STOPP-2, misuse) and Screening Tool to Alert to Right Treatment (START-2, underuse) criteria were cross-referenced and linked to the medication use (in Anatomical Therapeutic Chemical coding) and clinical problems. Survival analysis until death or first hospitalization was performed at 18 months after inclusion using Kaplan-Meier, with Cox regression to control for covariates.

RESULTS

Mean age was 84.4 (range 80-102) years. Mean number of medications prescribed was 5 (range 0-16). Polypharmacy (≥5 medications, 58%), underuse (67%) and misuse (56%) were high. Underuse and misuse coexisted in 40% and were absent in 17% of the population. A higher number of prescribed medications was correlated with more misused medications (r_s  = .51, P < 0.001) and underused medications (r_s  = .26, P < 0.001). Mortality and hospitalization rate were 8.9%, and 31.0%, respectively. After adjustment for number of medications and misused medications, there was an increased risk of mortality (HR 1.39, 95% CI 1.10, 1.76) and hospitalization (HR 1.26, 95% CI 1.10, 1.45) for every additional underused medication. Associations with misuse were less clear.

CONCLUSION

IP (polypharmacy, underuse and misuse) was highly prevalent in adults, aged 80 years and older. Surprisingly, underuse and not misuse had strong associations with mortality and hospitalization.

Evaluation of clinical information modeling tools

Objective Clinical information models are formal specifications for representing the structure and semantics of the clinical content within electronic health record systems. This research aims to define, test, and validate evaluation metrics for software tools designed to support the processes associated with the definition, management, and implementation of these models.

Methodology The proposed framework builds on previous research that focused on obtaining agreement on the essential requirements in this area. A set of 50 conformance criteria were defined based on the 20 functional requirements agreed by that consensus and applied to evaluate the currently available tools.

Results Of the 11 initiative developing tools for clinical information modeling identified, 9 were evaluated according to their performance on the evaluation metrics. Results show that functionalities related to management of data types, specifications, metadata, and terminology or ontology bindings have a good level of adoption. Improvements can be made in other areas focused on information modeling and associated processes. Other criteria related to displaying semantic relationships between concepts and communication with terminology servers had low levels of adoption.

Conclusions The proposed evaluation metrics were successfully tested and validated against a representative sample of existing tools. The results identify the need to improve tool support for information modeling and software development processes, especially in those areas related to governance, clinician involvement, and optimizing the technical validation of testing processes. This research confirmed the potential of these evaluation metrics to support decision makers in identifying the most appropriate tool for their organization.

OBJECTIVO Los Modelos de Información Clínica son especificaciones para representar la estructura y características semánticas del contenido clínico en los sistemas de Historia Clínica Electrónica. Esta investigación define, prueba y valida un marco para la evaluación de herramientas informáticas diseñadas para dar soporte en la en los procesos de definición, gestión e implementación de estos modelos.

METODOLOGIA El marco de evaluación propuesto se basa en una investigación previa para obtener consenso en la definición de requisitos esenciales en esta área. A partir de los 20 requisitos funcionales acordados, un conjunto de 50 criterios de conformidad fueron definidos y aplicados en la evaluación de las herramientas existentes.

RESULTADOS Un total de 9 de las 11 iniciativas identificadas desarrollando herramientas para el modelado de información clínica fueron evaluadas. Los resultados muestran que las funcionalidades relacionadas con la gestión de tipos de datos, especificaciones, metadatos y mapeo con terminologías u ontologías tienen un buen nivel de adopción. Se identifican posibles mejoras en áreas relacionadas con los procesos de modelado de información. Otros criterios relacionados con presentar las relaciones semánticas entre conceptos y la comunicación con servidores de terminología tienen un bajo nivel de adopción.

CONCLUSIONES El marco de evaluación propuesto fue probado y validado satisfactoriamente contra un conjunto representativo de las herramientas existentes. Los resultados identifican la necesidad de mejorar el soporte de herramientas a los procesos de modelado de información y desarrollo de software, especialmente en las áreas relacionadas con gobernanza, participación de profesionales clínicos y la optimización de la validación técnica en los procesos de pruebas técnicas. Esta investigación ha confirmado el potencial de este marco de evaluación para dar soporte a los usuarios en la toma de decisiones sobre que herramienta es más apropiadas para su organización.

Requirements for clinical information modelling tools

OBJECTIVE

This study proposes consensus requirements for clinical information modelling tools that can support modelling tasks in medium/large scale institutions. Rather than identify which functionalities are currently available in existing tools, the study has focused on functionalities that should be covered in order to provide guidance about how to evolve the existing tools.

METHODOLOGY

After identifying a set of 56 requirements for clinical information modelling tools based on a literature review and interviews with experts, a classical Delphi study methodology was applied to conduct a two round survey in order to classify them as essential or recommended. Essential requirements are those that must be met by any tool that claims to be suitable for clinical information modelling, and if we one day have a certified tools list, any tool that does not meet essential criteria would be excluded. Recommended requirements are those more advanced requirements that may be met by tools offering a superior product or only needed in certain modelling situations.

RESULTS

According to the answers provided by 57 experts from 14 different countries, we found a high level of agreement to enable the study to identify 20 essential and 21 recommended requirements for these tools.

CONCLUSIONS

It is expected that this list of identified requirements will guide developers on the inclusion of new basic and advanced functionalities that have strong support by end users. This list could also guide regulators in order to identify requirements that could be demanded of tools adopted within their institutions.