Development and Integration of Machine Learning Algorithm to Identify Peripheral Arterial Disease: Multistakeholder Qualitative Study

BACKGROUND

Machine learning (ML)-driven clinical decision support (CDS) continues to draw wide interest and investment as a means of improving care quality and value, despite mixed real-world implementation outcomes.

OBJECTIVE

This study aimed to explore the factors that influence the integration of a peripheral arterial disease (PAD) identification algorithm to implement timely guideline-based care.

METHODS

A total of 12 semistructured interviews were conducted with individuals from 3 stakeholder groups during the first 4 weeks of integration of an ML-driven CDS. The stakeholder groups included technical, administrative, and clinical members of the team interacting with the ML-driven CDS. The ML-driven CDS identified patients with a high probability of having PAD, and these patients were then reviewed by an interdisciplinary team that developed a recommended action plan and sent recommendations to the patient's primary care provider. Pseudonymized transcripts were coded, and thematic analysis was conducted by a multidisciplinary research team.

RESULTS

Three themes were identified: positive factors translating in silico performance to real-world efficacy, organizational factors and data structure factors affecting clinical impact, and potential challenges to advancing equity. Our study found that the factors that led to successful translation of in silico algorithm performance to real-world impact were largely nontechnical, given adequate efficacy in retrospective validation, including strong clinical leadership, trustworthy workflows, early consideration of end-user needs, and ensuring that the CDS addresses an actionable problem. Negative factors of integration included failure to incorporate the on-the-ground context, the lack of feedback loops, and data silos limiting the ML-driven CDS. The success criteria for each stakeholder group were also characterized to better understand how teams work together to integrate ML-driven CDS and to understand the varying needs across stakeholder groups.

CONCLUSIONS

Longitudinal and multidisciplinary stakeholder engagement in the development and integration of ML-driven CDS underpins its effective translation into real-world care. Although previous studies have focused on the technical elements of ML-driven CDS, our study demonstrates the importance of including administrative and operational leaders as well as an early consideration of clinicians' needs. Seeing how different stakeholder groups have this more holistic perspective also permits more effective detection of context-driven health care inequities, which are uncovered or exacerbated via ML-driven CDS integration through structural and organizational challenges. Many of the solutions to these inequities lie outside the scope of ML and require coordinated systematic solutions for mitigation to help reduce disparities in the care of patients with PAD.

Local Adaptation Without Work Intensification: Experimentalist Governance of Digital Technology for Mutually Beneficial Role Reconfiguration in Organizations

This 1.5-year ethnographic study of a U.S. medical center shows that avoiding loss of autonomy and work intensification for less powerful actors during digital technology introduction and integration presents a multisited collective action challenge. I found that technology-related participation problems, threshold problems, and free rider problems may arise during digital technology introduction and integration that enable loss of autonomy and work intensification for less powerful actors. However, the emergence of new triangles of power allows for novel coalitions between less powerful actors and newly powerful third-party actors that can help mitigate this problem. I extend the political science perspective of experimentalist governance to examine how a digital technology-focused, iterative collective action process of local experimentation followed by central revision can facilitate mutually beneficial role reconfiguration during digital technology introduction and integration. In experimentalist governance of digital technology, local units are given discretion to adapt digital technologies to their specific contexts. A central unit composed of diverse actors then reviews progress across local units integrating similar digital technology to negotiate a new shared understanding of mutually beneficial technology-related tasks for each group of actors. The central unit modifies both local routines and the technology itself in response to problems and possibilities revealed by the central revision process, and the cycle repeats. Here, accomplishing mutually beneficial role reconfiguration occurs through an experimentalist, collective action process rather than through a labor-management bargaining process or a professional-led tuning process.

Pragmatic AI-augmentation in mental healthcare: Key technologies, potential benefits, and real-world challenges and solutions for frontline clinicians

The integration of artificial intelligence (AI) technologies into mental health holds the promise of increasing patient access, engagement, and quality of care, and of improving clinician quality of work life. However, to date, studies of AI technologies in mental health have focused primarily on challenges that policymakers, clinical leaders, and data and computer scientists face, rather than on challenges that frontline mental health clinicians are likely to face as they attempt to integrate AI-based technologies into their everyday clinical practice. In this Perspective, we describe a framework for "pragmatic AI-augmentation" that addresses these issues by describing three categories of emerging AI-based mental health technologies which frontline clinicians can leverage in their clinical practice-automation, engagement, and clinical decision support technologies. We elaborate the potential benefits offered by these technologies, the likely day-to-day challenges they may raise for mental health clinicians, and some solutions that clinical leaders and technology developers can use to address these challenges, based on emerging experience with the integration of AI technologies into clinician daily practice in other healthcare disciplines.

Moving Violations: Pairing an Illegitimate Learning Hierarchy with Trainee Status Mobility for Acquiring New Skills When Traditional Expertise Erodes

We explore how members of a community of practice learn new tools and techniques when environmental shifts undermine existing expertise. In our 20-month comparative field study of medical assistants and patient-service representatives learning to use new digital technology in five primary care sites, we find that the traditional master-apprentice training model worked well when established practices were being conferred to trainees. When environmental change required introducing new tools and techniques with which the experienced members had no expertise, third-party managers selected newer members as trainers because managers judged them to be agile learners who were less committed to traditional hierarchies and more willing to deviate from traditional norms. This challenged community members’ existing status, which was based on the historical distinctions of long tenure and expertise in traditional tasks. In three sites, the introduction of this illegitimate learning hierarchy sparked status competition among trainees and trainers, and trainees collectively resisted learning new tools and techniques. In the other two sites, managers paired the new, illegitimate learning hierarchy with the opportunity for trainee status mobility by rotating the trainer role; here, trainees embraced learning in order to exit the lower-status trainee group and join the higher-status trainer group. Drawing on ideas of status group legitimacy and mobility, we suggest that managers’ pairing of an illegitimate learning hierarchy with the opportunity for trainee status mobility is a mechanism for enabling the situated learning of new techniques when traditional expertise erodes.

The Radical Flank Effect and Cross-occupational Collaboration for Technology Development during a Power Shift

This 12-month ethnographic study of an early entrant into the U.S. car-sharing industry demonstrates that when an organization shifts its focus from developing radical new technology to incrementally improving this technology, the shift may spark an internal power struggle between the dominant engineering group and a challenger occupational group such as the marketing group. Analyzing 42 projects in two time periods that required collaboration between engineering and marketing during such a shift, we show how cross-occupational collaboration under these conditions can be facilitated by a radical flank threat, through which the bargaining power of moderates is strengthened by the presence of a more-radical group. In the face of a strong threat by radical members of a challenger occupational group, moderate members of the dominant engineering group may change their perceptions of their power to resist challengers' demands and begin to distinguish between the goals of radical versus more-moderate challengers. To maintain as much power as possible and prevent the more-dramatic change in engineering occupational goals demanded by radical challengers, moderate engineers may build a coalition with moderate challengers and collaborate for incremental technology development.

Operating room: relational spaces and microinstitutional change in surgery

One of the great paradoxes of institutional change is that even when top managers in organizations provide support for change in response to new regulation, the employees whom new programs are designed to benefit often do not use them. This 15-month ethnographic study of two hospitals responding to new regulation demonstrates that using these programs may require subordinate employees to challenge middle managers with opposing interests. The article argues that relational spaces--areas of isolation, interaction, and inclusion that allow middle-manager reformers and subordinate employees to develop a cross-position collective for change--are critical to the change process. These findings have implications for research on institutional change and social movements.

Career satisfaction of women in surgery: perceptions, factors, and strategies

BACKGROUND

With the current and projected shortages of general surgeons, more attention is being paid to the increasing pool of women physicians. This study seeks to understand the variables leading to career satisfaction for women surgeons to better recruit, retain, and support them.

STUDY DESIGN

Eighteen semi-structured interviews of 12 female and 6 male surgeons 2 to 12 years into practice were qualitatively analyzed and converted to coded, categorized data. Significance was derived by Fisher's exact test. Participants were recruited by snowball sampling.

RESULTS

Our sample represents a highly satisfied group of female and male surgeons. Although both women and men describe with equal frequency having made career tradeoffs for personal and family time, and vice versa, women far more frequently than men cite reasons related to their personal time, predictable time, and family relationships as why they are currently satisfied with their career (34.1% versus 8.7%; p < 0.05). Both cite being satisfied by career content equally. When describing strategies used in developing a successful surgical career, women most frequently cite social networks as a key to success (88% versus 12% by men; p < 0.05), and men more frequently cite reasons related to training (29% versus 0% by women; p < 0.05) and compensation (24% versus 0% by women; p < 0.05).

CONCLUSIONS

Although both men and women make tradeoffs of career for family and family for career, women's perception of satisfaction comes from viewing their surgical career within the broader context of their lives. Women might be attracted to a career that acknowledges and values the whole person beyond the surgeon, and could benefit from work infrastructures that enhance networking.

The impact of the 80-hour resident workweek on surgical residents and attending surgeons

OBJECTIVE

To assess the impact of the 80-hour resident workweek restrictions on surgical residents and attending surgeons.

SUMMARY BACKGROUND DATA

The ACGME mandated resident duty hour restrictions have required a major workforce restructuring. The impact of these changes needs to be critically evaluated for both the resident and attending surgeons, specifically with regards to the impact on motivation, job satisfaction, the quality of surgeon training, the quality of the surgeon's life, and the quality of patient care.

METHODS

Four prospective studies were performed at a single academic surgical program with data collected both before the necessary workforce restructuring and 1 year after, including: 1) time cards to assess changes in components of daily activity; 2) Web-based surveys using validated instruments to assess burnout and motivation to work; 3) structured, taped, one-on-one interviews with an external PhD investigator; and 4) statistical analyses of objective, quantitative data.

RESULTS

After the work-hour changes, surgical residents have decreased "burnout" scores, with significantly less "emotional exhaustion" (Maslach Burnout Inventory: 29.1 "high" vs. 23.1 "medium," P = 0.02). Residents have better quality of life both in and out of the hospital. They felt they got more sleep, have a lighter workload, and have increased motivation to work (Herzberg Motivation Dimensions). We found no measurable, statistically significant difference in the quality of patient care (NSQIP data). Resident training and education objectively were not statistically diminished (ACGME case logs, ABSITE scores). Attending surgeons perceived that their quality of their life inside and outside of the hospital was "somewhat worse" because of the work-hour changes, as they had anticipated. Many concerns were identified with regards to the professional development of future surgeons, including a change toward a shift-worker mentality that is not patient-focused, less continuity of care with a loss of critical information with each handoff, and a decrease in the patient/doctor relationship.

CONCLUSION

Although the mandated restriction of resident duty hours has had no measurable impact on the quality of patient care and has led to improvements for the current quality of life of residents, there are many concerns with regards to the training of professional, responsible surgeons for the future.

Resistance to change in surgical residency: an ethnographic study of work hours reform

BACKGROUND

Although the practical challenges to work hour restrictions have been the focus of much discussion, cultural resistance to such change has received less attention. Surgical residency has its own unique social structure, and we hypothesized that challenges to this would provide impediments to successful implementation of duty hours reform.

STUDY DESIGN

We used ethnographic research methods to study the efforts at work hour restriction over a 15-month period before the introduction of the Accreditation Council for Graduate Medical Education regulations. These methods, validated for studying institutional change, build on intense periods of observation. Records of observations are then analyzed and coded to uncover cultural and political challenges. The frequency of successful hand-offs in sign-out situations between day and night float residents was measured as an objective index of success.

RESULTS

Practical issues were addressed initially by scheduling adjustments including creating a night float system. The hand-offs that this system required, however, were successful only 14% of the time. Subsequent steps to address the challenge to resident identity by top-down support of a new definition of professionalism increased the number of successful hand-offs to 39%. Finally, a reduction in a noted hierarchy violation led to successful hand-offs 79% of the time.

CONCLUSIONS

These results demonstrate that practical solutions alone may not be a sufficient basis for change in surgical residency. While we face other challenges to the traditional surgical culture, attention to social and political issues may enhance the success of our efforts.

Effect of work-hour reforms on operative case volume of surgical residents

PURPOSE

There has been much concern regarding the impact of work-hour reforms on the operative case volume of surgical residents. Operative case volume by PGY year and clinical rotation were examined to determine if changes in work hours affected residents' operative case volume.

METHODS

A careful but aggressive plan of work-hour reduction was devised for the residency of the authors' institution with the goal to decrease work hours while maintaining optimal patient care and resident education, including operative case volume. Changes made included hiring physician extenders (PEs), decreasing call schedules to every fourth night (with the next day free from clinical activities-Q4) or call from home (HC), and night float rotation coverage for services (NF). Case volume before (academic year 2002) and after (academic year 2003) changes were compared by PGY year, for all residents and for specific rotations-private general surgery, which had changes of PE, HC, and NF for PGY5; PE, Q4 and NF for PGY1 and 10% exemption for work hours; Churchill service (a resident-run ward teaching service), which had changes of PE and Q4 for PGY5 and PGY1 and 10% exemption for work hours; and vascular surgery, which had HC and NF for PGY5. Total case volume on these services was likewise compared. Statistical analysis was by student t test. Operative case volume was measured with data from the resident-entered information on the ACGME Surgical Operative Log (SOL). Case volume for PGY4 residents could not be compared over this time period because of lack of access to archived data by PGY year for graduated residents through the ACGME SOL. Work hours before and after rotation changes were measured with an intranet-based monitoring system. This article is a retrospective review of the affects of these work-hour changes on operative case volume.

RESULTS

Total case volume for the general surgical services (both private and Churchill) was unchanged over this period (5905 in 02, 5930 in 03), and likewise for the vascular service (1101 vs 1196). Overall, there was no change in mean operative volume per year for surgical residents in this program (231 cases in 2002, 246 cases in 2003; p = 0.61). For PGY5 residents, the case volume increased; 339 cases 02, 390 in 03, and p = 0.05. Mean case volume for PGY5 residents increased on the private general surgery service (136 in 02, 160 in 03, p = 0.03), but it remained stable on the Churchill service (137 in 02, 158 in 03, p = 0.39) and vascular service (65 in 02, 73 in 03, p = 0.42). For PGY3 residents, case volume remained stable (171 in 02, 187 in 03, p = 0.62), as it did for PGY2 and PGY1 residents (PGY2: 148 in 02, 121 in 03, p = 0.12; PGY1: 265 in 02, 246 in 03, p = 0.23). However, operative case volume for PGY1 residents did decrease on the private general surgery service (mean 52 cases per month 02, 43 cases per month 03, p = 0.07), while remaining stable on the Churchill service (mean 23 cases per month 02, 25 cases per month 03, p = 0.66). Average hours worked per week decreased significantly over the time period. Before work-hour reforms, residents' average work hours were as follows: PGY1 105, PGY2 97, PGY3 78.7, PGY4 111, and PGY5 92. After the changes, average work hours were PGY1 81.5, PGY2 77.7, PGY3 78.7, PGY4 75.5, and PGY5 75.9.

CONCLUSIONS

Work-hour limitation can be devised to maximize resident education, optimize patient care, and maintain resident operative volume. Although some changes (HC, PE, NF) seemed to increase the operative case volume for PGY5 residents, others had no effect (Q4, HC). There does not seem to be a clear relationship between types of changes and case volume. At the PGY1 level, Q4 and PE changes decreased operative experience on 1 rotation but not on another, although the difference in this decrease seems clinically insignificant. Individualization of changes to meet the needs of specific rotations seems more important than specific changes in coverage pattern. Perhaps the most important finding is that changes can be made to bring work hours into compliance without materially effecting operative case volume.