Hack weeks as a model for data science education and collaboration

Hands-On Neuroinformatics Education at the Crossroads of Online and In-Person: Lessons Learned from NeuroHackademy

NeuroHackademy ( https://neurohackademy.org ) is a two-week event designed to train early-career neuroscience researchers in data science methods and their application to neuroimaging. The event seeks to bridge the big data skills gap by introducing participants to data science methods and skills that are often ignored in traditional curricula. Such skills are needed for the analysis and interpretation of the kinds of large and complex datasets that have become increasingly important to neuroimaging research due to concerted data collection efforts. In 2020, the event rapidly pivoted from an in-person event to an online event that included hundreds of participants from all over the world. This experience and those of the participants substantially changed our valuation of large online-accessible events. In subsequent events held in 2022 and 2023, we have developed a "hybrid" format that includes both online and in-person participants. We discuss the technical and sociotechnical elements of hybrid events and discuss some of the lessons we have learned while organizing them. We emphasize in particular the role that these events can play in creating a global and inclusive community of practice in the intersection of neuroimaging and data science.

Hack your organizational innovation: literature review and integrative model for running hackathons

This article aims to offer a comprehensive overview of the existing literature on the hackathon phenomenon to offer scholars a common ground for future research and managers and practitioners research-based guidelines on best planning and running a hackathon. A review of the most relevant literature on hackathons was conducted to serve as the research basis for our integrative model and guidelines. This article synthesizes the research on hackathons to offer comprehensible guidelines for practitioners while also providing questions for future hackathon researchers. We differentiate between the different design characteristics of hackathons while noting their advantages and disadvantages, discuss tools and methodologies for successful hackathon setup and execution step-by-step, and provide recommendations to encourage project continuity.

One-off events? An empirical study of hackathon code creation and reuse

CONTEXT

Hackathons have become popular events for teams to collaborate on projects and develop software prototypes. Most existing research focuses on activities during an event with limited attention to the evolution of the hackathon code.

OBJECTIVE

We aim to understand the evolution of code used in and created during hackathon events, with a particular focus on the code blobs, specifically, how frequently hackathon teams reuse pre-existing code, how much new code they develop, if that code gets reused afterwards, and what factors affect reuse.

METHOD

We collected information about 22,183 hackathon projects from Devpost and obtained related code blobs, authors, project characteristics, original author, code creation time, language, and size information from World of Code. We tracked the reuse of code blobs by identifying all commits containing blobs created during hackathons and identifying all projects that contain those commits. We also conducted a series of surveys in order to gain a deeper understanding of hackathon code evolution that we sent out to hackathon participants whose code was reused, whose code was not reused, and developers who reused some hackathon code.

RESULT

9.14% of the code blobs in hackathon repositories and 8% of the lines of code (LOC) are created during hackathons and around a third of the hackathon code gets reused in other projects by both blob count and LOC. The number of associated technologies and the number of participants in hackathons increase reuse probability.

CONCLUSION

The results of our study demonstrates hackathons are not always "one-off" events as the common knowledge dictates and it can serve as a starting point for further studies in this area.

How do participants collaborate during an online hackathon? An empirical, quantitative study of communication traces

Starting as niche programming events, hackathons have since become a popular form of collaboration. Events are organized in various domains across the globe, aiming to foster innovation and learning, create and expand communities and tackle civic and environmental issues. While research around such events has grown in recent years, most studies are based on observations of a few individuals during an event and on post-hoc interviews during which participants report their experiences. Such studies are helpful but somewhat limited in that they do not allow us to study how individuals communicate at scale using technology. To address this gap, we conducted an archival analysis of communication traces of teams during a 48-h event. Our findings indicate that teams scaffold their communication around the design of an event, influenced by milestones set by the organizers. The officially selected communication platform's main use was to organize the event and the teams and to facilitate contact between participants and hackathon officials. We further investigated the balance of intra-team communication on the given platform and the potential use of additional communication tools.

Paths towards greater consensus building in experimental biology

In a recent editorial, the Editors-in-Chief of Journal of Experimental Biology argued that consensus building, data sharing, and better integration across disciplines are needed to address the urgent scientific challenges posed by climate change. We agree and expand on the importance of cross-disciplinary integration and transparency to improve consensus building and advance climate change research in experimental biology. We investigated reproducible research practices in experimental biology through a review of open data and analysis code associated with empirical studies on three debated paradigms and for unrelated studies published in leading journals in comparative physiology and behavioural ecology over the last 10 years. Nineteen per cent of studies on the three paradigms had open data, and 3.2% had open code. Similarly, 12.1% of studies in the journals we examined had open data, and 3.1% had open code. Previous research indicates that only 50% of shared datasets are complete and re-usable, suggesting that fewer than 10% of studies in experimental biology have usable open data. Encouragingly, our results indicate that reproducible research practices are increasing over time, with data sharing rates in some journals reaching 75% in recent years. Rigorous empirical research in experimental biology is key to understanding the mechanisms by which climate change affects organisms, and ultimately promotes evidence-based conservation policy and practice. We argue that a greater adoption of open science practices, with a particular focus on FAIR (Findable, Accessible, Interoperable, Re-usable) data and code, represents a much-needed paradigm shift towards improved transparency, cross-disciplinary integration, and consensus building to maximize the contributions of experimental biologists in addressing the impacts of environmental change on living organisms.

A community-based approach to image analysis of cells, tissues and tumors

Emerging multiplexed imaging platforms provide an unprecedented view of an increasing number of molecular markers at subcellular resolution and the dynamic evolution of tumor cellular composition. As such, they are capable of elucidating cell-to-cell interactions within the tumor microenvironment that impact clinical outcome and therapeutic response. However, the rapid development of these platforms has far outpaced the computational methods for processing and analyzing the data they generate. While being technologically disparate, all imaging assays share many computational requirements for post-collection data processing. As such, our Image Analysis Working Group (IAWG), composed of researchers in the Cancer Systems Biology Consortium (CSBC) and the Physical Sciences - Oncology Network (PS-ON), convened a workshop on "Computational Challenges Shared by Diverse Imaging Platforms" to characterize these common issues and a follow-up hackathon to implement solutions for a selected subset of them. Here, we delineate these areas that reflect major axes of research within the field, including image registration, segmentation of cells and subcellular structures, and identification of cell types from their morphology. We further describe the logistical organization of these events, believing our lessons learned can aid others in uniting the imaging community around self-identified topics of mutual interest, in designing and implementing operational procedures to address those topics and in mitigating issues inherent in image analysis (e.g., sharing exemplar images of large datasets and disseminating baseline solutions to hackathon challenges through open-source code repositories).

Evaluation of epidemiological lectures using peer instruction: focusing on the importance of ConcepTests

Background

In clinical research, the ability to properly analyze data is a necessary skill that cannot be learned simply by listening to lectures. Interactive classes, such as Peer Instruction (PI), are required to help medical students understand the concept of epidemiology for future valid research. In PI lectures, ConcepTests are conducted to confirm and deepen students’ understanding of the lecture material. Although it is important to evaluate PI lectures, there have been no studies conducted on PI lectures in epidemiology. This study employed the ConcepTest to evaluate PI lectures in a medical school epidemiology class to measure the efficiency of active learning techniques and the usefulness of ConcepTests in determining effective active learning approaches.

Methods

The PI lecture was conducted as part of an existing epidemiology class for fourth-year medical students at Okayama University on October 17, 2019. The lecture was conducted as follows. The lecturer taught the fundamental concepts of epidemiology and presented the ConcepTest to students. After answering the test, students were provided with the answer distribution, followed by peer discussion. After the discussion, students answered the ConcepTest again, and a new answer distribution was presented. Subsequently, the lecturer announced the correct answers and delivered a commentary. The ConcepTest comprised five questions, each related to fundamental concepts of epidemiology. Students’ responses to five ConcepTests were collected and analyzed by calculating the proportion of correct answers before and after the discussion, as well as PI efficiency to evaluate the PI lecture.

Results

Overall,121 students attended the epidemiology lecture. The proportion of correct answers before the discussion ranged from 0.217 to 0.458, and after the peer discussion they ranged from 0.178 to 0.767. The PI efficiency ranged from −0.051 to 0.657, and was higher than the theoretical value in three ConcepTests. The efficiency was about the same as the theoretical value in one ConcepTest, and lower than the theoretical value in another.

Conclusion

In this study, the efficiency of a PI lecture was determined by calculating the PI efficiency of each ConcepTest. The results showed that the educational efficiency of a ConcepTest in epidemiology lectures can be widely distributed, ranging from efficient to inefficient. Particularly in three ConcepTests, the proportion of correct answers after the discussion and the PI efficiency were higher than the theoretical value. This suggests that PI lectures can be useful in epidemiology education with the efficient use of ConcepTests.

Entrofy your cohort: A transparent method for diverse cohort selection

Selecting a cohort from a set of candidates is a common task within and beyond academia. Admitting students, awarding grants, and choosing speakers for a conference are situations where human biases may affect the selection of any particular candidate, and, thereby the composition of the final cohort. In this paper, we propose a new algorithm, entrofy, designed to be part of a human-in-the-loop decision making strategy aimed at making cohort selection as just, transparent, and accountable as possible. We suggest embedding entrofy in a two-step selection procedure. During a merit review, the committee selects all applicants, submissions, or other entities that meet their merit-based criteria. This often yields a cohort larger than the admissible number. In the second stage, the target cohort can be chosen from this meritorious pool via a new algorithm and software tool called entrofy. entrofy optimizes differences across an assignable set of categories selected by the human committee. Criteria could include academic discipline, home country, experience with certain technologies, or other quantifiable characteristics. The entrofy algorithm then yields the approximation of pre-defined target proportions for each category by solving the tie-breaking problem with provable performance guarantees. We show how entrofy selects cohorts according to pre-determined characteristics in simulated sets of applications and demonstrate its use in a case study of Astro Hack Week. This two stage candidate and cohort selection process allows human judgment and debate to guide the assessment of candidates’ merit in step 1. Then the human committee defines relevant diversity criteria which will be used as computational parameters in entrofy. Once the parameters are defined, the set of candidates who meet the minimum threshold for merit are passed through the entrofy cohort selection procedure in step 2 which yields a cohort of a composition as close as possible to the computational parameters defined by the committee. This process has the benefit of separating the meritorious assessment of candidates from certain elements of their diversity and from some considerations around cohort composition. It also increases the transparency and auditability of the process, which enables, but does not guarantee, fairness. Splitting merit and diversity considerations into their own assessment stages makes it easier to explain why a given candidate was selected or rejected, though it does not eliminate the possibility of objectionable bias.

Ad hoc efforts for advancing data science education

With increasing demand for training in data science, extracurricular or "ad hoc" education efforts have emerged to help individuals acquire relevant skills and expertise. Although extracurricular efforts already exist for many computationally intensive disciplines, their support of data science education has significantly helped in coping with the speed of innovation in data science practice and formal curricula. While the proliferation of ad hoc efforts is an indication of their popularity, less has been documented about the needs that they are designed to meet, the limitations that they face, and practical suggestions for holding successful efforts. To holistically understand the role of different ad hoc formats for data science, we surveyed organizers of ad hoc data science education efforts to understand how organizers perceived the events to have gone-including areas of strength and areas requiring growth. We also gathered recommendations from these past events for future organizers. Our results suggest that the perceived benefits of ad hoc efforts go beyond developing technical skills and may provide continued benefit in conjunction with formal curricula, which warrants further investigation. As increasing numbers of researchers from computational fields with a history of complex data become involved with ad hoc efforts to share their skills, the lessons learned that we extract from the surveys will provide concrete suggestions for the practitioner-leaders interested in creating, improving, and sustaining future efforts.

From the Wet Lab to the Web Lab: A Paradigm Shift in Brain Imaging Research

Web technology has transformed our lives, and has led to a paradigm shift in the computational sciences. As the neuroimaging informatics research community amasses large datasets to answer complex neuroscience questions, we find that the web is the best medium to facilitate novel insights by way of improved collaboration and communication. Here, we review the landscape of web technologies used in neuroimaging research, and discuss future applications, areas for improvement, and the limitations of using web technology in research. Fully incorporating web technology in our research lifecycle requires not only technical skill, but a widespread culture change; a shift from the small, focused "wet lab" to a multidisciplinary and largely collaborative "web lab."