Gold Standard for macromolecular crystallography diffraction data

The evolution of raw data archiving and the growth of its importance in crystallography

The hardware for data archiving has expanded capacities for digital storage enormously in the past decade or more. The IUCr evaluated the costs and benefits of this within an official working group which advised that raw data archiving would allow ground truth reproducibility in published studies. Consultations of the IUCr's Commissions ensued via a newly constituted standing advisory committee, the Committee on Data. At all stages, the IUCr financed workshops to facilitate community discussions and possible methods of raw data archiving implementation. The recent launch of the IUCrData journal's Raw Data Letters is a milestone in the implementation of raw data archiving beyond the currently published studies: it includes diffraction patterns that have not been fully interpreted, if at all. The IUCr 75th Congress in Melbourne included a workshop on raw data reuse, discussing the successes and ongoing challenges of raw data reuse. This article charts the efforts of the IUCr to facilitate discussions and plans relating to raw data archiving and reuse within the various communities of crystallography, diffraction and scattering.

Data reduction in protein serial crystallography

Serial crystallography (SX) has become an established technique for protein structure determination, especially when dealing with small or radiation-sensitive crystals and investigating fast or irreversible protein dynamics. The advent of newly developed multi-megapixel X-ray area detectors, capable of capturing over 1000 images per second, has brought about substantial benefits. However, this advancement also entails a notable increase in the volume of collected data. Today, up to 2 PB of data per experiment could be easily obtained under efficient operating conditions. The combined costs associated with storing data from multiple experiments provide a compelling incentive to develop strategies that effectively reduce the amount of data stored on disk while maintaining the quality of scientific outcomes. Lossless data-compression methods are designed to preserve the information content of the data but often struggle to achieve a high compression ratio when applied to experimental data that contain noise. Conversely, lossy compression methods offer the potential to greatly reduce the data volume. Nonetheless, it is vital to thoroughly assess the impact of data quality and scientific outcomes when employing lossy compression, as it inherently involves discarding information. The evaluation of lossy compression effects on data requires proper data quality metrics. In our research, we assess various approaches for both lossless and lossy compression techniques applied to SX data, and equally importantly, we describe metrics suitable for evaluating SX data quality.

A standard data format for 3DED/MicroED

Electron diffraction from three dimensional crystals, as a technique for solving molecular structures, is rapidly increasing in popularity. The development of methodology and software has borrowed, to great effect, from macromolecular X-ray crystallography. However, standardization lags behind the development of the technique, and practitioners are forced to work with inadequate data formats that are unable to capture a full description of their experiments. This creates obstacles that are increasingly difficult to overcome as experiments become ever faster and the need for data autoprocessing becomes more pressing. We present a data format standard based on best practice from macromolecular crystallography and demonstrate how the adoption of this standard enabled autoprocessing of datasets collected with a high-throughput detector system.

Kilohertz serial crystallography with the JUNGFRAU detector at a fourth-generation synchrotron source

Serial and time-resolved macromolecular crystallography are on the rise. However, beam time at X-ray free-electron lasers is limited and most third-generation synchrotron-based macromolecular crystallography beamlines do not offer the necessary infrastructure yet. Here, a new setup is demonstrated, based on the JUNGFRAU detector and Jungfraujoch data-acquisition system, that enables collection of kilohertz serial crystallography data at fourth-generation synchrotrons. More importantly, it is shown that this setup is capable of collecting multiple-time-point time-resolved protein dynamics at kilohertz rates, allowing the probing of microsecond to second dynamics at synchrotrons in a fraction of the time needed previously. A high-quality complete X-ray dataset was obtained within 1 min from lysozyme microcrystals, and the dynamics of the light-driven sodium-pump membrane protein KR2 with a time resolution of 1 ms could be demonstrated. To make the setup more accessible for researchers, downstream data handling and analysis will be automated to allow on-the-fly spot finding and indexing, as well as data processing.

Making your raw data available to the macromolecular crystallography community

A recent editorial in the IUCr macromolecular crystallography journals [Helliwell et al. (2019), Acta Cryst. D75, 455-457] called for the implementation of the FAIR data principles. This implies that the authors of a paper that describes research on a macromolecular structure should make their raw diffraction data available. Authors are already used to submitting the derived data (coordinates) and the processed data (structure factors, merged or unmerged) to the PDB, but may still be uncomfortable with making the raw diffraction images available. In this paper, some guidelines and instructions on depositing raw data to Zenodo are given.

EIGER2 hybrid-photon-counting X-ray detectors for advanced synchrotron diffraction experiments

The ability to utilize a hybrid-photon-counting detector to its full potential can significantly influence data quality, data collection speed, as well as development of elaborate data acquisition schemes. This paper facilitates the optimal use of EIGER2 detectors by providing theory and practical advice on (i) the relation between detector design, technical specifications and operating modes, (ii) the use of corrections and calibrations, and (iii) new acquisition features: a double-gating mode, 8-bit readout mode for increasing temporal resolution, and lines region-of-interest readout mode for frame rates up to 98 kHz. Examples of the implementation and application of EIGER2 at several synchrotron sources (ESRF, PETRA III/DESY, ELETTRA, AS/ANSTO) are presented: high accuracy of high-throughput data in serial crystallography using hard X-rays; suppressing higher harmonics of undulator radiation, improving peak shapes, increasing data collection speed in powder X-ray diffraction; faster ptychography scans; and cleaner and faster pump-and-probe experiments.

Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors

The JUNGFRAU 4-megapixel (4M) charge-integrating pixel-array detector, when operated at a full 2 kHz frame rate, streams data at a rate of 17 GB s^-1. To operate this detector for macromolecular crystallography beamlines, a data-acquisition system called Jungfraujoch was developed. The system, running on a single server with field-programmable gate arrays and general-purpose graphics processing units, is capable of handling data produced by the JUNGFRAU 4M detector, including conversion of raw pixel readout to photon counts, compression and on-the-fly spot finding. It was also demonstrated that 30 GB s^-1 can be handled in performance tests, indicating that the operation of even larger and faster detectors will be achievable in the future. The source code is available from a public repository.

AMX - the highly automated macromolecular crystallography (17-ID-1) beamline at the NSLS-II

The highly automated macromolecular crystallography beamline AMX/17-ID-1 is an undulator-based high-intensity (>5 × 1012 photons s-1), micro-focus (7 µm × 5 µm), low-divergence (1 mrad × 0.35 mrad) energy-tunable (5-18 keV) beamline at the NSLS-II, Brookhaven National Laboratory, Upton, NY, USA. It is one of the three life science beamlines constructed by the NIH under the ABBIX project and it shares sector 17-ID with the FMX beamline, the frontier micro-focus macromolecular crystallography beamline. AMX saw first light in March 2016 and started general user operation in February 2017. At AMX, emphasis has been placed on high throughput, high capacity, and automation to enable data collection from the most challenging projects using an intense micro-focus beam. Here, the current state and capabilities of the beamline are reported, and the different macromolecular crystallography experiments that are routinely performed at AMX/17-ID-1 as well as some plans for the near future are presented.

IUCrData launches Raw Data Letters

A new category of articles - Raw Data Letters - is introduced to IUCrData.

Using Powder Diffraction Patterns to Calibrate the Module Geometry of a Pixel Detector

The precision and accuracy of diffraction measurements with 2D area detectors depends on how well the experimental geometry is known. A method is described to measure the module geometry in order to obtain accurate strain data using a new Eiger2 4M CdTe detector. Smooth Debye–Scherrer powder diffraction rings with excellent signal to noise were collected by using a fine-grained sample of CeO2. From these powder patterns, the different components of the module alignment errors could be observed when the overall detector position was moved. A least squares fitting method was used to refine the detector module and scattering geometry for a series of powder patterns with different beam centers. A precision that is around 1/350 pixel for the module positions was obtained from the fit. This calibration was checked by free refinement of the unit cell of a silicon crystal that gave a maximum residual strain value of 2.1 × 10−5 as the deviation from cubic symmetry.

Discerning best practices in XFEL-based biological crystallography - standards for nonstandard experiments

Serial femtosecond crystallography (SFX) at X-ray free-electron lasers (XFELs) is a novel tool in structural biology. In contrast to conventional crystallography, SFX relies on merging partial intensities acquired with X-ray beams of often randomly fluctuating properties from a very large number of still diffraction images of generally randomly oriented microcrystals. For this reason, and possibly due to limitations of the still evolving data-analysis programs, XFEL-derived SFX data are typically of a lower quality than 'standard' crystallographic data. In contrast with this, the studies performed at XFELs often aim to investigate issues that require precise high-resolution data, for example to determine structures of intermediates at low occupancy, which often display very small conformational changes. This is a potentially dangerous combination and underscores the need for a critical evaluation of procedures including data-quality standards in XFEL-based structural biology. Here, such concerns are addressed.