TY  - JOUR
AU  - White, Thomas
AU  - Schoof, Tim
AU  - Yakubov, Sergey
AU  - Tolstikova, Aleksandra
AU  - Middendorf, Philipp
AU  - Karnevskiy, Mikhail
AU  - Mariani, Valerio
AU  - Henkel, Alessandra
AU  - Klopprogge, Bjarne
AU  - Hannappel, Juergen
AU  - Oberthür, Dominik
AU  - De Gennaro Aquino, Ivan
AU  - Egorov, Dmitry
AU  - Munke, Anna Carina
AU  - Sprenger, Janina
AU  - Pompidor, Guillaume
AU  - Taberman, Helena
AU  - Gruzinov, Andrey
AU  - Meyer, Jan
AU  - Hakanpaeae, Johanna
AU  - Gasthuber, Martin
TI  - Real-time data processing for serial crystallography experiments
JO  - IUCrJ
VL  - 12
IS  - 1
SN  - 2052-2525
CY  - Chester
PB  - [Verlag nicht ermittelbar]
M1  - PUBDB-2024-00039
SP  - 97-108
PY  - 2024
AB  - We report the use of streaming data interfaces to perform fully on-line data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than one second per frame.Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames (“hits”). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the “data deluge” problem from recent and future photon science experiments, in particular on calibration requirements, computing access patterns and the need for preservation of raw data.
LB  - PUB:(DE-HGF)16
C6  - 39714221
UR  - <Go to ISI:>//WOS:001395741400012
DO  - DOI:10.1107/S2052252524011837
UR  - https://bib-pubdb1.desy.de/record/600990
ER  -