Journal Article

PUBDB-2024-00039

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png Real-time data processing for serial crystallography experiments

White, T. (Corresponding author)DESY* ; Schoof, T.DESY* ; Yakubov, S.Extern* ; Tolstikova, A.DESY* ; Middendorf, P.CFEL*DESY* ; Karnevskiy, M.XFEL.EU*DESY* ; Mariani, V.XFEL.EU*DESY* ; Henkel, A.CFEL*Extern*XFEL.EU*EMBL*DESY* ; Klopprogge, B.CFEL*Extern*DESY* ; Hannappel, J.XFEL.EU*DESY* ; Oberthür, D.CFEL*XFEL.EU*DESY* ; De Gennaro Aquino, I.DESY* ; Egorov, D.CFEL*DESY* ; Munke, A. C.CFEL*XFEL.EU*DESY* ; Sprenger, J.CFEL*DESY* ; Pompidor, G.EMBL*DESY* ; Taberman, H.EMBL*DESY* ; Gruzinov, A.DESY* ; Meyer, J.CFEL*XFEL.EU*DESY* ; Hakanpaeae, J.DESY* ; Gasthuber, M.XFEL.EU*DESY*

2024
[Verlag nicht ermittelbar] Chester

This record in other databases:      

Please use a persistent id in citations: doi:10.1107/S2052252524011837  doi:10.3204/PUBDB-2024-00039

Abstract: We report the use of streaming data interfaces to perform fully on-line data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than one second per frame.Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames (“hits”). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the “data deluge” problem from recent and future photon science experiments, in particular on calibration requirements, computing access patterns and the need for preservation of raw data.

---

Contributing Institute(s):

1. Scientific computing (FS-SC)
2. Informationstechnologie (IT)
3. FS-CFEL-1 (Group Leader: Henry Chapman) (CFEL-I)
4. PETRA-D (FS-PETRA-D)

Research Program(s):

1. 623 - Data Management and Analysis (POF4-623) (POF4-623)
2. 6G9 - IDAF (DESY) (POF4-6G9) (POF4-6G9)
3. 6G3 - PETRA III (DESY) (POF4-6G3) (POF4-6G3)
4. ExPaNDS - EOSC Photon and Neutron Data Services (857641) (857641)

Experiment(s):

1. PETRA Beamline P11 (PETRA III)

Appears in the scientific report 2024

---

Database coverage:
; ; ; ; Article Processing Charges ; Clarivate Analytics Master Journal List ; Current Contents - Physical, Chemical and Earth Sciences ; DOAJ Seal ; Essential Science Indicators ; Fees ; IF < 5 ; JCR ; SCOPUS ; Science Citation Index Expanded ; Web of Science Core Collection

---