High speed cameras for X-rays: AGIPD and others

Experiments at high pulse rate Free Electron Laser (FEL) facilities require new cameras capable of acquiring 2D images at high rates, handling large signal dynamic ranges and resolving images from individual pulses. The Adaptive Gain Integrated Pixel Detector (AGIPD) will operated with pulse rates and separations of 27000/s and 220 ns, respectively at European XFEL. Si-sensors, ASICs, PCBs, and FPGA logic are developed for a 1 Mega-pixel camera with 200 μm square pixels with per-pulse occupancies ⩽ 104. Data from 3520 images/s will be transferred with 80 Gbits/s to a DAQ-system. The electronics have been adapted for use in other synchrotron light source detectors.


Free Electron Lasers a new challenge for science and detectors
The free electron laser (FEL) principle is used to generate high intensity coherent X-ray pulses at new generation light sources. Since 2009 and 2011 the Linac Coherent Light Source, LCLS, [1] and Spring-8 Angstrom Compact free electron LAser, SACLA, [2] are operated with pulse rates 60-120 Hz. The European XFEL [3], X-ray Free Electron Laser, starts user operation in 2016 and provides X-rays, < 25 keV, with a peak brilliance nine orders of magnitude above the brilliance generated at synchrotrons, a few times 10 34 photons s mm 2 rad 2 % [4]. The pulse rate of the European-XFEL will be 27 000 bunches/s. Each bunch will contain up to 10 12 photons delivered in less than 100 fs. At the European XFEL, a linear superconducting accelerator is used to accelerate the electron bunches used in the FEL process. The accelerating cavities used require a cool down phase after periods of applied RF-field and consequently the bunches will be delivered in so called trains of 2700 bunches with 220 ns bunch separation and a train repetition rate of 10 Hz.
The short and extremely high intensity bunches generated by XFELs allow the structure of target samples to be studied in a single bunch exposure. To do this successfully, detectors are required, which cope with a high incidence rate of simultaneously arriving photons, store the images of a bunch before the next bunch arrives and use as many bunches as possible. The scientific requirements for operating 2-dimensional cameras [5] has been defined for different experiment techniques: pump/probe, diffractive scattering and correlation spectroscopy and define a wide range of challenging properties. Key property values are: photon energies from 0.8-15 keV, radiation tolerance of 10 16 photons, pixel sizes as small as 100 µm, 0.25-16 Mega-pixels, local photon rates up to 3 · 10 6 photons/pixel and resolving single photons. Some requirements are extremely challenging and can not be satisfied by current developments.
Three consortia are developing cameras with different techniques. The key parameters are summerized in table 1. DSSC [6] will be based on a DEPMOS Sensor with Signal Compression.  LPD [7] will be a Large Pixel Detector and AGIPD, Adaptive Gain Integrated Pixel Detector, [8]. The latter will be described in detail in the next sections.
2 AGIPD, a two dimensional high speed X-ray camera AGIPD will be a planar modular 2-dimensional 1 Mega-pixel camera in the first development stage. The mechanical concept is shown in figure 1. The focal plane is split into 4 quadrants which can be shifted relative to each another leaving a hole for the intense non-scattered beam. A modular design matching sensor usage and detector assembly is used and quadrants are assembled from 4 modules where each module has 512 × 128 pixels. The final 1 Mega-pixel camera consists of 16 modules assembled in 4 quadrants. All electronics for readout follows the modular concept, while the control will be organized per quadrant, shared by 4 modules. AGIPD is designed to meet the following requirements: In the next sections the challenges and the solutions, which are being worked on, for the full frontend chain from sensor over ASICs, analogue and digital printed circuit boards (PCBs) to the data transfer and the off-detector data acquisition system are described.

The sensor for AGIPD
The required detection efficiency of > 90% for 12.4 keV photons is reached with 500 µm thick silicon sensors. The challenges for the sensor design arise from the wide field radiation dose of up to 1 GGy acquired during three years of operation and Bragg-reflections which can generate intense peaks with sizes much smaller than a pixel. Detailed radiation studies have led to the decision to use n-bulk silicon sensors with a common n + electrode on the entrance side of the sensor and a segmented p + -doping per pixel (figure 2).
The detector response to Bragg-reflections has been studied by generating the equivalent ionization densities, 3 · 10 16 /cm 3 , with laser light [9]. For a wavelength of 1015 nm electron-hole-pairs are produced with the same profile in sensor depth as with 12.4 keV photons. While the electrons and holes drift to the electrodes the charge clouds expand due to the Coulomb force. In figure 2 the measurement for different ionization densities and applied bias voltages are shown. To keep the charge for the high ionizations within 200 µm pixel a bias of 500 V is used.
Radiation damage induced by the 12.4 keV photons increases oxide-charges. Measurements and simulations have shown that for dosage below 1 GGy at the surface of the sensor the oxidecharges stay below 3×10 12 /cm 2 [10]. For this value it was possible to design a doping and insulator profile, verified with the simulation program TCAD, that keeps the dark current and the inter-pixel capacitance small, < 14.4 µA/sensor and < 312 fF, allowing operation with acceptable pixel-topixel cross-talk.
With the chosen modular focal plane design, each sensor contains 512 × 128 pixels. Scientific usage requires as small a pixel sizes as possible. But a compromise has to be made to allow sufficient space in the ASIC for storage cell structures. Pixel size 200 µm and total sensor size 10.8 × 2.8 cm 2 are chosen. This pixel size choice matches the charge spread for high pulse charge depositions.

An ASIC for fast image acquisition and a high dynamic range
The charge of the sensor is collected on a metalized p + -implant per pixel, bump bonded to an ASIC. Each ASIC will cover 64 × 64 pixels. Therefore each sensor is equipped with two rows of 8 -3 - ASICs. Figure 3 (left) illustrates the analogue signal chain of the ASIC [11,12]. The charge from the pixel is integrated onto a feedback capacitor. For each new image the integration is started with a small capacitor and high sensitivity. When the output of the integrator increases above a threshold a larger capacitor is added into the feedback. That can be repeated a second time so that three gain stages are generated. This concept, called "adaptive gain", is verified on an early protoytpe using injected charge. The generated output amplitudes are shown in figure 3 (right). The CDS stage subtracts the signal at the start of the integration from the result of the integration. This difference is written per pixel and image into capacitors. To minimize the number of ASIC connections to readout and control devices the gain level used is also stored as an analogue value. This allows both signal magnitude and gain values to be multiplexed out of the ASIC for 64 × 16 pixels and reduces the number of readout lines per sensor to 64 differential analogue lines. Design of the analogue storage system is challenging because of the requirement of maximizing the number of storage cells which must fit with the adaptive amplifier into the available pixel area whilst maintaining the required radiation hardness and minimizing charge leakage (droop). An optimized design of the switch [13] limits the signal droop to the 11 ms needed to multiplex all critical analogue values to a few %. The resulting design allows 352 storage cells to fit behind the pixel area which is less than the 2700 bunches per train. The shortfall in cells is alleviated by allowing cell usage to be overwritten by information from high quality images as defined by an external trigger signal.

Control system to operate the ASIC and select the best bunches
Operation of the ASIC has to be synchronized to both bunches and trains. For this an external control system sends on three digital control lines the necessary information: a 99 MHz system clock as multiple of the bunch clock, a start/reset and a data stream of 22 bits/bunch to flag with random delay bunches which can be overwritten. The external control system [14], which is common to all European XFEL detectors, has been developed using the crate standard MTCA.4, Micro Telecommunication Computing Architecture [15]. A FPGA in the quadrant control part of the camera head receives control line signals, performs free storage cell bookkeeping and instructs the ASICs of the quadrant which storage cell to use next. Bookkeeping information will be added to the main data stream so that offline the correlation between bunch number and storage cell is available.
An intelligent power supply system delivers the high currents, ≈ 500 A, for the 20 × 20 cm 2 camera head electronics and the high voltages, ≈ 500 V, for the sensors. Control system communication to the camera head quadrants and power system will be based on a 10/100 MbE using TCP/IP protocol. Within the detector head the slow information is distributed and collected by an I 2 C network. With a few digital control lines a part of the network, a branch, is selected and the use of multiple identical modules in the network is possible.

Mechanical design of the focal plane
For the focal plane 16 ASICs are bump bonded to one sensor (see figure 4). The control signals are introduced into the ASICs using wire bonds located along both long sides of the sensor. Due to large connection density required by the 64 differential analogue outputs per sensor a high density PCB is being developed. This will be realized as multilayer ceramic board onto which the ASICs are glued and through which heat is transferred. On the rear side space has to be shared between a connector with 500 pins and a cooling device. A silicone oil coolant flow will allow operation at temperature down to −50 • C whilst transporting the heat of the ASICs, ≈ 0.8 kW, out of the detector head. The challenge of transferring hundreds of signals and heat in a small area appears in many different developments of high speed X-ray cameras. Figure 4 shows a similar prototype solution for the Large Area Medipix-Based Detector Array, LAMBDA [16]. An additional challenge for the AGIPD design is the barrier implementation separating the focal plane components operating in vacuum, and the PCB electronics operating outside the vacuum and being air-cooled, see figure 1.
The final design profits from the technology developments during the last years.

Printed circuit boards for the camera head
The vacuum-air feed through, analogue signal processing and digital signal processing PCBs mounted in the AGIPD modules detector head assembly are described in this section. The vacuum-air PCB has two functions, it provides the vacuum seal and acts as a feed through for the 64 module ASIC readout channels, ASIC control signals, bias voltages, etc. The latter are implemented through surface mounted connectors on either side of the PCB. The analogue PCB system performs line receiving, pickup noise filtering, digitizing and driving the signal to a digital board. The compact mechanical design allowing mounting into the module assembly is achieved by using a single operational amplifier and organizing the analogue part on two parallel PCBs ( figure 5). The single stage filter suppresses the noise at the 33 MHz sampling frequency by 3dB and settles to better that 11 bits. Digitization is performed by a AD9257 fast serial output ADC typically used for medical imaging, which generates a data stream of up to 700 Mbit/s. An amplifier strip path layout width of 5 mm allows 16 paths to be implemented on each side of the two PCBs used in the final analogue PCB assembly which is ≈ 23.7 cm long. The two normal size PCBs are interconnected with a high density connector containing 120 individual shielded pairs. The digitization order is driven by ASIC-design and minimization of analogue value droop requirements. The digital electronic PCB assembly ( figure 6) [17] is required to: receive and decode the 64 × 700 Mbit/s = 45 Gbit/s data streams, sort the data and sent them out to the off-detector data -5 -2013 JINST 8 C01042  acquisition system. The core of the digital design is a Virtex-5 FPGA. Data sorting consists of three steps: first combining the value and the gain for a pixel, then sort to module images, and finally sort within the image to small geometrical regions. The resulting sorted data for the entire train are stored in memory. Sort processing takes 11 ms for the values and another 11 ms for the gains. Afterwards the data are read and formatted for transfer on a single 10 GbE line. In addition to module image pixel data, information allowing correlation of used storage cell and bunch number, and debug information used to identify misbehaving hardware are transferred to the DAQ-system. The protocol used to transfer data is UDP for which the receiver will never be in a busy-state and will listen continuously for packet delivery.
The protocol and packetizing of data is performed within the FPGA and transferred via four 3.125 Gbit/s-lines to a PHY chip which sends the data on short lines with 10 Gbit/s to a SFP+ transceiver cage, which depending on length to drive uses either TwinAx copper or optical transceivers. The performance of the 3.125 Gbit/s lines is excellent as shown by the open-eye diagram shown in figure 6. The total data transfer including the 10 GbE header overhead part has been measured by counting the number of error bits registered during soak tests. No error was observed during a full day of operation and an error rate of less than 10 −15 is calculated for the design.
DDR2 memory with a 128-bit wide data bus to the FPGA allowing maximum access rates of 65 Gbits/s is used, this is larger than the design receive and send sum rate of 55 Gbits/s and is sufficient for the implementation.
The digital part is designed as a functional mezzanine board, which can be reused in other high speed readout device projects developed at DESY, like LAMBDA and PERCIVAL [18], Pixelated Energy Resolving CMOS Imager Versatile And Large. Therefore four 10 GbE outputs are integrated, but it also allows to minimize the size for AGIPD by just populating one of the links. AGIPD will deliver to the off-detector DAQ-System 80 Gbit/s on 16 links, 10 GbE/UDP, and each is used for ≈ 5 Gbit/s .

Off-detector data acquisition (DAQ) and commonalities with other X-ray cameras
The off-detector DAQ is developed as a common component for use with all detectors at European XFEL beamlines. Of all detectors being integrated 2-dimensional cameras produce by far the largest maximum data bandwidths of ≈ 80 Gbit/s. For the cameras and potentially other systems (digitizers, fast ADCs, etc.) a data acquisition train builder card [7] is being developed to receive, sort and reject data of a 1 Mega-pixel or large cameras. It is designed in the ATCA standard, Advanced Telecom Computing Architecture [19]. Data from two AGIPD modules are recieved by an input train builder FPGA which performs additional module specific data sorting before storing to DDR memory. Using time scheduled cross-point switches final sorting to full 1-Mega-pixel images and full trains is performed from the input memory to the DDR2 memory of an output FPGA. The latter then sends the bunch ordered images and additional data to a PC-farm [20] using UDP for further processing and archiving.
Synchrotron radiation facility detector data rates and bandwidths are increasing significantly. DESY is participating in two such developments, LAMBDA and PERCIVAL. LAMBDA is aiming for a combination of high Z materials, Ge, GaAs or CdTe, and a MEDIPIX [21] based readout counting individual photons. The focal plane modules with 1636 × 512 pixels will be connected via 150 LVDS-outputs. For an imaging rate of 2 kHz a throughput of 25 Gbit/s is needed. PERCIVAL designs with the MAPS-technology 16 Mega-pixel cameras for low energy photons, > 250 eV. The ADCs within the sensor will generate 60 Gbits/s for a frame rate of 120 Hz. They are transferred on 150 LVDS lines. With all the differences of counting detectors or integrated ADCs there are similar challenges like for AGIPD: cooling, high density connectivity, high speed data handling and DAQ. On these similarities the project-consortia cooperate and reuse the developments from the others.

Summary and outlook
X-ray science is making increasing use of high speed imaging and data processing systems, consequently the design demands on electronics and data processing are increasing significantly. The AGIPD 2-dimensional camera development consortium is currently designing and implementing all stages required for a 1 Mega-pixel detector: sensors, ASICs, PCBs and FPGA-code. The offdetector DAQ-systems being developed for use with the AGIPD, LPD and DSSC cameras at European XFEL implement a common interface to be used for control, synchronization and data transfer. ATCA readout boards are being developed whose FPGAs and cross point switches allow data to be processed on-the-fly and transferred to a PC-farm for further data processing, rejection and archiving. Reuse of designs and design techniques between systems currently being developed and future projects will increasingly require sharing expertise and development solutions in regions of hard-, firm-and software. Operation of an AGIPD camera begins in 2016 at European XFEL when first beams are delivered.