See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/301228235

ATLAS NOTE Level-1 Data Driver Card Design Review Report

Technical Report · March 2015

CITATION READS

1 11,715

5 authors, including:

Theo Alexopoulos Christos Bakalis

National Technical University of Athens National Technical University of Athens
1,456 PUBLICATIONS 78,790 CITATIONS 3 PUBLICATIONS 2 CITATIONS

SEE PROFILE SEE PROFILE

Panagiotis Gkountoumis
National Technical University of Athens
127 PUBLICATIONS 6,795 CITATIONS

SEE PROFILE

All content following this page was uploaded by Christos Bakalis on 13 April 2016.

The user has requested enhancement of the downloaded file.

 Draft version 1.0

ATLAS NOTE
February 14, 2015

1 Level-1 Data Driver Card Design Review Report

2 T. Alexopoulosa , C. Bakalisa , P. Gkountoumisa , G. Iakovidisb , A. Koulourisa

a
3 National Technical University of Athens, Greece
b
4 Brookhaven National Laboratory, USA

5 Abstract

6 In this document we describe the overall design and specifications of the final Level-1 Data
7 Driver Card for the electronics review process of the New Small Wheel. Furthermore, the
8 roadmap to the final board through out the evaluation of several prototype cards will be
9 presented along with the various tests that have to be performed in order to validate the
10 proposed design. Finally, an updated time schedule and the manpower plan needed to cover
11 the design and production of the Level-1 Data Driver Cards will be given.

© Copyright 2015 CERN for the benefit of the ATLAS Collaboration.

Reproduction of this article or parts of it is allowed as specified in the CC-BY-3.0 license.
 February 14, 2015 – 12 : 37 DRAFT 1

12 Contents
13 1 Introduction 3

14 2 Overall design and specifications 5

15 2.1 Architecture, 3D models, dimensions, components - connectors, tooling used . . . . . . 5
16 2.2 Board architecture and materials (Layer stack, Differential width/gap, Differential impedance,
17 FR4, etc.) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
18 2.3 Power distribution and consumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
19 2.4 Location of the L1DDC cards on detector and services routing . . . . . . . . . . . . . . 13
20 2.4.1 Micromegas card placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
21 2.4.2 sTGC card placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
22 2.4.3 Micromegas services routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
23 2.4.4 sTGC services routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
24 2.5 Specific design for MM and sTGC, share most of the design . . . . . . . . . . . . . . . 16

25 3 Specification of the functions to be performed 18

26 3.1 Connectivity and compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
27 3.1.1 Interface with FE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
28 3.1.2 Interface with ADDC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
29 3.1.3 Interface with FELIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
30 3.2 GBTx ASIC bank connectivity and configuration . . . . . . . . . . . . . . . . . . . . . 21

31 4 Tests 21
32 4.1 Validation in radiation and magnetic field . . . . . . . . . . . . . . . . . . . . . . . . . 23
33 4.2 Reliability - SEU detection/correction . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
34 4.3 Debugging tools and programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
35 4.4 Procedures for the final L1DDC tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

36 5 L1DDC Prototype1 23
37 5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
38 5.2 Design choices and Dimensions (Layer stack, Differential width/gap, Differential impedance,
39 FR4 etc.) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
40 5.3 Power distribution and Power consumption (Component description, Ramp up sequence,
41 Voltage Levels) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
42 5.4 Component description and specifications (FPGA, Ethernet PHY, SFP) . . . . . . . . . . 33
43 5.5 Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
44 5.5.1 Clock domain (internal and external clocks) . . . . . . . . . . . . . . . . . . . . 38
45 5.5.2 Connectivity with FEs, ADDC and DAQ . . . . . . . . . . . . . . . . . . . . . 41
46 5.6 Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
47 5.6.1 Communication Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
48 5.6.2 System core - User logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
49 5.7 Adapter boards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
50 5.8 Tests and debugging (Tools and applications used) . . . . . . . . . . . . . . . . . . . . . 48
51 5.9 Issues - errors to be corrected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
52 5.10 Tests with VMM equipped frontend prototypes . . . . . . . . . . . . . . . . . . . . . . 53

53 6 Prototype 2 & 3 and roadmap to final board 54

54 7 Time schedule and manpower 55

 February 14, 2015 – 12 : 37 DRAFT 2

55 Appendices 57

56 A Adaptation of firmware to Artix7 FPGA 57

 February 14, 2015 – 12 : 37 DRAFT 3

57 1 Introduction
58 High speed serial data transmission has become the norm in data acquisition systems of HEP experiments
59 following the development of commercial high-speed serial protocols such as PCIexpress, 10G Ethernet,
60 infiniband, etc. serializers and deserializers are the key components of such systems. Off the shelf
61 components (COTS) are typically not sufficiently radiation resistant and, in particular, quite susceptible
62 to single event upsets (SEU) caused by energetic (E > 20 MeV) hadrons. For this reason the CERN
63 microelectronics group has been developing a radiation hard chipset, the GigaBit Transceiver (GBT) [1].
64 The GBT chipset consists of a 4.8 Gb/s serializer/deserializer ASIC, a versatile optical transceiver along
65 with a laser driver IC, a pin diode receiver IC and a Slow Controls Adaptor (SCA). This separation of
66 functions allows different combinations of transceivers, optical drivers, laser sources, and photodiodes
67 to be used according to the needs of a particular application. Event data, Timing, Trigger and Control
68 (TTC) signals, front-end configuration, and DCS data can be exchanged via a single fibre pair between
69 the front-end and USA15 using the GBT. It is assumed that the USA15 end is implemented in an FPGA
70 rather than a GBT ASIC.
71 The GBT ASIC can be viewed as multiplexing a number of serial links, called E-links, with 80, 160, or
72 320 Mb/s rates onto one fibre by transferring 2, 4, or 8 bits per E-link via an 80-bit word (optionally 116
73 bits) every 25 ns, for an aggregate GBT bandwidth of 3.2 Gb/s. One e-link, consisting of three pairs of
74 differential lines (6 wires) being the clock (Clk+ and Clk-), the Data In (Din+ and Din-) and Data Out
75 (Dout+ and Dout-). The electrical levels follow the LVDS standard.
76 The Slow Control Adapter (SCA) ASIC provides an interface to several protocols needed for the front-
77 end configuration and monitoring. The GBT-SCA implements different buses like the JTAG and the I2C
78 which allow the configuration of the ASICs while features an ADC for monitoring input signals. The
79 SCA features a specific E-link for the data communication with the GBTx. Both micromegas (MM) and
80 small Thin Gap Chambers (sTGC) place an SCA on each front-end board. Although this results in a
81 substantial number of SCAs, it is far more reliable, more compact and less expensive than connectors
82 and cables to a more central location.
83 The Level-1 Data Driver Card (L1DDC) card will serve as an intermediate stage between the FE boards
84 and the Felix interface system. The two technologies of NSW sTGC and Micromegas (MM) use the
85 same building blocks for the readout L1DDC cards.
86 The micromegas system of the NSW needs to provide muon segment candidates to the Sector Logic
87 (SL) within the Level-1 latency of 41 clock ticks of 40 MHz. This corresponds to a latency of 1025 ns.
88 Once the segment is confirmed by the SL the tracking primitives have to be readout and processed. The
89 micromegas system consists of ∼2.1 M channels.
90 The front-end ASIC providing the trigger and tracking primitives is the VMM. For the micromegas, eight
91 VMM ASICs will be installed in a front-end board called MMFE. This creates a front-end board capable
92 of reading 512 micromegas readout strips. The MMFE, apart of the VMM ASIC, will feature a GBT-SCA
93 for the configuration, calibration and monitoring of the MMFE. Moreover a custom readout ASIC will
94 be integrated in the MMFE capable to gather, format and serially transmit the information of the eight
95 VMM into a single e-link output. A separate e-link will be used for the configuration path. The MMFE is
96 also capable of driving the ART signals to a separate transmission line for every LHC bunch crossing of
97 25 ns. All the components of the MMFE are installed in a PCB of 215×60 mm2 . The ART signals of the
98 eight MMFE boards are multiplexed into a trigger board called ART Data Driver Card (ADDC) which
99 is not described in this document. The readout data of eight MMFE, are multiplexed through e-links
100 into a data driver card called Level-1 Data Driver Card (L1DDC). The L1DDC features one GBTx. It is
101 capable of driving the Level-1 data through the GBTx into a fiber to a network interface called FELIX. It
102 is also capable to configure the ADDC and the MMFE through dedicated to configuration E-links. The
103 L1DDC is built in a PCB of the same dimensions like the ADDC. An overview representation of the
 February 14, 2015 – 12 : 37 DRAFT 4

104 NSW electronics complex is shown in Figure 1. The total number of L1DDC boards is 1024 (512 boards
105 for the micromegas detectors and 512 boards for the sTGC detectors).

BE detector electronics (USA15)

Figure 1: Overview representation of the NSW micromegas electronics complexity. The MMFE frontend
cards are connected to the L1DDC (light green) cards via e-Links.

106 The front end electronics will be installed on the micromegas wedges radially along both sides. This
107 provides a way of equalizing the load on both sides of the detector and the cable routing. Figure 2
108 shows a multiplexing diagram of the micromegas frontend electronics boards. The L1DDC implements
109 the configuration e-links uniform along the radial direction with a bit-rate of 80 Mbps e-links while the
110 readout e-links, with a bitrate of 160 Mbps for the four outer MMFE and 320 Mbps for the inner four.
111 This is due to the higher particle rate which requires a higher bandwith in the inner radius of the detector.

112 2 Overall design and specifications

113 In this section the final L1DDC board will be described. There is a detailed description of the architec-
114 ture of the board and the components that will be used, the estimation of power consumption and the
115 placement on the detector side. Also, it is mentioned the design choices (like layer stack, differential
116 impedance, differential width/gap) and the materials that will be used in order to fabricate the final board
117 (dielectric constant, FR4).

118 2.1 Architecture, 3D models, dimensions, components - connectors, tooling used

119 There is a limitation of the overall available space for the placement of the board to the detector. Because
120 of this height limitation the final board was decided to have connectors only in the one side of the board
121 and not in both sides. The other side will accommodate only the components that need to be cooled
122 through the cooling channel. The final board will be 200 mm in length, 50 mm in width and about
123 18.4 mm in height. A representation of the placement of the L1DDC board on the detector is shown in
124 Figure 3. In this figure in the upper margin of the detector is placed the L1DDC board. In the top side
125 of the board the connectors are visible (the power connector, the miniSAS connectors and the VTRx
126 transceiver) and on the bottom side the GBTx ASIC along with the DC-DC converter, the foam that
 February 14, 2015 – 12 : 37 DRAFT 5

MMFE

fibre
r

fibre
80 Mbps e-Link
160 Mbps e-Link
320 Mbps e-Link
160 Mbps twin-ax

Figure 2: Frontend electronics multiplexing diagram. The front end (MMFE) electronics cards are con-
nected via e-Links to L1DDC and ADDC cards. In addition a communication is established between
L1DDC and ADDC cards. The output of the readout cards is connected to the FELIX distubution system
via fiber optics cables.

127 keeps in touch the Integrated Circuits with the cooling channel and finally the ground pins that hold the
128 board on the detector. On the bottom side, on top of the hot water output is placed the micromegas Front
129 End board (MMFE8).
130 The final L1DDC board will accommodate the following basic components: the GBTx ASIC, the VTRx
131 optical transceiver, the DC-DC converter and the connectors (for power and data). The GBTX is a
132 radiation tolerant chip that can be used to implement multi-purpose high speed (3.2 − 4.48 Gb/s user
133 bandwidth) bidirectional optical links for high-energy physics experiments, see Figure 4 where the top
134 and bottom views are shown.
135 It will provide three distinct data paths for Timing and Trigger Control (TTC), Data Acquisition (DAQ)
136 and Slow Control (SC) information to the front end electronics and the ADDC boards as shown in Fig-
137 ure 5. The GBTX can be electrically interfaced with the on detector electronics using different topolo-
138 gies. The simplest one consists of interconnecting the GBTX and a front-end device through a parallel
139 lane while the most sophisticated allows the GBTX to interface simultaneously with up to 40 front-end
140 devices via duplex local Electrical serial links (E-links). The use of one single parallel lane, the use
141 of multiple parallel lanes or individual serial connections are valid subsets of the E-link programmable
142 features. Each E-link normally consists of three signal lines (differential pairs):

143 • Differential Clock line (dClk+/dClk-): Clock driven by GBTX to front-end module.

144 • Differential Downlink data output (dOut+/dOut-): Data line from GBTX to the front-end module.
 February 14, 2015 – 12 : 37 DRAFT 6

Figure 3: On detector placement of the L1DDC board; top board on the side of MM wedge is shown.
The board between the two readout panels of the micromegas wedge is the FrontEnd MMFE-8. Also, in
this figure, the cross section of the various cables (High & Low voltage and Fiber cables) running along
the cooling cutouts is shown.

(a) Top side of GBTx (b) Bottom side of GBTx

Figure 4: The GBTx ASIC

Figure 5: Overall functionality of the GBTX ASIC

145 • Differential Uplink data input (dIn+/dIn-): Data line from front-end module to GBTX [6]

146 Figure 6 represents the general interconnection topology between the GBTX chip and the Front End
147 electronics using E-links. For each group the E-Link clock signals (dClk+/dClk-) can be programmed
 February 14, 2015 – 12 : 37 DRAFT 7

Figure 6: Elinks between the front end boards and the GBTx.

148 independently to get any of the following frequencies: 40 MHz, 80 MHz, 160 MHz or 320 MHz. If, for
149 example, the E-Link clocks are programmed to be 40 MHz for the 2 × data rate (80 Mb/s) the links and
150 its associated clocks basically run as a Double Data Rate (DDR) connection with the clock having rising
and falling transitions occurring in the middle of the bit period as shown in Figure 7 [6].

Figure 7: E-Link clocks and data outputs.

 February 14, 2015 – 12 : 37 DRAFT 8

151 The connectivity of the L1DDC board with the on detector electronics (Front Ends and ADDC boards)
152 will be through the Molex 36p miniSAS connectors (Part No 75783-0132) as shown in Figure 8a and 3M
153 mini SAS cables (Part No 8F36-AAA105) as shown in Figure 8b [9]. The small size of the connectors
154 (8.47 mm height and 17.8 mm width) is shown in Figure 9a and Table 9b along with the highly routable
155 cables is the reason for choosing them for the update of the NSW. The miniSAS 36 position cable
156 can accommodate up to 4 receiving and 4 transmitting differential pairs, as long as 4 receiving and 4
157 transmitting pins are dedicated to sidebands. The L1DDC board will communicate with the Read Out
158 Companion ASIC (ROC) of the front-end boards which is the assigned chip to collect the data from the
159 VMMs and transmit them to the L1DDC. The front-ends are equipped with a second ASIC (the Slow
160 Control Adapter SCA) which is responsible for controlling and monitoring. Both ASICs (ROC and SCA)
161 must communicate with the L1DDC through one E-link each.

(a) Molex 36 position Mini SAS connector. (b) 3M mini 36 position mini SAS cable.

Figure 8: Mini SAS connector and cable

(b) Circuit Size

(a) Mini SAS dimensions.

Figure 9: MiniSAS connector dimensions and specifications

162 The general architecture of the GBTX ASIC and its main external connections are displayed in Figure 10.
163 The GBTX connects to the GBLD laser driver ASIC and to the GBTIA trans-impedance amplifier ASIC.
164 The Clock and Data Recovery (CDR) circuit receives high speed serial data from the GBTIA. It recovers
165 and generates an appropriate high speed clock to correctly sample the incoming data stream. The serial
166 data is then de-serialized (that is converted to parallel form) and then DECoded, with appropriate error
167 corrections, and finally DeSCRambled (DSCR).
168 In the transmitter part the data to be transmitted is SCRambled (SCR), to obtain DC balance, and then
169 encoded with a Forward Error Correction (FEC) code, before being serialized and send to the GBLD laser
170 driver. The configuration of the GBLD can be performed via a simplified I2 C-Light connection from the
171 GBTX. A clock manager circuit takes care of generating and manage the different high speed and low
172 speed clocks, needed in the different parts of the GBTX. A programmable phase shifter is available to
173 generate 8 external user clocks with programmable frequency and phase. An external clock, or an on-
 February 14, 2015 – 12 : 37 DRAFT 9

Figure 10: GBTx full functionality and connectivity with the Front End boards & the rest of the DAQ
system.

174 package crystal oscillator, is used during start-up as a locking aid for the CDR circuit and as a clock
175 reference for the ASIC watchdog circuit. General control and monitoring logic takes care of controlling
176 the different parts of the chip according to the operation mode selected and the ASIC configuration
177 information. Initial configuration information is taken from the on chip e-Fuses that can then be modified
178 via the optical link itself or via an I2 C slave interface. A JTAG interface is available for boundary scan.
179 Connections to the front-end modules or ASICs are made through sets of local Electrical Links (E-
180 Links). Depending on the data rate and transmission media used, E-Links allow connections that can
181 extend up to a few meters. E-Links use Low-Voltage Differential Signalling, with signal amplitudes that
182 are programmable to suit different requirements in terms of transmission distances, bit rate and power
183 consumption. The E-Links are driven by a series of ePorts on the GBTX and are associated with E-link
184 ports in the front-end modules. The number of active E-Links and their data rate are programmable.
185 Parallel front-end interfaces with different bit widths are valid sub-sets of the flexible E-Links [6].
186 There are two options for miniSAS cables, one with sidebands as shown in Table 11b and one without
187 sidebands as shown in Table 11c. Also 3M fabricates custom miniSAS cables in any length in case it
188 needed. In our scheme for the connection to the Front End the same length of cables will be used. This
189 will lead to the same clock and data delay propagation.
190 The receiver ports of the GBTX ASIC uses the Scalable Low-Voltage Signalling for 400 mV (SLVS-400)
191 and the LVDS standard. The LVDS signals uses a differential voltage swing of 400 mV centred on 1.2 V.
192 The SLVS standard is also differential but with a reduced voltage swing of 200 mV, centred on 0.2 V as
193 shown in Figure 12.
194 The VTRx optical transceiver consists of two ASICs, the GBTIA and the GBLD. The GTIA (GigaBit
195 TransImpendance Amplifier) has a bit rate of 5 Gb/s (min) and a total jitter smaller than 40 ps P-P. Its
196 supply voltage is 2.5 V and its power consumption is 250 mW [8]. The GBLD (GigaBit Laser Diode) is
197 also a radiation tolerant ASIC fabricated in 130 nm. It has also a bit rate of 5 Gb/s (min), supply voltage
 February 14, 2015 – 12 : 37 DRAFT 10

(a) 3M 36 position Mini SAS cable dimensions

(c) 3M mini 36 position mini SAS cable No sidebands

(b) 3M mini 36 position mini SAS cable

Figure 11: Mini SAS connector and cable

Figure 12: Typical levels of LVDS ans SLVS signals.

198 of 2.5 V and its power consumption is about 325 mW [7]. The VTRx is the largest component that it
199 will be placed on the L1DDC board with width of 45.3 mm, a length of 14.5 mm and a height of 10 mm
200 see Figures 13a and 13b.
201 The GBTX ASIC, with an I2 C slave port which allows the writing and reading of the GBTX configuration
202 registers. This can be used when the GBTX is operated in any of its modes. The GBTX is equipped with
203 a standard I2 C slave interface, and is accessed by an I2 C master (FPGA), transmitting data with the
204 correct address as shown in Figure 14b. The GBTX contains a simplified I2 C master that can be used
205 to configure the GBLD. This master is designed specifically to write and read the 7 GBLD registers and
206 cannot be used for programming other devices. Seven registers in the GBTX are reserved for storing the
207 values to write to the GBLD. These can be accessed by a normal write to the GBTX by the IC interface
208 as shown in Figure 13a or by I2 C as shown in Figure 14b [6].
 February 14, 2015 – 12 : 37 DRAFT 11

(a) VTRx bottom side and dimensions. (b) VTRx side view

Figure 13: VTRx optical transceiver.

(a) IC Configuration of the GBLD.

(b) I2 C Configuration of the GBLD

Figure 14: GBLD configuration

209 2.2 Board architecture and materials (Layer stack, Differential width/gap, Differential
210 impedance, FR4, etc.)
211 For the final board a quick estimate shows that a 14 layer board will enough to route all the signals. For
212 the differential pairs a 4 mils width and 4 mils gap will be used and all the pairs will be isolated with
213 copper planes which will serve as the ground and power planes. An estimate, using the Saturn program
214 of SATURN PCB Toolkit [21] shows that a differential impedance of 99.222 Ohms can be achieved for
215 the outer layers, as shown in Figure 15a and a 100.200 Ohms for the inner layers, as shown in Figure 15b.
216 The material is a standard FR4, the dielectric constant is er = 4.6 and the copper weight is 0.25 oz. The
217 GBTX will be placed in the middle of the board. An estimate of the routing length for this topology for
218 all the differential pairs is 3500 mils. The L1DDC will have the same routing length for all the differential
219 pairs, in order to minimize the delay of the clocks send to the FEs. This can be easily achieved using
220 accordion technique for each pair or individually for every signal of the pair. So the deviation of the
221 length for all pairs will not be more than 10 mils.

222 2.3 Power distribution and consumption

223 The distribution of the Low Voltage (LV) power on the board will be done through a power connector
224 (one option is the two pin Molex 0015912025). The connector has a total height of 7 mm and width of
225 8.64 mm. The dimensions of the connector are shown in Figure 16.
226 The GBTX ASIC has a maximum power consumption of about 1, 414 mW, as shown in Table 17a and
227 Figure 17b respectively. The power consumption of GBLD is 325 mW and the power consumption of
228 GBTIA is 250 mW. Finally, if the power consumption of the DC-DC converter is added, an estimate of
229 the upper limit of the overall power consumption is 3.5 Watts
 February 14, 2015 – 12 : 37 DRAFT 12

(a) Inner Layers (b) Outer Layers

Figure 15: Differential Estimation with Saturn

(a) Power connector

(b) Connector dimensions

(c) Circuit Size Table

Figure 16: Molex power connector

230 So far there is no candidate for the site of the DC-DC converter. Radiation tests shown that the Linear
231 Technology LT8612 [2] and the Analog Devices ADP1755 LDO [3] have a relatively tolerance in
232 radiation environments. There is also the possibility to use the CERN radiation hard DC-DC converter.
233 This DC-DC has input voltage range of 5 V to 12 V, 4 A load capacity and achieves a 76% efficiency. It
234 contains a radiation tolerant ASIC with total ionizing dose up to 200 Mrad (Si) and displacement damage
235 up to 5 × 1014 n/cm2 .
 February 14, 2015 – 12 : 37 DRAFT 13

(a) Table with power consumption portions of the GBTX ASIC

(b) A power consumption representation pite of the GBTX

Figure 17: GBTx power consumption

(a)
(b)

Figure 18: CERN radiation hard DC-DC converter

236 2.4 Location of the L1DDC cards on detector and services routing
237 In this section the location and placement of the L1DDC boards and the routing services (miniSAS cables
238 between the front-end boards, ADDC boards and the L1DDC, fiber & power cables) will be described in
239 detail.

240 2.4.1 Micromegas card placement

241 The L1DDC cards, will be placed on the wedge sides as seen in Figure 20. Here we are dealing with a
242 wedge, which is equivalent to 4 planes. As mentioned above, there is one L1DDC per layer. This means
243 that we have to place eight L1DDC cards on the wedge. The configuration is as follows: As mentioned
244 earlier in this document, each L1DDC is connected to 1 ADDC card and 8 MMFE cards, which gives
245 a total of 9 connections per L1DDC card, as shown by the number of mSAS connectors in Figure 19 .
 February 14, 2015 – 12 : 37 DRAFT 14

246 With this in mind, it makes sense to place the L1DDC cards close to the middle of the wedge. As seen
247 in Figure 20, 4 L1DDC are placed on each side of the wedge; 2 on the top and 2 on the bottom. This
248 means that two of the four L1DDC are facing the IP side and the other two L1DDC are facing the HO
249 side. In Figure 21, the underlined labels show that the card is place on the other (i.e. not visible) side of
the wedge.

Figure 19: Top view of L1DDC 3D Model.

250

Figure 20: Micromegas wedge 3D model, top view. The labels indicate where all the L1DDC will be
placed. Underlined label signifies that placement of the boards is on the opposite (here not visible) side
of the wedge.

251 2.4.2 sTGC card placement

252 First option is to place the cards either on or close to the rim (i.e. large radii). Apparently such a choice
253 is based on the wish to maintain the cards accessible even during operation. However this is not ideal
 February 14, 2015 – 12 : 37 DRAFT 15

Figure 21: View of the Large MM Wedge. Note the TwinAx cables, roughly modelled, going from the
miniSAS connectors to the ”cable space” represented as gray space, under the wedge. Fiber cables are
also incorporated in this design, with varying colours for every board. In the middle of the picture, you
can see the 9 TwinAx cables, which indicate an L1DDC card in the opposite side of the wedge, not
visible in previous pictures.

254 in terms of general services routing and placement, which include numerous boxes to be placed on an
255 accessible place close to the rim.
256 Another option is to follow the MM scheme and place the cards on the wedge side. For the time being
257 a combination of the above is the guideline. As visible in Figure 22 & 23, the L1DDC cards are to be
258 placed on the side of the wedge close to the rim, on the positions indicated for the two types of sectors,
259 small sTGC & large sTGC, respectively.

260 2.4.3 Micromegas services routing

261 Some work and analysis has already been presented on the MM on-wedge services routing. The services
262 concerning the L1DDC are related to the connectors of the card. Namely: 9 miniSAS, 1 fiber pair, and 1
263 power cable.
264 The main volume of these cables is represented by the miniSAS connections, which are carried out by
265 the TwinAx cables. This main volume is routed ”off-wedge”, yet on-sector, in the space between the 2
266 MM wedges. This space can be considered as ”extension” of the spacer frame. The fibers are less space
267 demanding, but naturally require extra care for the routing. They are to be routed through the cooling
268 channels, next to the actual cooling cutouts.
269 There are few issues that might make the cable routing more difficult:

270 1. The cooling channel is not yet finalized, which could mean less or no space for the fibers.

271 2. The sector kinematic mounts. These attach the spacer frame to the main NSW structure. As the
272 spacer frame is still under design, the foreseen routing space might be unavailable at a certain
273 point.
 February 14, 2015 – 12 : 37 DRAFT 16

Figure 22: L1DDC placement on small TGC wedge.

Figure 23: L1DDC placement on large TGC wedge.

274 3. The L1DDC fibers’ routing is not fully designed yet. Keeping in mind their fragile nature, it should
 February 14, 2015 – 12 : 37 DRAFT 17

275 be studied more closely, in order to eliminate possible mishaps.

276 so far, the preliminary routing can be seen in Figure 21.

277 2.4.4 sTGC services routing

278 The sTGCs service routing of the L1DDC board will be similar to the MM case, but is not yet fully
279 studied.

280 2.5 Specific design for MM and sTGC, share most of the design
281 There is the option of using different L1DDC boards for the micromegas detectors and different for the
282 sTGCs detectors. This comes up because the sTGCs need actually only 3 miniSAS connectors with 10
283 pairs of differential pairs as shown in Table 1. Generally, the routing signals from the GBTX to the
284 mSAS are different in the micromegas and sTGCs detectors. So, finally, in the case of fabricating only
285 one board, an extra logic/circuitry must be added (multiple selection with external jumpers or 0 Ohm
286 resistors) in order to be compatible for both detectors.

Table 1: L1DDC elinks to sTGCs

287 The advantages of designing a different board are:

288 • sTGCs board will have a dramatically lower size, probably less than 100 mm in length. Of course
289 the width is defined at 50 mm. A preliminary example of the board for the sTGCs is shown in
290 Figure 24.

Figure 24: L1DDC final board for sTGCs

 February 14, 2015 – 12 : 37 DRAFT 18

291 • The routing length of the differential pairs will also be much smaller.

292 • A first approximation shows that accordion technique may be not necessary.

293 • A slightly less cost as it will contain less mSAS connectors (3 instead of 9).

294 • Signal integrity will not be affected by external jumpers or 0 Ohm resistors.

295 • Designing one board can cause a complicated routing scheme.

296 The disadvantages of designing a different board are:

297 • It will be needed to design, fabricate and test two different boards. These procedures must be
298 implemented not only for the final board but also for all the prototypes.

299 • 2 different boards will cost more than one in the fabrication procedure.

300 3 Specification of the functions to be performed

301 In this section the functions of the L1DDC board and the connectivity (connectors and cables) to the on
302 detector electronics and the electronics room (USA15) will be described. Data rates, protocols, commu-
303 nication standards-voltage levels and configuration of the GBTX also are explained.

304 3.1 Connectivity and compatibility

305 The L1DDC board on the one side will communicate with the FEs and ADDC boards. These boards
306 will accommodate custom ASICs and for that reason a full compatibility must be achieved. The voltage
307 levels of all ASICs have to be the same to avoid using extra Integrated Circuits or techniques. The E-Link
308 data rates are programmable in a per group basis and are given in Table 2. Within a group, the input and
309 output lanes can be set to work at different data rates and an unused group can be turned off to reduce
310 the power consumption in the corresponding internal logic. The bit shift in/out order for the E-Link data
311 inputs and outputs is MSB first.

Table 2: GBTx E-link rates

312 3.1.1 Interface with FE

313 As mentioned before, the communication between the front-ends and the L1DDC will be through the
314 miniSAS connectors and cables. As was described in previous sections the final front-end boards will
315 have 2 ASICS, the ROC ASIC which is responsible for transmitting the Level-1 data and the SCA which
316 is responsible for controlling and monitoring. The connection to each ASIC will be from a different elink
317 but from the same miniSAS cable.
318 The GBTx ASIC uses the SLVS standard in the transmitting side and the SLVS and LVDS standards in
319 the receiving sides. For this reason any board that will be connected with a L1DDC must be compatible
 February 14, 2015 – 12 : 37 DRAFT 19

320 with any of these 2 standards. The data rates in the elinks for the connection to the ROC ASIC will
321 be programmable at 80 Mbps, 160 Mbps and 320 Mbps. The SCA elinks bandwidth will be fixed at
322 80 Mbps.

323 3.1.2 Interface with ADDC

324 As with the front-ends, the connection with the ADDC will be through one more miniSAS connector.
325 As the L1DDC will send only the clock and configuration data to the ADDC board, the data rate will be
326 fixed at 80 Mbps. The final ADDC board will have two GBTX ASICs and for that reason two E-links
327 will be used.

328 3.1.3 Interface with FELIX

329 On the other side, the L1DDC will be connected to Front-End LInk eXchange (FELIX) through the
330 VTRx optical transceiver and a dual fibre as shown in Figure 1. This link can perform bidirectional
331 communication with the FELIX as it is responsible to deliver the Level-1 data to the USA15 but also
332 to send the TTC data to the front-ends and ADDC boards. The GBTX can implement three types of
333 frame, the 8B/10B frame, the GBT frame and the wide frame. The 8B/10B mode is only available in the
334 transmitter part of the GBTX. The 8B/10B frame format is shown in Figure 25. No error correction and
335 only very limited error detection capability is possible with this format. As the 8B/10B coding is DC
336 balanced no data scrambling is used in this mode. The 120-bit link frame consists of 12 8B/10B words
337 of which the first is always a comma character required for the receiver synchronization. The effective
number of user bits in this mode are 11 × 8 = 88, giving a coding efficiency of 88/120 = 73% [6].

Figure 25: 8B/10B frame

338
339 The 120-bit GBT frame format, sketched in Figure 26, is transmitted during a single LHC bunch crossing
340 interval (25 ns), resulting in a line rate of 4.8 Gb/s. Four bits are used for the frame Header (H) and
341 32 are used for Forward Error Correction (FEC). This leaves a total of 84 bits for data transmission
342 corresponding to a user bandwidth of 3.36 Gb/s. Of the 84-bits, 4 are always reserved for Slow Control
343 information (Internal Control (IC) and External Control (EC) fields), leaving 80-bits for user Data (D)
344 transmission. The D and EC fields use is not pre-assigned and can be used indistinguishably for Data
345 Acquisition (DAQ), Timing Trigger & Control (TTC) and Experiment Control (EC) applications. DC-
346 balance of the data being transmitted over the optical fibre is ensured by scrambling the data contained in
347 the SC and D fields. For forward error correction the scrambled data and the header are Reed-Solomon
 February 14, 2015 – 12 : 37 DRAFT 20

348 encoded before serialization. The 4-bit frame header is chosen to be DC balanced. The GBT frame is
349 shown in Figure 26 [6].

Figure 26: GBT frame

350 A wide frame mode format with only scrambling is available for the transmitter direction. It provides
351 an alternative for data transmission where the forward error correction functionality is traded off for
352 bandwidth. That is, in the wide frame mode the FEC field is not present, see Figure 27 and the space
353 taken by the FEC code in the GBT frame is used to transmit data. As a consequence, the data field
354 increases to 112 bits resulting in a total user bandwidth of 4.48 Gb/s, representing an increase of (112-
355 80)/80 = 40% of available bandwidth when compared with the GBT frame format. However this is
356 done at the cost of having no SEU error protection on the transmitted data. Frame bits D[111:80] are
357 scrambled separately to maintain DC balance of transmitted data [6].
358 For the communication of the L1DDC board with FELIX the GBT frame (the frame which implements
359 Reed-Salomon encoding for SEU protection) will be used.

360 3.2 GBTx ASIC bank connectivity and configuration

361 As mention in previous sections the L1DDC will communicate with the front-ends through two E-links.
362 The one E-link will be used for the communication with the ROC and the other for the communication
363 with SCA. The SCA E-link can work effectively with the lowest bandwidth of 80 Mbps. On the other
364 hand the inner portions of the detectors (LM1, SM1), which can achieve a high rate, and will need a
365 bandwidth of 320 Mbps. For this reason the configuration of the GBTX banks was arranged as it is
366 shown in Table 3. One GBTX bank can support only two E-links at 320 Mbps, 4 E-links at 160 Mbps
367 and 8 E-links at 80 Mbps. In the case of MMs detectors, the 4 inner front-end will occupy the 2 of the 5
368 GBTX banks (green color in Table 3). This means that these front-ends can achieve a rate of 320 Mbps.
369 The next 2 front-ends are connected to another bank of GBTX and can run at 160 Mbps or 320 Mbps.
370 The other 2 front-ends are connected along with one E-link for the ADDC to the 4th bank. These front-
371 ends can run to a maximum of 160 Mbps (blue color in Table 3). Finally the last GBTX bank is reserved
372 for the 8 configuration E-links of the SCA (yellow color in Table 3).
373 For the sTGCs detectors one front-end is connected to 2 banks of GBTX (green color in Table 3). these
374 front-ends can run at 320 Mbps. The other 2 banks are used for the connection to the next 2 front-ends
375 (blue color in Table 3) which can run at 160 Mbps. Finally, the last GBTX bank available is used for the
376 SCA configuration of the front-ends and run at 80 Mbps (yellow color in Table 3). The sTGCs connection
 February 14, 2015 – 12 : 37 DRAFT 21

Figure 27: Wide frame

Table 3: GBTx bank connectivity with micromegas and sTGCs front-ends

377 with the L1DDC will be trough one miniSAS cable and the differential pairs shown in Table 4 will be
 February 14, 2015 – 12 : 37 DRAFT 22

378 used.
379 GBTX also provides an extra E-link for Slow Control operations running only at 80 Mbps and it will be
380 used as a second E-link connection with the ADDC.

Table 4: sTGCs E-link connection

381 4 Tests
382 The on detector electronics will be exposed to a huge dose of radiation. For that reason every component
383 that will be housed on these boards must be radiation tolerant in order to achieve correct operation. Also,
384 techniques and methods must be implemented to ensure the transmitting signal integrity. Error correction
385 codes are used in the transmission of data and also triple mode redundancy is implemented inside the
386 ASICs.
387 On the other hand extensive tests, especially in radiative environments and magnetic fields, must be
388 performed in order to test the reliability of the boards and components.

389 4.1 Validation in radiation and magnetic field

390 There are no current tests of the L1DDC board in radiation and magnetic fields. The L1DDC prototype2
391 board will contain only radiation hard components and it will be tested in radiation and magnetic field
392 environments.

393 4.2 Reliability - SEU detection/correction

394 The GBTx ASIC has shown a good behaviour in radiation. On the other hand the DC-DC converters and
395 LDOs seem to be not so stable after they are exposed to radiation. For this reason there is a thought to use
396 the CERN radiation hard DC-DC converter to step down the voltage to the appropriate levels. GBTX,
397 as mentioned in section 3.1.3, when using the GBT frame for the transmission of the data to Felix
398 implements a forward error correction algorithm. This algorithm is a 2 times interleaved Reed-Salomon
399 RS(15,11) code with 4-bit symbols. The code is built by interleaving two Reed-Solomon encoded words
400 with 4-bit symbols, each capable of correcting a double symbol error. This, in practice, means that a
401 sequence of up to 16 consecutive corrupted bits can be corrected. Finally, all configuration registers in
402 the GBTX are fully protected against SEU’s with triple redundant registers with auto error recovery.
 February 14, 2015 – 12 : 37 DRAFT 23

403 4.3 Debugging tools and programs

404 For the next prototypes the software Hyperlinx version 8.2.1 of Mentor Graphics will be used in order
405 to simulate the integrity of the signals inside the board. Also, the HyperLynx Thermal will be used as
406 it performs a fast, complete, accurate 3D modeling of complex flow and thermal fields in PCB designs.
407 After the fabrication of the boards an extra thermal analysis can be performed with a thermal camera.
408 Extended hardware tests with evaluation boards and PCs can verify the stability and integrity of the
409 signals being transmitted. Evaluation boards also will be used to test the communication of the L1DDC.
410 For this reason firmware for the evaluation board will be implemented. Xilinx’s ISE design Suite and
411 ISE Simulation and ChipScope programs can be used.

412 4.4 Procedures for the final L1DDC tests

413 For the final L1DDC board a set of tests must be performed in order to decide if the board is accepted.
414 First of all power consumption, and thus the current, must be within a safe margin that will be defined.
415 The integrity of the data being transmitted from the FE boards through the L1DDC must also be tested.
416 For this reason evaluation boards as long as L1DDC prototypes can be used in order to send defined data
417 to the L1DDC. These data will propagate through the L1DCC and transmitted to a PC. There, using a
418 custom software we can compare the received data with the defined data. If there are frame errors the
419 specific board will be rejected. Also, L1DDC provides the configuration data and the clock to the FEs.
420 This clock can be at 40 MHz, 80 MHz and 160 MHz. Extended tests for these three frequencies must
421 be performed in order to evaluate the proper creation of the clock in order to ensure the simultaneously
422 propagation to the front-ends. For the external communication of the L1DDC fixed and tested cables
423 must be used.
424 Finally, some L1DDC final boards will be picked up randomly for continuous functionality in order to
425 test the integrity of the board for a long time operation.

426 5 L1DDC Prototype1

427 In this section the L1DCC prototype board will be described. Especially there is a detailed description
428 of the architecture of the board (dimensions, materials and rules that were followed), of the components
429 that were used and for design choices that were made.

430 5.1 Overview

431 The L1DDC Prototype1 is a multifunction board containing not only the radiation hard GBTx ASIC
432 but also a Xilinx’s FPGA as an alternative solution in case of GBTx malfunction or failure during the
433 operation. In Figure 28 is shown the full functionality of the L1DDC prototype1. Except the VTRx
434 optical transceiver, it has also two extra data outputs an SFP+ and a 1 Gb Ethernet in order to be com-
435 patible with any read out system. There is a second RJ45 connector which is responsible to receive the
436 trigger and time signals from an external source. The GBTX ASIC provides 8 SLVS clock outputs that
437 can be used as local timing references in the front-end electronics. These clocks are generated by the
438 phase-shifter circuit, and are fully synchronous with one of the GBTX 40 MHz clocks, depending on the
439 transceiver mode, (synchronous with LHC bunch crossing reference) and maintain with it a stable phase
440 relationship. The 8 reference clocks are both programmable in phase, 50 ps resolution and frequency 40,
441 80 160 and 320 MHz. These clocks are associated with differential drivers with programmable driving
442 strength and are routed to an extra miniSAS connector as shown in the lower right side of Figure 28. A
443 SMA connector was also used for driving external signals to the FPGA, left side of Figure 28. A SMA
444 pair was used in order to provide an external reference clock to the GBTx ASIC if this is needed. A
 February 14, 2015 – 12 : 37 DRAFT 24

Figure 28: L1DDC functionality

445 3D representation of the Top side of the L1DDC board using Altium Designer software can be seen in
446 Figure 29a and the bottom side in Figure 29b.

(a) Top layout (b) Bottom layout

Figure 29: 3D Representation of L1DDC prototype1

447

448 5.2 Design choices and Dimensions (Layer stack, Differential width/gap, Differential
449 impedance, FR4 etc.)
450 L1DDC was designed with a 14 layer stack as is shown in Figure 30. The selection between GBTx
451 and FPGA, which has as result some non active signals, gave the opportunity to use some non isolated
452 layers in the design. The board was designed using Mentor Graphics software version 9.2 and especially
453 Pads Logic for the schematic, Pads Layout for the layout and Pads Router for routing the signals. A
454 typical layout of the board is shown in Figure 31. There was no actual limitation in the dimensions of
455 the prototype board which led to the final dimension of 210 mm × 144 mm. The maximum height of the
 February 14, 2015 – 12 : 37 DRAFT 25

Figure 30: L1DDC stack layer properties

456 board is 2.2911 mm (90.2 mils) and the highest component is 10 mm (VTRx optical transceiver) which
457 led to an overall height of 12.2911 mm.
458 The target for the differential impedance was 100 Ohms for the internal and external layers. The param-
459 eters (material height, dielectric constant, copper height) chosen led to a very good approximation of the
460 targeted impedance as it shown in Table 5.
461 The placement of the GBTx and FPGA was chosen to be in the middle of the board in order to minimize
462 the routing length of the signals to the miniSAS connectors. This led to a maximum routing length of
463 4550 mils for FPGA and 4580 mils for the GBTx ASIC. On the other hand, this scheme has the penalty
464 of the deviation in the routing lengths of the differential pairs from each miniSAS connector to the
465 GBTX and FPGA. For that reason all differential pairs were designed to have the same length. This was
466 succeeded by using the accordion technique for the differential pair, as it is shown in Figure 32a and for
467 every signal independently, as it is shown in Figure 32b.
468 For the layout design of the L1DDC the clearance rules shown in Figure 33 were used. The GBTx ASIC
469 uses a 400pins BGA package. It’s ball diameter is 0.5 mm and it’s pitch is 0.8 mm see figure 34 for
470 GBTX ball pitch and dimensions. This leaves very little space between the vias and it was difficult to
471 pour the power and ground planes underneath the BGA. This is the reason that the vias were designed
472 with diameter size 18 mils and drill size 8 mils see Figure 35.
473 Five borads of the first prototype of the L1DDC were fabricated in 09/2104. In Figure 37a there is
474 the top side and in Figure 37b the bottom side of the of the L1DDC prototype1 after the fabrication.
475 Unfortunately due to the minimum number of one GBTx available only one of the 5 prototypes has the
476 GBTx ASIC.
477
 February 14, 2015 – 12 : 37 DRAFT 26

Figure 31: L1DDC layout

(b) Accordion used to indi-

(a) Accordion used to differential pairs vidual signal

Figure 32: Use of accordion

478 5.3 Power distribution and Power consumption (Component description, Ramp up se-
479 quence, Voltage Levels)
480 The distribution of the low voltage in the L1DDC is made through a 4 pin Molex power connector (Part
481 No 039303041), which is a quite small connector in size (width and height) as shown in Figure 38a . The
482 dimensions of the connector are shown in Figure 38c and Table 38d. The female connector is the 4 pin
483 Molex (Part No 39-01-4040) which is shown in Figure 38b.
The Linear Technology LT8612 DC-DC converter was used to step down the low voltage power to the
appropriate levels. This DC-DC converter has a wide input voltage range of 3.4 V to 42 V, an output
 February 14, 2015 – 12 : 37 DRAFT 27

Table 5: Differential impendance

Figure 33: Clearance rules in the L1DDC board

current of 6 A and a low dropout (under all conditions) voltage of 250 mV at 3 A [2]. Also, radiation
tests showed that it is quite a radiation tolerant chip. There is a recommended PCB layout of the LT8612
for the top layer of the board, as shown in Figure 39a which was used as a guideline in the L1DDC board.
Figure 39b represents the schematic of the DC-DC converter that was used in L1DDC prototype1 board.
In Figure 40a there is a recommended schematic design for the step down process from 3.9 V to 42 V
input voltage to the 3.3 V output voltage with the use of the LT8612 DC-DC Converter. In Figure 40b is
shown the design that was implemented in the L1DDC board. The output voltage is programmed with a
resistor divider between the output and the FB pin. Resistor values can be chosen according to:
 V 
out
R1 = R2 −1
0.970 V
484 where the R2 resistor is the ground resistor and R1 is the resistor connected to Vout .
 February 14, 2015 – 12 : 37 DRAFT 28

Figure 34: GBTX ball pitch and dimensions

Figure 35: Vias dimensions inside GBTx

Figure 36: Pouring internal planes

485 In order to step down the outputs of the DC-DC converters, to the appropriate levels, and to supply the
486 analog devices, parts LDOs were used. As mentioned in section 2.3 the AD1755 LDO was used. The
487 specific LDO has a maximum output current of 1.2 A and the input voltage range is 1.6 V to 3.6 V. Also,
 February 14, 2015 – 12 : 37 DRAFT 29

(a) Top Side of the L1DDC (b) Bottom Side of the L1DDC

Figure 37: L1DDC Prototype1

(a) 3D representation of the male power connector (b) 3D representation of the female power connector

(c) Dimensions of power connector (d) Table with dimensions

Figure 38: Molex Inc power connector

488 the ADP1755 has a very low drop-out voltage, 105 mV at 1.2 A load and an adjustable output voltage
489 option with soft start from 0.75 V to 3.3 V.
490 Artix7 has 8 DC supply voltages (VCCINT, VCCAUX, VCCBRAM, VCCO, VCCBATT, VMGTAVTT,
491 VMGTAVCC, VCCADC). It’s DC characteristics are shown in table 6.
492 On the other hand, Artix7 has a recommended power-on sequence (VCCINT, VCCBRAM, VCCAUX,
493 and VCCO) to achieve minimum current draw and ensure that the I/Os are 3-stated at power-on. The
494 recommended power-off sequence is the reverse of the power-on sequence. If VCCINT and VCCBRAM
495 have the same recommended voltage levels, SO both can be powered by the same supply and ramped
 February 14, 2015 – 12 : 37 DRAFT 30

(b) LT8612 layout on L1DDC

(a) LT8612 recommended layout

Figure 39: ADP1755 schematics

(a) LT8612 recommended schematic (b) Schematic implemented

Figure 40: LT8612 schematics

496 simultaneously. If VCCAUX and VCCO have the same recommended voltage levels, then both can be
497 powered by the same supply and ramped simultaneously. For VCCO voltages of 3.3 V in HR I/O banks
498 and configuration bank 0:

499 • The voltage difference between VCCO and VCCAUX must not exceed 2.625 V for longer than
500 TVCCO2VCCAUX for each power-on/off cycle to maintain device reliability levels.

501 • The TVCCO2VCCAUX time can be allocated in any percentage between the power-on and power-
502 off ramps.

503 The recommended power-on sequence to achieve minimum current draw for the GTP transceivers is
504 VCCINT, VMGTAVCC, VMGTAVTT OR VMGTAVCC, VCCINT, VMGTAVTT. Both VMGTAVCC
505 and VCCINT can be ramped simultaneously. The recommended power-off sequence is the reverse of
 February 14, 2015 – 12 : 37 DRAFT 31

Table 6: Artix 7 DC characteristics

506 the power-on sequence to achieve minimum current draw. If these recommended sequences are not met,
507 current drawn from VMGTAVTT can be higher than specifications during power-up and power-down.
508 When VMGTAVTT is powered before VMGTAVCC and VMGTAVTT − VMGTAVCC > 150 mV and
509 VMGTAVCC < 0.7 V, the VMGTAVTT current draw can increase by 460 mA per transceiver during
510 VMGTAVCC ramp up. The duration of the current draw can be up to 0.3× TMGTAVCC (ramp time
511 from GND to 90% of VMGTAVCC). The reverse is true for power-down. When VMGTAVTT is powered
512 before VCCINT and VMGTAVTT−VCCINT > 150 mV and VCCINT < 0.7 V, the VMGTAVTT current
513 draw can increase by 50 mA per transceiver during VCCINT ramp up. The duration of the current draw
514 can be up to 0.3× TVCCINT (ramp time from GND to 90% of VCCINT). The reverse is true for power-
515 down [4].
516 To maintain this power on sequence, an enabling logic for DC-DC converters and LDOs was designed.
517 On the first stage the DC-DC converters that power on the VCCINT(1.0 V), VCCBRAM (1.0 V) and the
518 power supply (3.3 V) for the LDOs are enabled. In the second stage the VCCAUX (1.8 V) and VCCADC
519 (1.8 V Analog) LDO are enabled. Finally in the third stage all the other power supplies (MGTAVVT
520 (1.2 V), VCCO (1.2 V or 2.5 V, 2.5 V, 1.5 V, 3.3 V), GBTx 1.5 V Digital and 1.5 V Analog) are enabled.
521 The 3 stage process for the ramp-up sequence of the power for the Arix7 FPGA is shown in Figure 41.
522 The output voltage of the ADP1755 can be set over a range of 0.75 V to 3.3 V by connecting a resistive
523 voltage divider from Vout to ADJ. The output voltage is calculated using the following equation:
!
R1
Vout = 0.5 V × 1 +
R2
524 where: R1 is the resistor from Vout to ADJ and R2 is the resistor from ADJ to GND, see Figure 42a. In
525 Figure 42b there is the final schematic that was used in L1DDC for the supply of the selectable voltage
 February 14, 2015 – 12 : 37 DRAFT 32

Figure 41: Power on sequence of Artix7 FPGA

526 (2.5 V and 1.2 V) for the FPGA banks. This LDO is supplied from an 3.3 V DC-DC converter and it is
527 enabled in the 3rd stage from the 1.8 V LDO.

(a) ADP1755 with adjustable

output voltage, 0.75 V to 3.3 V

(b) ADP1755 schematic in L1DDC

Figure 42: ADP1755 schematics

528

529 The manufacturer suggests a specific layout for top layer, as it is shown in Figure 43a and bottom layer,
530 as it is shown in Figure 43b, which was used as a guideline in the L1DDC board [3]. The final layout of
531 the ADP1755 that was used for the top and bottom layers are shown in Figures 43c and 43d respectively.
532

533 5.4 Component description and specifications (FPGA, Ethernet PHY, SFP)
534 As an alternative routing path, in case of failure of the GBTx ASIC, a Xilinx FPGA was used. The Artix7
535 xc7a200t-3fbg484 was quite cheap and full fills all the specifications of the board. It consists of 215,360
536 logic cells, it has 4 GTP transceivers and 285 user IOs, as shown in Table 7a and table 7b.
537 There is also a MICRON SPI flash memory (Part No N25Q256A13EF840E) of 256 Mb to store program-
538 ming permanently [15]. The FPGA and the FLASH are programmed through a JTAG (Join Test Action
539 Group) connector with the use of the Xilinx DLC10 programming cable and the iMpact software. There
 February 14, 2015 – 12 : 37 DRAFT 33

(a) Top Layout (b) Bottom Layout

(c) Top Layout on L1DDC (d) Bottom Layout on L1DDC

Figure 43: ADP1755 recommended layout

Figure 44: Xilinx DLC10 programming cable

540 are 2 JTAG connectors on the L1DDC board one for the Artix7 FPGA and one for the GBTx ASIC. The
541 JTAG of the FPGA has a voltage level of 3.3 V Figure 45a and the one for the GBTx ASIC 1.5 V. In
542 Figure 45b there is a representation of the schematics for the GBTX JTAG connector that was used in
543 L1DDC prototype1 board.
544 The FPGA has 484 BGA package with 0.6 mm ball diameter and 1 mm pitch and has a sum of 7 banks
545 (Banks 13,14,15,16,34,35 and one MGT bank 216) see Figure 46.
546 The banks are powered as follow:
547 • Bank 35 is powered with 1.5 V and is used to communicate with the GBTx.

548 • Bank 14 is powered with 3.3 V and is used to control the multiplexers and for the SFP+ commu-
 February 14, 2015 – 12 : 37 DRAFT 34

(a) artix7 Feature Summary

(b) Artix7 GTPs and IOs

Table 7: Artix7 Characteristics

(a) Artix7 JTAG schematic

(b) GBTx JTAG schematic

Figure 45

549 nication.

550 • Bank 13 is powered with 2.5 V and is used to communicate with Ethernet PHY chip and VTRx
551 transceiver.

552 • Banks 34, 14, 15, 16 are powered with a selectable power of 2.5 V or 1.2 V and are used as E-links
553 (differential pairs) for communication with the on detector electronics.

554 Finally in Figure 47a is shown the I/O pin description of the Artix7 xc7a200t-3fbg484 FPGA and in
555 Table 47b is represented the explanation of those pins.
556 For the decoupling of the FPGA most of the recommendations mentioned in the Xilinx ”7 Series FPGAs
557 PCB Design and Pin Planning Guide” were implemented, see Table 8 [11]. All the 0201 and some
558 0420 size capacitors were placed underneath (bottom side of the board) the FPGA, minimizing the dis-
 February 14, 2015 – 12 : 37 DRAFT 35

Figure 46: Artix7 xc7a200t-3fbg484 available banks

(b) Artix7 xc7a200t-3fbg484 FPGA pin explanation

(a) Artix7 xc7a200t-3fbg484 FPGA I/O map-
ping

Figure 47: Artix7 xc7a200t-3fbg484 FPGA pinout

559 tance from the corresponding pins. All other capacitors (0402, 0603 and 0805) including the tantalum
560 capacitors were placed very close to the FPGA.
561 On the other hand for the decoupling of the GBTx ASIC the recommendations of the GBT group were
562 followed as shown in Figure 48a. The decoupling logic was simplified, leading in only three distinct
563 power supplies the 1.5 V Analog, the 1.5 V Digital and the 3.3 V for the efuses as shown in Figure 48b.
564 The Ethernet PHY chip Marvell 88e1111 [12] was used to implement the physical layer for the Ethernet
565 communication of the L1DDC. The Ethernet PHY works with the Serial Media Independent Interface
566 (SGMII) without clock and supports the 1 Gb Ethernet. The schematic that was implemented in the
567 L1DDC prototype1 board is shown in Figure 49. The configuration pins CONGIG[6:0] are set as shown
568 in Table 9a. The mapping of the configuration pins was chosen from the Table 9b.
569 The Ethernet PHY is connected to the Bel Fuse Inc RJ45 connector with magnetics as is shown in the
570 schematics Figure 50.
 February 14, 2015 – 12 : 37 DRAFT 36

Table 8: PCB decoupling recommendation for Artix7 FPGA

(a) PCB decoupling recommendation for GBTx ASIC

(b) PCB final decoupling for GBTx ASIC

Figure 48: GBTX decoupling

571 An alternative communication can be implemented through a standard SFP+ optical transceiver. The
572 connector that was used is a standard SFP connector of TE Connectivity (Part No 1888247-1). The
573 schematic of the connector for the optical transceiver is shown in Figure 51a. There is a hard-wired
574 selection for the SFP RS0 and SFP RS1 as long a selectable power of 2.5 V and 3.3 V for compatibility
575 tests with the VTRx transceiver. Finally the signal SFP TX DISABLE can be set to ground through an
576 hard-wired connector.
 February 14, 2015 – 12 : 37 DRAFT 37

Figure 49: Ethernet PHY 88e1111 schematics

(a) Configuration pins in L1DDC (b) Pin to constant mapping

Table 9: Ethernet configuration

577 For the VTRX optical transceiver the same standard SFP connector of TE Connectivity (Part No 1888247-
578 1) was used. Figure 51b shows the schematic of the connector for the VTRX optical transceiver.
579 There was also another issue concerning the incompatibility between the voltage levels as the GBTx
580 ASIC works with 1.5 V and the VTRx with 2.5 V. For that reason the voltage levels had to be translated
581 from the one voltage to the other and for that purpose the Texas Instruments I2 C voltage level translator
582 was used (Part No PCA9306DCTR) see Figure 52.
February 14, 2015 – 12 : 37 DRAFT 38

Figure 50: Ethernet RJ45 connector schematic

(a) SFP+ optical transceiver connector schematic

(b) VTRX optical transceiver connector schematic

Figure 51: SFP+ and VTRX connector schematic

Figure 52: I2 C voltage translator

 February 14, 2015 – 12 : 37 DRAFT 39

583 5.5 Functionality

584 In this subsection the functionality of the L1DDC board is described in great detail. Decisions and
585 choices for the functionality of the board and the selection of the components are also mentioned. Finally,
586 abstracts of the schematics and layout are represented.

587 5.5.1 Clock domain (internal and external clocks)

588 There are two user clocks, a 40 MHz single ended of Abracon Corporation (Part No ASV-40.000 MHz-
589 EJ-T) and a 200 MHz LVDS differential clock of Silicon Laboratories Inc (Part No 510BBA200M000BAG).
590 For the FPGA GTP reference clock a 25 MHz Crystal (Part No ABM8-25.000 MHz-B2-T) was used in
591 conjunction with a the Ethernet generator Integrated Circuit of IDT, Integrated Device Technology Inc
592 (Part No 844021BG-01LF) which performs a multiplication of the input frequency by 5 times as shown
593 in Figure 54 [16]. For the GTP transceivers a second 200 MHz LVDS differential clock of Silicon
594 Laboratories Inc (Part No 510BBA200M000BAG) was used as the second reference clock.

Figure 53: Creation of the 125 MHz reference clock for the Ethernet

Figure 54: Ethernet clock generator

595 For the GBTx the reference clock is an external clock assigned to the chip through two SMA connectors.
596 Micron SPI FLASH memory supports 54 MHz (MAX) clock frequency (supported for all protocols
597 in DTR mode). The EMCCLK pin of the Artix7 FPGA was connected to a 50 MHz clock of AVX
 February 14, 2015 – 12 : 37 DRAFT 40

598 Corp/Kyocera Corp with Part No : KC2520B50.0000C10E00. This pin is an optional external clock
599 (External Master Configuration Clock) input for running the configuration logic in a master mode (versus
600 the internal configuration oscillator). In master modes the FPGA can optionally switch to EMCCLK as
601 the clock source, instead of the internal oscillator, for driving the internal configuration engine [20].
602 Finally the GBTX has a selectable supply voltage of 1.5 V and 3.3 V to power the efuses. The selection is
603 implemented with the use of a relay of the TE Connectivity (Part No IM01GR) and is controlled through
604 a MOSFET from the FPGA. The 3.3 V can be also set manually through hard-wired connector. Figure 55
shows the schematic that was implemented in the L1DDC board.

Figure 55: Efuse power selection

605

606 5.5.2 Connectivity with FEs, ADDC and DAQ

607 As mentioned in previous chapters the connectivity with the front-ends and the ADDC boards will be
608 through miniSAS connectors and it will use 2 E-links for every connection. The pin assignment at the
miniSAS connector for the front-ends is shown in Figure 56a and for the ADDC to Figure 56b. The

(a) Elink connectivity to front-end (b) Elink connectivity to ADDC

Figure 56: Elink connectivity to on detector electronics

609
610 selection between the FPGA and the GBTx is made through a NXP 3.3 V, 4 differential channel, 2:1
611 multiplexer/demultiplexer switch as shown in Figure 57 [13]. The selection of the switch is controlled
 February 14, 2015 – 12 : 37 DRAFT 41

612 by the FPGA. The input signals from the E-links that are routed to the FPGA are translated into the LVDS
613 standard through the Texas Instrument 2x2 LVDS CROSSPOINT SWITCH (Part No SN65LVDT122)
614 [10]. This translator is designed for signaling rates up to 1.5 Gbps and has a total jitter smaller than 65 ps.

Figure 57: Use of multiplexers-demultplexers to choose functionality

615

616 On the DAQ side the FPGA is connected to a SFP+ connector and to a RJ45 Ethernet connector. Also,
617 the FPGA is connected with an extra pair of GTP transceivers to the custom VTRx for testing purposes
618 or in case of failure of the GBTx. In Figure 58 there is a schematic representation of this multi-functional
619 connectivity for the read out data. On the other hand the GBTx is connected to both the VTRx and SFP+.
620 With this schema there is an alternative route for GBTx in case of VTRx failure or for testing purposes,
621 as shown in Figure 59b. For that purpose the NXP (Part No CBTL02043A) 2 bidirectional differential
622 channel, 2 : 1 multiplexer/demultiplexer was used. This multiplexer/demultiplexer is a High-speed signal
623 switching for PCIe Gen3 8 Gbit/s and has a high bandwidth of 10 GHz at −3 dB [14]. It has also a low
624 intra-pair skew of 5 ps typical and a low inter-pair skew of 35 ps maximum [14].

Figure 58: DAQ side of the L1DDC board

625 Finally, FPGA implements a master I2 C channel for the connection to the GBTX ASIC. This I2 C is
626 responsible for configuring the GBTX registers. These registers can be programmed also by the IC
 February 14, 2015 – 12 : 37 DRAFT 42

(a) FPGA communication through SFP+ or VTRx (b) GBTx communication through SFP+

Figure 59: DAQ alternative routes for FPGA and GBTx

627 channel when the GBTX is working in the transceiver mode. The schematics files of L1DDC board are
628 at the L1DDC Schematics link and the layout files to L1DDC Layout link.

629 5.6 Firmware

630 The largest fraction of the code was taken from CERN GBT group and the CACTUS project. The
631 firmware for the Artix7 FPGA was compiled with the Xilinx ISE 14.2 design software with the help of
632 the PlanAhead and Core Generator tools. For the simulation of the code the ISIM software was also
633 used. To test the integrity of the code Xilinx ML605 and Xilinx SP605 evaluation boards and CERN
634 GLIB board were used. Figure 60 shows the distinct portions of code for the Artix7 firmware that will
635 be used. Basically 5 main blocks can be determined. There are 3 blocks for the communication to the
636 external devices, the on board GBTX configuration, the system and user logic and finally the external
637 component control.

638 5.6.1 Communication Protocols

639 For the communication of the L1DDC through the internet the IPBus UDP code was used. The code had
640 to be migrated from Virtex6 family to the Artix7 family. There is also an attempt to use a custom UDP
641 protocol independent of the IPBus. For the communication with the Mini2 front-end boards the SRS
642 communication protocol will be used.

643 5.6.2 System core - User logic

644 The GBTx functionality that has to be emulated in the FPGA, is shown in Figure 61. The data received
645 from the elinks along with the SC field are passed through the scrambler which is responsible to maintain
646 the DC-balance. After that the scrambled data and the header are Reed-Solomon encoded before serial-
647 ization. The forward error correction algorithm used is a 2 times interleaved Reed-Salomon RS(15,11)
648 code with 4-bit symbols. The 4-bit frame header is chosen to be DC balanced. After the Reed-Salomon,
649 interleaving takes place and finally the data are serialized and transmitted through the Ethernet (RJ45) or
650 SFP+ (optical fibre).
651 The user logic includes a custom packet generator in order to test the L1DDC board Ethernet commu-
652 nication standalone. In this case no connection to the front-end boards is required and the data can be
 February 14, 2015 – 12 : 37 DRAFT 43

Figure 60: FPGA firmware block diagram

Figure 61: FPGA core functionality

653 generated by the FPGA. Sending the data to a PC the integrity of the transmitting signals can be verified.

654 5.7 Adapter boards

655 The mini2 front-end has only microHDMI connectors and for that reason an adapter board must be
656 used to connect it with the L1DDC prototype board. Also the mini2 boards need to be powered up
657 through these microHDMI connectors. On the other hand the MMFE8 boards use a custom STTL135
658 standard powered of 1.2 V and not of 1.35 V which is the default power scheme. The GBTx supports
659 the SLVS and LVDS standards for the receiving signals. For all the above reasons, an adapter board
 February 14, 2015 – 12 : 37 DRAFT 44

660 with the name BBALA (Bucharest Brookhaven Arizona L1DDC Adapter) was fabricated as is shown
661 in Figure 62. The BBALA has microHDMI connectors and miniSAS from the one side and the signals
662 from both connectors are routed in the same miniSAS connector on the other side. Furthermore there
663 are LVDS translators that convert the signals to the LVDS standard. This coversion is only one way and
664 from the miniSAS only side to the miniSAS and microHDMI side. Finally the BBALA has a power
665 distribution circuit in order to provide the power supply to mini2 through the microHDMI connectors.
666 The power is selectable between 3.3 V and 4 V in case there is a drop down in the voltage due to long
667 cables. Summarizing, the BBALA adapter board:

668 • Can power on the mini2 front-ends as it provides a selectable power of 3.3 V and 4 V through the
669 micro HDMI connectors.

670 • Can translate the signals going to micro HDMI - mini SAS side into LVDS signals.

671 • Can be used as an intermediary in order to connect a board with only microHDMI connectors to
672 the L1DDC which has only miniSAS connectors.

673 • Gives the opportunity for external configuration of the mini2 FEs through a standard I2 C protocol.

(a) BBALA layout

(b) BBALA top

(c) BBALA bottom

Figure 62: BBALA adapter board

 February 14, 2015 – 12 : 37 DRAFT 45

674 For the power distribution circuit the LT8612 DC-DC converter was also used. There are three DC-DC
675 converters, two of them are with selectable output of 3.3 V and 4 V for powering the mini2 front-ends as
676 mentioned before. A third DC-DC converter with 3.3 V output was used to power on the Texas Instrument
677 (SN65LVDT122) LVDS translators. The power connector is the same that was used for L1DDC Molex
678 Inc (Part No 0039303041). The BBALA adapter board was designed with 8 layers as shown in Figure 63
679 with differential pairs at 4 mils gap/4 mils width and the material used was standard FR4.

Figure 63: BBALA stack layer properties

680 The Mini2 front-end board supports only LVDS standard for the input and output signals. The GBTx
681 housed on the L1DDC uses the SLVS standard for the transmitting signals. For that reason the commu-
682 nication with the mini2 board can never be accomplished. With the use of BBALA adapter the output
683 LVDS signals of the GBTx are translated into the LVDS and read by the mini2 as shown in Figure 64.
684 Figure 65a shows the top side and Figure 65b shows the bottom side of the fabricated BBALA board.
685 For more informations about the schematics of BBALA you can refer to BBALA Schematics link and
686 about the BBALA layout to BBALA Layout link.
687 The ADDC board will communicate with the front-ends using the same miniSAS connectors and cables.
688 For the communication to the mini2 front-ends ADDC needs also an adapter board as mini2 has only
689 micro HDMI connectors and ADDC only miniSAS connectors. Moreover, the ADDC is designed to
690 communicate with 8 VMM2 ASICS. Mini2 accommodates only 2 VMM2s and for that reason 4 mini2
691 boards must be connected to the same ADDC miniSAS connector. Finally ADDC will provide a clock
692 to the 8VMMs but for the mini2 board the same clock from ADDC must be distributed to the 4 mini2s.
693 For all the above reasons the BBAA (Brookhaven Bucharest ADDC Adapter) board was fabricated. The
694 dimensions of the BBAA Adapter board are 50 mm width and 120 mm length. It consists of 6 layers as
695 shown in Figure 66.
696 The differential pairs, contrary to other boards, were designed with 5 mils width and 5 mils gap. The
697 differential impedance for the outer and inner layers was also calculated to achieve the 100 Ohm, as is
698 shown in figure 67a and figure 67b respectively.
699 Figure 68a is the final layout, Figure 68b is the 3D representation of the top side and Figure 68c is
700 the bottom side of of the BBAA adapter board. For the fanout of the clock the Microsemi Consumer
 February 14, 2015 – 12 : 37 DRAFT 46

Figure 64: Mini2 connectivity to L1DDC with the use of BBALA adapter

(a) BBALA top side after fabrication

(b) BBALA bottom side after fabrication

Figure 65: BBALA board

701 Medical Product Group clock buffer 1:4 750MHZ (Part No ZL40214LDG1) was used. The BBAA has
702 also a LT8612 DC-DC converter with an output of 3.3 V in order to power the clock fanout. Finally, as
703 in the L1DDC and BBALA boards the same molex power connector was used. 6 BBAA adapter boards
 February 14, 2015 – 12 : 37 DRAFT 47

Figure 66: Stack layer of BBAA

(a) Inner layers (b) Outer layers

Figure 67: BBAA differential impedance calculation

704 were fabricated in October of 2014. Figure 68d shows the top side of the BBAA adapter board and
705 Figure 68e shows the bottom side of the fabricated BBAA adapter board. The schematics files of BBAA
706 board is at the BBAA Schematics link and the layout files to BBAA Layout link.

707 5.8 Tests and debugging (Tools and applications used)

708 For testing and debugging the firmware, Xilinx ChipScope Pro 14.2 & ISE simulation were used. For the
709 estimation of thermal dissipation a thermal camera FLIR E5 was also used. The thermal camera shows
710 in detail the components that dissipate more power, because of the high current, and need to be taken
 February 14, 2015 – 12 : 37 DRAFT 48

(a) BBAA layout

(b) BBAA Top (c) BBAA Bottom

(d) Top side after fabrication (e) Bottom Side

Figure 68: BBAA Adapter Board after fabrication

711 into consideration in the next prototypes (adding another component) or need cooling. In Figure 69a is
712 shown the thermal analysis of the top side and in Figure 69b the bottom side of the L1DDC.
713 In order to test the efficiency and stability of the L1DDC board, regarding the propagation and reception
714 of signals, a series of tests were performed, ranging from simple ones, to a bit more sophisticated in
715 nature. The main goal was to accurately test the ability of the L1DDC board to receive the LVDS signals.
716 Another goal was to test the ability of the BBALA adapter board to propagate all signals efficiently
717 through its micro HDMI and miniSAS connectors.
718 The tools used for these tests were the under testing L1DDC boards, together with the BBALA adapter
719 boards, a Xilinx Spartan SP605 Evaluation Board, a Tektronix DPO 4504 Digital Phosphor Oscilloscope,
720 several micro HDMI to micro HDMI connection cables and miniSAS cables of 0.5 m. Xilinx’s ISE 14.3
721 Project Navigator was used to develop VHDL code which programmed the FPGAs of the SP605 and
722 L1DDC boards.
723 The main testing procedure involved transmitting frames and clock signals from the Spartan SP605 test
724 board, to the L1DDC board via the LVDS standard. The signal was propagated from the SP605 board
725 via an FMC adapter card connected onto the board. The micro HDMI cable connected to the FMC card
726 was in turn connected to the BBALA adapter card. A miniSAS cable connected to the corresponding
727 connector then transferred the LVDS frames or clock signal towards the L1DDC board. The signal
 February 14, 2015 – 12 : 37 DRAFT 49

(a) L1DDC Top Side (b) L1DDC Bottom Side

Figure 69: L1DDC Thermal Analysis. In the top side the components that dissipate the larger amount
of thermal are the translators (shown with a white color in the top and lower right corner), and the one
LDO (lower left corner) which correspond to a approximately 50o C. The GBTX housed in the middle of
the board also dissipates a large amount of thermal and is more visible on the bottom side of the board.

728 was read out using a high end oscilloscope by Tektronix. The result was visible on the oscilloscope’s
729 screen, were the final conclusions could be made on the quality and efficiency of the signal propagation
730 throughout the entire propagating channel. In Figure 70 the testing setup is shown. A: Spartan SP605
731 Test Board together with the FMC adapter. B: BBALA adapter board. C: L1DDC Board. D: Tektronix
Oscilloscope.

Figure 70: Testing setup

732
733 In Figure 71 there is a closer look of the oscilloscope L1DDC pin connectivity. The frames which the
734 oscilloscope is reading out from the pin is clearly visible on the screen.
735 This testing procedure was carried out for all BBALA boards, and for all L1DDC boards. At first a simple
 February 14, 2015 – 12 : 37 DRAFT 50

Figure 71: Oscilloscope-L1DDC pin connectivity

736 system clock was sent out from the SP605 board and finally a frame was transmitted. This frame was
737 transmitted with 4 different frequencies, 40 MHz, 80 MHz, 160 MHz and 320 MHz which corresponds
738 to 25 ns, 12.5 ns, 6.25 ns and 3.125 ns respectively. The use of multiple transmission frequencies allowed
739 further survey of the mechanics of the signal propagation through the boards and cables, by also taking
740 into consideration the factor of transmission speed. In general, no major issues occurred, indicating the
741 versatility and efficiency of the L1DDC throughout a broad range of frequencies. The only concern was
742 the fact that for higher frequencies (320 MHz), the measured waveform from the oscilloscope seemed
743 (optically) relatively more distorted, introducing a read out error regarding the frequency and voltage
744 measurements of the signal. However, these concerns were deemed as minor ones, mainly because
745 the inclusion of the oscilloscope in the testing apparatus as is, introduces significant systematic error.
746 Nevertheless, the fact of the usage of the oscilloscope to test the efficiency of signal propagation called
747 for the development of more sophisticated testing techniques, which would leave out the use of the
748 oscilloscope. More information on this can be found at the rest of this section.
749 The main advantage of the above testing technique was the speed and ease of interchanging the connec-
750 tions in the BBALA board throughout the test, thus allowing fast and clear checking of all the BBALA
751 micro HDMI and miniSAS connectors. In addition, the FPGA of the L1DDC was programmed so that
752 every miniSAS connector of the board could receive data, which would then be driven towards a corre-
753 sponding pin. This allowed for fast interchanging at the miniSAS connectors of the L1DDC board as
754 well, so that every single connector could be checked for its integrity. Another basic test, was to transmit
755 the data from the L1DDC card, towards the BBALA adapter board, and from the adapter board back
756 to the L1DDC. In Figure 72 there is a setup that involves transmitting data from the miniSAS cable A,
757 towards the BBALA board, and finally back to the same L1DDC board, via the miniSAS cable labeled
758 as B. The result is read out from the pin by the Oscilloscope.
759 The final basic test, was to simply send data from one L1DDC miniSAS connector, via a miniSAS cable,
760 towards a second connector. In Figure 73 is shown a simple testing set-up. The data is being transmitted
761 from connector A, back to connector B. This test allows for simply checking the integrity of the L1DDC
762 connectors alone.
 February 14, 2015 – 12 : 37 DRAFT 51

Figure 72: Testing set-up with BBALA adapter

Figure 73: On board checking

763 Naturally, the above tests involved the use of an oscilloscope for reading out the signal waveform from
764 the L1DDC pins. The inclusion of the oscilloscope introduced a systematic error to the tests. Also,
765 the fact that the waveforms at higher frequencies seemed distorted at the oscilloscope, called for tests
766 of slightly different nature. To conclude, it was crucial to determine whether the L1DDC board could
767 clearly distinguish the binary digit 0 from 1, in a broad frequency spectrum. For instance, if a transmitted
768 frame from the SP605 board could be read correctly from the L1DDC board, for a variety of frequencies
769 from high to low, then the performance of the L1DDC would finally be deemed as adequate. A test of
 February 14, 2015 – 12 : 37 DRAFT 52

770 this nature, would also determine even more accurately the integrity of the interconnections between the
771 boards, as, e.g, if a specific connector introduces high voltage distortion and/or jitter, then the L1DDC
772 would fail to read the frames correctly. So the philosophy of this test would be to transmit a simple and
773 specific frame from the SP605 test board, through the BBALA, to the L1DDC, and determine whether
774 the L1DDC board recognizes the frame correctly.
775 This method allows for raw checking the transmission procedure, as if a bus and/or connector introduces
776 high distortion, then several bits would eventually change during the propagation and none of the frames
777 would be read out from the L1DDC board. This procedure is repeated for 100 checking periods, and for
778 every erroneously read frame, an error counter increases by 1. After 100 checks, the checking process
779 terminates, and the final number of error count is finally determined by the L1DDC. The error count
780 number is being read by the user by sending a long (period of about 1 second) clock like pulse back
781 to the SP605 board, which is driven to a LED. Zero errors would turn on the LED permanently. This
782 program also includes a reset button, thus allowing for quick checking every single connector of the
783 BBALA and L1DDC board. Figure 74 is shows the schematic of the checking procedure. Between the
784 SP605 and the L1DDC boards, the BBALA adapter board is interpolated, allowing the signals to be
785 propagated.

Figure 74: schematic of the checking procedure

786 This procedure was instantiated for 4 different transmission frequencies (40 MHz, 80 MHz, 160 MHz
787 and 320 MHz), so the L1DDC board’s ability to read out frames correctly in all frequencies was tested
788 thoroughly. The results were more than satisfactory, as absolutely no errors occurred when the signals
789 were propagated through all channels and connectors, for all four frequencies. This final test concluded
790 all the checking procedures, thus proving the flawless ability of the BBALA adapter board in propagating
791 the signals, and the perfection of the L1DDC board in reading out raw data.

792 5.9 Issues - errors to be corrected

793 During the debugging process of the L1DDC prototype1 board, a number of issues were found. These
794 issues along with some improvements need to be taken into account in the next prototypes. Issues found
795 until now:

796 • There was an error with a MOSFET decal. This MOSFET powers on the led indicators of the
797 LDOs. Moreover, the same decal was used for the selection of 3.3 V and 1.5 V for the efuses of
798 the GBTx through a relay.

799 • There is a need to feed the MGTAVVT supply voltage from an extra LDO and not directly from
800 the DC-DC converter in order to avoid extra noise that add the DC-DC converters.
 February 14, 2015 – 12 : 37 DRAFT 53

801 • The stack layer will change in order to isolate all internal layers with differential pairs with power
802 and ground planes.

803 • The signals SFP TX FAULT and SFP TX DISABLE were not routed to the FPGA and need to
804 be corrected in the next prototypes. Signal SFP TX DISABLE can be hard-wired (set to ground)
805 through an external jumper.

806 • Also the signals SFP RS0 and SFP RS1 must have an alternative power supply of 2.5 V in case of
807 connection to the VTRx transceiver.

808 • An extra hard-wired clock will be added for the reference clock of the GBTx.There will be a choice
809 (through jumper or 0Ohm resistors) for the hardwired or an external clock.

810 5.10 Tests with VMM equipped frontend prototypes

811 The functionality and verification tests of the L1DDC first prototype (Prototype 1) includes the communi-
812 cation with the actual frontend boards. VMM2 equipped boards have been developed and manufactured,
813 eg the “Mini-2” and the “MMFE-8”.
814 The Mini-2 frontend board features two VMM2 ASICs along with a Xilinx Spartan 6 FPGA which
815 emulates the readout companion ASIC. The interface to the detector is done with a Panasonic 130-pin
816 connector compatible to most of the sTGC detectors (though adapter boards) and small or medium size
817 micromegas detector prototypes. The interface to the L1DDC is done through a uHDMI connector
818 driving signals through an adapter board (BBALA) to the mSAS interfaces of the L1DDC. This board
819 was developed to provide a way to test small detector prototypes and to qualify the VMM2 ASIC.
820 The MMFE-8 board features up to eight VMM2 ASICs along with a Artix-7 FPGA which emulates
821 the readout companion ASIC. The interface to the detector is done with a Zebra 512-pin connector
822 compatible to large sTGC detectors1 and micromegas detector prototypes. The interface to the L1DDC
823 is done through a mSAS connector driving signals through an adapter board (BBALA) to the mSAS
824 interfaces of the L1DDC2 . This board is the first prototype of the final frontend board used by the two
825 detector technologies.
826 Testing the frontend boards can be done with the connectivity as shown in the Figure 75. The readout
827 can be test wither by the VMM2 internal pulser or attaching the fronend boards to detectors sitting in
828 a cosmic stand facility. Multiple L1DDC boards can be synchronized through an external board called
829 CTF (Clock Trigger Fanout board). In order to realize the tests with the actual detector or the internal
830 VMM2 pulser, the L1DDC Artix-7 firmware will be adjusted in the needs of the setup. The DAQ/DCS
831 software (mmddf-micromegas data driver framework) is capable of controlling and acquiring the data of
832 many L1DDC boards.

833 6 Prototype 2 & 3 and roadmap to final board

834 The planning for the prototype2 L1DDC is to remove the FPGA functionality from the board. This will
835 lead as to a more simplified board which is closer to the final product. In detail, removing the FPGA will
836 lead us in:

837 • A fully radiation hard board which can be tested in radiation and magnetic fields.

838 • Estimating the precise overall power consumption of the board and its heat dissipation.
1
The final sTGC connector will be a different one
2
Although the connectors are the same the adapter board is used to make the level of the signals between the L1DDC and
the MMFE-8v1 compatible. No adapter board will be used in the final NSW
 February 14, 2015 – 12 : 37 DRAFT 54

Figure 75: Schematic of the VMM frontends readout with the L1DDC board. The two different type
frontend boards can be connected through an adapter board (BBALA) to the L1DDC which by its turn
can configure and readout the VMM2 ASICs. The communication to the DAQ/DCS PC is established
through UDP protocol to the already developed software (mmddf).

839 • Smaller routing signals and probably less layers in the layout.

840 • Estimating the actual jitter/delay in the e-links as there will be no intermediate components.

841 • Giving a smaller failure probability as the design will be much easier.

842 • Estimating the final dimensions of the board and the component placement.

843 • Understanding better the on-detector placement.

844 • Optimizing the final cost.

845 7 Time schedule and manpower

846 The time plan of the L1DDC prototypes as well as the mass production is oulined in Figure 10. As shown
847 there, the year 2015 will be devoted to development of prototype 2 & 3, where prototype 3 will serve
848 as the final design of the L1DDC card. Prototype 1 has been already delivered and tested at some level
849 where few design issues have been identified and have been currently implemented to the layout of the
850 prototype 2. The mass production & testing of the L1DDC cards will be done during the year 2016.
851 The available manpower for the R&D and mass production/testing for the years 2015 & 2016 is at the
852 level of 5 FTE. This workload is distributed as follows:

853 • 1 Electronics Engineer and 1 Electrical Engineer

854 • 1 Senior Physicist

855 • 2 Graduate students

856 • 4 Undergraduate students

 February 14, 2015 – 12 : 37 DRAFT 55

Table 10: Time Schedule for the R&D and mass production of the L1DDC boards

857 Acknowledgments
858 The present work was co-funded by the European Union (European Social Fund ESF) and Greek national
859 funds through the Operational Program ”Education and Lifelong Learning” of the National Strategic
860 Reference Framework (NSRF) 2007-2013, ARISTEIA-1893-ATLAS MICROMEGAS.
 56

861 Appendices
862 A Adaptation of firmware to Artix7 FPGA
863 There are a lot of firmware examples implemented by the Cactus project team of CERN including SFP+,
864 I2 C, UDP, IP protocols, 8B/10B and Reed Salomon encodings. The targeted evaluation boards are the
865 Xilinx kc705, ml605, sp605 and Avnet v5fxt. These boards have a Virtex5, Kintex7, Virtex6 and Spar-
866 tan6 FPGA families. Also there is a GBT emulation firmware for CERN GLIB board from the GBT
867 project group. The GLIB board uses also the Virtex6 FPGA family. None of these families are compati-
868 ble with the Artix7 FPGA family and for that reason the firmware has to be modified.
869 For that reason the IPbus firmware for GLIB was used. For the Ethernet communication GLIB uses
870 the SGMII (Serial Gigabit Media Independent Interface) without clock. For Ethernet implementation,
871 GLIB uses the Xilinx IP (Intellectual Property) Etrhernet MAC wrapper (v6 emac v2 3). This core
872 implements the data wrapping and uses the SGMII interface in order to connect to the GTXE1 transceiver
873 to transform the parallel data into a serial stream. Both IP cores are unsuitable for the Artix7 family and
874 must be modified.
875 For the migration to Artix7 the V6 emac core was replaced by the TEMAC core. The Ethernet Medium
876 Access Controller (MAC) is responsible for the Ethernet framing protocols and error detection of these
877 frames. The MAC is independent of, and can be connected to, any type of physical layer [18]. The
878 function block of the TEMAC is shown in Figure 76.

Figure 76: TEMAC functional block

879 This core supports only the GMII (Giagabit Media Interface) and must be modified to SGMII before
880 it is connected to the Gigabit Transceiver. The IP core responsible for this operation is the The Eth-
881 ernet 1000BASE-X PCS/PMA or SGMII core. This core provides the functionality to implement the
882 1000BASE-X PCS and PMA sublayers or used to provide a GMII to SGMII/SGMII to GMII bridge
883 when used with a device-specific transceiver Figure 77 [17].
884 Finally, 7 family does not support the GTXE1 transceiver of Virtex6 and for this reason was replaced by
 57

Figure 77: Functional Block Diagram Using Device-Specific Transceiver

885 the GTP transceiver. Figure 78 shows the Transceiver Wizard Wrapper [19].

Figure 78: Transceiver Wizard Wrapper

886

887 References
888 [1] ATLAS Collaboration, New Small Wheel Technical Design Report, CDS ATLAS-TDR-020.

889 [2] Linear Technology, LT8612 Datasheet, LT8612.

890 [3] Analog Devices, ADP1754/1755 Datasheet, ADP1755.

891 [4] Xilinx, DC and AC Switching Characteristics, ds181 Artix 7 DataSheet.

 58

892 [5] Xilinx, 7 Series FPGAs SelectIO Resources, ug471 7Series SelectIO.

893 [6] GBT Project, GBTx Manual, gbtxManual.

894 [7] GBT Project, GBLD Specifications, GBLD specification Apr2012.

895 [8] GBT Project, GBTIA Specifications, gbtiaSpecsP01.

896 [9] 3M, 3M Mini SAS 8F36 datasheet, mini SAS cable datasheet.

897 [10] Texas Instruments, LVDS Translators, SN65LVDT122 Datasheet.

898 [11] Xilinx, 7 Series FPGAs PCB Design and Pin Planning Guide, ug483.

899 [12] Marvell, 7 Series FPGAs PCB Design and Pin Planning Guide, 88e1111.

900 [13] NXP, 3.3 V, 4 differential channel, 2 : 1 multiplexer/demultiplexer switch for PCI Express Gen3,
901 CBTL04083A-CBTL04083B.

902 [14] NXP, 3.3 V, 2 differential channel, 2 : 1 multiplexer/demultiplexer switch for PCI Express Gen3,
903 CBTL02043A-CBTL02043B.

904 [15] MICRON, Serial NOR Flash Memory, N25Q256A13EF840E.

905 [16] IDT, Integrated Device Technology Inc, CRYSTAL-TO-LVDS CLOCK GENERATOR, 844021BG-
906 01LF.

907 [17] Xilinx, LogiCORE IP Ethernet 1000BASE-X PCS/PMA or SGMII v11.1, ug155.

908 [18] Xilinx, LogiCORE IP Tri-Mode Ethernet MAC v5.4, pg051.

909 [19] Xilinx, LogiCORE IP 7 Series FPGAs Transceivers Wizard v2.4, ug769.

910 [20] Xilinx, 7 Series FPGAs Configuration, ug470.

911 [21] SATURN PCB Design INC., Saturn PCB Toolkit, PCB Toolkit.

View publication stats