Efficient sharing of computing and storage resources is critical for data centers and high-performance computing clusters. For this purpose, specialized servers are linked to each other via switches. The first level of connection takes place in the server rack, where servers are interconnected via a top of the rack (TOR) switch using passive and/or active cables.
Layer 1 and Layer 2 switches (also called leaf and spine switches) are used to share resources between racks. As they are usually more than 5 meters away from each other, more expensive and power-consuming optical transceiver modules are used for the connections.
active cables
When passive direct-attached copper (DAC) cables are used, the “host” chip of the networking interfaces sitting at each end of the DAC must accommodate for the loss and distortion of the passive copper channel.
This sets very stringent requirements on the host chip design and drastically limits the cable reach when operating at 50 Gpbs or 100 Gpbs signaling rates. For this reason, active cables are becoming increasingly popular.
types of active cables
AECs and AOCs include a retimer chip that considerably relaxes requirements on the host chip
Due to the varying reach objectives, there are several different types of active cables: active copper cables (ACC), active electrical cables (AEC), and active optical cables (AOC).
A simple linear amplifier and an analog equalizer are typically used in ACCs, whereas AECs and AOCs include a retimer chip that considerably relaxes requirements on the host chip compared to DAC-cable-based on-clock recovery (CR).
chip-to-module (C2M) interface
The so-called chip-to-module (C2M) interface is identical to one of the optical transceiver modules. Contrary to classical pluggable optical transceivers, there is no need to test the optical client “side”, the link between the two ends of the cable is a cable vendor implementation, only the interface to the host (chip-to-module interface).
Both the IBTA (InfiniBand Trade Association) and IEEE (Institute of Electrical and Electronics Engineers) Standards Association provides compliance test procedures to ensure interoperability between active cables and the networking interface of server and switches.
FEC-aware active cable testing
With the advent of PAM4 modulation, a combination of digital equalizers and forward error correction (FEC) must be used to ensure error-free transmission. As a result, the complexity of measurements and test approaches have significantly increased compared to NRZ-based systems.
Verifying the performance of transmitters requires emulating a reference adaptive equalizer that accommodates the combined effects of the transmitter limitations and the transmission channel. Metrics such as vertical eye closure (VEC) and eye height (EH) are measured on a reconstructed eye.
performance metrics
When testing the performance of receivers, measuring the raw bit error ratio (BER) is not sufficient anymore at this data speed. Additional performance metrics are needed to qualify system performance:
- Frame loss ratio (FLR): it counts how many Ethernet frames were discarded because errors couldn’t be corrected. It is related to the BER after FEC.
- FEC margin: This describes how far the system operates from the point where the FEC can’t correct bit errors within a frame.
Transmitter test
The transmitter test consists in characterizing the output of the module through the mated connector with a module compliance board (MCB). An oscilloscope is the main instrument for transmitter testing. It captures the signal and processes it by emulating the reference receiver described below.
This reference receiver includes:
- A fourth-order Butterworth filter (40 GHz BW) as a specified receiver noise filter
- Receiver input-referred noise (emulating the intrinsic noise of a realistic receiver)
- A two-gain-stage continuous-time filter (CTLE)
- A four-tap decision-feedback equalizer (DFE)
EH and VEC values
Eye measurement will then use the optimal equalizer setting, to run the near-end EH and near-end VEC
EH and VEC are measured after the DFE. The values of EH and VEC are the values obtained after the reference receiver equalization. For this, the CTLE DC must be optimized together with the DFE taps. The values of eye height need to comply with the specification for EH (min 15 mV) where it also has a minimum value of VEC.
This measurement process can be done using Keysight’s electrical 802.3ck test app (D90103CKC or N1091CKCA). Eye measurement will then use the optimal equalizer setting, to run the near-end EH and near-end VEC.
Receiver test
The receiver test consists in testing the module interface with the worst allowable signal coming out of a host.
The receiver test setup consists of Keysight M8040A 64 Gbaud high-performance bit error ratio tester (BERT), a frequency-dependent attenuator or inter-symbol-interference (ISI) board, and a module compliance board (MCB). The pattern generator de-emphasis must be first optimized for the test channel and reference receiver.
stressed input test
The actual amplitude and random jitter (RJ) values to achieve the stressed input test calibration targets
After this, its amplitude and jitter profile is tuned to meet the specified VEC and EH before performing the stressed input test.
The Stressed eye calibration results show the target EH and VEC to be achieved and the measured values after the CTLE optimization. Besides, the actual amplitude and random jitter (RJ) values to achieve the stressed input test calibration targets.
Keysight Active Cable Test Solution
Below is a complete setup for active time-domain (ATD) testing of active cables. It consists of an M8040A bit error ratio tester (BERT), M8195A arbitrary waveform generator (AWG), ISI board, DCA sampling scope (N1000A DCA-X mainframe & N1060A precision waveform analyzer), test software (M8070B, N1010A, M8091CKPA & N1091CKCA) & accessories (cables, adapters, etc.).
The ATD method of testing contemporary high-performance data center interconnects as illustrated in this post, is undergoing review by the IBTA as an effective and more relevant form of a comprehensive test than those methods that preceded this FLR-based ATD technique.
Key benefits
A summary of key benefits of this configuration are as follows:
- Evaluation of FLR through up to 8 lanes of striped FEC-encoded signal generation provides highly spec-relevant performance metrics on the performance of the interconnect which are impossible to gain with single-lane test configurations.
- For the first time, an effective means of visualizing error per frame distribution (FEC tail) is provided in this test configuration enabling accurate estimation of the system FEC margin.
- Automated closed-loop signal impairment calibration SW coordinates the time-consuming process of achieving precision scope-based feedback to BERT settings to ensure electrical limits are reached reliably and consistently from run to run or site to site.