Testing Data Center Infrastructure

BiS Team November 7, 2020

5 minutes read

Madhukar-Tripathi — Madhukar Tripathi, Head –MARCOM & OPTICAL, PRODUCTS, ANRITSU INDIA PVT. LTD

The spread of cloud services, Social Networking Sites (SNS), and video streaming is increasing the number of businesses using metropolitan data centers (Metro DC) as interconnects. As Metro DC and data center interconnects (DCI) continue to grow, network installation and maintenance (I&M) is becoming increasingly complex due to the diversifying communications standards, while faster speeds and increased security administration demand more efficient work practices.

Issues in DC Installation and Maintenance

Previously, to leverage the merits of reduced power costs and disaster prevention countermeasures, data centers (DC) have been co-location spaces in large-scale suburban buildings. However, more recently, smaller-scale DC in city buildings (Metro DC) are more commonly using mesh network connections configured to operate as one virtual DC. This type of Metro DC costs relatively less to install even in a city building and does not require a large installation space.

Electronic Manufacturing Towards Hyper Automation

Also Read

Requires evaluation of various communications standards and interface circuits

Due to different applications and cost advantages, Metro DC installations are adopting different standards such as fibre channel (FC), 100GbE, 400GbE, etc., for WDM. At Metro DC I&M, various interfaces and communications standards must be tested; providing a separate tester for each standard not only increases work costs but also reduces work efficiency. An all-in-one network tester supporting each communication standard has been urgently needed.

Requires efficient I&M under severe security administration

Data center sites are subject to severe security administration requiring I&M work to be performed during restricted time periods. Performing tests of the various communications standards for Metro DCI requires trained and experienced operators with good knowledge about communications and network I&M.

Requires optical fiber maintenance

Metro DCI network configurations use fast high-capacity optical fibers. In addition to conventional optical loss measurements, fast DCI optical fibers must be evaluated for optical reflections. The IEEE802.3 100GBASE-LR4 and 10GBASE-LR standards specify an optical fiber connector reflection loss of not more than 26 dB. Generally, SPC and UPC fiber connectors have a reflection loss of about 40 dB and 50 dB, respectively, but if the connector end face is contaminated with oil, dust, etc., the optical loss increases due to the increased optical reflections caused by the contamination, resulting in degraded communications quality. Appropriate maintenance at optical fiber I&M is key to preventing communications problems.

ERP — Contaminated Optical Fiber End Face

Data Center ToR/Leaf Level Construction/Maintenance

Top of Rack (TOR) architecture is where the cabling between switch and server stays within a rack. This has the benefit of reducing the overall amount of cabling with the downside of reduced efficiency in the usage of Ethernet switch ports which are limited to within a rack.

Following measurements at Data Center (DC) Interconnect I&M are conducted.

BER: BER is one of the basic measurements and includes include Frame Loss and Sequence Error tests.
Latency: Network latency is the time it takes for data or a request to go from the source to the destination. Latency in networks is measured in milliseconds. The closer your latency is to zero, the better.
Throughput: Throughput measures how many packets arrive at their destinations successfully & is measured in bits per second. Packet loss, latency, and jitter are all related to slow throughput speed.

BER Test: BER test consist of many tests such as Generation and detection of test patterns, Count of errors in received test pattern, Pattern generation: Unframed (Layer 1), Framed Ethernet (MAC) header (Layer 2), Framed Ethernet (MAC) header with IP header (Layer 3) or Framed Ethernet (MAC) header, Framed with IP header and TCP/UDP header (Layer 4), User-defined header pattern (14 byte to 256 byte), Detection of sequence errors, loss of sequence synchronization, Frame loss count and frame loss seconds.

Throughput measurement results are calculated for:

Utilization layer, Physical layer, Physical layer excluding preamble, Link layer, Network layer and Data layer
Min./Max./Avg. values

Performance (M.2100 type) parameters: ES, SES, ALS, UAT, AVT, EFS Test patterns: PRBS 9, PRBS 11, PRBS 15, PRBS 20, PRBS 23, PRBS 29, PRBS 31, HF test pattern, CRPAT, JTPAT, SPAT, 55 Hex, Fox, 32-bit user programmable User-defined resolution: 1, 2, 5, 10, 15, 30 s, 1, 5, 10, 15, 30 min, 1, 2, 4, 6, 12 h Event log: Major measurement events incl. errors and alarms are logged with 1-second resolution

FEC Analysis:

Moderns Data Center (DC) are equipped to handle high data rate to meet consumer high bandwidth demand. FEC Analysis is one of the important measurements in DC now a days.

400G Ethernet FEC Analysis Ethernet equipment interfaces use optical modules. With some exceptions, pass/fail verification of optical modules up to 100G is performed simply by measuring the bit error rate (BER) for a fixed period of repeated sending and receiving of signals to confirm the error-free status. Similarly, network throughput and latency are measured using a BERT. However, the 400G Ethernet PMD layer is switching to PAM4 and FEC (Forward Error Correction) to correct errors occurring at optical transceivers and networks transmitting extremely fast signals, and to assure both communications quality and lower costs. FEC is a technology for correcting errors within the correctable range, and assures high reliability as well as extended transmission distance.

Service Activation Test (Y.1564).

This consist of service configuration and service performance test. These tests include

Color-aware and non-color-aware in combinations (IP DSCP or VLAN PCP)
Test modes: One-way (uni- or bi-directional, symmetrical or asymmetrical), Round-trip
Verification against service acceptance criteria: Information rate, Frame transfer delay, Frame delay variation, Frame loss rate, Availability Optional GPS timing synchronization All services tested simultaneously at CIR
Duration 15 min, 2 h, 24 h, user programmable
Results: Pass/Fail indication, IR (Min./Avg./Max.), FL (Count/ FLR), FTD, FDV (Min./Avg./Max./Current (during measurement)), AVAIL (%), Unavail (s).

Service Disruption Measurement.

Service disruption measurement activated as part of BER test includes

Max./Avg. service disruption time, Resolution: 0.1 μs
Number of service disruptions
Disruption Type: Packet, LOS

Fibre Channel Test:

Installing the Fibre Channel option supports Fibre Channel tests from 1GFC to 16GFC.

Latency measurement
BER tests including service disruption measurement
Performance Test
Buffer Credit environment
Confirming optimum network parameter values

Anritsu Metro DC I&M Solutions

Anritsu’s range of testers for resolving the above-described data center I&M issues includes the Network Master Pro MT1040A/MT1000A, Access Master MT9085series, and Fiber Maintenance Master MT9090A listed below.

Key consideration in selection of testing tool

To assure and improve network quality, operators perform periodic equipment upgrades and maintenance, so they prefer measuring instruments with a variety of interfaces to help assure efficient work in the shortest time.

Here are some key points to consider when selecting any Data Centre Installation & Maintenance tool.

1. Data rates & interfaces: Any testing tool must have support of various data rates & interfaces used in todays industry.

Below table shows some interface and supported technology.

2. Easy to use GUI- Graphical Unser Interface (GUI). Tester must have very user-friendly GUI. This helps to reduce learning curve and expedite testing. This ensure Data Centre is performing well and SLA with customers are met.

3. Automation in measurement process: Many times, field engineer/ technician is not well trained for proper usage/ testing of various test required during installation or maintenance of Data Centre. In such scenario trained engineer can easily configure test instrument to perform required test. Testing procedure is saved in instrument and just one button click is required to test. This helps to achieve better network efficiency & performance.

Traditionally, the operator creates a job ticket explaining the required tests to be completed with simple and clear stepby-step requirements for the technician running the tests. Commonly the job ticket is provided either as a paper hardcopy or PDF, this work procedure can now be registered in the MT1000A/MT1040A using the free Scenario Edit Environment Kit MX100003A (SEEK) software for PC, with an intuitive GUI helping lighten the operators workload and ensuring all required testing is completed by the engineer. Automation supports creation of test scenarios without expertise in program language minimizing complexity and user error.

4. Data storage/ sharing: Testing instrument must have enough storage of measurement data- trace and interfaces to transfer/ share this data for further analysis and reporting. v) All -in- One feature: Test instrument must be able to perform field and lab test, and multi-function, multi feature type so that CAPEX is limited i.e. cost saving. Anritsu Network Master Pro MT1040A supports lower to higher ( 10M to 400G) data rate, Ethernet, OTN test and OTDR test as well in single unit. Fiber Connector inspection is also possible using Video Probe.