AI Data Center Optics: Validation & Testing Handbook 2026

What test equipment vendors don't write. What transceiver vendors don't write. What you need — from incoming sample to RMA evidence package.

For Tier-2 AI data center operators, neoclouds, and system integrators.

Reading the module#

Every test you run on a transceiver, every validation check you perform on a deployed link, and every triage decision you make in production rests on one foundation: the data the module reports about itself. That data lives in the module's EEPROM and, on modern modules, in the CMIS data model layered on top. Knowing how to read it — and what to ignore — is the single most leveraged skill in this handbook. It also separates engineers who can diagnose a failing link from engineers who can only swap modules and hope.

2.1 · The CMIS data model#

CMIS organizes module data into a memory map with a familiar two-byte addressing scheme: a Page selector, then a byte offset within the page. The Page selector distinguishes the kind of data you're reading; the offset addresses a specific field. Pages are organized into Banks for modules whose lane count exceeds what fits in a single page.

The pages an operator interacts with most often, and what each contains:

2.2 · The Datapath State Machine#

A CMIS module is not a stateless device. It moves through a defined state machine from the moment power is applied to the moment it carries traffic at full rate. Understanding the state machine matters because the most common module failure mode in 2026 is a module that is stuck in an intermediate state — link doesn't come up, no error message that obviously identifies why, and the engineer ends up replacing modules and cables in sequence trying to find the problem.

The CMIS Datapath State Machine has five operational states an operator should recognize:

The five operational states. A module that won't advance from DPInit to DPActivated is the most common module-side failure pattern in 2026 — and almost always reveals firmware mismatch, ApSel mismatch, or host SerDes misconfiguration before it ever reveals a defective module.

 

A module that fails to advance from DPInit to DPActivated almost always indicates one of three causes: (1) a CMIS firmware mismatch where the module's firmware advertises a feature or behavior the switch's CMIS parser doesn't recognize, (2) an Application Selection (ApSel) request from the host that the module's advertising doesn't actually support — host trying to configure an 8×100G mode on a module that only advertises 4×200G, for instance, or (3) host-side SerDes configuration that doesn't match what the module expects on its electrical input.

None of these will tell you, in a clean error message, what is wrong. The engineer who knows the state machine reads Page 11h, sees the DPInit state, and immediately knows where to look. The engineer who doesn't replaces three modules before figuring it out.

2.3 · VDM — the diagnostic gold mine#

If you read only one CMIS page during ongoing operations, read the VDM pages. Versatile Diagnostic Monitoring is where modern modules expose the data that lets you diagnose problems before they become outages.

The fields VDM exposes that you cannot get anywhere else:

• Per-lane Pre-FEC BER statistics — not just the current value, but the running minimum, maximum, and average over a configurable window. The trend in these fields is what catches a degrading link weeks before it crosses a threshold.
• Per-lane SNR — for PAM4 modules, the signal-to-noise ratio per lane is a leading indicator of channel margin. SNR drops typically precede BER rises by hours to days.
• Historical bins of pre-FEC BER — the module accumulates how often the BER has fallen into each of several bins over time, giving you a distribution rather than a snapshot.
• Host-side and media-side per-lane telemetry separated — letting you isolate problems to one side of the channel.
• Lane equalizer settings — what the DSP is doing to compensate for channel impairments. A lane whose equalizer is at extreme settings is a lane that's working hard.

The combination of VDM and the standard DOM telemetry is the operator's diagnostic toolkit. Without VDM you have aggregate counters and a single-point BER sample. With VDM you have per-lane trends and statistical distributions. The difference between a fabric that catches problems and a fabric that surprises you is largely the difference between operators who use VDM and operators who don't know it's there.

2.4 · Firmware as a validation event#

One last note before we close out the foundation. Module firmware is updateable in the field on every modern CMIS module, via the CDB pages. This is mostly a feature — vendors fix bugs, improve performance, and address newly-identified compatibility issues by shipping firmware updates. But it has implications for operations.

A module's behavior under your validation tests is specific to its firmware version at the time of testing. A module that passes sample testing on firmware version 1.2.4 may behave differently on firmware 1.3.0. The implication: every firmware update is, in principle, a re-validation event. Most of the time the new firmware behaves equivalently or better; occasionally it doesn't.

Why operators run their own testing#

Sample testing is the program that closes the operator's gap before procurement commits. This chapter is about the why — the four real questions a sample-testing program has to answer, the decision between an in-house program and a contract lab, and the moment when running your own testing pays for itself versus the moment it doesn't. The how comes in Chapters 4 through 8.

3.1 · When sample testing pays for itself#

Running a serious sample-testing program costs money: equipment, engineer time, lab space, the modules themselves. The investment makes sense in some contexts and not others.

Sample testing pays for itself when:

• You're evaluating a new vendor. The cost of a bad vendor decision is measured in production failures and emergency replacement spend; the cost of testing 30 modules to learn whether to make the bet is small by comparison.
• You're evaluating a new SKU from a known vendor. A vendor whose 100G product was excellent does not automatically have a great 800G product. Sample-test every new SKU.
• You're deploying at scale. The break-even calculation runs in your favor as deployment size grows. A 50-module deployment can absorb mistakes; a 5,000-module deployment cannot.
• You're at a step-change in technology. First 800G deployment, first LPO deployment, first 1.6T deployment. Step-change technology has higher first-lot variance; sample testing is how you discover it before production.
• The application is high-stakes. An AI training fabric where a fabric-induced training-job interruption costs tens of thousands of dollars per hour justifies more aggressive testing than a development environment.

Sample testing does not pay for itself when:

• The deployment is small and the SKU is mature. Buying twenty 100G QSFP28 modules for a storage fabric expansion, from a vendor you've used for three years, on a platform you know well — sample testing is overhead.
• You have an accepted contract lab partner. If a contract lab tests on your behalf and produces reports you trust, replicating their work in-house is cost without benefit.
• You're under a vendor support agreement that explicitly covers your use case. If the vendor has tested your specific configuration and contractually warrants it, your testing program is verifying their claims rather than doing the original work.

3.2 · In-house, contract lab, or trust-but-verify#

The three structural options for an operator-side testing program:

Most operators end up running a hybrid: full in-house testing for new vendors and step-change SKUs, contract-lab testing for occasional rigor on important decisions, trust-but-verify for routine reorders from established suppliers. The breakdown depends on operator size and risk tolerance, not on a universal answer.

Sample size — the math, and why "five units" doesn't pass#

Every sample-testing program comes back to one number: how many units do you actually need to test? The answer most operators converge on without thinking — five or ten — is a number with no statistical meaning. This chapter walks through what the standards actually require, what hyperscalers actually do, and what a defensible Tier 2/3 program looks like.

If you observe zero defects in a sample of size n, the upper 95% confidence bound on the true defect rate is approximately 3/n. Sampling five units and seeing them pass tells you very little. Sampling thirty tells you something. Sampling one hundred tells you something useful.

4.1 · The standard everyone should know#

ANSI/ASQ Z1.4 — formerly MIL-STD-105 — is the sampling standard the rest of the manufacturing world uses for incoming-inspection decisions on lot acceptance. The General Inspection Level II tables map lot size to sample size to accept/reject criteria at a given Acceptable Quality Level (AQL). At AQL 1.0, a 3,200-unit lot requires inspecting 125 units; you accept if 3 or fewer fail and reject at 4 or more. Reference Card 02 shows the relevant AQL tables visually.

The reason Z1.4 matters: it's defensible. A vendor can't argue "you only tested 5 units of my 3,200-unit lot, that's not statistically meaningful" if you tested 125 — that's the standard. The same vendor will accept the rejection at 4 failures because they understand the statistical floor. Operators who size sample-testing programs against Z1.4 produce decisions that survive vendor escalation. Operators who run "five units we had handy" produce decisions that don't.

4.2 · The brutal arithmetic at n=5#

Let's translate the standard's reluctance into something concrete. Suppose you order 5 sample modules from a new vendor and all 5 pass your tests. What does that tell you about the vendor's actual defect rate?

The exact binomial 95% confidence interval for "0 failures in 5 trials" admits population defect rates from 0% up to roughly 45%. That is, your data is statistically consistent with a vendor whose actual production has a 45% defect rate. Five passes in a row is not improbable from such a vendor — it has roughly a 5% chance of giving you exactly that result. You cannot tell the difference between a vendor with a 1% defect rate and a vendor with a 30% defect rate from a sample of 5.

The same arithmetic, presented differently for sample sizes that show up in operator practice:

The implication is straightforward: if you want to make any defensible statement about a vendor's defect rate, you need at least 30 samples, and ideally 100. Below that, you are running a smoke test and calling it qualification.

4.3 · What hyperscalers actually do#

Hyperscaler optical-procurement programs run statistical sampling at scale. The pattern: AQL 1.0 sampling per Z1.4 General Inspection II at every receiving lot, with full first-article protocols at vendor onboarding (typically 200–500 units across multiple lots) and ongoing Cpk tracking against published baselines. Their published failure data — Alibaba, Microsoft, Meta — derives from this discipline; the data exists because the discipline exists. Tier-2 operators don't need the same depth, but the pattern is the model: structured sampling, statistical pass/fail criteria, baseline tracking. The next section describes the Tier-2 adaptation.

4.4 · The realistic Tier 2/3 sampling pattern#

You are not Microsoft. You don't have a 50,000-module fleet, you don't have an in-house optical lab, and you don't have engineers whose full-time job is qualification. What does a defensible sample-testing program look like at your scale?

The realistic pattern, which we have seen converge across Tier 2 cloud operators and well-engineered Tier 3 AI startups:

For each new vendor or new SKU: a first-article qualification of 30 units. Thirty units is the smallest sample size that gives statistically defensible coverage at AQL 2.5%, the minimum acceptable AQL for production AI fabric. Test all 30 units to a documented protocol covering identification, DOM, FEC, thermal corner, and 72-hour burn-in. Zero failures across 30 units gives you a defensible 9.5% upper-bound on defect rate and an actionable distribution to compare against future lots.

For first production lots (the first 100–500 units after qualification): 100% inspection on a lightweight protocol. Identification, DOM read, link-up verification, FEC baseline. No burn-in, no thermal corner. Catches gross failures before they reach deployment.

This program is not hyperscaler-grade. It does not catch every issue a million-port fleet would. It does, however, provide statistically defensible vendor evaluation, scales with operator size, and fits into a budget that small ops teams can defend. The full implementation guidance — what equipment, what tests, what protocols — is the rest of Part 2.

4.5 · The decision: how many samples for your program#

Bringing the whole chapter together:

• 5 samples is engineering smoke testing. Useful for quick-and-dirty checks; not a procurement decision.
• 10 samples is informal supplier evaluation. Useful for ranking vendors against each other on a quick screen; not for absolute quality claims.
• 30 samples is the floor for defensible Tier 2/3 first-article qualification at AQL 2.5%.
• 100 samples is the comfortable floor for confident Cpk-based decisions, and what we'd recommend for any high-stakes vendor evaluation.
• 300+ samples approaches hyperscaler-grade coverage at AQL 0.65%. Generally only practical for first-production-lot 100% inspection rather than first-article evaluation.

The next chapter answers the obvious follow-up: what equipment do you need to run the testing on those 30 or 100 modules?

Test equipment — what you need, what you can skip#

The honest answer to "what equipment do I need" is "less than the test-equipment vendors will tell you, and more than the budget owner will want to hear." This chapter is the equipment hierarchy: the minimum that lets you run a real testing program, the recommended tier for serious Tier 2/3 operations, and the equipment that earns its place only at hyperscaler scale or in a contract lab.

5.1 · Equipment by test purpose#

Before the tier breakdown, here is the equipment landscape mapped to what each instrument actually catches. Most operators inherit equipment recommendations from vendors with an interest in selling them. The honest mapping is below.

5.2 · What you can do with just a switch and hosts#

Before you buy any optical test equipment, understand what your existing infrastructure can already do. Modern switches and the modules themselves expose a remarkable amount of test capability at zero additional equipment cost.

Together, these capabilities — using only a switch, two hosts, basic patch cables, and a fiber inspection scope — catch what we estimate to be roughly 70% of vendor-quality issues an instrumented lab would catch. The remaining 30% — TDECQ measurement, stressed-receiver testing, accelerated aging in a controlled chamber, eye-diagram analysis at the physical layer — are real but bounded. They are also exactly the cases where a contract-lab relationship pays for itself.

5.3 · The realistic Tier 2/3 lab#

Putting the budget reality together: a sub-$50K lab does 90% of what a serious operator-side testing program needs to do. That's a real number, not aspirational. The lab consists of:

• 2 switches matching deployment platform (~$15K each, used or refurbished from secondary market — $20K total)
• 2 hosts with NICs (~$5K each — $10K)
• VIAVI FBP-P5000i fiber inspection scope ($2K)
• VIAVI MP-60 power meter ($2K)
• OTDR — VIAVI OTDR-1 entry-level ($8K)
• Cleaning supplies and references ($1K)
• Patch cables — known-good characterized set ($2K)
• Mating-cycle fixture, in-house built (~$500)
• Test rack, environmental sensors, cabling ($3K)

Total: roughly $48,500. Plus the engineer time to set it up and operate it.

What this lab does not do: TDECQ measurement (send to contract lab when needed), stressed-receiver eye analysis (same), full environmental chamber thermal cycling (same), formal qualification certification documents (no operator-side lab provides these — you go to a NIST-traceable lab if a customer demands it).

What this lab does do: every sample-testing protocol described in Chapter 7. Every bring-up validation procedure described in Part 3. Field failure analysis on the modules that matter. Cross-vendor interop testing in a controlled environment. The bulk of what an operator-side testing program is for.

5.4 · When to use a contract lab#

A contract-lab relationship complements rather than replaces in-house capability. You use the contract lab for the cases where:

• The test requires equipment you don't have. TDECQ measurement is the canonical example — almost no Tier 2/3 operator owns a TDECQ-capable oscilloscope, but specific procurement decisions sometimes require it.
• You need third-party documentation. A NIST-traceable test report carries weight in vendor disputes that an in-house report does not. If you anticipate contesting an RMA or escalating a vendor quality issue, contract-lab documentation is worth its cost.
• You need a reference comparison. Some contract labs maintain historical databases of measured modules across vendors; sending your samples for comparison against their database can yield insights your own data cannot.
• The deployment is high-stakes and you want defense-in-depth. First 800G deployment, first 1.6T deployment, first deployment of a vendor with no track record. Contract-lab validation supplements your own testing.

Reputable optical test labs in North America include those operated by major test-equipment vendors (VIAVI, Keysight have services arms), academic labs at universities with photonics programs, and specialized commercial labs. Costs run from a few thousand dollars for a quick characterization to tens of thousands for a full qualification battery on a sample lot.

Reading factory test reports critically#

A factory test report is the vendor's claim about a unit. It is not third-party verification; it is the vendor grading their own homework. A serious operator reads factory test reports the way an experienced editor reads a press release — looking for what the document is and is not saying, and looking for the tells that indicate the document was generated rather than measured. This chapter is the field guide.

6.1 · What a serious factory test report contains#

The minimum content a factory test report must contain to be useful:

6.2 · The OCP Optics Telemetry Specification as a procurement floor#

In 2025, the Open Compute Project published the Optics Telemetry Software Specification Rev 0.9, which defines what factory test data should be embedded in the module itself for in-deployment retrieval. The specification mandates:

• Non-volatile circular event buffers in the module that retain operational history
• Summary flags accessible to the host for remote-end diagnostics
• CDB-based remote register reads for vendor and host integration
• Per-unit factory test data accessible via standardized CDB commands

The OCP Optics Telemetry Spec is operator-friendly in a way that previous specs were not — it puts operator-relevant test data in a place the operator can read it. We recommend operators specify OCP Optics Telemetry Spec Rev 0.9 (or later) compliance as a procurement requirement. This single contractual line item resolves multiple problems: it standardizes what factory data is available, it ensures the vendor's test program is rigorous enough to populate the spec's required fields, and it makes per-unit verification straightforward post-deployment.

6.3 · The red flags#

Reference Card 07 enumerates seven tells of fabricated factory test reports — identical timestamps across "different" units, identical calibration coefficients, suspiciously round numbers, zero variance, missing metadata, pass-only results, and absent test conditions. The card is the bench artifact; engineers reviewing 30 reports for procurement decisions can apply it directly. The principle: real reports have realistic noise and varied timestamps; fabricated reports look uncannily clean.

6.4 · The 60-second scan#

Real operators do not study factory test reports in detail; they scan them in roughly 60 seconds. The scan looks for:

A report that fails any of those five checks needs a follow-up conversation with the vendor before the lot is accepted. A report that passes all five gets the assumption of good faith — though in-house verification of a small sample is still appropriate.

6.5 · Challenging a suspicious report#

What do you do when a report has multiple red flags? The professional path:

Running the tests#

The actual test protocols — what to run, in what order, with what equipment. This chapter is the procedural core of Part 2. The protocols here are designed for the realistic Tier 2/3 lab described in Chapter 5, with notes on what changes if you have a contract lab or full instrumentation. Every test is calibrated against the standards it relates to; where the standards-prescribed procedure is impractical, we describe the alternative most operators actually use.

7.1 · The first-article qualification protocol#

For a new vendor or new SKU, this is the full 30-unit protocol that produces defensible data for a procurement decision. Total elapsed time: 5–7 days for the testing, plus burn-in time. Engineer time: 2–3 days of active work spread across the elapsed window.

Phase 1: Receiving and identification (Day 1, ~2 hours)

For each of the 30 units:

• Visual inspection — packaging intact, module body undamaged, latches functional
• Read EEPROM via switch CLI — vendor name, part number, serial number, date code
• Verify identification matches the order (vendor and part number) and the unit (serial number on label matches serial in EEPROM)
• Read CMIS Page 02h calibration data — store for later red-flag analysis
• Read Application Selection table from Page 00h — record advertised configurations
• Endface inspection at 200× via fiber inspection scope — record any contamination found before first mate

Outcome: a per-unit identification record, a clean baseline endface state, and any red-flag detections logged.

Phase 2: Initial DOM and link characterization (Day 1, ~3 hours)

Set up the test bench: two switches matching deployment platform, characterized fiber jumpers between them, both modules of a link from the same lot. For each pair (15 pairs from 30 units):

• Insert modules, confirm Datapath State Machine reaches DPActivated
• Read DOM — temperature, TX power per lane, RX power per lane, voltage, laser bias per lane
• Compare DOM-reported values to external power meter measurement on a representative subset of lanes (catches calibration faults)
• Configure traffic — line-rate PRBS31Q from one host to the other, both directions
• After 5 minutes of sustained traffic, read FEC counters — pre-FEC BER per lane, corrected codewords, uncorrectable codewords
• Read VDM Pages 14h–17h for per-lane SNR and BER statistics
• Stop traffic, read interface counters — CRC, FCS, alignment errors should be zero

Outcome: a per-pair characterization record covering identification, initial DOM at room temperature, and a 5-minute FEC baseline. Any pair that fails to reach DPActivated, shows DOM outside expected ranges, or accumulates uncorrectable codewords in the 5-minute window is flagged for individual investigation.

Phase 3: Thermal corner testing (Day 2, ~6 hours)

Test the same 15 pairs at three thermal corners:

• Cold corner — module case temperature in the 0–5°C range. Achievable by reducing rack ambient or by running the modules at idle (they self-cool to near ambient).
• Hot corner — module case temperature in the 65–70°C range. Achievable by reducing switch fan speed via NOS commands and elevating ambient toward 30–35°C.
• Nominal — module case temperature in the 40–50°C range. The "Day 1" condition.

At each corner, allow the modules to thermally stabilize (15–20 minutes), then re-run the DOM and FEC characterization from Phase 2. The acceptance criterion is not that the values are identical across corners — they won't be, and shouldn't be — but that they remain inside spec at every corner and that the per-lane spread doesn't widen unacceptably.

Failed corners include: a module that drops to a degraded link state, a lane whose pre-FEC BER crosses the action threshold, a temperature reading that doesn't track the corner condition (suggests internal thermal interface fault), or a unit that fails to recover when conditions return to nominal.

Phase 4: Burn-in (Days 3–5, 72 hours, automated)

Configure all 15 pairs with sustained line-rate PRBS31Q traffic. Configure telemetry to log DOM, FEC counters, and interface error counters at one-minute cadence. Allow the burn-in to run unattended for 72 hours.

At the end of 72 hours, the per-pair acceptance is:

• Zero uncorrectable codewords across the entire window
• Zero CRC or FCS errors
• Zero link flaps
• Pre-FEC BER trend flat or improving (rising trend ≥ 0.5 decades is a fail even if the absolute value is in spec)
• DOM stable within ±0.3 dB on power, ±3°C on temperature at constant ambient

Outcome: 15 pairs each with 72 hours of telemetry, scored against the acceptance criteria. Pairs that fail any criterion are pulled for individual investigation; the rest pass to Phase 5.

Phase 5: Mating cycle and recovery (Day 6, ~3 hours)

For a subset (typically 5 of the 15 pairs, randomly selected): subject the optical interfaces to a controlled number of mating cycles using a fixture or careful manual procedure. The IEC 61300-2-2 standard specifies 500 cycles for original spec; 200 cycles for the 2023 single-mode patchcord revision. Most operators run 100 cycles as a practical compromise — enough to expose poor mechanical quality, not enough to consume excessive engineer time.

After mating cycles, re-inspect the endface for damage and re-run the DOM and FEC characterization. Modules whose endfaces show damage, whose insertion loss has increased meaningfully, or whose DOM values have shifted are flagged.

Phase 6: Statistical analysis and decision (Day 7, ~4 hours)

Compile the data from Phases 1–5 into a per-unit data record and a population summary. Compute Cpk for each measurement against the relevant spec limit. Identify outliers, assess their cause, and produce the procurement decision document — which is what the §4.5 decision framework in Chapter 4 covers.

7.2 · The follow-on lot inspection protocol#

Once a vendor and SKU have passed first-article qualification, follow-on lots are inspected on a lighter protocol.

Per Z1.4 General Inspection Level II at AQL 4.0 (the realistic Tier 2/3 level) for typical lot sizes: sample 32 units from a 1,200-unit lot, 50 units from a 3,200-unit lot. From each sampled unit:

• Identification (5 minutes per unit)
• DOM read at room temperature (5 minutes)
• Link-up and 5-minute FEC baseline (10 minutes)

Total per-unit time: 20 minutes. Per-lot total: ~10–17 hours of engineer-attended time. Catches gross defects, batch-level issues, and any drift from the qualified vendor baseline. Does not catch the marginal cases that first-article qualification catches with its longer protocols, which is why first-article testing comes first.

7.3 · Switch-resident PRBS — the protocol details#

The switch-resident BERT capability deserves its own section because it is the most under-utilized capability in operator-side testing.

Every CMIS-compliant module supports PRBS generation and checking via Page 13h. The host-side commands to drive it vary by NOS, but the underlying capability is universal:

SONiC: sfputil show prbs reads current state; sfputil set prbs configures pattern and enables generator/checker.

NVOS (NVIDIA): nv show interface eth1 transceiver prbs reads state; nv set interface eth1 transceiver prbs pattern PRBS31Q configures.

Cumulus: PRBS support via ethtool extensions and platform-specific utilities.

Arista EOS: platform fap prbs commands on Broadcom-based platforms; equivalent on Cisco-based.

Pattern selection matters. The patterns most relevant to optical testing:

• PRBS31Q — 2³¹-1 sequence on PAM4. Worst-case modulation for testing 100G-per-lane and 200G-per-lane PAM4 channels. The standard test pattern for 400G/800G optical link characterization.
• PRBS13Q — Shorter sequence, easier for some BERT instruments. Allowed by IEEE 802.3ck for 100G/lane PAM4 BERT.
• SSPRQ — Short Stress Pattern Random Quaternary. The official TDECQ test pattern. Stresses the channel differently than PRBS31Q.
• PRBS31 (NRZ) — For 25G NRZ links (100G QSFP28). The PAM4 patterns don't apply.

Test duration matters. A PRBS test running for 1 second on an 800G link covers 8×10¹¹ bits — enough to characterize a BER of 10⁻¹¹ with reasonable confidence. Running for 60 seconds covers 4.8×10¹³ bits — enough to characterize 10⁻¹³. For first-article testing, run PRBS for at least 5 minutes per condition; for production verification, 60 seconds is usually sufficient.

7.4 · Thermal corner testing without an environmental chamber#

An environmental chamber costs $30,000 and runs for years. Most Tier 2/3 operators don't have one. The practical alternatives that catch most thermal defects:

The fan-modulation method

Modern switches expose fan control via NOS commands. Reducing fan speed elevates module case temperature — a 32-port 800G switch with full population at 17W per module produces enough heat that reducing fans to 50% drives module case temperature into the 65–70°C range within 15 minutes.

The procedure:

• Populate test switch with modules under test
• Run line-rate traffic to ensure modules dissipate full power
• Reduce switch fan speed to a controlled level (specific NOS command varies)
• Monitor module case temperature via DOM
• When target temperature is reached and stable, run the test protocol
• After test, restore normal fan operation and verify recovery

This catches: thermal interface defects, lane-imbalance issues that worsen at high temperature, FEC margin reductions at temperature, and unit-to-unit variation in thermal behavior. Does not catch: cold-corner issues (the switch can't drive temperatures below ambient), thermal cycling fatigue (you're not cycling repeatedly), and humidity effects.

The HVAC method

If the test rack can be moved to a controlled environment — a closet that runs cold, an outdoor area that runs hot — you can test cold corner and very hot corner without a chamber. Practical, occasionally necessary, but operationally awkward.

When the chamber is necessary

Some tests genuinely require an environmental chamber. Programmable thermal cycling (Telcordia GR-468 §3.3.2.2 specifies -40°C to +85°C cycling with ≥10 minute dwell), accelerated aging tests (running modules at elevated temperature for weeks to compress lifetime), and the standards-compliant version of corner-condition validation. For these, a contract lab is the right choice.

7.5 · Mating cycle testing#

Most optical and copper module failures are not measurement failures — they're mechanical. A mating cycle test exposes mechanical quality: the strength of the latch, the precision of the connector alignment, the durability of the optical endface against repeated mating, and the integrity of the EEPROM contact pins.

The standards: IEC 61300-2-2 defines mating durability testing for optical connectors, with a typical 500-cycle severity for data-center applications. IEC 61753-021-2:2023 reduces this to 200 cycles for single-mode patchcords specifically. EIA-364-1000 covers the controlled-environment mechanical assessment for connector-to-cage mating.

The realistic operator-side test: a fixture that mates and unmates a module 100 times, with periodic verification of:

• Endface inspection at 200× (look for damage progression)
• Insertion loss measurement (look for IL drift)
• EEPROM read consistency (look for contact-pin wear)
• Latch retention force (look for mechanical fatigue)

Fixtures can be built in-house from servo motors and a programmable microcontroller for under $1,000, or purchased as commercial test fixtures for $5,000–10,000.

Receiving inspection and physical hygiene#

Most production link incidents trace to physical-layer issues that should have been caught at receiving. The economics: a fiber inspection scope costs $1,500, and connector and endface contamination is consistently among the largest single categories of avoidable link issues — catching it at the connector, before a module reaches a switch port, is one of the highest-return checks in the program. The discipline is structured but not complicated.

8.1 · The ten-minute receiving check#

For every shipment of optics or cables, the receiving check before stocking:

1. Box and ESD bag integrity. Damaged bags or absent ESD packaging is grounds for refusing the shipment — modern modules can be ESD-damaged in handling without showing visible defects.
2. Spot-quantity check. Count units against the packing list. Note serial-number ranges. Flag any out-of-sequence serials that suggest mixed lots.
3. Visual inspection of each module. Housing damage, bent connector pins, foreign material on the optical port. Set aside any unit that looks anomalous for closer inspection.
4. Endface inspection on a 10% sample of modules and 100% of any shipped cables. Use a fiber inspection scope; pass/fail criteria per IEC 61300-3-35 by zone. Reject any unit with visible scratches in the core zone or failure in any zone above the standard's pass criteria.
5. Read the EEPROM (Page 00h) on a sample of 5–10 modules to confirm vendor, P/N, S/N, firmware version match the packing list and the order. Mis-shipped SKUs catch here, before they reach a deployment.
6. Tag and store with the receiving record (date, lot, vendor, P/N, S/N range). The tag follows the unit through bring-up and into the validation record.

The full ten-minute discipline is the sample-protocol floor for every lot regardless of vendor or relationship. It catches the obvious problems (damaged in transit, mis-shipped, wrong firmware) before any time is invested in deeper testing.

8.2 · Endface inspection, cleaning, and the first-mate rule#

Endface inspection is the highest-ROI receiving activity. A $1,500 fiber scope catches contamination, scratches, and damage that produce 0.1–0.5 dB additional attenuation per affected lane — enough to push a marginal link into post-FEC errors. The discipline: inspect before first mate, period.

IEC 61300-3-35 defines pass/fail by zone. Core zone (the fiber core itself) — no defects allowed. Cladding zone — limited defect count and size. Contact zone (the area that physically mates) — moderate tolerance. Adhesive and outer zones — wide tolerance. Modern fiber scopes apply this automatically and produce a pass/fail with the captured image.

8.3 · Polarity, APC vs UPC, and IHS vs RHS#

Three physical-compatibility traps catch deployments at receiving repeatedly. None of them require depth to avoid; all require attention.

The receiving check that catches all three of these takes the same ten minutes as the basic procedure — the discipline is to actually look, not to add more steps.

Cabling and fiber plant validation#

An operator who validates modules but not the cabling they plug into is solving half the problem. The fiber plant is the channel; the modules are the endpoints. A degraded plant produces problems indistinguishable from degraded modules — and replacing modules to fix a cabling problem is one of the most expensive ways to learn about your fabric. This chapter covers what to validate before populating modules.

9.1 · Why validate the plant separately#

Most operators inherit cabling. The data center was built years ago, the patch panels were installed by someone who's no longer here, and the fiber plant has been used for previous generations of equipment. Before installing modern 400G or 800G modules, the plant must be validated against the requirements of those modules — not against the requirements of whatever ran on it last.

The risks of skipping plant validation:

• OM3 multimode where you assume OM4 — reach is shorter than your design
• G.652.D fiber where you need G.657 bend-insensitive — bends will cause loss
• Patch panels with degraded connectors — accumulated insertion loss eats your link budget
• Splices you didn't know existed — adding 0.3 dB each to your loss budget
• Wrong polarity in places — links won't come up
• Mixed APC/UPC in places — same problem

The validation program below addresses all of these.

9.2 · OTDR traces — what they tell you#

An Optical Time-Domain Reflectometer (OTDR) sends an optical pulse down a fiber and analyzes the reflections that return. From the reflection pattern, the OTDR reports:

• Total fiber length
• Per-event insertion loss (each connector, each splice, each patch panel)
• Reflectance at each event
• Macrobend-induced loss
• Localized fiber damage

An OTDR trace looks like a sloped line punctuated by step-changes at events. The slope is fiber attenuation (typical 0.35 dB/km for OS2 at 1310nm, 2.3 dB/km for OM4 at 850nm); the steps are losses at events. A clean trace has small, predictable steps; a problematic trace has unexpectedly large steps, anomalous reflectance peaks, or non-linear slopes that indicate fiber damage.

For each link in your fabric, the OTDR trace should show:

• Total length matching design (within reasonable tolerance — fiber routes have actual lengths, not theoretical)
• Per-connector loss ≤ 0.5 dB (≤ 0.35 dB for high-grade installations)
• Per-splice loss ≤ 0.3 dB (fusion splices)
• Total link loss within IEEE-spec budget for the variant you'll deploy
• No anomalous reflectance peaks indicating damaged or mismatched connectors

9.3 · Insertion loss budget — the math#

For each interconnect variant, IEEE 802.3 publishes a total channel insertion loss budget. The link must fit inside that budget with margin. Typical budgets:

Per-element typical loss values:

• OS2 single-mode fiber at 1310nm: 0.35 dB/km
• OS2 single-mode fiber at 1550nm: 0.22 dB/km
• OM4 multimode fiber at 850nm: 2.3 dB/km
• Standard MPO/LC mated connector: 0.35 dB
• High-grade mated connector (Panduit Ultra, etc.): 0.10–0.15 dB
• Fusion splice: 0.10–0.30 dB
• Mechanical splice: 0.30–0.70 dB
• Patch panel pass (connector + jumper + connector): 0.75 dB total

9.4 · The plant acceptance protocol#

Before populating a fabric with modules, accept the plant against criteria:

9.5 · Common findings and what to do#

What plant validation typically catches:

Power, thermal, and infrastructure validation#

Modules don't operate in isolation; they operate in racks, drawing power and producing heat that the rack and the data center hall must absorb. Before a fabric goes into production, the infrastructure validation confirms that the rack can handle what you're about to plug in. Skip this and you discover at week three that a rack is thermally over-design and modules are throttling — at which point fixing it requires either re-spreading equipment across more racks or upgrading the room cooling.

10.1 · The PDU headroom check#

Each rack has a power budget defined by its PDU. The compute and switching equipment in the rack must fit inside that budget, with margin for transient loads.

Module-side power contributions for a fully populated 32-port 800G switch:

• Switch ASIC: 400–600 W
• 32 modules at 17 W each (DSP optical): 544 W
• Fans: 100–200 W
• Power supply overhead: 100 W
• Total switch: ~1,150–1,450 W

For a rack with 4 such switches plus compute, the switch contribution alone is roughly 5 kW. Add GPUs, CPUs, and storage, and even moderate-density AI racks reach 15–25 kW. Many older racks were designed for 8–12 kW; populating them with modern 800G fabric exceeds the design.

The validation is straightforward: total your projected per-rack draw, compare to PDU rating, ensure 25% margin (transients, future expansion, derating for elevated temperature). Racks that don't have margin need either equipment redistribution or PDU upgrade before commissioning.

10.2 · Thermal validation under load#

Power becomes heat. Heat is removed by airflow. The validation question: at full populated load, does the rack stay within thermal design?

Module case temperature is the proximate measure. Per Chapter 12's DOM baselines, the module's own thermal limits are typically 70–75°C case temperature. The rack must keep modules well inside that limit even at full load.

The validation procedure:

• Populate the rack with target equipment
• Run line-rate traffic across all interconnects (PRBS or representative workload)
• Monitor module case temperatures via DOM, sampled every minute
• Run for at least 30 minutes after temperatures stabilize
• Verify no module exceeds 65°C case temperature at sustained load (10°C below limit gives margin for ambient excursions)

If modules exceed the threshold, the issues are usually one of:

• Rack airflow obstruction — missing blanking panels, cable bundles blocking exhaust, equipment mounted backward (front-to-back airflow vs back-to-front mismatch)
• Room cooling insufficient — the CRAC is undersized for the load you've added
• Hot spot at the rack — the rack is in a hot aisle that's recirculating warm exhaust
• Module thermal interface defect — single module runs hotter than its peers; replace and re-test

10.3 · Airflow shadowing and cable density at 800G#

At GB200-class rack densities the thermal failure mode is rarely a whole-rack cooling shortfall — the room CRAC is usually sized correctly. It is local. Two effects dominate, and both produce a signature that is easy to misread as a module defect.

Add to the thermal-validation procedure in §10.2: capture per-module case temperature across every populated slot, not a rack aggregate, and flag any module more than roughly 8 °C above its slot-row peers for an airflow-routing check.

10.4 · Ambient conditions#

Ambient temperature and humidity at the rack intake set the floor for everything else. ASHRAE specifies envelope classes (A1 through A4) for data center conditions; AI deployments should generally aim for ASHRAE A1 or A2 (18–27°C intake at 30–60% RH).

Out-of-spec ambient makes every threshold in this handbook unreliable. A module running 30°C above ambient is fine when ambient is 22°C; the same delta is a thermal alarm when ambient is 32°C. Validation that proves a fabric works at one ambient doesn't prove it works at another.

Verify before commissioning:

• Rack intake air temperature within 18–27°C (A2 envelope), measured during peak load conditions
• Humidity 30–60% RH (avoid below 30% — static discharge risk; avoid above 70% — condensation risk)
• No visible dust accumulation on intake surfaces
• HVAC system has 25% capacity margin against current load

10.5 · The infrastructure pre-flight checklist#

Before installing modules into a rack:

• ☐ PDU capacity verified ≥ projected load + 25% margin
• ☐ All blanking panels in place; no recirculation paths
• ☐ Cable management does not obstruct airflow exhaust
• ☐ Cable bundles routed clear of module faceplates; no bundle shadowing a populated cage
• ☐ Switch airflow direction matches rack layout (front-to-back in cold-aisle/hot-aisle layouts)
• ☐ Rack intake air temperature at peak load measured and within envelope
• ☐ Per-module case temperature captured across all populated slots; no module >8 °C above its slot-row peers
• ☐ Rack-level monitoring active (PDU draw, intake/exhaust temperatures)
• ☐ HVAC capacity confirmed for added thermal load

Each item that fails the check needs to be remediated before module installation, not after.

The five-check bring-up procedure#

Each module that enters production should pass five checks, in order, with each check gating the next. The procedure is the same across all NOS stacks; the commands differ. This chapter is the master procedure with side-by-side NOS commands. The experienced engineer runs all five in about 4 minutes per module.

11.1 · Why five checks, in this order#

The check sequence is designed so that each step proves a property the next step depends on. Skipping forward produces confusing diagnostics:

• Check 1 (presence/identity) gates everything — if the module isn't recognized, nothing else matters
• Check 2 (link state/rate) requires presence; if link doesn't come up, DOM and FEC don't apply
• Check 3 (DOM) requires link-up; idle modules don't produce meaningful telemetry
• Check 4 (FEC) requires DOM showing healthy levels; FEC errors on a module with bad RX power tells you the module, not the link, has a problem
• Check 5 (interface counters) is the final gate; clean counters under traffic mean the link is production-ready

11.2 · Check 1 — Presence and identity#

Verify the module is detected, the EEPROM is readable, and the identification matches what was ordered.

Verify in output:

• Vendor name matches the order
• Part number matches the SKU
• Serial number is present and unique (not duplicating another module)
• Module type matches expectation (e.g., "800G OSFP DR8" — not a 400G module mistakenly shipped)
• Date code reasonable (not stock from years ago)

Failure modes: Module not detected (reseat; verify cage power; check for bent pins). Wrong part identified (CMIS parser update needed on switch). EEPROM corrupted/unreadable (faulty module — RMA).

11.3 · Check 2 — Link state and negotiated rate#

Confirm the module links up at the expected speed, FEC mode, and lane configuration.

Verify:

• Link state: up
• Operational speed matches configured
• FEC mode correct (KP4 for 400G/800G PAM4; KR4 or none for 100G NRZ)
• Duplex: full
• Lane count matches form factor (8 for 800G OSFP; 4 for 400G QSFP-DD)

Failure modes: Link down despite presence (wrong cable polarity, fiber type, or far-end module off). Link up but wrong speed (auto-neg mismatch, firmware mismatch). Link up but wrong FEC (CMIS FEC negotiation failed — common on new AECs).

11.4 · Check 3 — DOM read, per-lane#

Verify optical/electrical telemetry is in healthy ranges per Chapter 12. Critically, per-lane, not aggregate.

Verify against Chapter 12 baselines:

• Temperature in expected range for variant at current ambient
• TX power per lane within IEEE min/max, within typical operating range
• RX power per lane above sensitivity floor, within saturation ceiling
• Lane-to-lane TX spread ≤ 1 dB
• Lane-to-lane RX spread ≤ 2 dB
• Laser bias within expected range for laser type

11.5 · Check 4 — FEC baseline#

Pre-FEC BER per lane below install target (Chapter 4 of Part 2 referenced these earlier; the operating thresholds for production are in Chapter 12 of this part). Critical: run Check 4 only after at least 5 minutes of line-rate traffic. An idle link shows zero errors which means nothing.

Verify:

• Pre-FEC BER per lane below install target for variant (per Table 14 in Chapter 12)
• Corrected codewords non-zero and accumulating — this is expected; zero means no traffic ran
• Uncorrectable codewords zero absolute
• Per-lane BER spread within 1 decade across lanes — a single lane far above its siblings is a lane-specific defect, not uniform aging

11.6 · Check 5 — Interface counter clean-start#

Final gate: confirm zero CRC, FCS, and symbol errors over a 15-minute traffic window after counter clear.

Procedure: clear counters, run line-rate traffic for 15 minutes, read counters. Acceptance: zero CRC, zero FCS, zero alignment errors, symbol errors below threshold (Chapter 17 covers the exact thresholds).

A link that passes Checks 1–4 but accumulates CRC errors in Check 5 has a subtle channel issue — usually a marginal connector, an MPO with alignment problems, or a cable with internal defect.

11.7 · Breakout-specific procedure#

For breakout deployments (one parent port broken into multiple child legs), each leg is an independent link and gets independent checks.

The procedure: configure the parent port for breakout mode (NOS-specific commands), wait for the child interfaces to appear (typically named with sub-port suffixes — Ethernet0/1, Ethernet0/2 in SONiC; et-0/0/0:0, et-0/0/0:1 in Junos), and run Checks 1–5 on each child interface independently.

Common breakout-specific issues:

• Parent port doesn't enter breakout mode — usually a NOS configuration issue, not module
• Some legs link, others don't — fan-out fiber assembly or partial breakout firmware mode
• FEC negotiation differs across legs — check that the same FEC mode is configured on all four child interfaces and on the far-end equivalents
• One leg accumulates errors; others clean — single-lane defect in the parent module

11.8 · The four-minute scan#

The experienced engineer runs the five checks as a tight sequence:

1. Insert module; wait 30 seconds (Check 1)
2. Read EEPROM identification — verify or fail (30 seconds)
3. Wait for link-up; read link state (30 seconds, Check 2)
4. Read DOM per-lane; eyeball against expected ranges (60 seconds, Check 3)
5. Start traffic; wait 5 minutes; read FEC and interface counters (Checks 4 and 5 together)

Total: roughly 8 minutes including the 5-minute traffic wait. The engineer can run multiple modules in parallel — start one, walk to next, return after 5 minutes — and process 5–10 modules per hour at pace.

DOM baselines — what healthy looks like#

DOM (Digital Optical Monitoring) is the per-lane operational telemetry the module exposes — temperature, TX power, RX power, laser bias current. Healthy values vary by vendor and SKU; what matters operationally is establishing baseline at install and alerting on excursion from baseline rather than against absolute thresholds.

12.1 · Reading DOM output#

DOM lives on CMIS Page 11h (per-lane data) plus Page 00h temperature. Most switch CLIs surface it via a single command — Cisco show interfaces transceiver detail, Arista show interfaces transceiver dom, NVIDIA mlxlink, SONiC sfputil show eeprom. The structure is the same across platforms: per-lane TX power (dBm), RX power (dBm), bias current (mA), plus module-level case temperature (°C). Capture this output at bring-up for every link — that capture is the baseline.

The four DOM parameters and their operational meaning:

• Module temperature — case temperature of the module package. Healthy range typically 10–65 °C with typical operation around 35 °C in a properly cooled rack. Sustained >70 °C means investigate airflow first; modules don't usually fail thermally before their environment does.
• TX power per lane — optical output power. Healthy range typically −5 to +3 dBm depending on optic class. The right-tail aging indicator: TX power drops over time as the laser ages. Alert on Δ > 1 dB drift from install baseline rather than on absolute threshold.
• RX power per lane — optical input power received from the far end. Healthy range typically −8 to +2 dBm. RX low does not mean module fault — it usually means channel: endface contamination, connector wear, fiber damage. Inspect / clean / re-mate before any module swap.
• Laser bias current per lane — drive current the laser is consuming to maintain output. Healthy range typically 25–80 mA depending on laser type. The leading aging indicator: bias rises gradually over the module's lifetime as the laser device degrades. Bias slope shifts 3–6 months ahead of TX power dropping measurably. Alert on the slope, not the value.

For complete healthy / warning / alarm thresholds visualized as zones, see Reference Card 04 (DOM Quick Reference). The card is the bench-side artifact; this chapter is how to interpret what it shows.

12.2 · Establishing the install baseline#

The install baseline is captured immediately after bring-up validation passes — once the link is up, error-free under PRBS, with traffic flowing in steady state. The baseline is per-lane and per-link; a 32-port switch produces ~256 baseline records (32 modules × 8 lanes plus module-level temperature).

What to capture in the baseline record:

• Module S/N, switch ID, port, lane index
• Each of the four DOM parameters at the moment of capture
• Ambient and rack inlet temperature at capture time
• Workload state — should be steady-state for representative DOM, not bring-up transient
• Capture timestamp

The baseline becomes the per-link reference for production telemetry. When a module's TX power drifts >1 dB from its baseline, that's the alert — not when it crosses a fleet-wide absolute threshold. Vendor thresholds vary; baseline-relative alerting catches per-module degradation that absolute thresholds miss.

12.3 · Drift detection and the trend#

A reading inside the healthy zone is fine in isolation. A reading drifting toward warning over weeks is the alert. The two trend patterns that matter:

TX power drifting downward without bias drifting up is rare and indicates a module-internal problem worth raising with the vendor. Temperature drifting upward without environmental change usually indicates dust accumulation in airflow paths or fan degradation in the rack — investigate the rack before the module.

Burn-in for production deployment#

Burn-in exists because some defects manifest only on a timescale of hours to days. A module that passes Chapter 11's five checks at minute zero may begin failing at hour 50 — a thermal interface that needs heating cycles to settle, a marginal solder joint that requires sustained thermal expansion to fail, a firmware state-machine edge case that triggers only after specific event sequences. Burn-in is the discipline that catches these.

13.1 · Why 72 hours is the minimum#

Burn-in duration relates directly to what classes of defect you catch:

Recommendation by deployment class:

• Storage / management fabric: 24 hours minimum
• Compute fabric (general): 48 hours minimum
• AI training fabric: 72 hours minimum (the standard)
• First deployment of new-generation technology (first 800G in your fabric): 7 days for a sample (10–20% of modules), 72 hours for the rest

13.2 · Traffic profile#

Burn-in at idle doesn't burn anything in. Sustained line-rate traffic is required.

For Ethernet fabrics:

• PRBS31Q test pattern via switch-resident BERT (best stress for PAM4)
• Synthetic iperf3 or trex traffic at line rate
• Real application traffic from a test load generator

For InfiniBand / RoCE fabrics:

• ib_send_bw running bidirectionally
• ib_write_bw for RDMA write stress
• Full perftest suite — every test type

What does not count as burn-in: ping traffic, link-up idle, short benchmarks that complete in minutes.

13.3 · Telemetry during burn-in#

One-minute polling of the following, persisted to a time-series store for later analysis:

• Per-lane TX power, RX power
• Module temperature
• Per-lane laser bias current
• Per-lane pre-FEC BER
• Per-lane corrected codewords (cumulative)
• Uncorrectable codewords (cumulative)
• Interface CRC, FCS errors (cumulative)
• Symbol errors on copper (cumulative)
• Link state changes (event-driven)
• Ambient rack temperature

13.4 · Acceptance criteria#

At end of burn-in window, all of the following must hold:

• Zero uncorrectable codewords across the entire window
• Zero CRC / FCS errors
• Zero link flaps
• Zero temperature excursions above action threshold
• Pre-FEC BER trend flat or improving — a link whose pre-FEC BER rose 0.5 decades or more during burn-in is a fail even if final value is in spec
• DOM stable within ±0.3 dB on TX power, ±3°C on temperature at constant ambient

A link failing any criterion does not ship to production. The process: remove, root-cause, replace, re-run burn-in on the replacement from hour zero. Do not partial-credit the burn-in clock.

13.5 · The textbook says one-minute polling. Reality says…#

13.6 · The parallel-burn-in pattern#

Burn-in is wall-clock time, not engineer time. The optimization: run as many modules in parallel as your test infrastructure supports. A bring-up bench that can host 64 modules simultaneously processes 64 modules in 72 hours; sequential burn-in of the same 64 modules would take 192 days.

Parallel burn-in requires:

• Sufficient test switches to host the modules — usually limited by switch port count
• Telemetry infrastructure that can poll all modules simultaneously without overloading any switch's management plane
• Process discipline to start, monitor, and accept many concurrent burn-ins

The realistic Tier 2/3 commissioning bench: 2–4 switches, supporting 64–256 modules in parallel, processing several thousand modules per quarter through 72-hour burn-ins.

Fabric-level validation#

Per-link validation proves each link is healthy. Fabric-level validation proves that the links, taken together, deliver what the workload needs: uniform path utilization, predictable performance under collective communication, no surprises when GPUs arrive. This chapter is the procedure for validating the whole fabric — ECMP hash quality, RDMA performance, fabric topology consistency, and the proof-without-GPUs problem.

ECMP hash polarization is the most common fabric-level defect on Ethernet AI fabrics. The aggregate counters look fine; per-next-hop counters tell the story. The coefficient of variation across your hash distribution is the single number that captures health.

14.1 · The CV measurement#

The fundamental ECMP health metric: coefficient of variation of per-path byte counts under representative traffic.

Procedure:

1. Run representative workload across the fabric (or synthetic traffic that mimics it) for 10–15 minutes
2. For each ECMP group (e.g., a leaf's uplinks to spines), collect per-link byte counters
3. Compute: CV = standard deviation of byte counts / mean of byte counts

Acceptance:

• CV < 0.15 (15%): Healthy. Distribution is good.
• CV 0.15 – 0.30: Warning. Investigate hashing.
• CV > 0.30: Fail. ECMP is not distributing properly.

14.2 · Common ECMP failure modes#

14.3 · The fabric-ready checklist#

When all of these pass, the fabric is ready to accept GPUs:

When GPUs arrive, the first nccl-tests run should closely match pre-GPU RDMA numbers. If it doesn't, the delta is either in the GPU stack (NCCL version, CUDA version, driver mismatch) or GPU-specific (PCIe topology, NVLink config) — not in the fabric. The network team is not the bottleneck.

The validation report#

Every link that passes bring-up generates a validation record. The record is the artifact that joins the factory test report, the bring-up data, and (later) the production telemetry into a single lifetime record per module. This chapter describes what goes in the record, what schema to use, and how the record gets joined to your inventory and operations systems.

15.1 · What goes in the validation record#

Per module / per link:

• Identification: vendor, part number, serial number, revision, manufacturing date code
• Deployment location: rack, switch hostname, port, position in fabric topology
• Configuration: speed, FEC mode, lane count, far-end module identification
• Bring-up evidence: timestamp and result for each of Chapter 11's five checks
• DOM at acceptance: TX power per lane, RX power per lane, temperature, voltage, laser bias per lane
• FEC at acceptance: pre-FEC BER per lane, corrected/uncorrectable codeword totals
• Burn-in results: start/end timestamps, duration, final BER per lane, BER trend, max temperature, flap count, error totals
• Validator: who or what system performed the validation, when
• Status: production-ready / re-test required / failed

This record is the baseline against which all future telemetry is trended. A module showing pre-FEC BER of 1.2e-9 in production has a different meaning depending on whether its acceptance baseline was 1.0e-9 (modest drift) or 5.0e-11 (significant drift).

15.2 · The YAML schema#

A representative validation record:

link_id: "rack42-leaf01:Eth1 -> rack42-spine01:Eth25"
timestamp: "2026-05-12T14:33:00Z"
interconnect:
  vendor: "Vitex"
  part_number: "VO-8CSR8CM-AA"
  serial_number: "VTX20260512001234"
  revision: "1.2"
  cable_length_m: 10
configuration:
  speed: "800G"
  fec_mode: "KP4"
  lanes: 8
bring_up:
  check_1_presence: pass
  check_2_link_state: pass (800G, KP4, full, 1.2s to link)
  check_3_dom:
    tx_power_dbm: [0.8, 0.9, 1.1, 0.8, 0.9, 1.0, 0.9, 0.8]
    rx_power_dbm: [-1.2, -1.1, -1.0, -1.2, -1.1, -1.0, -1.3, -1.2]
    temperature_c: 62.3
    pass: true
  check_4_fec:
    pre_fec_ber_per_lane: [1.2e-9, 1.5e-9, 1.1e-9, 1.3e-9,
                           1.4e-9, 1.2e-9, 1.1e-9, 1.3e-9]
    uncorrectable: 0
    pass: true
  check_5_interface:
    crc: 0
    fcs: 0
    pass: true
burn_in:
  start: "2026-05-12T15:00:00Z"
  end: "2026-05-15T15:00:00Z"
  duration_hours: 72
  final_ber_per_lane: [1.3e-9, 1.4e-9, 1.2e-9, 1.2e-9,
                       1.4e-9, 1.3e-9, 1.2e-9, 1.3e-9]
  ber_trend: "flat"
  flaps: 0
  uncorrectable_total: 0
  crc_total: 0
  max_temperature_c: 66.1
  pass: true
validator: "network-ops-shift-b"
status: "production_ready"
factory_report_ref: "VTX-FR-202604-N1234.pdf"

The exact format isn't sacred; YAML, JSON, or a database schema work equivalently. What matters is the binding to module serial number, deployment location, factory report reference, and the data fields above.

15.3 · Joining the records#

The validation record becomes useful when it's joined to other data sources:

• Factory test report — the per-unit report from the vendor, indexed by serial. Joining gives you the comparison point: how does the module's behavior at acceptance compare to its factory measurement?
• Production telemetry — the running stream of DOM and FEC data once the module is deployed. The validation record is the baseline; the telemetry is the trend.
• Inventory system — what module is in what rack/switch/port, and when did it get there.
• RMA and FFA records — when modules are returned, the validation record is part of the evidence package.

15.4 · The lightweight tracking system#

You don't need a full CMDB to do this well. The minimum that works:

• A directory of validation records, one file per module serial
• An inventory spreadsheet or simple database mapping serial to deployment location
• Telemetry storage (Prometheus, InfluxDB, or commercial equivalent) that ingests the production stream and tags by serial
• Process discipline that updates the inventory record on every install, swap, and retirement

Operators with thousands of modules sometimes graduate to dedicated DCIM tools; operators with hundreds usually do fine with the lightweight approach. The volume threshold is roughly 5,000 modules, where manual update of inventory records starts to break down.

Production telemetry#

Validation does not end at bring-up. The fabric runs for years, and every link in it accumulates a continuous stream of data that — read correctly — tells you which links are healthy, which are drifting, and which are about to fail. This chapter covers what to log, why, and at what cadence. The goal is not maximalist instrumentation; it is the smallest set of measurements that catches the largest fraction of preventable failures.

16.1 · The four telemetry classes#

Production telemetry on an interconnect link falls into four classes, each with different characteristics and different operational use:

An operator running all four classes, joined by link identifier and timestamp, has the data to diagnose effectively any production issue. An operator running only one class — DOM, typically — has the data to alert on gross failures but to diagnose almost nothing.

16.2 · The minimum telemetry set#

If you are building telemetry from nothing, this is the minimum. Each metric, polled at the indicated cadence, with the indicated retention.

16.3 · The OCP Optics Telemetry Specification#

OCP's Optics Telemetry Specification is the operator-friendly reference for what factory test reports and ongoing telemetry should expose. Hyperscalers drove its creation; Tier-2 operators benefit from the same standard. When evaluating vendor telemetry depth, OCP-spec compliance is the floor. See §21.1 for the standard reference.

16.4 · Where the data lives#

Telemetry storage is its own discipline. Three tiers in operator practice:

The simplest setup that works at Tier 2/3 scale: SONiC or Cumulus exporting via gNMI, Prometheus collecting and retaining 90 days, periodic export to S3 for long-term. Total cost: a few thousand dollars per year in storage; a few engineer-weeks to set up.

16.5 · Workload-aware monitoring#

DOM telemetry interpretation depends on what the workload is doing. A module showing TX-power drop during all-reduce-heavy training looks different from the same module during inference idle. Capture workload state alongside telemetry — at minimum, a marker for "training under sustained workload" vs "idle" vs "inference active" — so trend analysis correctly distinguishes drift from workload-correlated transients. This is the single change that makes telemetry actionable rather than noisy.

Counter interpretation and alerting#

Telemetry that nobody alerts on is decoration. The job of an alerting system is to distinguish normal operation from incipient failure, fire on the latter, and not fire on the former. Get this right and you catch problems before they become outages. Get it wrong and your engineers ignore the alerts, which is a worse outcome than not alerting at all.

A pre-FEC BER snapshot at Day 30 reads as healthy. The 30-day trend at the same moment shows the slope. The slope is what you alert on, not the value. Operators who alert on instantaneous BER chase noise; operators who alert on trend catch failures.

17.1 · Corrected versus uncorrectable — what to alert on#

Before any threshold discussion, one distinction has to be clear, because getting it wrong is the most common alerting mistake operators make on PAM4 fabrics. Corrected codewords are not a fault signal. On a 100G-per-lane PAM4 link, the FEC is doing continuous correction work as a normal condition of operation — a healthy link under load produces a steady stream of corrected codewords, and that count climbing is expected, not alarming. An operator who pages on "corrected codewords increasing" will page constantly on healthy links and quickly learn to ignore the alerts.

What actually warrants attention is one of three things: trend acceleration — the corrected-codeword rate or the pre-FEC BER rising faster than its established baseline slope; lane imbalance — one lane of an eight-lane module correcting far more than its siblings, which points at a specific lane defect rather than uniform aging; and uncorrectable codewords — any non-zero count, because uncorrectables are post-FEC errors that reach the host as real data corruption. Corrected codewords are evidence the link is working as designed. Uncorrectables, lane spread, and slope are the signals. Alert on those.

17.2 · The threshold ladder#

For each metric, three thresholds in sequence:

17.3 · The alert hierarchy#

The four-tier alert structure that we see actually work in operator practice:

17.4 · Correlation and noise reduction#

An alert on every threshold crossing produces alert fatigue. The alerting system has to do work to suppress redundant noise.

Correlation rules that pay for themselves:

• Suppress link-down alerts during planned maintenance. The maintenance window calendar should be a feed into the alerting system; alerts during a window go to a tracker, not to a pager.
• Group alerts by switch and rack. Twenty link-down alerts from the same switch is one event ("switch X went down") not twenty events. Pager fatigue is real; group correlated alerts.
• Alert on rate, not on instantaneous values. A pre-FEC BER sample of 1e-6 from a normally healthy link could be measurement noise; sustained 1e-6 over 5 minutes is a real condition. Hold-down windows reduce false alerts.
• Workload-context filtering. A link with elevated BER during an AllReduce phase, returning to baseline during forward-pass, is normal-for-workload. Subtract the workload-induced component before alerting.
• Vendor-aware thresholds. If you've established that vendor X's modules normally run 1.5e-9 pre-FEC BER and vendor Y's run 1e-10, applying the same threshold to both produces noise on vendor X. Calibrate per-vendor.

Triage and live module swap#

When an alert fires, the engineer's job is to determine fast: is this the module, the cabling, the far-end module, or the platform? The answer determines what gets replaced. Replacing the wrong thing wastes time, costs an extra module from spare inventory, and leaves the actual problem unfixed. This chapter is the triage discipline that produces the right answer in minutes rather than hours.

The triage discipline. Every module swap that fixes a fiber-plant problem becomes a no-fault-found RMA — and the original problem stays in your fabric. Channel-side investigation always precedes module replacement.

18.1 · The five-minute triage#

Five questions answered in roughly five minutes, in this order:

18.2 · The triage decision tree#

The mapping from triage findings to action:

18.3 · Live module swap procedure#

Modern modules support hot-insertion. The procedure for swapping a module on a live fabric, with minimal workload disruption:

1. Drain the link. If the link is part of an ECMP group, taking it down forces traffic to the other paths. Temporarily disable the interface on the local switch; verify ECMP redistributes correctly; verify aggregate fabric throughput is unaffected.
2. Take a final snapshot. Read DOM, VDM, lane states, recent FEC counters. Snapshot the validation record — useful for FFA analysis.
3. Power-down the port. NOS-specific commands (interface shut, no power-on, etc.) — drops cage power before mechanical extraction.
4. Extract the module. Release latch, withdraw cleanly. Inspect endface immediately for evidence of damage or contamination. Bag the module in antistatic packaging with the validation record reference.
5. Inspect the cage and far-end endface. Cage pins look intact? Far-end endface clean? This is where you catch issues that would have been re-introduced with the replacement.
6. Insert the replacement. Verify DPActivated state, run Checks 1–5 from Chapter 11. Run a 15-minute traffic window with counter monitoring before re-enabling in ECMP.
7. Re-enable in fabric. Bring the interface up; verify routing reconverges; verify ECMP picks up the path.
8. Update records. Inventory shows new serial in this slot. Old serial moves to "extracted, pending FFA". Validation record for the new module is created.

Total time for an experienced engineer: 20–30 minutes per swap, of which most is the 15-minute post-insertion traffic window.

18.4 · The "the module isn't actually broken" case#

A non-trivial fraction of replaced modules — by some operators' tracking, 15–25% — work fine when re-tested at a vendor's failure analysis facility. The labels for this in the industry are "no fault found" (NFF) and "no trouble found" (NTF). The vendor returns the module, often charges restocking, and points out it tested clean.

NFF/NTF rates above 10% indicate a triage discipline problem. The contributing factors:

• Modules replaced based on noisy alerts that resolved on their own
• Modules replaced because the platform reported "transceiver fault" without further investigation
• Modules replaced as part of "swap them all" responses to a cluster failure that was actually a power or thermal issue
• Modules replaced for problems that were actually in the cable or far-end module
• Modules replaced for a CMIS parser or revision mismatch — the host NOS mishandled the module's management interface and reported a fault that the module did not have
• Modules replaced for a retimer or DSP firmware mismatch with the host — a firmware alignment, not a hardware defect
• Modules replaced for a power-class negotiation mismatch — the host throttled a module that was never given the power class it needed
• Modules replaced for a breakout profile mismatch — the switch port was configured for a different application than the module advertised

The fix is the triage discipline above. The five-minute triage takes longer than just swapping the module — but reduces NFF/NTF dramatically and saves the inventory cost of the unnecessary swap, the engineer time of the swap itself, and the future time spent diagnosing the actual root cause that didn't get fixed.

18.5 · Common triage mistakes#

Four mistakes account for most wasted triage time:

1. Swapping the module first. RX-low symptoms point at channel four times in five. Inspect, clean, re-mate before the module comes out — the 90 seconds saves on average 40% of RMA freight.
2. Reading values, not trends. A reading inside healthy is fine in isolation. The slope is the alert. Pull telemetry history before deciding.
3. Skipping firmware confirmation. DPInit-stuck modules look like hardware faults but are usually firmware mismatch. Confirm both ends are on the qualified firmware before any swap.
4. Not capturing evidence before action. The DOM and VDM snapshot before extraction is the input to the FFA report and the contemporaneous record. Capture first, then act — even when the action is "swap immediately because the link is critical." Sixty seconds is rarely the constraint.

Re-validation, retirement, and the RMA/FFA evidence package#

A link validated at bring-up is not validated for life. Conditions change — firmware updates, ambient drift, cable disturbances during adjacent maintenance, slow component aging. The discipline of periodic re-validation maintains the assurance that bring-up established. Proactive retirement removes modules from production before their failure becomes a workload event. And when modules do fail, the RMA evidence package and the FFA loop close the cycle by feeding what you learn back to the vendor — and back into the next procurement decision.

19.1 · The re-validation triggers#

A previously-validated link should be re-validated when any of the following occurs:

19.2 · The re-validation protocol#

Lighter than full bring-up but heavier than a routine telemetry read. The re-validation procedure:

• Read DOM and VDM; compare to validation-record baseline
• Run 5 minutes of line-rate traffic; verify clean FEC and interface counters
• Compute pre-FEC BER; compare to baseline (significant drift is the signal)
• Read laser bias; compare to baseline (rise indicates aging)
• Update the validation record with the re-validation timestamp and results

Total time per link: roughly 10 minutes including the 5-minute traffic window. Multiplied across a 5% annual sample of a 5,000-module fleet, that's 250 modules × 10 minutes = ~42 hours per year of engineering time. Easily affordable for the assurance it provides.

19.3 · The retirement decision#

Modules don't last forever. Some retire because they failed; others retire because they're approaching end-of-life and proactive replacement is cheaper than reactive failure response. The decision factors:

19.4 · The lifetime tracking record#

The validation record from Chapter 15, augmented with periodic re-validation entries and ultimately a retirement entry, becomes the lifetime record per module. A representative complete lifetime record:

serial: VTX20260512001234
vendor: Vitex
part_number: VO-8CSR8CM-AA
lifecycle:
  - event: factory_test
    date: "2026-04-15"
    reference: "VTX-FR-202604-N1234.pdf"
  - event: receiving_inspection
    date: "2026-05-09"
    pass: true
  - event: first_article_qualification_member
    date: "2026-05-10"
    qualification_lot: "Q2-2026-Vitex-800G-DR8"
    pass: true
  - event: bring_up_validation
    date: "2026-05-12"
    deployment: "rack42-leaf01:Eth1"
    burn_in_hours: 72
    pass: true
  - event: deployed
    date: "2026-05-15"
  - event: re_validation
    date: "2026-11-12"
    trigger: "firmware_update_v1.3.0"
    findings: "ber_baseline_unchanged"
  - event: re_validation
    date: "2027-05-10"
    trigger: "annual_periodic"
    findings: "bias_+8%_from_baseline_normal"
  - event: re_validation
    date: "2028-05-09"
    trigger: "annual_periodic"
    findings: "bias_+18%_flag_for_retirement_planning"
  - event: proactive_retirement
    date: "2028-09-15"
    reason: "bias_aging_trajectory"
    final_state_snapshot: "snapshot_20280915.yaml"
    disposition: "vendor_FFA_program"

This record persists for the module's lifetime. When the module is finally returned to the vendor for FFA, the record is the evidence package that accompanies it. When the same vendor comes up for procurement renewal, the records of their fielded modules are part of the evaluation.

19.5 · RMA and FFA — the distinction#

RMA — Return Merchandise Authorization. The administrative process for returning a failed unit to the vendor. Vendor issues a return authorization, you ship the unit back with the documentation, vendor sends a replacement (or credit) and tests the returned unit. Most vendor support agreements include RMA SLAs measured in days for replacement turnaround.

FFA — Field Failure Analysis. The technical process the vendor uses to determine why the returned unit failed. The deliverable is a written report — the FFA report — documenting root cause. Some vendors do this routinely as part of the RMA flow; others do it only on request; others charge for it.

An operator who only does RMAs is replacing modules without learning. An operator who pursues FFA on a sample of returns is building knowledge: which failure modes are common, which are batch-correlated, which expose vendor process issues. Specify FFA as a contractual deliverable on at least 5–10% of RMAs. Some vendors include this; others charge a few hundred dollars per FFA report; few refuse it outright.

19.6 · The RMA evidence package#

An RMA backed by evidence is processed faster, taken more seriously, and produces better FFA reports than an RMA backed by "this one's broken." The evidence package per returned module:

• The validation record from bring-up — original factory test report reference, acceptance baseline, validation results
• The lifetime telemetry record — DOM trends, FEC trends, link state history
• The triage findings — what was investigated, what was ruled out, what evidence pointed to module fault
• The deployment context — switch, port, far-end module, recent maintenance, ambient conditions
• The failure description — what happened, when, what the symptoms were
• (If available) the final DOM and VDM snapshot taken before extraction

The evidence package gives the vendor what they need to do real FFA: a complete history of the module from manufacture to failure. Without it, the vendor's FFA capacity is bench testing — the module on a tester, looking at it cold. With it, the FFA finding feeds back into the lifetime record above and into the procurement decision next time around.

Quick reference — pass/fail thresholds at a glance#

The threshold values referenced throughout the handbook, consolidated in one place. Indicative for an 800G OSFP IHS module under typical AI-fabric conditions; specific vendor & SKU thresholds may vary. Always baseline at install and alert on excursion from baseline rather than against absolute thresholds.

20.1 · DOM thresholds (per lane)#

20.2 · FEC & counter thresholds#

20.3 · Sample-testing pass/fail#

20.4 · Fabric-readiness gates#

Standards and references#

The handbook draws on a small set of standards and operator-published data. The list below is intentionally short — these are the documents an operator references in practice, not the complete bibliography of optical networking.

21.1 · The standards that matter operationally#

21.2 · Operator-published reliability data we reference#

21.3 · Glossary and acronyms#

See Chapter 22 for the operationally relevant glossary.

Glossary#

Terms used throughout the handbook. Definitions are operational rather than rigorous — where the rigorous version differs, that's noted.

Sources and references#

A short list of the data and documents cited throughout this handbook. Operator-published reliability data is the most useful body of evidence on optical-link AFR; vendor MTBF claims and academic papers fill in around it.

23.1 · Operator-published reliability data#

Tian, Z. et al. Alibaba HPN: A Data Center Network for Large Language Model Training. SIGCOMM 2024. (Source of the 0.68% per-access-link AFR figure.)

Roy, A. et al. Inside the Social Network's (Datacenter) Network. ACM SIGCOMM 2015. Microsoft optical reliability follow-on papers, 2022–2023. (Source of the optical-vs-copper AFR ratio framing.)

Meta AI. The Llama 3 Herd of Models. 2024. (Table 5 — training interruption attribution data, including network and cable.)

23.2 · Standards bodies and platform documentation#

IEEE 802.3 (multiple amendments through 2024) — Ethernet physical layer specifications. OIF CMIS 5.x — Common Management Interface Specification. OCP Optics Telemetry Specification, current revision. IEC 61300-3-35 — fiber endface inspection. NVIDIA LinkX (current revision), Cisco Transceiver Module Group, Juniper Hardware Compatibility, Arista Transceiver Optical Interoperability — current platform compatibility documents.

23.3 · A note on data sources#

The reliability and failure-mode data in this handbook is aggregated from: vendor failure-field-analysis reports we have visibility into; operator-published papers cited above; conversations with operators across Tier-2 AI clusters, neoclouds, and system integrators over the past 24 months. Specific numerical claims (e.g., "35% of failures are connector-related") should be read as indicative orders of magnitude — operator fleets vary substantially with workload, vendor mix, and deployment vintage. The methodologies in this handbook are the most directly transferable content; the numerical context exists to motivate them.