Home > Tech Gallery > Technology Feast | A Brief Discussion on 100G Optical Modules in Data Centers
Time: August 27th, 2024
Traditional data centers are mainly based on a 10G network architecture. In order to adapt to the large-scale deployment of services such as AI, deep learning, and big data computing, the next-generation data center architecture is evolving towards 25G/100G network architecture. In China, Internet giants such as BAT have already achieved large-scale deployment.
Building a 25G/100G data center requires a large number of 100G optical modules, which account for a high proportion of the network construction cost. What are the 100G optical module standards and how should we choose? Today, we will briefly sort out the 100G optical module standards and packaging formats for data centers.
Building a 25G/100G data center requires a large number of 100G optical modules, which account for a high proportion of the network construction cost. What are the 100G optical module standards and how should we choose? Today, we will briefly sort out the 100G optical module standards and packaging formats for data centers.
100G optical module standard organization
Before we start sharing the optical module standards, let's first understand the standardization organizations of optical modules. The definition of optical modules is mainly based on two key organizations, namely IEEE and MSA (Multi Source Agreement), which complement and learn from each other.
Everyone knows that IEEE is the Institute of Electrical and Electronics Engineers, and 802.3 is a working group under IEEE. Many 10G, 40G, 100G, and 400G optical module standards are proposed by the IEEE 802.3 working group.
MSA is a multi-vendor specification. Compared with IEEE, it is a non-official organization. Different MSA protocols will be formed for different optical module standards, which can be understood as the behavior of the industry's corporate alliance. In addition to defining the structural packaging of optical modules (including external dimensions, electrical connectors, pin assignments, etc.), MSA also defines electrical interfaces and optical interfaces, thus forming a complete optical module standard.
A long time ago, the optical module industry chain was very chaotic. Each manufacturer had its structural packaging, and the optical modules they developed were large and small, and the interfaces were also varied. In order to solve this problem, the MSA multi-source agreement came into being. All manufacturers followed the standards proposed by MSA to unify the structural packaging and related interfaces of optical modules, like the standardization of mobile phone charging ports. For 100G, the standards defined by MSA include 100G PSM4 MSA, 100G CWDM4 MSA, and 100G Lambda MSA.
Everyone knows that IEEE is the Institute of Electrical and Electronics Engineers, and 802.3 is a working group under IEEE. Many 10G, 40G, 100G, and 400G optical module standards are proposed by the IEEE 802.3 working group.
MSA is a multi-vendor specification. Compared with IEEE, it is a non-official organization. Different MSA protocols will be formed for different optical module standards, which can be understood as the behavior of the industry's corporate alliance. In addition to defining the structural packaging of optical modules (including external dimensions, electrical connectors, pin assignments, etc.), MSA also defines electrical interfaces and optical interfaces, thus forming a complete optical module standard.
A long time ago, the optical module industry chain was very chaotic. Each manufacturer had its structural packaging, and the optical modules they developed were large and small, and the interfaces were also varied. In order to solve this problem, the MSA multi-source agreement came into being. All manufacturers followed the standards proposed by MSA to unify the structural packaging and related interfaces of optical modules, like the standardization of mobile phone charging ports. For 100G, the standards defined by MSA include 100G PSM4 MSA, 100G CWDM4 MSA, and 100G Lambda MSA.
100G Optical Module Standards
100G interconnection scenarios of different distances, IEEE and MSA have defined more than ten 100G standards, but the mainstream ones are the following six standards.
|
Standard
|
Formulating bodies
|
Connector
|
Type of fiber
|
Transmission distance
|
|
100GBASE SR10
|
IEEE 802.3
|
24core MPO, 10 transmitter and 10 receiver
|
Multimode fiber, 850 nm
|
OM3 100M
OM4 150m
|
|
100GBASE-SR4
|
IEEE 802.3
|
12-core MPO, 4 transmitter and 4 receiver
|
Multimode fiber, 850 nm
|
OM4 100m
|
|
100GBASE-LR4
|
IEEE 802.3
|
Duplex Lc, 1 transmitter and 1 receiver
|
single mode fiber 1295.56~1309.14nm
|
10KM
|
|
100GBASE-ER4
|
IEEE 802.3
|
Duplex LC, 1 transmitter and 1receiver
|
single mode fiber 1295.56~1309.14nm
|
40KM
|
|
100G PSM4
|
MSA
|
12-core MPO, 4 transmitter and 4receiver
|
single mode fiber 1310nm
|
500m
|
|
100G CWDM4
|
MSA
|
Duplex LC, 1 transmitter and 1 receiver
|
single mode fiber 1271nm-1331nm
|
2KM
|
▲ Table 1- Mainstream standards for 100G optical modules
The standards starting with 100GBASE are all proposed by IEEE 802.3.
▲FIG-1
As shown in the figure above: In 100GBASE-LR4, LR stands for long reach, i.e. 10 km, and 4 stands for four channels, i.e. 4*25G, which together form a 100G optical module that can transmit 10 km.
The naming convention for - R is as follows:
The naming convention for - R is as follows:
|
PMD Type
|
Transmission distance
|
Remarks
|
|
KR
|
Tens of centimeters
|
k, backplane, signal transmission between backplanes
|
|
CR
|
A few meters
|
C, copper, coaxial cable connection
|
|
SR
|
Tens of meters
|
S，short， Short distance, high-speed optical modules generally use multimode optical fibers
|
|
DR
|
500 m
|
D,
Note: PSM4 has a transmission range of 500 meters, but it does not belong to the IEEE standard system
|
|
FR
|
2 km
|
F,
Note: 100 G CWD4 is also 2 km, but it is not an IEEE standard, it is defined by the MSA.
|
|
LR
|
10 km
|
L, long, long distance
|
|
ER
|
40 km
|
E, extended distance, extended relative to LR
|
|
ZR
|
80 km
|
Non IEEE Standard
|
▲ Table 2- R Glossary (Noun Explanation)
In addition to the 100GBASE series standards proposed by IEEE, why did MSA propose the PSM4 and CWDM4 standards?
100GBASE-SR4 and 100GBASE-LR4 are defined by IEEE Commonly used 100G interface specifications. However, for the internal interconnection scenarios of large data centers, the distance supported by 100GBASE-SR4 is too short to meet all interconnection requirements, while the cost of 100GBASE-LR4 is too high. Therefore, MSA brought medium-distance interconnection solutions to the market, and PSM4 and CWDM4 are the products of this revolution.
Of course, the capabilities of 100GBASE-LR4 completely cover CWDM4, but in the 2km transmission scenario, the CWDM4 solution is lower in cost and more competitive.
100GBASE-SR4 and 100GBASE-LR4 are defined by IEEE Commonly used 100G interface specifications. However, for the internal interconnection scenarios of large data centers, the distance supported by 100GBASE-SR4 is too short to meet all interconnection requirements, while the cost of 100GBASE-LR4 is too high. Therefore, MSA brought medium-distance interconnection solutions to the market, and PSM4 and CWDM4 are the products of this revolution.
Of course, the capabilities of 100GBASE-LR4 completely cover CWDM4, but in the 2km transmission scenario, the CWDM4 solution is lower in cost and more competitive.
The following figure is the schematic diagram of 100GBASE-LR4 and 100G CWDM4:
▲ FIG 2- 100GBASE-LR4 schematic diagram
▲ FIG 3- 100G CWDM4 schematic diagram
LR4 and CWDM4 are similar in principle, both use optical devices MUX and DEMUX to multiplex four parallel 25G channels into one 100G optical fiber link. However, there are several differences between the two:
1. MUX/DEMUX devices used in LR4 are more expensive
1. MUX/DEMUX devices used in LR4 are more expensive
CWDM4 defines a CWDM interval of 20nm. Since the wavelength temperature drift characteristic of the laser is approximately 0.08nm/°C, the wavelength variation within the working range of 0~70°C is approximately 5.6nm, and some isolation bands must be left in the channel itself.
Channel 1: 1264.5~1277.5nm
Channel 2: 1284.5~1297.5nm
Channel 3: 1304.5~1317.5nm
Channel 4: 1324.5~1337.5nm
LR4 defines a LAN-WDM spacing of 4.5nm.
LR4 defines a LAN-WDM spacing of 4.5nm.
Channel 1: 1294.53~1296.59nm
Channel 2: 1299.02~1301.09nm
Channel 3: 1303.54~1305.63nm
Channel 4: 1308.09~1310.19nm
The lower the requirements for optical MUX/DEMUX devices, which can save costs.
2. The laser used in LR4 is more expensive and consumes more power
The lower the requirements for optical MUX/DEMUX devices, which can save costs.
2. The laser used in LR4 is more expensive and consumes more power
CWDM4 uses DML (Direct Modulated Laser), while LR4 uses EML (Electro- absorption Modulated Laser).
DML is a single laser, while EML is two devices, one is DML and the other is an EAM modulator, together they are called EML. The principle of DML is to achieve signal modulation by modulating the injection current of the laser. Since the injection current will change the refractive index of the laser active area, causing wavelength drift (chirp) and thus dispersion, it is difficult to perform high-speed signal modulation and the transmission is not far enough. 10KM is a bit beyond the capability of DML, so EML is the only choice.
Note: Chirp refers to a signal whose frequency changes (increases or decreases) over time. This signal sounds similar to the chirping of birds.
3. LR4 requires an additional TEC (Thermo Electric Cooler)
DML is a single laser, while EML is two devices, one is DML and the other is an EAM modulator, together they are called EML. The principle of DML is to achieve signal modulation by modulating the injection current of the laser. Since the injection current will change the refractive index of the laser active area, causing wavelength drift (chirp) and thus dispersion, it is difficult to perform high-speed signal modulation and the transmission is not far enough. 10KM is a bit beyond the capability of DML, so EML is the only choice.
Note: Chirp refers to a signal whose frequency changes (increases or decreases) over time. This signal sounds similar to the chirping of birds.
3. LR4 requires an additional TEC (Thermo Electric Cooler)
Because the interval between adjacent channels of LR4 is only 4.5nm, the laser needs to be placed on the TEC for temperature control. The TEC Driver chip needs to be placed on the circuit, and the laser also needs to be integrated into the TEC material. In this way, the cost of LR4 is increased compared to CWDM4.
Based on the above three points, the cost of optical modules of the 100GBASE-LR4 standard is higher, so the 100G CWDM4 standard proposed by MSA fills the gap caused by the high cost of 100GBASE-LR4 within 2 km.
In addition to CWDM4, PSM4 is also a medium-distance transmission solution. So what are the advantages and disadvantages of PSM4 compared to CWDM4?
The 100G PSM4 specification defines a point-to-point 100 Gbps link over 8 single-mode fibers (4 transmit and 4 receive), with each channel transmitting at 25 Gbps. Each signal direction uses four independent channels of the same wavelength. Therefore, two transceivers typically communicate via an 8- 8-fiber MTP/MPO single-mode patch cord. The transmission distance of PSM4 is up to 500 meters.
Based on the above three points, the cost of optical modules of the 100GBASE-LR4 standard is higher, so the 100G CWDM4 standard proposed by MSA fills the gap caused by the high cost of 100GBASE-LR4 within 2 km.
In addition to CWDM4, PSM4 is also a medium-distance transmission solution. So what are the advantages and disadvantages of PSM4 compared to CWDM4?
The 100G PSM4 specification defines a point-to-point 100 Gbps link over 8 single-mode fibers (4 transmit and 4 receive), with each channel transmitting at 25 Gbps. Each signal direction uses four independent channels of the same wavelength. Therefore, two transceivers typically communicate via an 8- 8-fiber MTP/MPO single-mode patch cord. The transmission distance of PSM4 is up to 500 meters.
▲ FIG 4- PSM4 schematic diagram
To summarize, as shown in thetablebelow, due to the use of wavelength division multiplexers, the cost of CWDM4 optical modules is higher than that of PSM4 optical modules. However, CWDM4 only needs two single-mode fibers for bidirectional transmission and reception, which is far less than the eight single-mode fibers of PSM4. As the distance increases, the total cost of the PSM4 solution rises very quickly. In actual applications, it is necessary to decide whether to use PSM4 or CWDM4 based on the interconnection distance.
|
CWDM4 optical module
|
PSM4 optical module
|
|
|
Optical emitter
|
Four CWDM directly modulated lasers (with a wavelength interval of 20nm)
|
Four integrated silicon photon modulators and one distributed feedback laser.
|
|
Four wavelength CWDM wavelength division multiplexer
|
required
|
Not required
|
|
Interface type
|
Duplex LC interface
|
MPO/MTP interface (8-core)
|
|
Distance
|
< 2 KM
|
< 500 M
|
▲Table 4-CWDM4 vs PSM4
After talking about the 100G medium and long-distance optical module standards, let's take a look at the 100G short-distance optical module.
100G short-distance optical module standards: 100GBASE-SR10 and 100GBASE-SR4. In order to meet the 100G demand in the market, the 100GBASE-SR10 standard was first proposed and applied to 100G short-distance interconnection.
100GBASE-SR10 standard uses 10 x 10Gbps parallel channels to achieve 100Gbps point-to-point transmission. The rate of electrical signals is 10G, and the rate of optical signals is also 10G. It uses NRZ modulation and 64B/66B encoding. Because IEEE 802.3 proposed the 100GBASE-SR10 standard as early as 2010, the electrical interface of the switch ASIC chip ( Application Specific Integrated Circuit ) could only support 10G at most, that is, CAUI-10 ( 10 channels x 10Gbps ).
100G short-distance optical module standards: 100GBASE-SR10 and 100GBASE-SR4. In order to meet the 100G demand in the market, the 100GBASE-SR10 standard was first proposed and applied to 100G short-distance interconnection.
100GBASE-SR10 standard uses 10 x 10Gbps parallel channels to achieve 100Gbps point-to-point transmission. The rate of electrical signals is 10G, and the rate of optical signals is also 10G. It uses NRZ modulation and 64B/66B encoding. Because IEEE 802.3 proposed the 100GBASE-SR10 standard as early as 2010, the electrical interface of the switch ASIC chip ( Application Specific Integrated Circuit ) could only support 10G at most, that is, CAUI-10 ( 10 channels x 10Gbps ).
▲ FIG 5- 100GBASE-SR10 schematic diagram
As the electrical interface rate of the switch ASIC chip increases from 10Gpbs to 25G bps, the electrical interface standard is upgraded from CAUI-10 (10 channels x 10Gbps) to CAUI-4 (4 channels x 25Gbps), and the channels are reduced from SR10’s 10 parallel channels to 4 parallel channels, the number of optical module components is reduced, the cost is reduced, the module size is reduced, and the power consumption is reduced.
The reduction in the size of optical modules allows switches to provide a higher density of 100G interfaces per 1U space. Based on the above advantages, 100GBASE-SR4 has replaced 100GBASE-SR10 to become the mainstream 100G short-distance optical module standard.
▲ FIG 6- 100GBASE-SR4 schematic diagram
100G optical module packaging
It is not enough to have only the optical interface and electrical interface specifications of the optical module. A complete optical module solution also requires a matching structural package. The packaging formats of 100G optical modules mainly include CFP, CFP2, CFP4, and QSFP28.
CFP was first proposed, and the 100GBASE-SR10 standard was used for short-distance transmission, and 100GBASE-LR4 was used for long-distance transmission. The first-generation CFP long-distance transmission solution is as follows. Because the electrical interface capacity is only CAUI-10, a built-in gearbox (10:4 serialized in the figure below) is required to achieve the conversion of 10 x 10Gbps and 4 x 25Gbps electrical signals. Later, as the electrical signal was upgraded to CAUI-4, the second-generation CFP (CFP2/CFP4) long-distance transmission solution did not require a built-in gearbox.
CFP was first proposed, and the 100GBASE-SR10 standard was used for short-distance transmission, and 100GBASE-LR4 was used for long-distance transmission. The first-generation CFP long-distance transmission solution is as follows. Because the electrical interface capacity is only CAUI-10, a built-in gearbox (10:4 serialized in the figure below) is required to achieve the conversion of 10 x 10Gbps and 4 x 25Gbps electrical signals. Later, as the electrical signal was upgraded to CAUI-4, the second-generation CFP (CFP2/CFP4) long-distance transmission solution did not require a built-in gearbox.
▲ FIG 7- The first generation of CFP optical module long-distance solution
However, the size of CFP is too large. As the integration of optical modules becomes higher and higher, the subsequent development direction is to reduce the size and power consumption. CFP has evolved to CFP2, CFP4, and then to QSFP28. Compared with CFP4, QSFP28 is smaller in size and consumes less power. The smaller size of QSFP28 enables the switch to have a higher port density (typically, each board can deploy 36 100G interfaces). Currently, QSFP28 is the mainstream packaging format for 100G optical modules in data centers.
▲ FIG 8- CFP/CFP2/CFP4/QSFP28 optical module size comparison
To summarize, regarding how to choose 25G/100G data center internal interconnect optical modules, it is recommended that you refer to the following standards:
● 100G short-reach interconnection scenarios (TOR-LEAF) of no more than 100 meters use 100GBASE-SR4 QSFP28 optical modules;
● 100G mid-range interconnection scenario (LEAF-SPINE) of 100m to 500m, using 100G PSM4 QSFP28 optical module;
● 100G medium- and long-distance interconnection scenarios ( LEAF-SPINE, SPINE-CORE ) from 500m to 2km, using 100G CWDM4 QSFP28 optical modules ;
● Long-distance interconnection scenarios exceeding 2 km (CORE-MAN), use 100GBASE-LR4 QSFP28 optical modules.
The following table 5 presents an explanation of professional terms.
|
Optical communication terminology
|
Noun Explanation
|
Full name
|
|
100G Lambda MSA
|
Single channel 100Gbps optical interface multi-source protocol
|
100G Lambda Multi-Source Agreement
|
|
ASIC
|
Application specific integrated circuit
|
Application Specific Integrated Circuit
|
|
CAUI
|
100G Ethernet electrical interface
|
100Gbps Attachment Unit Interface
|
|
CDR
|
Clock data recovery
|
Clock Data Recovery
|
|
CFP
|
100G pluggable optical module
|
Centum Form-factor Pluggable
|
|
CWDM4
|
Four channel coarse wavelength division multiplexing
|
Coarse Wavelength Division Multiplexing 4
|
|
DeMux
|
Optical DE multiplexer (splitter)
|
DE multiplexer
|
|
DML
|
Direct modulation laser
|
Direct Modulated Laser
|
|
EAM
|
Electro absorption modulator
|
Electro Absorption Modulator
|
|
EML
|
Electro absorption modulated laser
|
Electro-absorption Modulated Laser
|
|
IEEE
|
Institute of Electrical and Electronics Engineers
|
Institute of Electrical and Electronics Engineer
|
|
LAN-WDM
|
(Local Area Network) Wavelength Division Multiplexing (WDM)
|
(Local Area Network) Wavelength Division Multiplexing
|
|
MD
|
Monitoring photodiodes
|
Monitor Diode
|
|
MMF
|
Multimode fiber
|
Multi-Mode Fiber
|
|
MSA
|
Multi source protocol
|
Multi Source Agreement
|
|
Mux
|
Optical multiplexer (multiplexer)
|
Multiplexer
|
|
NRZ
|
Non zeroing
|
Non Return Zero
|
|
PSM4
|
Four parallel single-mode channels
|
Parallel Single Mode 4 lane
|
|
QSFP28
|
Four channel small pluggable optical module
|
Quad Small Form-factor Pluggable 28
|
|
SMF
|
Single-mode fiber
|
Single-Mode Fiber
|
|
TEC
|
Semiconductor thermoelectric cooler
|
Thermo Electric Cooler
|
|
TIA
|
Trans-impedance amplifier
|
Trans-impedance Amplifier
|
▲Table 5
Summary
This document provides an overview of 100G optical modules in data centers. It discusses the evolution of data center architecture towards 25G/100G networks to meet the demands of AI, deep learning, and big data computing. The document explains the standardization organizations for optical modules, including IEEE and MSA, and highlights the different standards proposed by these organizations. It also compares various 100G optical module standards such as CWDM4, PSM4, LR4, and SR4 based on their transmission distances and cost considerations. Additionally, it discusses the packaging formats for 100G optical modules: CFP, CFP2/CFP4, and QSFP28.
Related Blog:
Exploration of Data Center Automated Operation and Maintenance Technology: Zero Configuration of Switches
Technology Feast | How to De-Stack Data Center Network Architecture
Related Blog:
Exploration of Data Center Automated Operation and Maintenance Technology: Zero Configuration of Switches
Technology Feast | How to De-Stack Data Center Network Architecture
Featured blogs
- Network Basics: Access point vs Bridge Mode | Key Differences
- Network 101: Difference between Wired LAN and Wireless LAN
- Educational Technology Network Solution with University AP for Smart Campus
- Technology Feast | How to De-Stack Data Center Network Architecture
- Technology Feast | Internet Data Center Network 25G Network Architecture Design
- Digital Innovation in Education: Ruijie's Optical Ethernet 3.0 Solution
- Technical Feast | A Brief Analysis of MMU Waterline Settings in RDMA Network
- What Is Wi-Fi 7?
- Technology Feast: Routing Protocol Selection for Large Data Centre Networks
- Technology Feast | The "Giant Sword" of Data Center Network Operation and Maintenance
