TCP/IP Network and Fibre Channel Data Links between the Data Centers

The maximum distance between the data centers for this type of configuration is currently limited by the maximum distance supported for the networking type or Fibre Channel link type being used, whichever is shorter.

Ethernet switches can support varying distances for the inter switch link between the data centers, depending upon the type of GBIC and fibre cabling used. Inter switch distances of up to 100 KM are supported in Extended Clusters. Check with the network switch vendor for the distances supported for the inter switch link and the hardware and cabling requirements.

Although FDDI is not commonly used, it is supported in Extended Clusters. FDDI offers a total ring length of 100 kilometers, so this will allow up to 50 kilometers between data centers for both two and three data center solutions using FDDI networks.

There can be a maximum of 500 meters between the Fibre Channel switches in the two data centers if Short wave GBICs are used. This distance can be increased to 10 kilometers by using Long wave Fibre Channel GBICs in the switches. The distance can be increased to 80 kilometers if Finisar (long haul) GBICs are used for the Inter Switch Links (ISL) between the Fibre Channel switches. WDM links can also be used for the connection between the Fibre Channel switches in the two Primary data centers and can provide ISL connections of up to 100 kilometers in length (see below for configuration requirements).

There must be at least two TCP/IP networking links, routed geographically differently between each Primary data center to prevent the “backhoe problem.” The backhoe problem can occur when all cables are routed through a single trench and a tractor on a construction job severs all cables and disables all communications between the data centers. It is allowable to have only a single network link routed from each Primary data center to the Arbitrator data center, however in order to survive the loss of the network link between a Primary data center and the Arbitrator data center, the network routing should be configured so that a Primary data center can also reach the Arbitrator via a route which passes through the other Primary data center.

There must be at least two Fibre Channel Data links, routed geographically differently between each data center. In three data center configurations, no Fibre Channel Data links are required for the Arbitrator data center.

Redundant Fibre Channel switches are required in each data center, unless the switch offers built in redundancy.

Refer to the SWD Streams documents for supported Fibre Channel switches An Extended Fabric license may be required if the inter-switch links (ISL) between the switches is greater than 10 kilometers. For optimum data replication performance, it is suggested to tune the buffer credits properly for the ISL used for data replication between the data centers.

If CVM or CFS is being used and all data replication links are lost between the data centers, but the network links remain functional, it is likely that all mirror copies that the CVM master cannot contact will be detached from the disk group. This means that applications running on nodes which are not in the same data center as the CVM Master node may hang, as their local mirror copies are detached and they cannot reach the remote mirror copies. SLVM handles this scenario differently, since it will arbitrate (during which time all writes to the shared volumes will temporarily hang) and allow only one of the nodes to continue writing to the shared volumes (the writes on the other node will continue to hang until the data replication links are re established).

The maximum distance supported between the data centers for DWDM and CWDM configurations is 100 kilometers.

Both the TCP/IP networking and Fibre Channel Data can go through the same WDM box.

Since WDM hardware is typically designed to be fault tolerant, it is acceptable to use only one WDM box (in each data center) for the links between each data center. However, for the highest availability, it is recommended to have redundant WDM boxes (in each data center) used for the links between each data center. If using a single WDM box for the links between each data center, it must be ensured that no Single Points of Failure (SPOFs) exist for that WDM box, and the redundant standby fiber link feature of the WDM box must be configured. If the WDM box supports multiple active WDM links, that feature can be used instead of the redundant standby feature.

At least two fiber optic links are required between each Primary data center, each fiber link routed geographically differently to prevent the “backhoe problem.” It is allowable to have only a single fiber link routed from each Primary data center to the Arbitrator data center, however in order to survive the loss of a link between a Primary data center and the Arbitrator data center, the network routing should be configured so that a Primary data center can also reach the nodes in the Arbitrator data center via a route passing through the other Primary data center.

The network switches can be 100Base T (TX or FX), 1000Base T (TX or FX), 10 Gigabit Ethernet or FDDI. The connections between the network switches and the WDM boxes must be fiber optic.

Direct Fabric Attach mode must be used for the Fibre Channel switch ports connected to the WDM link. Redundant Fibre Channel switches are required in each data center, unless the switch offers built in redundancy.

Refer to the SWD Streams documents for supported Fibre Channel switches An Extended Fabric license may be required if the ISL link between the switches is greater than 10 kilometers. For optimum data replication performance, it is suggested to tune the buffer credits properly for the inter switch links (ISL) used for data replication between the data centers.

It is also possible to have a combination of separate network links and WDM links used for Fibre Channel data, or WDM links used for networking and Fibre Channel links for data; however it is probably much more cost effective to use the WDM links for both networking and Fibre Channel data.

For this document, Synchronous Optical Network (SONET) an ANSI standard, and Synchronous Digital Hierarchy (SDH) an ITU T standard, can be considered to be roughly equivalent and interchangeable. For brevity, the remainder of this document will refer only to SONET, however applies equally to SDH. SONET can support a point to point or a ring topology. In ring topologies, one working ring handles all data traffic and another protection ring remains on standby. In the event of a failure, SONET includes the capability to automatically detect the failure and transfer control to the protection ring in less than one second. This capability of SONET can be described as a self healing network technology.

Dual cluster lock disks are not supported, which means that Two Data Center configurations are not supported. This is because in some cases, SONET providers cannot guarantee physically distinct paths for the SONET links between the data centers, therefore should a single failure result in a failure of all SONET links between the data centers, a split brain cluster could result with dual cluster lock disks.

The maximum distance supported between the data centers for Extended Clusters using SONET is 100 kilometers.

Both the TCP/IP networking and Fibre Channel Data can go through the same SONET box.

Since SONET hardware is typically designed to be fault tolerant, it is acceptable to use only one SONET box (in each data center) for the links between each data center. However, for the highest availability, it is recommended to have redundant SONET boxes (in each data center) used for the links between each data center. If using a single SONET box for the links between each data center, it must be ensured that no SPOFs exist for that SONET box, and it must support redundant SONET links (either active/standby or active/active).

	NOTE: DWDM requires dark fiber, CWDM can use multimode fiber.

At least two SONET links are required between each Primary data center, each link routed geographically differently to prevent the “backhoe problem.” It is allowable to have only a single link routed from each Primary data center to the Arbitrator data center, however in order to survive the loss of a link between a Primary data center and the Arbitrator data center, the network routing should be configured so that a Primary data center can also reach the nodes in the Arbitrator data center via a route passing through the other Primary data center. If a SONET ring topology is used, it requires dual rings, and each ring should pass through all three data centers.

Fibre Channel expects that the ordering of packets is preserved for the Inter switch links, however ordering is not guaranteed by SONET. Therefore Fibre Channel Gateway / SAN Extension devices are typically used between the Fibre Channel switches and the SONET box to preserve the packet ordering. Redundant Fibre Channel switches are required in each data center, unless the switch offers built in redundancy.

Refer to the SWD Streams documents for supported Fibre Channel switches. An Extended Fabric license may be required if the ISL link between the switches is greater than 10 kilometers. For optimum data replication performance, it is suggested to tune the buffer credits properly for the inter switch links (ISL) used for data replication between the data centers.

It is also possible to have a combination of separate network links and SONET links used for Fibre Channel data, or SONET links used for networking and Fibre Channel links for data; however it is probably much more cost effective to use the SONET links for both networking and Fibre Channel data.