Parallel Sysplex

Overview

Parallel Sysplex InfiniBand (PSIFB) links are the latest Coupling Facility links. They were announced in February 26, 2008 in US Hardware Announcement 108-154, “IBM System z10 Enterprise Class – The forward-thinking mainframe for the twenty-first century” as well as in the May 6, 2008 US Hardware Announcement 108-269, “IBM System z10 Enterprise Class: Helping to meet global 24x7 demands for information services with improvements for Internet access and coupling,” IFB3 protocol is described in the July 12, 2011 hardware announcement 111-121, “zEnterprise 196 enhancements deliver faster access to data.

There are several ways of classifying IFB links. It can be Double Data Rate (DDR) or Single Data Rate (SDR). The IFB links can also be characterized by the amount of lanes of communication within each physical link. For example, 12x links provide 12 lanes of communication within each physical link. 1x links provide a single lane of communication within each physical link.


Value

There are many reasons for migrating to PSIFB links.

Like other coupling links, external InfiniBand coupling links support time synchronization signals for Server Time Protocol (STP).


Performance

Specifically, a comparison of the effect of the links relative performance can be seen in the table below.

This table assumes:

  1. The CF activity rate to the coupling facility is 9 operations per million instructions, or 9 operations/second per MIPS of the server. This is typical for a high end data sharing environment. If the activity rate is less, then the values shown in the table should be proportionally less.
  2. Heuristic Synchronous to Asynchronous conversion is not taken into account. This would limit the coupling overhead to about 18%.
  3. Shown is data for a non-System-Managed duplexing environment.
Host z10 BC z10 EC z114 z196 zBC12 zEC12      
CF                  
z10 BC ISC3 16 18 17 21 19 24      
z10 BC 1x IFB 13 14 14 17 18 19      
z10 BC 12x IFB 12 13 13 15 15 17      
z10 BC ICB4 10 11 - - - -      
z10 EC ISC3 16 17 17 21 19 24      
z10 EC 1x IFB 13 14 14 17 17 19      
z10 EC 12x IFB 11 12 12 14 14 16      
z10 EC ICB4 10 10 - - - -      
z114 ISC3 16 18 17 21 19 24      
z114 1x IFB 13 14 14 17 17 19      
z114 12x IFB 12 13 12 15 15 17      
z114 12x IFB3 - - 10 12 12 13      
z196 ISC3 16 17 17 21 19 24      
z196 1x IFB 13 14 13 16 16 18      
z196 12x IFB 11 12 11 14 14 15      
z196 12x IFB3 - - 9 11 10 12      
zBC12 ISC3 16 17 17 21 19 24      
zBC12 1x IFB 14 15 14 18 17 20      
zBC12 12x IFB 13 13 12 15 14 17      
zBC12 12x IFB3 - - 10 11 11 12      
zEC12 ISC3 16 17 17 21 19 24      
zEC12 1x IFB 13 13 13 16 16 18      
zEC12 12x IFB 11 11 11 13 13 15      
zEC12 12x IFB3 - - 9 10 10 11      
                   

Configurations

Coupling Link types

Coupling links provide a path to transmit/receive Coupling Facility (CF) data as well as Server Time Protocol (STP) timekeeping messages. The CF data may be exchanged between z/OS and the CF or between CFs. For a zEnterprise (z114, z196), the coupling options are:

Note 1: The InfiniBand link data rates do not represent the performance of the link. The actual performance is dependent upon many factors including latency through the adapters, cable lengths, and the type of workload. Systems Lab Services can assist your migration to PSIFB coupling links by providing services to assess the impact of the migration or to assist with the implementation of the migration.

Note 2: The list of qualified DWDM vendors is available on Resource Link, located under the “Hardware products for server” on the Library page.)

The support types of IFB links for the System z servers are:

Link 1x IFB 12x IFB
System z9 N Y
System z10 Y Y
zEnterprise Y Y


Number of links 

  #fanouts 1x IFBs (4 ports / fanout) (HCA3-O LR)
Max 12 fanouts *
12x IFB(2 ports / fanout)
Max 16 fanouts *
ISC-3 (spread over 2 fanouts) IC
z114          
M05
(1 drawer)
4 16 8 48 32
M10
(2 drawer)
8 32 16 48 32
z196          
M15
(1 book)
8 32 16 48 32
M32
(2 book)
16 48 32 48 32
M49
(3 book)
20 48 32 48 32
M66
(4 book)
24 48 32 48 32
M80
(4 book)
24 48 32 48 32
zEC12          
H20 8 32 16 48 32
H43 16 64 32 48 32
H66 20 64 32 48 32
H89 24 64 32 48 32
HA1 24 64 32 48 32
zBC12          
H06 4 16 8 32 32
H13 4 32 16 32 32

* Maximum 16 HCA-O fanouts per server. 

Each fanout can support:

In addition, up to 48 ISC-3 links can be configured spread over two fanouts. 

For example:
The maximum number of possible external CF links on a z114 M10 is:
48 ISC-3 + 24 1x IFB3s (6 fanouts) = 72

The maximum number of possible external CF links on an M80 is:
48 ISC-3 + 48 1x IFB3s (12 fanouts) + eight 12x IFB3s (4 fanouts) = 104 links


More information

More information regarding Parallel Sysplex using InfiniBand can be obtained from:

Redbooks

White Paper

Contact IBM

Browse System z


Hardware

IBM provides world-class IBM mainframe technology to help today's enterprises respond quickly to evolving business conditions and with extreme flexibility. From automation to advanced virtualization technologies and open industry standards, IBM mainframes help deliver competitive advantages for enterprises contributing and succeeding on a smarter planet.

Operating Systems

IBM System z supports multiple operation systems:

Solutions

IBM's technology, solutions and industry expertise can help you find the competitive edge with a sharper understanding of your customers. Our System z solutions combine the foundation of IBM hardware, software and middleware with flexible financing and packaging options to help your business meet and overcome the challenges of doing business in the on demand world. IBM can help you develop a customer-centric view—and assist you in delivering the right solution and the right products.


Server Time Protocol

Time Synchronization for the Next Generation