Parallel Sysplex

Overview

Parallel Sysplex InfiniBand (PSIFB) links are the latest Coupling Facility links. They were announced in February 26, 2008 in US Hardware Announcement 108-154, “IBM System z10 Enterprise Class – The forward-thinking mainframe for the twenty-first century” as well as in the May 6, 2008 US Hardware Announcement 108-269, “IBM System z10 Enterprise Class: Helping to meet global 24x7 demands for information services with improvements for Internet access and coupling,” IFB3 protocol is described in the July 12, 2011 hardware announcement 111-121, “zEnterprise 196 enhancements deliver faster access to data.”

There are several ways of classifying IFB links. It can be Double Data Rate (DDR) or Single Data Rate (SDR). The IFB links can also be characterized by the amount of lanes of communication within each physical link. For example, 12x links provide 12 lanes of communication within each physical link. 1x links provide a single lane of communication within each physical link.

Value

There are many reasons for migrating to PSIFB links.

• The 12x links are supported up to 150 meters. This extends high speed coupling beyond the 10 meters supported by ICB-4 links.
• The 1x links are supported up to 10 kilometers (km) unrepeated. Further distances are supported with qualified DWDM vendors. The list of qualified venders can be found on ResourceLink
• Up to 128 coupling CHPIDs per server
• Up to 32 subchannels (devices) per IFB link

Like other coupling links, external InfiniBand coupling links support time synchronization signals for Server Time Protocol (STP).

Performance

Specifically, a comparison of the effect of the links relative performance can be seen in the table below.

This table assumes:

1. The CF activity rate to the coupling facility is 9 operations per million instructions, or 9 operations/second per MIPS of the server. This is typical for a high end data sharing environment. If the activity rate is less, then the values shown in the table should be proportionally less.
2. Heuristic Synchronous to Asynchronous conversion is not taken into account. This would limit the coupling overhead to about 18%.
3. Shown is data for a non-System-Managed duplexing environment.

Host	z10 BC	z10 EC	z114	z196	zEC12
CF
z10 BC ISC	16	18	17	21	24
z10 BC 1x IFB	13	14	14	17	19
z10 BC 12x IFB	12	13	13	15	17
z10 BC ICB4	10	11	-	-	-
z10 EC ISC	16	17	17	21	24
z10 EC 1x IFB	13	14	14	17	19
z10 EC 12x IFB	11	12	12	14	16
z10 EC ICB	10	10	-	-	-
z114 ISC3	16	18	17	21	24
z114 1x IFB	13	14	14	17	19
z114 12x IFB	12	13	12	15	17
z114 12x IFB3	-	-	10	12	13
z196 ISC3	16	17	17	21	24
z196 1x IFB	13	14	13	16	18
z196 12x IFB	11	12	11	14	15
z196 12x IFB3	-	-	9	11	12
zEC12 ISC3	16	17	17	21	24
zEC12 1x IFB	13	13	13	16	18
zEC12 12x IFB	11	11	11	13	15
zEC12 12x IFB3	9	9	9	10	11

Configurations

Coupling Link types

Coupling links provide a path to transmit/receive Coupling Facility (CF) data as well as Server Time Protocol (STP) timekeeping messages. The CF data may be exchanged between z/OS and the CF or between CFs. For a zEnterprise (z114, z196), the coupling options are:

• Internal coupling links (ICs) can be used for internal communication between Coupling Facilities (CFs) defined in LPARs and z/OS images on the same server.
• InterSystem Channel-3 (ISC-3) links continue to be offered as a migration aid
  • Up to 2 Gbps (200 MBps)
  • Unrepeated distance up to 10 km, up to 20 km with RPQ 8P2197 (at 1 Gbps)
  • Repeated distance with DWMs up to 100 km (62 miles). RPQ 8P2340 is required for repeated fiber distances in excess of 100 km.
  The z114 and z196 are the last servers to offer ordering of ISC-3
• 12x InfiniBand
  • Uses HCA2-O or HCA3-O fanout features
  • Up to 6 GBps (Note 1)
  • Up to 150 meters (492 feet)
  • Up to 16 CHPIDs across two ports
  • IFB3 protocol used when connected to a zEnterprise at driver 93 or later with four or less CHPIDs per port
• 1x InfiniBand
  • Uses HCA2-O LR or HCA3-O LR fanout features
  • Up to 5 Gbps (500 MBps) (Note 1). Either 2.5 or 5 Gbps when attached to a supported DWDM.
  • Unrepeated distance up to 10 km, up to 20 km with RPQ 8P2197
  • Repeated distance with DWMs (Note 2) up 10 100 km (62 miles). RPQ 8P2340 is required for repeated fiber distances in excess of 100 km.
  • Up to 16 CHPIDs across two ports with HCA2-O LR fanout feature
  • Up to 16 CHPIDs across four ports with HCA3-O LR fanout feature
  • Up to 32 subchannels (devices) per CHPID

Note 1: The InfiniBand link data rates do not represent the performance of the link. The actual performance is dependent upon many factors including latency through the adapters, cable lengths, and the type of workload. Systems Lab Services can assist your migration to PSIFB coupling links by providing services to assess the impact of the migration or to assist with the implementation of the migration.

Note 2: The list of qualified DWDM vendors is available on Resource Link, located under the “Hardware products for server” on the Library page.)

The support types of IFB links for the System z servers are:

Link	1x IFB	12x IFB
System z9	N	Y
System z10	Y	Y
zEnterprise	Y	Y

Number of links 

	#fanouts	1x IFBs (4 ports / fanout) (HCA3-O LR) Max 12 fanouts *	12x IFB(2 ports / fanout) Max 16 fanouts *	ISC-3 (spread over 2 fanouts)	IC
z114
M05 (1 drawer)	4	16	8	48	32
M10 (2 drawer)	8	32	16	48	32
z196
M15 (1 book)	8	32	16	48	32
M32 (2 book)	16	48	32	48	32
M49 (3 book)	20	48	32	48	32
M66 (4 book)	24	48	32	48	32
M80 (4 book)	24	48	32	48	32
EC12
H20	8	32	16	48	32
H43	16	64	32	48	32
H66	20	64	32	48	32
H89	24	64	32	48	32
HA1	24	64	32	48	32

* Maximum 16 HCA-O fanouts per server. 

Each fanout can support:

• 4 1x IFBs on an HCA3-O LR
• 2 12x IFBs on an HCA3-O

In addition, up to 48 ISC-3 links can be configured spread over two fanouts. 

For example:
The maximum number of possible external CF links on a z114 M10 is:
48 ISC-3 + 24 1x IFB3s (6 fanouts) = 72

The maximum number of possible external CF links on an M80 is:
48 ISC-3 + 48 1x IFB3s (12 fanouts) + eight 12x IFB3s (4 fanouts) = 104 links

More information

More information regarding Parallel Sysplex using InfiniBand can be obtained from:

Redbooks

• Getting Started with InfiniBand on System z10 and System z9
• SG24-5444 IBM System z Connectivity Handbook

White Paper

• Coupling Facility Configuration Options