|
The ability to detect and quickly recover from a massive-scale hardware failure is of paramount importance to businesses that make use of real-time data processing systems. General Parallel File System (GPFS) provides a number of features that facilitate the implementation of highly-available GPFS environments capable of withstanding catastrophic hardware failures. By maintaining a redundant replica of the file system's data at another (geographically separated) location, we allow the system to sustain its processing using the secondary replica of the data in the event of a total failure in the prime environment. In this paper, we present an overview of the disaster recovery features available in the 2.2 release of GPFS and provide detailed hands-on guidance on implementing the various types of disaster-tolerant configurations supported in this release.
Overview and terminology
Businesses that depend on real-time data processing systems are vulnerable to a profound negative impact from natural or unnatural disasters such as fires, tornadoes, earthquakes, or power failures. A catastrophe can permanently disable the customer's data processing infrastructure and, without proper backup procedures, result in a permanent loss of business-critical data. To help minimize the impact from such unplanned hardware outages, many businesses are implementing preventive measures enabling them to quickly recover from a disastrous failure and enable the near continual availability of mission-critical applications and the associated data. GPFS 2.2 provides you with a number of features for the support of your disaster recovery solution.
On a very high level, a disaster-resilient GPFS cluster environment is typically made up of two (sometimes three) distinct hardware sites that operate in a coordinated fashion and are separated by a substantial geographic distance. Each site houses some number of GPFS nodes and a storage resource holding a complete replica of the file system. In the event of a catastrophic hardware failure that disables the operation of an entire site, GPFS is designed to failover to the remaining subset of the cluster and continue serving the data using the replica of the file system that survived the disaster. In this paper, we examine several methods for maintaining the secondary replica, which include synchronous mirroring (through the use of IBM TotalStorage® Enterprise Storage Server® (ESS) Peer-to-Peer Remote Copy (PPRC) or logical GPFS replication) along with a non-synchronous approach that utilizes ESS FlashCopy.
Certain high-end storage subsystems such as ESS implement support for volume-level geographic mirroring of data. For example, the PPRC feature of the ESS enables users to establish a persistent mirroring relationship between pairs of Logical Units (LUNs) on two subsystems connected over an ESCON or a fiber-channel link. All updates performed on the set of primary (or source) LUNs appear in the same order on the secondary (target) disks in the target subsystem. The PPRC mechanism provides that if the source volume fails, the target holds an exact bitwise replica of the source's content as seen at the time of the failure. This solution is described in the section Active/passive GPFS configurations with ESS Peer-to-Peer Remote Copy.
The existing data and metadata replication features of GPFS can be similarly used to implement synchronous mirroring between a pair of geographically separated sites. The usage of logical replication-based mirroring can be seen as an attractive alternative to PPRC, in particular because it offers a generic solution that relies on no specific support from the disk subsystem beyond the basic ability to read and write data blocks. This solution is described in the section Active/active GPFS configurations employing logical replication and quorum tiebreakers.
The primary advantage of both synchronous mirroring methods lies in the minimization of the risk of permanent data loss. Both methods provide us with two consistent up-to-date replicas of the file system, each available for recovery should the other one fail. However, inherent to all solutions that synchronously mirror data over a wide-area network link is the latency penalty necessarily induced by the replicated write I/Os. This makes both mirroring methods prohibitively inefficient for certain types of performance-oriented applications.
An alternative technique involves taking periodic point-in-time copies of the file system using a facility such as the ESS FlashCopy®. The copy is subsequently transferred to a remote back-up location using PPRC and/or written to tape. The key difference between this and the synchronous mirroring techniques is that the creation of the secondary replica takes place asynchronously with respect to the regular file system activity against the primary replica, effectively eliminating the write penalty associated with synchronous mirroring. This solution is described in the section Online backup with ESS FlashCopy.
This paper is written for Information Technology professionals who have some experience with GPFS, ESS PPRC, and FlashCopy. For more information on these technologies, please refer to the sources listed in the References section. The reader is assumed to have some familiarity with the standard administrative concepts of GPFS and understand the node quorum rules enforced by GPFS to help protect the integrity of on-disk data in the event of a network partition.
| 
