BRMS Integration with TS75x0 Virtual Tape Libraries

Background

In a nutshell, the user needs to decide whether BRMS or the VTL will manage the duplications and media.

BRMS has always provided limited support for virtual tape libraries (VTL). As long as they provided an emulated media library interface, BRMS would use them as a media library device. However, it was not able to take advantage of some of the more advanced features. For example, to duplicate data from a VTL to a physical media library (MLB), the DUPTAP or DUPMEDBRM command was used. Data would be read from the VTL, processed by the i5 partition, then written to an attached device. This would use CPU cycles and SAN bandwidth.

The TS75x0 units had the option of attaching a tape library directly to it, effectively bypassing the i5. The TS75x0 would copy directly from disk to the tape library, with no involvement from the i5. However, the result would be two identical 'tapes' with the same volume serial number. This duplication/migration would be unknown to BRMS (and many other media management systems). Recovery reports would not include the physical duplicates, nor would BRMS be able to manage the locations and movement patterns of them. If the user were to insert the physical tape in an MLB or tape drive, BRMS would think the tape had moved from the VTL, and would no longer perform recoveries from the VTL unless the user manually told BRMS that the tape had been updated. To complicate matters, if BRMS didn't think the tape was in the VTL, it would not attempt to reuse it after expiration, thereby never freeing up the space allocated for it.

The i5 assumes that labeled tapes have unique serial numbers. Attempting to write software to bypass this limitation is impractical, as it would also require a rewrite of the BRMS database.

The new VTL integration allows BRMS to direct the duplication of media from the virtual tape library to a physical tape library. When a duplication is requested, BRMS will direct which tapes for the VTL to use for input and output, and update the necessary files in QUSRBRM to reflect that a duplication has occurred. To facilitate tape movement for offsite storage, BRMS can also direct the VTL to eject tapes for movement. The VTL will migrate the data from disk to physical tapes, updating the labels correctly. BRMS will track the tapes and their data as duplicates - they can belong to separate media classes and move policies, and therefore move and expire independently. This allows BRMS to reuse the allocated storage on the VTL while maintaining the movement pattern of the physical copies.

Restrictions

• The VTL is unable to span tapes when writing to the physical tapes. If the tape stream on the VTL is too large to fit on a single physical cartridge, the VTL will not span to a second cartridge and adjust the headers on the continuation cartridge. To avoid this situation, configure the size of volumes on the VTL to be smaller than the capacity of the physical tapes. When the i5 writes to the VTL and spans tapes, it will correctly modify the tape headers to reflect this. During duplication to the physical media library, the VTL will simply copy the continuation volume to new physical media, despite the less efficient use of media.
• If you plan to duplicate from the VTL to a physical tape and expect that the VTL media will not fit entirely on a physical tape, use DUPMEDBRM or DUPTAP without the BRMS+VTL integration enabled. This will increase SAN traffic and workload on the i5, but it is the only way to ensure the tapes are recoverable after the duplication.
• Not all DUPMEDBRM parameters are functional during VTL to Physical migration. Only entire tapes can be duplicated. Any parameters specifying individual file sequences or ranges, resuming failed duplications, or appending to output volumes will result in entire tapes being duplicated to scratch media (DUPMEDBRM VOL(*SCHHST))
• Remote duplication is not supported (DUPMEDBRM FROMSYS()).
• All volume ejects from the MLB should be managed by the VTL. All BRMS ejects of tapes will be routed through the VTL. However, non-BRMS ejects will not be intercepted. It is the user’s responsibility to The BRMS eject design issues ejects sequentially, relying on the device to manage the ejects. IBM MLB's receive the eject requests and return control to BRMS immediately, but the VTL may honor requests synchronously. The implication of this, is that the VTL may eject tapes from the MLB one at a time instead of in batch.
• The VTL does not honor physical drive allocation the same way as the i5 does. The i5 uses the SCSI "reserve" command at the hardware level whereas the VTL relies on the presence of media in a drive. The implication of this, is that the VTL will refuse to use a drive if there is media mounted which the VTL did not mount. For this reason, it is recommended that no drives in the physical media library are shared between the VTL and the i5.
• If physical media is loaded in a drive not controlled by the VTL, the VTL will not be able to access it. This is consistent with existing media library limitations - it is the user's responsibility to ensure media required by the VTL be available in a slot (i.e. not mounted on a drive). Even if the media is mounted on a drive, the VTL will think another process is using the drive and will block until the media is unmounted.
• The VTL cannot recover from a mount failure of physical media. Since BRMS is not notified of the mount failure, it cannot recover and select alternate media.
• Enhanced Cache is not supported and must be turned off.
• BRMS Integration is only compatible with the CVT3 code. This is default on the TS5730. The CVT3 code can be loaded on a TS75x0, and this is a supported configuration.
• All virtual tapes in the TS75x0 must have unique 6-character barcodes. When asked to export a virtual tape, the barcode BRMS uses is not qualified with the virtual library name, therefore the TS75x0 is unable to differentiate between volume serials and will pick the first one it finds.
• The CVT3 does not support sharing physical drives with the i5.

Encryption Considerations

Encryption adds a layer of complexity to the VTL configuration. It is important to understand and test the configuration to ensure your data is recoverable.

• The VTL has the option of encrypting all of the data sent to it. It is encrypted and decrypted on the VTL unit, and is transparent to the i5. This option allows encrypting *SAVSYS and anything else saved. When this data is duplicated to tape, the physical tape will NOT be encrypted.
• If BRMS is controlling the encryption (available starting at V6R1) the data sent to the VTL will be encrypted (*SAVSYS cannot be encrypted - all of the normal encryption rules apply). The output tape will also be encrypted, and BRMS will maintain the keys required to decrypt from the physical tape.
• If the saves to the VTL are not encrypted, and encryption is enabled on export, the data on the physical tapes may not be accessible to non-VTL devices. These tapes cannot be used directly for recovery without the VTL.
• BRMS does not track the keys if the VTL issues and manages the keys.

Hardware Configuration and Verification

Several steps are required to configure BRMS to use the VTL. Ensure the following is true before continuing with the BRMS-VTL configuration:

1. The VTL is configured and attached to the i5.
2. The physical library (MLB) must be attached to the i5.
3. Use INZBRM *DEVICE to allow BRMS to discover the devices.
4. Set the locations as appropriate for your BRMS implementation. We recommend that each MLB have a different location, and media policies be configured with the Storage Location parameter configured to select volumes specifically from each media library.
5. Enroll both VTL and MLB volumes into BRMS, ensuring the locations are correct.
6. Attach the MLB to the VTL and perform any necessary configurations on the VTL.
7. Perform test backups and recoveries to both the VTL and MLB to ensure the configuration is correct.

BRMS Configuration Overview

1. Configure and authenticate users.
2. Tell BRMS which VTL and MLB devices are managed.
3. Perform backups and duplications.
4. Test recoveries from the duplicate media.

Requirements

1. 57xxSS1 opt 33 "Portable App Solutions Environnent"
2. 5733SC1 *BASE "IBM Portable Utilities for i5/OS"
3. 5733SC1 opt 1 "OpenSSH, OpenSSL, zlib"
4. 57xxBR1 BRMS Base and Advanced Features.
5. PTFs:
   • V6R1M0 – SI33263 or supersede
   • V5R4M0 – SI33260 or supersede
   • V5R4M0 – SI33259 or supersede
6. Download the Java Secure Channel from http://www.jcraft.com/jsch/ (link resides outside of ibm.com) and place it in an IFS directory on the i5.

The user profiles performing the configurations and duplications must have *ALLOBJ authority.

The CVT3 minimum build level is 1465.

Configure and authenticate users

BRMS uses scp and ssh to communicate with the VTL. In order to use these functions, authentication between a user profile on the i5 and a user profile on the VTL must be configured and keys exchanged. This configuration is only necessary when changing profiles or when new keys must be generated - it is not necessary to perform these steps for every BRMS duplication or tape eject.

BRMS maintains a coupling between the VTL, physical MLB, and a user profile on the VTL. When duplication is performed from the VTL to the physical MLB, it will use the VTL user profile to transfer files, execute commands, etc. All of the user profiles on the i5 which will perform duplications from the VTL will need to authenticate to the configured VTL user profile. BRMS on each i5 must share a single VTL profile. In other words, all of the i5 users which will perform duplications on a specific i5 must authenticate with a single user on the VTL. All of the i5 users on a different i5 that will perform duplications, can authenticate with another user.

It is recommended that each i5 have its own dedicated VTL user profile.

The interface to configure BRMS is handled through program calls to QBRM/Q1AOLD. In the future, BRMS may incorporate these program calls into panels, menus, etc.

This guide details the steps to achieve this. It is not a guide to SSL, and may not be applicable if you have a more complex or limited configuration. All of these configuration steps can be performed manually on the VTL using ssh to access a command line. Various ssh clients are available for a number of operating systems. For additional information on SSL and ssh on the i5, visit:

http://www.ibm.com/servers/enable/site/porting/tools/openssh.html

For clarity, our examples will use the following terms. Please substitute your own values where appropriate.

i5admin - the profile on the i5 which will be performing the configuration and/or backups.
i5adminpwd - Password for the i5admin
i5user - the profile on the i5 which will be performing backups.
i5usrpwd - Password for the i5user
i5name - the name of the i5. An IP address can also be used.
vtlname - the name of the VTL on the network. An IP address can also be used.
vtlroot - the profile on the VTL which has authority to create profiles, directories, etc.
vtlrootpwd - password for vtlroot
vtluser - the profile on the VTL which i5user will authenticate with during duplications.
vtlpwd - password for vtluser
/home - base directory for user's home directories

1. Create or identify a user profile on the i5 which will perform the duplications. In our examples we will use i5user. During the setup, i5user must have a password - it can be removed later. Assign this profile to group profile QBRMS so it has the authorizations to perform the duplications.
2. Issue the following command to configure BRMS to use the VTL:
   
   CALL QBRM/Q1AOLD PARM('VTL'
   

   '*SETUP'	    <== Which function to execute
   'i5user'	    <== i5 user to authenticate with the VTL user
   'vtldev' 	    <== BRMS name for the VTL device
   'vtlmlb' 	    <== BRMS name for the physical MLB
   'vtlname '	    <== Name or IP of the VTL on the network
   'vtlroot'	    <== Admin profile on the VTL (i.e. 'root')
   'vtlrootpwd' 	    <== Password for admin profile on the VTL
   'vtluser'	    <== VTL profile to create and authenticate with
   'vtluserpwd' 	    <== VTL user's password
   '/jsch0.1.39.jar')  <== Path to the Java Secure Channel jar
   

   
   

   This will create the user profile vtluser, generate keys for vtluser and i5user, exchange the public keys, and configure BRMS to reroute duplication requests for these devices to the VTL.

3. Perform the following command to view the current configuration:
   
   CALL QBRM/Q1AOLD PARM('VTL'
   

   '*DISPLAY')	<== Tells BRMS which function to execute
   
   You will see a display like this:
   POLLING INTERVAL: 15.
   TIMEOUT INTERVAL: 300.
   SOURCE     TARGET     USER VTL PATH.
   ---------- ---------- --------------------------------.
   CVT2       CVTMLB     temp1      rochcvt2a      /poll4work.
   

4. To authenticate additional users, issue the following command for each user:
   
   CALL QBRM/Q1AOLD PARM('VTL'
   

   '*ADDUSER'	    <== Tells BRMS which function to execute
   'i5user' 	    <== i5 user to authenticate with the VTL user
   'vtlname'	    <== Name or IP of the VTL on the network
   ‘vtluser' 	    <== VTL profile to create and authenticate with
   'vtluserpwd' 	    <== VTL user's password
   '/jsch0.1.39.jar')  <== Path to the Java Secure Channel jar
   

5. To test that the configuration is correct, sign on to the i5 as the user you authenticated with (i5 user) and issue the following command:
   
   CALL QBRM/Q1AOLD PARM('VTL'
   

   '*TESTCVT'	    <== Tells BRMS which function to execute
   'vtldev' 	    <== BRMS name for the VTL device
   '/jsch0.1.39.jar')  <== Path to the Java Secure Channel jar
   

   A successful completion message will indicate that it worked.
   

Additional BRMS configuration interfaces

Occasionally additional steps may be required to configure BRMS to use the VTL correctly. The following lists the interfaces which perform individual steps. The "*setup" and "*adduser" interfaces perform these steps in various sequences.

Add additional VTL configurations

If you have additional VTL / MLB pairs, or wish to change the user, name of VTL, or polling directory of an existing pair, use the following interface:
CALL QBRM/Q1AOLD PARM('VTL'

'*ADD'			<== Tells BRMS which function to execute
'vtldev' 		<== BRMS name for the VTL device
'vtlmlb' 		<== BRMS name for the physical MLB
'vtluser' 		<== VTL profile for BRMS to use
'vtlname '		<== Name or IP of the VTL on the network
'*default') 		<== Optional new polling directory

Changed an existing VTL configuration

Use the "*ADD" interface above, specifying the BRMS VTL and MLB names, with different user, VTLname and polling directory, to change an existing VTL configuration.

Removing an existing VTL configuration

'*REMOVE'	<== Tells BRMS which function to execute
'vtldev' 	<== BRMS device name for the VTL (can be *ALL)
‘vtlmlb') 	<== BRMS name for the physical MLB

(if the VTL device is *ALL, this must not be specified)

Change polling directory

BRMS communicates with the VTL by copying a file to the VTL into a specific directory. The default directory is /poll4work. If you wish to change this, you will need to create the alternate directory on the VTL, then notify both BRMS and the VTL to use the different directory.

Create new polling directory:

CALL QBRM/Q1AOLD PARM('VTL'

‘CRTCVTDIR'	<== Tells BRMS which function to execute
'vtlname '	<== Name or IP of the VTL on the network
'vtlroot'	<== Admin profile on the VTL (i.e. 'root')
'vtlrootpwd' 	<== Password for admin profile on the VTL	
'/newdir'	<== Name of new directory (can also be*DEFAULT for /poll4work)
'/jsch0.1.39.jar') <== Path to the Java Secure Channel jar

Notify BRMS to use the new directory

Notifying BRMS to use a different polling directory is done via the '*ADD' interface described above.

Notify the VTL to user the new directory

1. Use ssh to log into the VTL.
2. Stop the comm service (ve stop comm) - CAN BE DISRUPTIVE TO OPERATIONS
3. Change the environment variable P4WFULLPATH to the new path.
4. Start the comm service (ve start comm)

Change the polling interval:

BRMS polls the VTL on a regular basis to determine the status of the duplications and ejects. This interval can be changed with the following interface:

CALL QBRM/Q1AOLD PARM('VTL'

'*INTERVAL'	<== Tells BRMS which function to execute
'30')		<== Polling interval, in seconds

Change the timeout period:

It the VTL's duplication or eject process terminates without updating it's status, BRMS may wait forever. To avoid that, set the timeout period to tell BRMS how long to wait before it fails the operation. Note that this is measured from the START of the command.

CALL QBRM/Q1AOLD PARM('VTL'

'*TIMEOUT'	<== Tells BRMS which function to execute
'300')		<== Timeout period, in minutes.

*CRTCVTUSR'	<== Tells BRMS which function to execute
'vtlname'	<== Name or IP of the VTL on the network
'vtlroot'	<== Admin profile on the VTL (i.e. 'root')
'vtlrootpwd'	<== Password for admin profile on the VTL
'vtluser'	<== VTL profile to create and authenticate with
'vtluserpwd'	<== VTL user's password
'/jsch0.1.39.jar') <== Path to the Java Secure Channel jar

'*RMVCVTUSR'	<== Tells BRMS which function to execute
'vtlname '	<== Name or IP of the VTL on the network
'vtlroot'	<== Admin profile on the VTL (i.e. 'root')
'vtlrootpwd' 	<== Password for admin profile on the VTL	
'vtluser' 	<== VTL profile to create and authenticate with
'Y' or 'N'	<== Remove user's home directories?
'/jsch0.1.39.jar') <== Path to the Java Secure Channel jar

'*CVTUSRPWD'    <== Tells BRMS which function to execute
'vtlname '	<== Name or IP of the VTL on the network
'vtlroot'	<== Admin profile on the VTL (i.e. 'root')
'vtlrootpwd' 	<== Password for admin profile on the VTL
'vtluser' 	<== VTL profile to create and authenticate with
'vtluserpwd' 	<== VTL user's new password
'/jsch0.1.39.jar') <== Path to the Java Secure Channel jar

Generate keys for a user on the VTL

Create keys on the VTL for the specified user profile. The keys will be created into ~/.ssh

CALL QBRM/Q1AOLD PARM('VTL'

'*GENCVTKEY'  <== Tells BRMS which function to execute
'vtlname '	<== Name or IP of the VTL on the network
'vtluser' 	<== VTL profile to create and authenticate with
'vtluserpwd' 	<== VTL user's password
'/jsch0.1.39.jar') <== Path to the Java Secure Channel jar

Generate keys for a user on the i5

Create keys for the current user profile. The keys will be created into ~/.ssh

CALL QBRM/Q1AOLD PARM('VTL'

'*GENLCLKEY')  <== Tells BRMS which function to execute

'*GENLCLKEY' <== Tells BRMS which function to execute
'*current')	<== Generate keys for the current user

Generate keys for another profile on the i5. The keys will be created into ~/.ssh

CALL QBRM/Q1AOLD PARM('VTL'

'*GENLCLKEY' <== Tells BRMS which function to execute
'i5user')	<== Generate keys for the i5user

Exchange public keys with a user on the VTL

After generating new keys on the i5, they must be placed on the VTL to authenticate for the BRMS integration.

CALL QBRM/Q1AOLD PARM('VTL'

'*EXCHKEYS'	<== Tells BRMS which function to execute
'i5user'	<== i5 user's keys to exchange (or *CURRENT)
'vtlname '	<== Name or IP of the VTL on the network
'vtluser' 	<== VTL profile to create and authenticate with
'vtluserpwd' 	<== VTL user's new password
'/jsch0.1.39.jar')  <== Path to the Java Secure Channel jar

Eject a tape from the VTL-managed media library

To cause the VTL to eject a tape from the managed media library, issue the following command. Note that the fifth parameter contains the results of the action.

CALL QBRM/Q1AOLD PARM('VTL'

'*EJECT'	<== Tells BRMS which function to execute
'vtlmlb'	<== Device to eject the tape from
'volser'	<== Volume serial to eject
' ') 		<== Return value:
			0 = device not managed
			1 = success
			9 = failure

Troubleshooting

There are several tools available to determine why an operation involving the VTL isn't working.

The export process is "waiting for a drive"

There is no support for sharing drives between the VTL and other hosts. However, if you choose to use this configuration to minimize drives, be aware that the VTL will occasionally enter a state where it will always be “waiting for a drive”.

The VTL uses a minimally-compatible drive allocation scheme. If it is sharing a physical drive with another system, and media is left in the drive, the VTL will assume it has been allocated. The default end option on the i5 backup commands is *REWIND, leaving the media mounted for another use. However, the VTL will refuse the use the drive until the media is unmounted. Use the i5's CHKTAP or the web interface to clear the drive. The export process will continue if this was the problem.

In some situations, after the media has been removed from the drive during an export operation, the VTL will become confused and enter a state which requires the physical tape library to be reset. To reset the library, open the VE Console, right-click on the physical tape library, and select “RESET”. Respond to the following prompt verifying you are aware that any existing exports will be terminated.

View the VTL system logs

The VTL contains two system logs which may be valuable. Both can be accessed by signing into the VTL using ssh, and issueing the following commands:

tail -n 200 /var/log/messages <== Will display last 200 entries of /var/log/messages

The other log is:

dmesg | tail -n 200 <== Will display the last 200 entries in the dmesg log

In either case, look for messages which may indicate tapes are not found, devices are not responding, etc.

Note that it is also possible to copy the /var/log/messags log to the i5 if the keys have been exchanged. Use the following command from an i5 command prompt to copy the log into /tmp/brms/vtllog:

QSH CMD('/QOpenSys/usr/bin/scp

vtluser@vtlname:/var/log/messages /tmp/brms/vtllog')

Substitute vtluser and vtlname with the VTL user the current user has authenticated with, and vtlname is the network name or IP of the VTL.

These files can become quite large - use the following command in an ssh session to clear them (this will also rebuild the files if they are deleted):

logrotate -f /etc/logrotate.conf

Also, BRMS has a software log at /tmp/brms/flighrec and /tmp/brms/flightrec.bku. Note that these logs are intended for BRMS developers to debug problems, and may not be too useful for anyone else. However, it may yield clues to why a process fails.

Run the configuration commands in debug mode to view more detail

Many interfaces have an option to view debug information. Simply add the parameter "Y" to enable it .. for example:

CALL QBRM/Q1AOLD PARM('VTL'

'*ADDUSER'	<== Tells BRMS which function to execute
'i5user' 	<== i5 user to authenticate with the VTL user
'vtlname '	<== Name or IP of the VTL on the network
'vtluser' 	<== VTL profile to create and authenticate with
'vtluserpwd' 	<== VTL user's password
'/jsch0.1.39.jar' <== Path to the Java Secure Channel jar
'Y') 		<== Enable debug

You receive an error when exchanging keys

If you receive the following error message:

com.jcraft.jsch.JSchException: java.lang.ArrayIndexOutOfBoundsException

when exchanging keys, the known hosts file on the i5 may be corrupt. This file is located in:

/home/i5user/.ssh/known_hosts

Where i5user is the user profile on the i5 which you are attempting to authenticate with. This file contains entries preceded with a host name or IP address. During our testing, we have encountered several situations where the first octet of the IP addresses may be missing. For example, instead of “10.10.10.5”, the value in the file would be “.10.10.5”. To correct this situation, remove or repair that entry in the file.