| Previous | Contents | Index |
When there are multiple virtual circuits between two OpenVMS systems it is possible for the VMS$VAXCLUSTER to VMS$VAXCLUSTER connection to use any one of these circuits. All lock traffic between the two systems will then travel on the selected virtual circuit.
Each port has a "LOAD CLASS" associated with it. This load class helps to determine which virtual circuit a connection will use. If one port has a higher load class than all others then this port will be used. If two or more ports have equally high load classes then the connection will use the first of these that it finds. Prior to enhancements found in V7.3-1 and later, the load class is static and normally all CI and DSSI ports have a load class of 14(hex), while the Ethernet and FDDI ports will have a load class of A(hex). With V7.3-1 and later, the load class values are dynamic.
For instance, if you have multiple DSSI busses and an FDDI, the VMS$VAXCLUSTER connection will chose the DSSI bus as this path has the system disk, and thus will always be the first DSSI bus discovered when the OpenVMS system boots.
To force all lock traffic off the DSSI and on to the FDDI, for instance, an adjustment to the load class value is required, or the DSSI SCS port must be disabled.
In addition to the load class mechanisms, you can also use the "preferred path" mechanisms of MSCP and TMSCP services. This allows you to control the SCS connections used for serving remote disk and tape storage. The preferred path mechanism is most commonly used to explicitly spread cluster I/O activity over hosts and/or storage controllers serving disk or tape storage in parallel. This can be particularly useful if your hosts or storage controllers individually lack the necessary I/O bandwidth for the current I/O load, and must thus aggregate bandwidth to serve the cluster I/O load.
For related tools, see various utilities including LAVC$STOP_BUS and
LAVC$START_BUS, and see DCL commands including SET PREFERRED_PATH.
15.6.1.2.3 Cluster Communications Control Tools and Utilities?
In most OpenVMS versions, you can use the tools:
These tools permit you to disable or enable all SCS traffic on the on the specified paths.
You can also use a preferred path mechanism that tells the local MSCP disk class driver (DUDRIVER) which path to a disk should be used. Generally, this is used with dual-pathed disks, forcing I/O traffic through one of the controllers instead of the other. This can be used to implement a crude form of I/O load balancing at the disk I/O level.
Prior to V7.2, the preferred path feature uses the tool:
In OpenVMS V7.2 and later, you can use the following DCL command:
$ SET PREFERRED_PATH |
The preferred path mechanism does not disable nor affect SCS operations on the non-preferred path.
With OpenVMS V7.3 and later, please see the SCACP utility for control
over cluster communications, SCS virtual circuit control, port
selection, and related.
15.6.2 Cluster System Parameter Settings?
The following sections contain details of configuring cluster-related
system parameters.
15.6.2.1 What is the correct value for EXPECTED_VOTES in a VMScluster?
The VMScluster connection manager uses the concept of votes and quorum to prevent disk and memory data corruptions---when sufficient votes are present for quorum, then access to resources is permitted. When sufficient votes are not present, user activity will be blocked. The act of blocking user activity is called a "quorum hang", and is better thought of as a "user data integrity interlock". This mechanism is designed to prevent a partitioned VMScluster, and the resultant massive disk data corruptions. The quorum mechanism is expressly intended to prevent your data from becoming severely corrupted.
On each OpenVMS node in a VMScluster, one sets two values in SYSGEN: VOTES, and EXPECTED_VOTES. The former is how many votes the node contributes to the VMScluster. The latter is the total number of votes expected when the full VMScluster is bootstrapped.
Some sites erroneously attempt to set EXPECTED_VOTES too low, believing that this will allow when only a subset of voting nodes are present in a VMScluster. It does not. Further, an erroneous setting in EXPECTED_VOTES is automatically corrected once VMScluster connections to other nodes are established; user data is at risk of severe corruptions during the earliest and most vulnerable portion of the system bootstrap, before the connections have been established.
One can operate a VMScluster with one, two, or many voting nodes. With any but the two-node configuration, keeping a subset of the nodes active when some nodes fail can be easily configured. With the two-node configuration, one must use a primary-secondary configuration (where the primary has all the votes), a peer configuration (where when either node is down, the other hangs), or (preferable) a shared quorum disk.
Use of a quorum disk does slow down VMScluster transitions somewhat -- the addition of a third voting node that contributes the vote(s) that would be assigned to the quorum disk makes for faster transitions---but the use of a quorum disk does mean that either node in a two-node VMScluster configuration can operate when the other node is down.
If you choose to use a quoum disk, a QUORUM.DAT file will be automatically created when OpenVMS first boots and when a quorum disk is specified -- well, the QUORUM.DAT file will be created when OpenVMS is booted without also needing the votes from the quorum disk.
In a two-node VMScluster with a shared storage interconnect, typically each node has one vote, and the quorum disk also has one vote. EXPECTED_VOTES is set to three.
Using a quorum disk on a non-shared interconnect is unnecessary---the use of a quorum disk does not provide any value, and the votes assigned to the quorum disk should be assigned to the OpenVMS host serving access to the disk.
For information on quorum hangs, see the OpenVMS documentation. For information on changing the EXPECTED_VOTES value on a running system, see the SET CLUSTER/EXPECTED_VOTES command, and see the documentation for the AMDS and Availability Manager tools. Also of potential interest is the OpenVMS system console documentation for the processor-specific console commands used to trigger the IPC (Interrrupt Priority Level %x0C; IPL C) handler. AMDS, Availability Manager, and the IPC handler can each be used to clear a quorum hang. Use of AMDS and Availability Manager is generally recommended over IPC, particularly because IPC can cause CLUEXIT bugchecks if the system should remain halted beyond the cluster sanity timer limits.
The quorum scheme is a set of "blade guards" deliberately
implemented by OpenVMS Engineering to provide data integrity---remove
these blade guards at your peril. OpenVMS Engineering did not
implement the quorum mechanism to make a system manager's life more
difficult--- the quorum mechanism was specifically implemented to
keep your data from getting scrambled.
15.6.2.2 Explain disk (or tape) allocation class settings?
The allocation class mechanism provides the system manager with a way to configure and resolve served and direct paths to storage devices within a cluster. Any served device that provides multiple paths should be configured using a non-zero allocation class, either at the MSCP (or TMSCP) storage controllers, at the port (for port allocation classes), or at the OpenVMS MSCP (or TMSCP) server. All controllers or servers providing a path to the same device should have the same allocation class (at the port, controller, or server level).
Each disk (or tape) unit number used within a non-zero disk (or tape) allocation class must be unique, regardless of the particular device prefix. For the purposes of multi-path device path determination, any disk (or tape) device with the same unit number and the same disk (or tape) allocation class configuration is assumed to be the same device.
If you are reconfiguring disk device allocation classes, you will want
to avoid the use of allocation class one ($1$) until/unless you have
Fibre Channel storage configured. (Fibre Channel storage specifically
requires the use of allocation class $1$. eg: $1$DGA0:.)
15.6.2.2.1 How to configure allocation classes and Multi-Path SCSI?
The HSZ allocation class is applied to devices, starting with OpenVMS V7.2. It is considered a port allocation class (PAC), and all device names with a PAC have their controller letter forced to "A". (You might infer from the the text in the "Guidelines for OpenVMS Cluster Configurations" that this is something you have to do, though OpenVMS will thoughtfully handle this renaming for you.)
You can force the device names back to DKB by setting the HSZ allocation class to zero, and setting the PKB PAC to -1. This will use the host allocation class, and will leave the controller letter alone (that is, the DK controller letter will be the same as the SCSI port (PK) controller). Note that this won't work if the HSZ is configured in multibus failover mode. In this case, OpenVMS requires that you use an allocation class for the HSZ.
When your configuration gets even moderately complex, you must pay careful attention to how you assign the three kinds of allocation class: node, port and HSZ/HSJ, as otherwise you could wind up with device naming conflicts that can be painful to resolve.
The display-able path information is for SCSI multi-path, and permits the multi-path software to distinguish between different paths to the same device. If you have two paths to $1$DKA100, for example by having two KZPBA controllers and two SCSI buses to the HSZ, you would have two UCBs in a multi-path set. The path information is used by the multi-path software to distinguish between these two UCBs.
The displayable path information describes the path; in this case, the SCSI port. If port is PKB, that's the path name you get. The device name is no longer completely tied to the port name; the device name now depends on the various allocation class settings of the controller, SCSI port or node.
The reason the device name's controller letter is forced to "A" when you use PACs is because a shared SCSI bus may be configured via different ports on the various nodes connected to the bus. The port may be PKB on one node, and PKC on the other. Rather obviously, you will want to have the shared devices use the same device names on all nodes. To establish this, you will assign the same PAC on each node, and OpenVMS will force the controller letter to be the same on each node. Simply choosing "A" was easier and more deterministic than negotiating the controller letter between the nodes, and also parallels the solution used for this situation when DSSI or SDI/STI storage was used.
To enable port allocation classes, see the SYSBOOT command SET/BOOT, and see the DEVICE_NAMING system parameter.
This information is also described in the Cluster Systems and
Guidelines for OpenVMS Cluster Configurations manuals.
15.6.3 Tell me about SET HOST/DUP and SET HOST/HSC
The OpenVMS DCL commands SET HOST/DUP and SET HOST/HSC are used to connect to storage controllers via the Diagnostics and Utility Protocol (DUP). These commands require that the FYDRIVER device driver be connected. This device driver connection is typically performed by adding the following command(s) into the system startup command procedure:
On OpenVMS Alpha:
$ RUN SYS$SYSTEM:SYSMAN SYSMAN> IO CONNECT FYA0/NOADAPTER/DRIVER=SYS$FYDRIVER |
On OpenVMS VAX:
$ RUN SYS$SYSTEM:SYSGEN SYSGEN> CONNECT FYA0/NOADAPTER |
Alternatives to the DCL SET HOST/DUP command include the console SET HOST command available on various mid- to recent-vintage VAX consoles:
Access to Parameters on an Embedded DSSI controller:
SET HOST/DUP/DSSI[/BUS:{0:1}] dssi_node_number PARAMS
|
Access to Directory of tools on an Embedded DSSI controller:
SET HOST/DUP/DSSI[/BUS:{0:1}] dssi_node_number DIRECT
|
Access to Parameters on a KFQSA DSSI controller:
SHOW UQSSP ! to get port_controller_number PARAMS SET HOST/DUP/UQSSP port_controller_number PARAMS |
These console commands are available on most MicroVAX and VAXstation 3xxx series systems, and most (all?) VAX 4xxx series systems. For further information, see the system documentation and---on most VAX systems---see the console HELP text.
EK-410AB-MG, _DSSI VAXcluster Installation and Troubleshooting_, is a good resource for setting up a DSSI VMScluster on OpenVMS VAX nodes. (This manual predates coverage of OpenVMS Alpha systems, but gives good coverage to all hardware and software aspects of setting up a DSSI-based VMScluster---and most of the concepts covered are directly applicable to OpenVMS Alpha systems. This manual specifically covers the hardware, which is something not covered by the standard OpenVMS VMScluster documentation.)
Also see Section 15.3.3, and for the SCS name of the OpenVMS host see
Section 5.7.
15.6.4 How do I rename a DSSI disk (or tape?)
If you want to renumber or rename DSSI disks or DSSI tapes, it's easy---if you know the secret incantation...
From OpenVMS:
$ RUN SYS$SYSTEM:SYSGEN SYSGEN> CONNECT FYA0/NOADAPTER SYSGEN> ^Z $ SET HOST/DUP/SERV=MSCP$DUP/TASK=PARAMS <DSSI-NODE-NAME> ... PARAMS> STAT CONF <The software version is normally near the top of the display.> PARAMS> EXIT ... |
From the console on most 3000- and 4000-class VAX system consoles... (Obviously, the system must be halted for these commands...)
Integrated DSSI:
SET HOST/DUP/DSSI[/BUS:[0:1]] dssi_node_number PARAMS |
KFQSA:
SET HOST/DUP/UQSSP port_controller_number PARAMS |
For information on how to get out into the PARAMS subsystem, also see the HELP at the console prompt for the SET HOST syntax, or see the HELP on SET HOST /DUP (once you've connected FYDRIVER under OpenVMS).
Once you are out into the PARAMS subsystem, you can use the FORCEUNI option to force the use of the UNITNUM value and then set a unique UNITNUM inside each DSSI ISE---this causes each DSSI ISE to use the specfied unit number and not use the DSSI node as the unit number. Other parameters of interest are NODENAME and ALLCLASS, the node name and the (disk or tape) cluster allocation class.
Ensure that all disk unit numbers used within an OpenVMS Cluster disk
allocation class are unique, and all tape unit numbers used within an
OpenVMS Cluster tape allocation class are also unique. For details on
the SCS name of the OpenVMS host, see Section 5.7. For details of SET
HOST/DUP, see Section 15.6.3.
15.6.5 Where can I get Fibre Channel Storage (SAN) information?
Review the VMScluster documentation, and the System Management documentation. The following are the key points, but are likely not the only things you will need to change.
OpenVMS Cluster support is directly integrated into the operating system, and there is no way to remove it. You can, however, remote site-specific tailoring that was added for a particular cluster configuration.
First: Create restorable image BACKUPs of each of the current system disks. If something gets messed up, you want a way to recover, right?
Create standalone BACKUP kits for the OpenVMS VAX systems, and create or acquire bootable BACKUP kits for the OpenVMS Alpha systems.
Use CLUSTER_CONFIG or CLUSTER_CONFIG_LAN to remove the various system roots and to shut off boot services and VMScluster settings.
Create as many architecture-specific copies of the system disks as required. Realize that the new systems will all likely be booting through root SYS0---if you have any system-specific files in any other roots, save them.
Relocate the copies of the VMScluster common files onto each of the new system disks.
Reset the console parameters and boot flags on each system for use on a standalone node.
Reset the VAXCLUSTER and NISCS_LOAD_PEA0 parameters to 0 in SYSGEN and in MODPARAMS.DAT.
Clobber the VMScluster group ID and password using SYSMAN.
Reboot the systems seperately, and run AUTOGEN on each.
Shut off MOP services via NCP or LANCP on the boot server nodes.
Permanent seperation also requires the duplication of shared files. For a list of the files commonly shared, please see the most current version of SYS$STARTUP:SYLOGICALS.TEMPLATE, and specifically a version from OpenVMS V7.2 or later. The following files are typically shared within a cluster:
Filename: default directory (in common root) and file type:
SYSUAF SYS$SYSTEM:.DAT
SYSUAFALT SYS$SYSTEM:.DAT
SYSALF SYS$SYSTEM:.DAT
RIGHTSLIST SYS$SYSTEM:.DAT
NETPROXY SYS$SYSTEM:.DAT
NET$PROXY SYS$SYSTEM:.DAT
NETOBJECT SYS$SYSTEM:.DAT
NETNODE_REMOTE SYS$SYSTEM:.DAT
QMAN$MASTER SYS$SYSTEM: (this is a set of related files)
LMF$LICENSE SYS$SYSTEM:.LDB
VMSMAIL_PROFILE SYS$SYSTEM:.DATA
VMS$OBJECTS SYS$SYSTEM:.DAT
VMS$AUDIT_SERVER SYS$MANAGER:.DAT
VMS$PASSWORD_HISTORY SYS$SYSTEM:.DATA
NETNODE_UPDATE SYS$MANAGER:.COM
VMS$PASSWORD_POLICY SYS$LIBRARY:.EXE
LAN$NODE_DATABASE SYS$SYSTEM:LAN$NODE_DATABASE.DAT
|
Also see the topics on "cluster divorce" in the Ask The Wizard area.
For additional information on the OpenVMS Ask The Wizard (ATW) area and for a pointer to the available ATW Wizard.zip archive, please see Section 3.9.
Information on changing node names is included in Section 5.7.
15.6.7 Details on Volume Shadowing?
This section contains information on host-based volume shadowing; on
the disk mirroring capabilities available within OpenVMS.
15.6.7.1 Does volume shadowing require a non-zero allocation classes?
Yes, use of host-based Volume Shadowing requires that the disk(s) involved be configured in a non-zero allocation class.
Edit SYS$SYSTEM:MODPARAMS.DAT to include a declaration of an non-zero allocation class, such as setting the host allocation class to the value 7:
ALLOCLASS = 7 |
Then AUTOGEN the system, and reboot.
You should now be able to form the shadow set via a command such as the following:
$ MOUNT dsa1007: /SHADOW=($7$dkb300:,$7$dkb500:) volumelabel |
When operating in an OpenVMS Cluster, this sequence will typically change the disk names from the SCSNODE prefix (scsnode$dkann) to the allocation-class prefix ($7$dkannn). This may provide you with the opportunity to move to a device-independent scheme using logical name constructs such as the DISK$volumelabel logical names in your startup and application environments; an opportunity to weed out physical device references.
Allocation class one is used by Fibre Channel devices; it can be best to use another non-zero allocation class even if Fibre Channel is not currently configured and not currently planned.
| Index | Contents |