Creating a Storage Infrastructure with LVM

Building Volume Groups for RAC on Mirrored Disks

The procedure described in this section uses physical volume groups for mirroring of individual disks to ensure that each logical volume is mirrored to a disk on a different I/O bus. This kind of arrangement is known as PVG-strict mirroring. It is assumed that your disk hardware is already configured in such a way that a disk to be used as a mirror copy is connected to each node on a different bus than the bus that is used for the other (primary) copy.

For more information on using LVM, refer to the HP-UX Managing Systems and Workgroups manual.

Creating Volume Groups and Logical Volumes

If your volume groups have not been set up, use the procedure in the next sections. If you have already done LVM configuration, skip ahead to the section ““Installing Oracle Real Application Clusters”.”

Selecting Disks for the Volume Group

Obtain a list of the disks on both nodes and identify which device files are used for the same disk on both. Use the following command on each node to list available disks as they are known to each system:

# lssf /dev/dsk/*

In the following examples, we use /dev/rdsk/c1t2d0 and /dev/rdsk/c0t2d0, which happen to be the device names for the same disks on both ftsys9 and ftsys10. In the event that the device file names are different on the different nodes, make a careful note of the correspondences.

Creating Physical Volumes

On the configuration node (ftsys9), use the pvcreate command to define disks as physical volumes. This only needs to be done on the configuration node. Use the following commands to create two physical volumes for the sample configuration:

# pvcreate -f /dev/rdsk/c1t2d0  # pvcreate -f /dev/rdsk/c0t2d0

Creating a Volume Group with PVG-Strict Mirroring

Use the following steps to build a volume group on the configuration node (ftsys9). Later, the same volume group will be created on other nodes.

1. First, set up the group directory for vgops:

   # mkdir /dev/vg_ops

2. Next, create a control file named group in the directory /dev/vg_ops, as follows:

   # mknod /dev/vg_ops/group c 64 0xhh0000

   The major number is always 64, and the hexadecimal minor number has the form

   0xhh0000

   where hh must be unique to the volume group you are creating. Use the next hexadecimal number that is available on your system, after the volume groups that are already configured. Use the following command to display a list of existing volume groups:

   # ls -l /dev/*/group

3. Create the volume group and add physical volumes to it with the following commands:

   # vgcreate -g bus0 /dev/vg_ops /dev/dsk/c1t2d0  # vgextend -g bus1 /dev/vg_ops /dev/dsk/c0t2d0

   The first command creates the volume group and adds a physical volume to it in a physical volume group called bus0. The second command adds the second drive to the volume group, locating it in a different physical volume group named bus1. The use of physical volume groups allows the use of PVG-strict mirroring of disks and PV links.

4. Repeat this procedure for additional volume groups.

Building Mirrored Logical Volumes for RAC with LVM Commands

After you create volume groups and define physical volumes for use in them, you define mirrored logical volumes for data, logs, and control files. It is recommended that you use a shell script to issue the commands described in the next sections. The commands you use for creating logical volumes vary slightly depending on whether you are creating logical volumes for RAC redo log files or for use with Oracle data.

Creating Mirrored Logical Volumes for RAC Redo Logs and Control Files

Create logical volumes for use as redo log and control files by selecting mirror consistency recovery. Use the same options as in the following example:

# lvcreate -m 1 -M n -c y -s g -n redo1.log -L 4 /dev/vg_ops

The -m 1 option specifies single mirroring; the -M n option ensures that mirror write cache recovery is set off; the -c y means that mirror consistency recovery is enabled; the -s g means that mirroring is PVG-strict, that is, it occurs between different physical volume groups; the -n redo1.log option lets you specify the name of the logical volume; and the -L 4 option allocates 4 megabytes.

	NOTE: It is important to use the -M n and -c y options for both redo logs and control files. These options allow the redo log files to be resynchronized by SLVM following a system crash before Oracle recovery proceeds. If these options are not set correctly, you may not be able to continue with database recovery.

If the command is successful, the system will display messages like the following:

Logical volume “/dev/vg_ops/redo1.log” has been successfully created with character device “/dev/vg_ops/rredo1.log” Logical volume “/dev/vg_ops/redo1.log” has been successfully extended

Note that the character device file name (also called the raw logical volume name) is used by the Oracle DBA in building the RAC database.

Creating Mirrored Logical Volumes for RAC Data Files

Following a system crash, the mirrored logical volumes need to be resynchronized, which is known as “resilvering”.

If Oracle does not perform “resilvering” of RAC data files that are mirrored logical volumes, choose a mirror consistency policy of “NOMWC”. This is done by disabling mirror write caching and enabling mirror consistency recovery. With “NOMWC”, SLVM performs the resynchronization.

Create logical volumes for use as Oracle data files by using the same options as in the following example:

# lvcreate -m 1 -M n -c y -s g -n system.dbf -L 28 /dev/vg_ops

The -m 1 option specifies single mirroring; the -M n option ensures that mirror write cache recovery is set off; the -c y means that mirror consistency recovery is enabled; the -s g means that mirroring is PVG-strict, that is, it occurs between different physical volume groups; the -n system.dbf option lets you specify the name of the logical volume; and the -L 28 option allocates 28 megabytes.

If Oracle performs resilvering of RAC data files that are mirrored logical volumes, choose a mirror consistency policy of “NONE” by disabling both mirror write caching and mirror consistency recovery. With a mirror consistency policy of “NONE”, SLVM does not perform the resynchronization.

	NOTE: Contact Oracle to determine if your version of Oracle RAC allows “resilvering” and to appropriately configure the mirror consistency recovery policy for your logical volumes.

Create logical volumes for use as Oracle data files by using the same options as in the following example:

# lvcreate -m 1 -M n -c n -s g -n system.dbf -L 28 /dev/vg_ops

The -m 1 option specifies single mirroring; the -M n option ensures that mirror write cache recovery is set off; the -c n means that mirror consistency recovery is disabled; the -s g means that mirroring is PVG-strict, that is, it occurs between different physical volume groups; the -n system.dbf option lets you specify the name of the logical volume; and the -L 28 option allocates 28 megabytes.

If the command is successful, the system will display messages like the following:

Logical volume “/dev/vg_ops/system.dbf” has been successfully created with character device “/dev/vg_ops/rsystem.dbf” Logical volume “/dev/vg_ops/system.dbf” has been successfully extended

Note that the character device file name (also called the raw logical volume name) is used by the Oracle DBA in building the OPS database.

Creating RAC Volume Groups on Disk Arrays

The procedure described in this section assumes that you are using RAID-protected disk arrays and LVM’s physical volume links (PV links) to define redundant data paths from each node in the cluster to every logical unit on the array.

On your disk arrays, you should use redundant I/O channels from each node, connecting them to separate controllers on the array. Then you can define alternate links to the LUNs or logical disks you have defined on the array. If you are using SAM, choose the type of disk array you wish to configure, and follow the menus to define alternate links. If you are using LVM commands, specify the links on the command line.

The following example shows how to configure alternate links using LVM commands. The following disk configuration is assumed:

8/0.15.0 /dev/dsk/c0t15d0  /* I/O Channel 0 (8/0) SCSI address 15 LUN 0 */
8/0.15.1 /dev/dsk/c0t15d1  /* I/O Channel 0 (8/0) SCSI address 15 LUN 1 */
8/0.15.2 /dev/dsk/c0t15d2  /* I/O Channel 0 (8/0) SCSI address 15 LUN 2 */
8/0.15.3 /dev/dsk/c0t15d3  /* I/O Channel 0 (8/0) SCSI address 15 LUN 3 */
8/0.15.4 /dev/dsk/c0t15d4  /* I/O Channel 0 (8/0) SCSI address 15 LUN 4 */
8/0.15.5 /dev/dsk/c0t15d5  /* I/O Channel 0 (8/0) SCSI address 15 LUN 5 */
 
10/0.3.0 /dev/dsk/c1t3d0   /* I/O Channel 1 (10/0) SCSI address 3 LUN 0 */
10/0.3.1 /dev/dsk/c1t3d1   /* I/O Channel 1 (10/0) SCSI address 3 LUN 1 */
10/0.3.2 /dev/dsk/c1t3d2   /* I/O Channel 1 (10/0) SCSI address 3 LUN 2 */
10/0.3.3 /dev/dsk/c1t3d3   /* I/O Channel 1 (10/0) SCSI address 3 LUN 3 */
10/0.3.4 /dev/dsk/c1t3d4   /* I/O Channel 1 (10/0) SCSI address 3 LUN 4 */
10/0.3.5 /dev/dsk/c1t3d5   /* I/O Channel 1 (10/0) SCSI address 3 LUN 5 */

Assume that the disk array has been configured, and that both the following device files appear for the same LUN (logical disk) when you run the ioscan command:

/dev/dsk/c0t15d0
/dev/dsk/c1t3d0

Use the following procedure to configure a volume group for this logical disk:

LVM will now recognize the I/O channel represented by /dev/dsk/c0t15d0 as the primary link to the disk; if the primary link fails, LVM will automatically switch to the alternate I/O channel represented by /dev/dsk/c1t3d0. Use the vgextend command to add additional disks to the volume group, specifying the appropriate physical volume name for each PV link.

Repeat the entire procedure for each distinct volume group you wish to create. For ease of system administration, you may wish to use different volume groups to separate logs from data and control files.

	NOTE: The default maximum number of volume groups in HP-UX is 10. If you intend to create enough new volume groups that the total exceeds ten, you must increase the maxvgs system parameter and then re-build the HP-UX kernel. Use SAM and select the Kernel Configuration area, then choose Configurable Parameters. Maxvgs appears on the list.

Creating Logical Volumes for RAC on Disk Arrays

After you create volume groups and add PV links to them, you define logical volumes for data, logs, and control files. The following are some examples:

# lvcreate -n ops1log1.log -L 4 /dev/vg_ops  
# lvcreate -n opsctl1.ctl -L 4 /dev/vg_ops
# lvcreate -n system.dbf -L 28 /dev/vg_ops
# lvcreate -n opsdata1.dbf -L 1000 /dev/vg_ops

Oracle Demo Database Files

The following set of files is required for the Oracle demo database which you can create during the installation process.

^* The size of the logical volume is larger than the Oracle file size because Oracle needs extra space to allocate a header in addition to the file's actual data capacity.

Create these files if you wish to build the demo database. The three logical volumes at the bottom of the table are included as additional data files, which you can create as needed, supplying the appropriate sizes. If your naming conventions require, you can include the Oracle SID and/or the database name to distinguish files for different instances and different databases. If you are using the ORACLE_BASE directory structure, create symbolic links to the ORACLE_BASE files from the appropriate directory. Example:

# ln -s /dev/vg_ops/ropsctl1.ctl /u01/ORACLE/db001/ctrl01_1.ctl

For more information about Oracle directories, refer to the Oracle Server for HP 9000 Installation and Configuration Guide. More information about the maximum sizes of data files is found in the Oracle 8i Reference (Oracle Part Number A76961-01), Chapter 4, “Database Limits.”

After creating these files, set the owner to oracle and the group to dba with a file mode of 660. The logical volumes are now available on the primary node, and the raw logical volume names can now be used by the Oracle DBA.

Displaying the Logical Volume Infrastructure

To display the volume group, use the vgdisplay command:

# vgdisplay -v /dev/vg_ops 

Exporting the Logical Volume Infrastructure

Before the Oracle volume groups can be shared, their configuration data must be exported to other nodes in the cluster. This is done either in Serviceguard Manager or by using HP-UX commands, as shown in the following sections.

Exporting with Serviceguard Manager

In Serviceguard Manager, choose Disks and File Systems, then choose Volume Groups. Open the Actions menu and choose Distribute. Follow the prompts in the dialog box to select the volume group that is to be distributed to one or more additional nodes. Enter the name of each node that is to receive the volume group and select Add. When the list is complete, press OK. Serviceguard Manager automatically configures the volume group for use on the other nodes.

	NOTE: In distributing the volume group, Serviceguard Manager does not import PV links on other nodes. Therefore, to complete the distribution, you use the vgextend command on all nodes to add the alternate PV links to the volume group you have distributed. Use the vgdisplay command to verify the existence of the correct groups of PV links on all nodes.