Managing the Shared Storage

Making LVM Volume Groups Shareable

Normally, volume groups are marked to be activated in shared mode when they are listed with the OPS_VOLUME_GROUP parameter in the cluster configuration file or in Serviceguard Manager. which occurs when the configuration is applied. However, in some cases you may want to manually make a volume group sharable. For example, if you wish to add a new shared volume group without shutting down the cluster, you can use the manual method to do it online. However, when convenient, it's a good practice to bring down the cluster and reconfigure it to include the new volume group.

Use the vgchange command on each node to ensure that the volume group to be shared is currently inactive on all nodes. Example:
# vgchange -a n /dev/vg_ops
On the configuration node, use the vgchange command to make the volume group shareable by members of the cluster:
# vgchange -S y -c y /dev/vg_ops
This command is issued from the configuration node only, and the cluster must be running on all nodes for the command to succeed. Note that both the -S and the -c options are specified. The -S y option makes the volume group shareable, and the -c y option causes the cluster id to be written out to all the disks in the volume group. In effect, this command specifies the cluster to which a node must belong in order to obtain shared access to the volume group.

Making a Volume Group Unshareable

If you wish to unmark a previously marked shared volume group:

Remove the volume group name from the ASCII cluster configuration file.

Enter the following command:

# vgchange -S n -c n /dev/volumegroup

The above example marks the volume group as non-shared and not associated with a cluster.

Activating an LVM Volume Group in Shared Mode

Activation and deactivation of shared volume groups is normally done through a control script. If you need to perform activation from the command line, you can issue the following command from each node to activate the volume group in shared mode. (The node on which you first enter the command becomes the server node.)

# vgchange -a s -p /dev/vg_ops

The following message is displayed:

Activated volume group in shared mode.
This node is the Server.

When the same command is entered on the second node, the following message is displayed:

Activated volume group in shared mode.
This node is a Client.




	NOTE: Do not share volume groups that are not part of the RAC configuration.

Deactivating a Shared Volume Group

Issue the following command from each node to deactivate the shared volume group:

# vgchange -a n /dev/vg_ops

Remember that volume groups remain shareable even when nodes enter and leave the cluster.




	NOTE: If you wish to change the capacity of a volume group at a later time, you must deactivate and unshare the volume group first. If you add disks, you must specify the appropriate physical volume group name and make sure the `/etc/lvmpvg` file is correctly updated on both nodes.

Making Changes to Shared Volume Groups

You may need to change the volume group configuration of RAC shared logical volumes to add capacity to the data files or to add log files. No configuration changes are allowed on shared LVM volume groups while they are activated. The volume group must be deactivated first on all nodes, and marked as non-shareable. Use the following procedure (examples assume the volume group is being shared by node 1 and node 2, and they use the volume group vg_ops):

Ensure that the Oracle RAC database is not active on either node.
From node 2, use the vgchange command to deactivate the volume group:
# vgchange -a n /dev/vg_ops
From node 2, use the vgexport command to export the volume group:
# vgexport -m /tmp/vg_ops.map.old /dev/vg_ops
This dissociates the volume group from node 2.
From node 1, use the vgchange command to deactivate the volume group:
# vgchange -a n /dev/vg_ops
Use the vgchange command to mark the volume group as unshareable:
# vgchange -S n -c n /dev/vg_ops
Prior to making configuration changes, activate the volume group in normal (non-shared) mode:
# vgchange -a y /dev/vg_ops
Use normal LVM commands to make the needed changes. Be sure to set the raw logical volume device file's owner to oracle and group to dba, with a mode of 660.
Next, still from node 1, deactivate the volume group:
# vgchange -a n /dev/vg_ops
Use the vgexport command with the options shown in the example to create a new map file:
# vgexport -p -m /tmp/vg_ops.map /dev/vg_ops
Make a copy of /etc/lvmpvg in /tmp/lvmpvg, then copy the file to /tmp/lvmpvg on node 2. Copy the file /tmp/vg_ops.map to node 2.
Use the following command to make the volume group shareable by the entire cluster again:
# vgchange -S y -c y /dev/vg_ops
On node 2, issue the following command:
# mkdir /dev/vg_ops
Create a control file named group in the directory /dev/vg_ops, as in the following:
# mknod /dev/vg_ops/group c 64 0xhh0000
The major number is always 64, and the hexadecimal minor number has the form
0xhh0000
where hh must be unique to the volume group you are creating. Use the next hexadecimal number that is available on your system, after the volume groups that are already configured.
Use the vgimport command, specifying the map file you copied from the configuration node. In the following example, the vgimport command is issued on the second node for the same volume group that was modified on the first node:
# vgimport -v -m /tmp/vg_ops.map /dev/vg_ops /dev/dsk/c0t2d0/dev/dsk/c1t2d0
Activate the volume group in shared mode by issuing the following command on both nodes:
# vgchange -a s -p /dev/vg_ops
Skip this step if you use a package control script to activate and deactivate the shared volume group as a part of RAC startup and shutdown.

Adding Additional Shared LVM Volume Groups

To add capacity or to organize your disk resources for ease of management, you may wish to create additional shared volume groups for your Oracle RAC databases. If you decide to use additional shared volume groups, they must conform to the following rules:

Volume groups should include different PV links to each logical unit on the disk array.
Volume group names must be the same on all nodes in the cluster.
Logical volume names must be the same on all nodes in the cluster.

Changing the VxVM or CVM Storage Configuration

You can add VxVM disk groups to the cluster configuration while the cluster is running. To add new CVM disk groups, the cluster must be running.

If you are creating new CVM disk groups, be sure to determine the master node on which to do the creation by using the following command:

# vxdctl -c mode

One node will identify itself as the master. Create disk groups from this node.

Similarly, you can delete VxVM or CVM disk groups provided they are not being used by a cluster node at the time.




	NOTE: If you are removing a disk group from the cluster configuration, make sure that you also modify or delete any package control script that imports and deports this disk group. If you are removing a CVM disk group, be sure to remove the `STORAGE_GROUP` entries for the disk group from the package ASCII file.