Jump to content United States-English
HP.com Home Products and Services Support and Drivers Solutions How to Buy
» Contact HP
More options
HP.com home
Managing Serviceguard NFS for Linux > Chapter 3 Sample Configurations

Configuring Multiple Adoptive Nodes for Cascading Failover

» 

Technical documentation

Complete book in PDF
» Feedback
Content starts here

 » Table of Contents

 » Index

This configuration has two packages and three servers. One server is the primary node for both packages. The other servers are adoptive nodes for the two packages. Figure 3-5 illustrates this configuration. Dotted lines indicate which servers are adoptive nodes for the packages. Figure 3-6 illustrates the configuration after host thyme fails.

Figure 3-5 Cascading Failover with Three Servers

Cascading Failover with Three Servers

Figure 3-6 shows the cascading failover configuration after host thyme has failed. Host basil is the first adoptive node configured for pkg01, and host sage is the first adoptive node configured for pkg02.

Figure 3-6 Cascading Failover with Three Servers After One Server Fails

Cascading Failover with Three Servers After One Server Fails

Cluster Configuration File for Three-Server Cascading Failover

This section shows the cluster configuration file (cluster.conf) for this configuration example. The comments are not shown.

CLUSTER_NAME             Cascading

QS_HOST                  qs
QS_POLLING_INTERVAL      300000000

NODE_NAME                thyme
  NETWORK_INTERFACE      eth0
    HEARTBEAT_IP         192.100.112.146
  NETWORK_INTERFACE      eth1

NODE_NAME                basil
  NETWORK_INTERFACE      eth0
    HEARTBEAT_IP         192.100.112.168

NODE_NAME                sage
  NETWORK_INTERFACE      eth0
    HEARTBEAT_IP         192.100.112.184
  NETWORK_INTERFACE      eth1
  NETWORK_INTERFACE      eth2

HEARTBEAT_INTERVAL       1000000
NODE_TIMEOUT             5000000

AUTO_START_TIMEOUT       600000000
NETWORK_POLLING_INTERVAL 2000000

MAX_CONFIGURED_PACKAGES  4

Sample Configuration for Legacy Packages

Configuration files of two legacy packages are described in the following section. These samples are applicable to Red Hat environment. For SLES user can replace all occurrences of “/usr/local” with “/opt”.

Package Configuration File for pkg01

This section shows the package configuration file (pkg01.conf) for the package pkg01 in this sample configuration. The comments are not shown.

PACKAGE_NAME               pkg01
PACKAGE_TYPE               FAILOVER
FAILOVER_POLICY            CONFIGURED_NODE
FAILBACK_POLICY            MANUAL
NODE_NAME                  thyme
NODE_NAME                  basil
NODE_NAME                  sage
AUTO_RUN                   YES
NODE_FAIL_FAST_ENABLED     NO
RUN_SCRIPT                 /usr/local/cmcluster/pkg01/pkg01.cntl
RUN_SCRIPT_TIMEOUT         NO_TIMEOUT
HALT_SCRIPT                /usr/local/cmcluster/pkg01/pkg01.cntl
HALT_SCRIPT_TIMEOUT        NO_TIMEOUT
SERVICE_NAME               nfs1.monitor
SERVICE_FAIL_FAST_ENABLED  NO
SERVICE_HALT_TIMEOUT       300
SUBNET                     192.100.112.0
Package Control Script for pkg01

This section shows the package control script (pkg01.cntl) for the package pkg01 in this sample configuration. Only the user-configured part of the script is shown; the executable part of the script and most of the comments have been omitted.

PATH=/sbin:/usr/bin:/usr/sbin:/etc:/bin:usr/local/
cmcluster/bin
VGCHANGE="vgchange -a y"                # Default
VG[0]="nfsu01"
LV[0]=/dev/nfsu01/lvol1; 
FS[0]=/hanfs/nfsu011; 
FS_TYPE[0]="ext3"; 
FS_MOUNT_OPT[0]="-o rw"
# FS_UMOUNT_COUNT=""
# FS_MOUNT_RETRY_COUNT=""
IP[0]="192.100.112.243"
SUBNET[0]="192.100.112.0"
HA_APP_SERVER="pre-IP"
SERVICE_NAME[0]="nfs1.monitor"
SERVICE_CMD[0]="/usr/local/cmcluster/pkg01/nfs.mon"
SERVICE_RESTART[0]="-r 0"
NFS Toolkit Configuration File for pkg01

This section shows the NFS Toolkit configuration file (hanfs.conf) for the package pkg01 on this sample configuration:

XFS[0]="-o rw *:/hanfs/nfsu011"
QUOTA_MON=YES
LOCK_MIGRATION=NO
Package Configuration File for pkg02

This section shows the package configuration file (pkg02.conf) for the package pkg02 in this sample configuration. The comments are not shown.

PACKAGE_NAME               pkg02
PACKAGE_TYPE               FAILOVER
FAILOVER_POLICY            CONFIGURED_NODE
FAILBACK_POLICY            MANUAL
NODE_NAME                  thyme
NODE_NAME                  sage
NODE_NAME                  basil
AUTO_RUN                   YES
NODE_FAIL_FAST_ENABLED     NO
RUN_SCRIPT                 /usr/local/cmcluster/pkg02/pkg02.cntl
RUN_SCRIPT_TIMEOUT         NO_TIMEOUT
HALT_SCRIPT                /usr/local/cmcluster/pkg02/pkg02.cntl
HALT_SCRIPT_TIMEOUT        NO_TIMEOUT
SERVICE_NAME               nfs2.monitor
SERVICE_FAIL_FAST_ENABLED  NO
SERVICE_HALT_TIMEOUT       300
MONITORED SUBNET                192.100.112.0
MONITORED_SUBNET_ACCESS	  FULL
Package Control Script for pkg02

This section shows the package control script (pkg02.cntl) for the package pkg02 in this sample configuration. Only the user-configured part of the script is shown; the executable part of the script and most of the comments have been omitted.

PATH=/sbin:/usr/bin:/usr/sbin:/etc:/bin:usr/local/
cmcluster/bin
VGCHANGE="vgchange -a y"                # Default
VG[0]="nfsu02"
LV[0]=/dev/nfsu02/lvol1; 
FS[0]=/hanfs/nfsu021; 
FS_TYPE[0]="ext3"; 
FS_MOUNT_OPT[0]="-o rw"
# FS_UMOUNT_COUNT=""
# FS_MOUNT_RETRY_COUNT=""
IP[0]="192.100.112.244"
SUBNET[0]="192.100.112.0"
HA_APP_SERVER="pre-IP"
SERVICE_NAME[0]="nfs2.monitor"
SERVICE_CMD[0]="/usr/local/cmcluster/pkg02/nfs.mon"
SERVICE_RESTART[0]="-r 0"
NFS Control Script for pkg02

This section shows the NFS control script (hanfs.sh) for the package pkg02 on this sample configuration on the user-configured part of the script is shown:

XFS[0]="-o rw *:/hanfs/nfsu021"
QUOTA_MON=YES
LOCK_MIGRATION=NO

Sample Configuration for Modular Packages

Configuration files of two Modular packages are described in this section. These samples are applicable to Red Hat environment. For SLES user can replace all occurrences of “/usr/local” with “/opt”.

Package Configuration File for pkg01

This section shows the package configuration file (pkg01.conf) for the package pkg01 in this sample configuration. The comments are not shown.

package_name 			    pkg01
package_type      	 failover
failover_policy           	configured_node
failback_policy          	manual
node_name 			     thyme 
node_name 			     sage
auto_run                   yes
node_fail_fast_enabled     no
script_log_file 		/usr/local/cmcluster/pkg01/log 
TKIT_DIR 			     /usr/local/cmcluster/pkg01
XFS[0]				       "-o rw *:/hanfs/nfsu011"
QUOTA_MON				   YES 
LOCK_MIGRATION			NO
monitored_subnet 		192.100.112.0
monitored_subnet_access 	full 
service_name               nfs1.monitor
service_cmd     "$SGCONF/scripts/tkit/nfs/tkit_module.sh nfs_monitor"
service_restart            none
service_fail_fast_enabled  no
service_halt_timeout       300
external_script            $SGCONF/scripts/tkit/nfs/lock_migration.sh
ip_subnet 	    192.100.112.0
#ip_subnet_node
ip_address 	    192.100.112.243
vgchange_cmd 	"vgchange -a y"                
vg					         nfsu01
fs_name				     /dev/nfsu01/lvol1	
fs_directory			 /hanfs/nfsu011
fs_type				ext3
fs_mount_opt			"-o rw"
# fs_umount_opt
# fs_fsck_opt

If you want to prevent a shared adoptive node from adopting both packages together, specify the cmmodpkg command with the package control option (-d) in the external script. The external script template is provided at /usr/local/cmcluster/conf/examples/external_script.template in Red Hat and at /opt/cmcluster/conf/examples/external_script.template in SLES. Copy and rename the external_script.template to the package directory with the following command: # cp /usr/local/cmcluster/conf/examples/external_script.template \ /usr/local/cmcluster/pkg01/external_script.sh

Add the additional external script into the package configuration file. For Example:

external_script /usr/local/cmcluster/pkg01/external_script.sh

Modify the external script in the adoptive node only. Specify the cmmodpkg command with the package control option (-d) in the function start_command. As below:

function start_command
{
    sg_log 5 "start_command"
	   cmmodpkg –d –n `hostname` pkg02 &
    return $?
}

Also, add the cmmodpkg command with package control option (-e ) in the function stop_command for re-enabling the package (pkg02) when the package (pkg01) is halted. For example:

function stop_command
{
    sg_log 5 "stop_command"
	   cmmodpkg –e –n `hostname` pkg02 &
    return $?
}

This package control function can prevent an adoptive node from becoming overloaded when multiple packages fail over. If an adoptive node becomes overloaded, it can fail. In this example, if a host is an adoptive node for both pkg01 and pkg02, disabling of pkg02, in the external script for pkg01, would prevent the host that is running pkg01 from adopting pkg02, and once the package pkg01 is halted in the node the pkg02 is enabled by using the command cmmodpkg –e in the function stop_command. The ampersand (&) causes the cmmodpkg command to run in the background.

The cmmodpkg command in the background allows the control script to complete and finish bringing up the package. There is a small window of time, during which if one package has begun to fail over but the cmmodpkg command has not executed, the other package can fail over and the host will adopt it. In other words, if two packages fail over at approximately the same time, a host may adopt both packages, even though the package control option is specified. If you omit the cmmodpkg -d command from the NFS control script, host sage can adopt both pkg01 and pkg02 if their primary nodes fail.

NOTE: The above changes in the external script must be done only on the adoptive node and NOT on the external script in the primary node. If there are more than 2 packages the user may repeat the cmmodpkg command as many times required with other package names. User may consider making FAILBACK_POLICY as AUTOMATIC to free the adoptive node as early as possible so that adoptive node is ready to accept pkg02 in case it fails.
Package Configuration File for pkg02

This section shows the package configuration file (pkg02.conf) for the package pkg01 in this sample configuration. The comments are not shown.

package_name 			 pkg02
package_type      	failover
failover_policy    configured_node
failback_policy    manual
node_name 			basil 
node_name 			sage
auto_run                   yes
node_fail_fast_enabled     no
script_log_file 		/usr/local/cmcluster/pkg02/log 
TKIT_DIR 			/usr/local/cmcluster/pkg02
XFS[0]				"-o rw *:/hanfs/nfsu021"
QUOTA_MON				YES 
LOCK_MIGRATION			NO
monitored_subnet 		192.100.112.0
monitored_subnet_access 	full 
service_name               nfs2.monitor
service_cmd     "$SGCONF/scripts/tkit/nfs/tkit_module.sh nfs_monitor"
service_restart            none
service_fail_fast_enabled  no
service_halt_timeout       300
external_script            $SGCONF/scripts/tkit/nfs/lock_migration.sh
ip_subnet 	192.100.112.0
#ip_subnet_node
ip_address 	192.100.112.244
vgchange_cmd 	"vgchange -a y"                
vg					nfsu02
fs_name				    /dev/nfsu02/lvol1	
fs_directory			/hanfs/nfsu021
fs_type				ext3
fs_mount_opt			"-o rw"
# fs_umount_opt
# fs_fsck_opt

To prevent a shared adoptive node from adopting both packages together, specify the cmmodpkg command with the package control option (-d) in the external script. The external script template is provided at /usr/local/cmcluster/conf/examples/external_script.template in Red Hat and at /opt/cmcluster/conf/examples/external_script.template in SLES. Copy and rename the external_script.template to the package directory with the following command.

# cp /usr/local/cmcluster/conf/examples/external_script.template \ /usr/local/cmcluster/pkg02/external_script.sh

Add the additional external script into the package configuration file. For example:

external_script /usr/local/cmcluster/pkg02/external_script.sh

Modify the external script in the adoptive node. Specify the cmmodpkg command with the package control option (-d) in the function start_command. As below:

function start_command
{
 sg_log 5 "start_command"
	cmmodpkg –d –n `hostname` pkg01 &
 return $?
}

Also, add cmmodpkg command with package control option (-e ) in the function stop_command for re-enabling the package pkg0 when the package pkg02 is halted. For example:

function stop_command
{
 sg_log 5 "stop_command"
	cmmodpkg –e –n `hostname` pkg01 &
 return $?
}

This package control function can prevent an adoptive node from becoming overloaded when multiple packages fail over. If an adoptive node becomes overloaded, it can fail. In this example, if a host is an adoptive node for both pkg01 and pkg02, disabling of pkg01, would prevent the host that is running pkg02 from adopting pkg01, and once the package pkg02 is halted in the node the pkg01 is enabled by the command cmmodpkg –e in the function stop_command. The ampersand (&) causes the cmmodpkg command to run in the background.

The cmmodpkg command in the background allows the control script to complete and finish bringing up the package. There is a small window of time, during which if one package has begun to fail over but the cmmodpkg command has not executed, the other package can fail over and the host will adopt it. In other words, if two packages fail over at approximately the same time, a host may adopt both packages, even though the package control option is specified. If the cmmodpkg -d command is omitted from the NFS control script the host sage can adopt both pkg01 and pkg02, if their primary nodes fail.

NOTE: The above changes in the external script must be done only in the adoptive node and not on the external script in the primary node. If there are more than 2 packages the user may repeat the cmmodpkg command as many times required with other package names. Consider making FAILBACK_POLICY as AUTOMATIC to free the adoptive node as early as possible so that adoptive node is ready to accept pkg02 in case it fails.
Printable version
Privacy statement Using this site means you accept its terms Feedback to webmaster
© Hewlett-Packard Development Company, L.P.