Customizing Services and Roles

Overview of the HP XC Services Configuration

HP XC System Software includes a predefined set of services that are delivered using node role assignments; however, a third-party software installation might require you to add a service that is not part of the default HP XC services model. On an HP XC system, you use the cluster_config utility to assign node roles to nodes. Each node role delivers a specific set of system services to the node. Both this document and the HP XC System Software Installation Guide describe the set of predefined node roles and the services provided by them.

You can customize the services configured on your system by adding a new service to an existing role or by creating a new role and adding a new service to it. You Customize services by using the cluster_config utility and customized sconfigure, gconfigure, nconfigure, and cconfigure configuration scripts that you write.

Table 4-3 provides a brief description of the scripts, utilities, and files that are involved in customizing services.

Service Configuration Sequence of Operation

To understand the relationship between the cluster_config utility and the service configuration scripts, it is important to know the sequence of events that occur during cluster_config processing:

Assigning Roles with the cluster_config Utility

The cluster_config utility invokes a text-based user interface for assigning roles (and thus services) to nodes. This interface accounts for scalability both in its initial default configuration (server placement) and how it assists you to configure your system.

After you use the cluster_config utility to assign roles and to configure external Ethernet connections, you are prompted for the optional expert services configuration portion of the cluster_config processing, which is delivered in the configuration scripts.

The output of the role assignment process is a list of servers assigned per service. The list of servers assigned per service is made available to the gconfigure scripts. Each gconfigure script assigns clients across all the servers passed to it, if possible, as a means of achieving scalability.

The *config.d Directories

As part of the system configuration process, the cluster_config utility searches the configuration script directories listed in Table 4-4 and runs the configuration scripts found in those directories. Table 4-4 lists the directories in the order in which they are accessed and provides the argument used to run the script.

To see the sconfig, gconfig, nconfig, and cconfig scripts that are delivered as part of the default services configuration mode, llook in the /opt/hptc/etc/*config.d/ directories.

Configuration Scripts

This section describes the conventions common to all configuration scripts and describes the following specific configuration scripts:

Common Conventions

When a node list is passed as an argument to the configuration scripts, the node list is passed in reverse numeric order, which means that the head node is first if it is present in the list.

The naming convention for script file names follows this syntax; all script names begin with a capital letter C:

C[0-9][0-9]service_name

For example, the gconfig script name for the Console Management Facility (CMF) is C50cmf. The capital letter C is required in all script names, and cmf is the service name. The number 50 represents the order in which the script runs during cluster_config processing. This number also enables you to control dependency issues. For example, if a second service depends on CMF being configured, the script name for the second service uses a number greater than 50 to ensure it is configured after CMF.

Depending upon the configuration requirements of a service, it is possible to have a C50cmf script in the gconfig.d directory and a C50cmf script in the nconfig.d directory or in any other combination of configuration directories.

It is also permissible for the script names for one or more services to use the same numeral. For this example, if a script named C50dbserver also exists in the gconfig.d directory, the scripts are executed in alphabetical order, with C50cmf executed first.

If you are adding a service, look in the script-specific directories, in /opt/hptc/etc/*config.d, and in the roles_services_ini file to ensure that a script name and service is not already in use.

The sconfigure Scripts

The sconfigure scripts in the /opt/hptc/etc/sconfig.d directory modify the golden image and are intended to configure systemwide functions, policies, and behaviors. They are intended to configure system parameters that are not related to specific services and servers. Security policies often fall into this category.

An sconfigure script often modifies files that become part of the golden image for the entire system or that modify the database in some way.

An sconfigure script is executed inside the cluster_config utility as follows:

script_path sconfigure golden_client script_path sunconfigure golden_client

The head node is the golden client, and only one golden client is supported. Each script in this directory is executed unconditionally during the sconfigure process. The sconfigure scripts return 0 (zero) on success and return a nonzero value on failures. You can stop the configuration process on a nonrecoverable sconfigure error, which is indicated by the sconfigure script exiting with a return code of 255. Alternatively, you can use config_die( ) in ConfigUtils.pm to return 255. Any exit code other than 255 continues with the configuration process.

The sconfigure scripts can include a user interface to aid in configuration and to suggest more optimal configuration choices to the user.

The sunconfigure action is not performed automatically.

The gconfigure Scripts

The gconfigure scripts in the /opt/hptc/etc/gconfig.d directory modify the golden image and set up client-to-server assignments and other global service configuration. A gconfigure script often modifies global configuration files and modifies the database to include client/server relationships for services.

The gconfigure action occurs only if one or more servers have been assigned the service. Thus, gconfigure scripts are executed conditionally.

The gconfigure scripts return 0 (zero) on success and return a nonzero value on failures. You can stop the configuration process on a nonrecoverable gconfigure error, which is indicated by the gconfigure script exiting with a return code of 255. Alternatively, you can use (config_die) in ConfigUtils.pm to return 255. Any exit code other than 255 will continue with the configuration process.

A gconfigure script can have a user interface to aid in configuration and can suggest more optimal configuration choices.

A gconfigure script is executed inside the cluster_config utility as follows:

script_path gconfigure server1 server2 server3 ... script_path gunconfigure server1 server2 server3 ... script_path gunconfigure

Combining the gunconfigure server with a server list unconfigures the specified nodes as servers for the service, but the service will still be active in the system (on other nodes). Server lists are derived from the servers specified during role assignment within the cluster_config utility. A gunconfigure script with no server list means that the service is being entirely unconfigured and will not be part of the new configuration being performed. The gunconfigure scripts are run only if some current server or servers are being removed from service.

In general, a gconfigure script assigns clients across all servers passed to it if possible. There are exceptions, such as services that must run on the head node only. Nothing precludes each gconfigure script from suggesting a more optimal allocation within its user interface. Such behavior is not expected to be typical, however.

Most services can build service-specific client lists within their gconfigure scripts completely by using a provided API. Client information that comes from the database to enable this consists of the following:

• Nodes identified in the database by the setting of node attributes (including the predefined attributes, such as na_all)

• Service-specific client selection flags (such as sa_no_client_on_server)

You can use node attributes (such as na_all) to include or exclude nodes, and you can use service attributes (such as sa_no_client_on_server) to refine the list with particular patterns. There is a small set of convenience function APIs to return per-service client lists. These sources of information (database, node attributes, service attributes) are sufficient in almost all cases to build client lists.

Writing gconfigure Scripts

The information in this section provides information about how to write a gconfigure script. A sample gconfigure script is provided in the /opt/hptc/templates/gconfig.d/gconfig_template.pl file for your reference.

The gconfigure and other configuration scripts often use the Perl Set::Node package, which is derived from the Set::Scalar package (from Perl's CPAN) to facilitate working with sets of nodes with the usual set operators (union, intersection, difference, and so on). In object-oriented terms, Set::Node is a Set::Scalar.

For more information about the Set::Scalar package from which Set::Node inherits, enter the following command to view the manpage:

# man Set::Scalar

The main difference between the Set::Scalar and Set::Node packages is that node_set->members() returns nodes in numerically sorted order, and scalar_set->members() returns members in random order. When order does not matter but efficiency does matter, use node_set->unsorted_members(). Set::Node also adds the select_by_name and select_by_attribute member functions.

Example 4-1 shows a sample gconfig script that performs client selection and client-to-server assignments.

Example 4-1 Sample gconfig Script: Client Selection and Client-to-Server Assignment

my($service) = 'my_new_service'; sub gconfigure { my(@servers) = @_; cs_reset_service($service,@servers); 1 my($client_flags) = ''; # "all selected nodes", default 2 my($assignment_flags) = ''; # "client may be assigned to itself", default 3 # Make a server set from the server list my($serverset) = Set::Node->select_by_name(@servers); 4 # Get my client list # Client list is "all nodes" my($clientset) = cs_get_all_clients 5 ($service,$serverset,$client_flags); # Assign client nodes to server nodes and modify database my(%map) = cs_assign_clients_to_servers 6 ($service,$serverset,$clientset, $assignment_flags); }

1	This instruction clears any previous client/server assignments for this service in the database.
2	The $client_flags are used by cs_get_all_clients(...) to include or to exclude specific nodes as clients. The default is all selected nodes (that is, exclude none). Service attributes for client selection are as follows: sa_no_client_on_server The clients are all nodes except for the nodes in the server set. The servers are not also clients. sa_clients_match_servers The client set is exactly the same as the server set. sa_clients_are_servers_of.service The clients of this service are the same set as the servers of some related service. sa_all_selected_nodes This is the default, which is the same as the double quote character (“); it means all nodes of the HP XC system.
3	The $assignment_flags are used to determine the pattern of the client-to-server assignments. The default is a server may be assigned to itself as a client. The following service attributes control a client's assignment to itself as a server and are valid for client and server assignments: sa_do_not_assign_to_self sa_must_assign_to_self sa_may_assign_to_self This is the default, which is the same as the double quote character (“). The following service attributes are valid for client and server assignment patterns: sa_block Assigns equal (as equal as practical) contiguous blocks of client nodes to each server subject to the additional assign to self specification. sa_block_cyclic.N - (for example, sa_block_cyclic.8) Assigns a contiguous block of N client nodes to each server in round-robin style, subject to the additional assign to self specification. sa_cyclic Is the same as sa_block_cyclic with N = 1, assigning round-robin one client at a time, subject to the additional assign to self specification. sa_locality Assigns clients to servers that have similar traffic patterns to limit the amount of network traffic. sa_default_assignment Assigns the default assignment. The default assignment is the assignment policy HP decided is the most useful for most of the services. NOTE: If necessary, you can combine patterns using a pipe character (|). For example, the following schedules clients to servers in blocks so that no server has itself as a client: sa_block|sa_do_not_assign_to_self
4	This instruction converts the server list @servers to a set of servers; a transformation of the list to a set for later use. Sets of nodes can be selected by their names or by node attributes (such as na_all and na_compute).
5	This instruction returns the set of all client nodes of the service (across the entire system). Only the client selection flags are required; the client and server assignment flags are not required.
6	This instruction assigns the clients across the server set and enters the server and client assignments into the database. This API requires the client and server assignment and scheduling flags.

The API calls are intended to make typical cases as easy and uniform as possible; the API calls are not intended to cover every possible scenario. Occasionally, you might have to resort to more basic Set::Node (Set::Scalar) operations to get the set you want (such as all nodes except the head node).

Nothing precludes a gconfig script from creating its client list in a proprietary way when it is necessary and appropriate to do so.

The API calls include the processing of the node attribute disable flags, such as:

• na_disable_server.service

• na_disable_client.service

Nodes with na_disable_server.service assigned for a service are excluded from the server list passed into the gconfig script as servers. Nodes with na_disable_client.service assigned for a service are not returned as potential clients of that service. In general, these flags are something the gconfig script does not need to handle explicitly.

Nothing precludes each gconfig script from offering optimal choices to you through its user interface. The APIs provided for server and client selection and mappings make no assertions regarding the interrelationship of the servers that are configured.

The nconfigure Scripts

An nconfigure script in the /opt/hptc/etc/nconfig.d directory modifies individual server nodes and sets up any server-specific configuration required for a particular service. An nconfigure script is executed only on nodes that are servers of the service.

An nconfigure script is executed inside the cluster_config utility as follows:

script_path nconfigure server script_path nunconfigure server

The server is the node name on which the script is running. The nconfigure action is performed only on nodes that are servers of the service. The nconfigure scripts cannot have a user interface.

The nconfigure scripts return 0 (zero) on success and return a nonzero value on failure. Any failure causes the node to be put into single-user mode so that the error can be investigated.

The nunconfigure action is not performed automatically, but it should be available to be performed manually.

The cconfigure Scripts

A cconfigure script in the /opt/hptc/etc/cconfig.d directory sets up client-specific configuration required for a particular service.

The cconfigure scripts are executed inside the cluster_config utility as follows:

script_path cconfigure client server1 server2 ... script_path cunconfigure client server1 server2 ...

The client is the node name on which the script is running. The servers are the nodes providing this service to this client. Typically, this is a single server. The cconfigure action is performed only on nodes that are clients of the service. The cconfigure scripts return 0 (zero) on success and return a nonzero value on failure. Any failure causes the node to be put into single-user mode so that the error can be investigated. The cconfigure scripts cannot have a user interface.

The cunconfigure action is not performed automatically, but it should be available to be performed manually.

Understanding Global Configuration Scripts

Imagine a role containing two or more services. With the present roles interface, every node for which this role is specified is selected as a potential server for these services. Imagine further that 10 nodes are assigned with this role. This potentially assigns these services to these 10 nodes. Eventually, these 10 nodes are passed as potential servers to the gconfig scripts for these services.

This results in two or more gconfig scripts receiving the same 10 nodes on which to schedule services. The instruction to each gconfig script is to use all of the servers it is passed, if possible.

Two or more services might not have compatible or consistent uses for the same set of 10 server nodes passed into them. Also, each service might benefit from optimizing its service assignments relative to the other services on the same set of nodes; however, there are no mechanisms to achieve this.

Another possible behavior of the system that you might want is that each gconfig script could identify that it needs only 5 of the 10 servers to satisfy optimal scale out requirements, and these scripts could choose that each use a disjoint 5 of 10 servers passed in order to spread the load. At present, for services that are part of the same role, there are no mechanisms in place to achieve this.

Advance Planning

You use the cluster_config utility to assigns roles, and hence, services, to nodes. To add a new service to the roles model, perform one of the following:

Ask yourself the following questions when you are adding a customized service to your HP XC system:

If you have a systemwide policy or behavior, an sconfigure script may be all you need. Any changes made in sconfigure scripts become part of the golden image.

Typically, if the service needs to run on specific servers and clients, you perform these tasks:

In general, your customized service configuration might require gconfigure, nconfigure, and cconfigure scripts for any new service, but you only have to provide the scripts actually needed to perform the configuration. For example, many services need no special client configuration and do not require a cconfigure script.

Editing the roles_services.ini File

Adding a new service to the roles model requires you to do one of the following:

You can accomplish both these tasks by editing the /opt/hptc/config/roles_services.ini file and adding the new service name to an existing role or a new role. This file is organized into stanzas as follows:

Creating a service.ini File

Services that are not assigned to the common role need a corresponding service.ini file that defines its configuration needs to the cluster_config utility. The service.ini files for all services are located in the /opt/hptc/config/etc/services directory.

The file name is based on the name of the service; for example, the cmf.ini and supermond.ini files are the .ini files for the cmf and supermond services, respectively. Services assigned to the common role and services for which no corresponding .ini file can be found are configured according to the default.ini file.

Evaluate any new service regarding whether or not you need to create a service-specific service.ini file with the appropriate parameters.

The default.ini file is available and may be suitable for new services. Review its contents to make sure it is compatible with your new service before allowing the cluster_config utility to use it as the default for your new service.

Example 4-2 shows the contents of the service.ini file.

Example 4-2 Sample service.ini FIle

[Config] # Is the service included in the default configuration? [0/1] service_recommended = 0 or 1 # How many nodes can each server handle optimally? # (Use 1000000 if there should be only one server per system.) optimal_scale_out = some_value # Must the service be run on the head node? [0/1] # Is it advantageous to run the service on the head node? [0/1] # (No more than one of these should be 1.) head_node_required = 0 or 1 head_node_desired = 0 or 1 # Must a server node have an external connection? [0/1] # Is it advantageous for a server to have an external connection? [0/1] # (No more than one of these should be 1.) external_connection_required = 0 or 1 external_connection_desired = 0 or 1 # Does the service need exclusive use of its server? [0/1] exclusivity_desired = 0 or 1 # If yes, a name to group other services to the same server # exclusivity_group = some_name

Description of service.ini File Parameters

The parameters in the service.ini file are defined as follows:

The [A]nalyze menu option of the cluster_config utility examines the parameters for each service.ini file and generates a report similar to the following:

[L]ist Nodes, [M]odify Nodes, [A]nalyze, [H]elp, [P]roceed, [Q]uit: A

Recommended Cluster Configuration
----------------------------------------------------------------
     3    Total nodes
     1    Nodes assigned service roles
     0    Exclusive servers
     1    Non-exclusive servers
     3    Compute nodes


                   Role  HN   HN   Ext  Ext  Exc
Recommend Assigned Rec   Req  Rec  Req  Rec  Rec  Role
----------------------------------------------------------------
     3         3                                  compute
     1         1                                  disk_io
     1         1                                  external (optional)
     1         0                                  login (optional)
     1         1                                  management_hub
     1         1                                  management_server
     1         0                                  nis_server (optional)
     1         1                                  resource_management

The column headings in the middle of the report correspond to parameters in the service.ini file as follows:

An N in these columns indicates a note; a W indicates a warning. When notes and warnings are present, more detailed descriptions are displayed immediately after the table.

All services requesting exclusive servers (exclusivity_desired=1) and specifying the same exclusivity group (exclusivity_group=group1, for instance) are grouped together on the same exclusive group of server nodes during the default initial configuration and configuration analysis if the HP XC system is of sufficient size. These hints are ignored on small HP XC systems.

Adding a New Service

The first step in adding a service is to decide whether you need to add the service to an existing role or if you need to create a new role and add the new service to it. This section describes both procedures.

Creating a New Role and Adding a Service to It

Use the following procedure to create a role for a new service, and then add that service to that role. This procedure creates a new role called newrole and creates a new service called newservice to illustrate their placement in the role_services.ini file.

	Note: Only perform this procedure when you have exclusive use of the HP XC system and when no jobs are running on the system because the procedure requires that you reimage nodes.

1. Log in as the superuser on the head node.

2. Make a copy of the current roles_services.ini file for safekeeping:

   # cd /opt/hptc/config # cp role_services.ini role_services.ini.ORIG

3. Edit the roles_services.ini file as follows:

   1. Add the name of the new role to the stanza that lists all the roles:

      roles = <<EOT node_management . . . common newrole EOT

   2. Add the name of the new service to the stanza that lists all the services:

      services = <<EOT cmf . . . swmlogger newservice EOT

   3. At the end of the file, create a new stanza specifying the role and the service:

      newrole = <<EOT newservice EOT

   4. Save the file.

4. Create the service.ini file if the service is not assigned to the common role. The file is named /opt/hptc/config/etc/services/newservice.ini in this example.

5. Ensure that the configuration scripts for newservice (gconfig, sconfig, nconfig, or cconfig) are in the appropriate /etc/hptc/etc/*config.d directory.

6. Back up the system management database:

   # managedb backup file_name

7. Run the cluster_config utility with the --init option:

   # /opt/hptc/config/sbin/cluster_config --init

   The cluster_config utility identifies the new role from the roles_services.ini file and makes it available to be assigned on any of the nodes.

   Be sure to use the [A]nalyze menu option and review the output. Ensure that the role is assigned the correct number of servers. If the assigned number of servers is insufficient, edit the service.ini file accordingly and repeat this step.

8. Reimage the applicable node or nodes as described in Chapter 10.

Verifying a New Service

Follow this procedure to verify the correct configuration of the new service: