Patch Management and Software Change Management Strategies

Patch management is a complex topic. Because of the complexity, there is not one right way to perform patch management. If you ask 10 patching experts to describe their approach to patch management, you will likely get 10 different answers. You must determine which approach to patch management works best in your situation based on your particular environment and your constraints.

This section discusses software change management and recommendations, as well as the three basic patch management strategies among others:

Proactive patch management strategy
Reactive patch management strategy
Security patch management strategy (Advanced Topic)

You may find that one of these strategies is a good fit for your organization. In most cases, a customized combination works well. For example, you could select a reactive patching strategy for most patching, but proactively patch your most update-sensitive areas. Security patch strategies often do not fit within the proactive or reactive strategies. In these cases, you need to follow a different strategy. Again, there is more than one path to creating an acceptable patch management strategy.

Establishing a Software Change Management Strategy

This section outlines a set of patch management strategies based on use and tolerance for down time. There is always a risk that software patches that have been successfully tested in a controlled environment will cause problems when applied to a new configuration. For this reason, it is important to limit the number of changes made to a target system.

The first step in defining your strategy is to determine what level of software change management you want to implement. HP has developed three strategies for dealing with software change management in mission critical environments. These strategies are based on operational requirements. The same concepts apply just as well to non-mission critical environments.

The following are three strategies for software change management. These strategies are described in Table 4-1: “Operational Factor and Patch Management Strategy Matrix”:

Restrictive
Conservative
Innovative

Table 4-1 Operational Factor and Patch Management Strategy Matrix

Patch Management Strategy	New Features	Unplanned Down Time	Impact on Core Business	Self-Maintenance
Restrictive	No	Unacceptable	High	No
Conservative	No	Unacceptable	Medium	No
Innovative	Yes	Acceptable	Low	Yes

The process of selecting an appropriate software change management strategy seeks to align behavior with the key business objectives of the systems involved. The goals of evaluating an operation and choosing an appropriate strategy include:

Reduced risk
Increased system and application availability
Reduced maintenance time

There are four operational factors that should determine your appropriate strategy:

New features
Do you need to introduce new operating system or application features into the operating environment?
Unplanned down time
What is your tolerance for the operation being unavailable outside the scheduled maintenance windows?
Impact on core business
How are business functions affected by down time?
Self-maintenance
This is an indication of whether or not all system planning and maintenance activities are performed inhouse without vendor or third-party involvement.

Recommendations for Software Change Management

The following are recommendations for software change management that correspond to each software change strategy. They cover the following five areas:

Operating System and Applications
Includes versions of the operating system as well as the applications running in the environment.
Proactive Patching
Includes all patching activities for which no symptoms or problems are currently evident.
Reactive Patching
Performed in response to a visible system problem.
Change Management
Covers all processes and standards used to manage data center operations.
Test Environment
Includes systems, software, and equipment used to support the production operations. The test environment is used to evaluate changes before they are put into production.

Table 4-2: “Recommendations Based on Strategy” offers recommendations to help you implement your chosen software change management strategy.

Table 4-2 Recommendations Based on Strategy

Strategy

OS & Applications

Proactive Patching

Reactive Patching

Change Management

Test Environment

Restrictive

Stable release, available for one year or more.

Use only thoroughly tested patches with the highest level of exposure.

Make fewest changes possible to restore function.

Perform full diagnostic analysis before attempting a solution.

Formal plan with explicit roles and responsibilities.

Prepared plan to back out changes, if necessary.

Documented disaster recovery plan that is updated and tested at least yearly.

Dedicated equipment that matches production environment, including simulated loads.

Conservative

Stable release, available for six months or more.

Use only thoroughly tested patches with substantial exposure.

Make fewest changes possible to restore function.

Perform full diagnostic analysis before attempting a solution.

Formal plan with explicit roles and responsibilities.

Prepared plan to back out changes, if necessary.

Dedicated equipment that matches production environment.

Innovative

Stable release, available for two months or more.

Carefully review patches for risks and benefits.

Focus on restoration of function.

Limit number of concurrent changes.

Established roles and responsibilities.

Test or development equipment or off hours on production environment.

Consideration of HP Patch Rating

Regardless of the type of patching strategy you choose to implement, you should include a policy detailing when it is appropriate to select patches for each HP patch rating. Based on rating alone, it is always appropriate to select a patch rating of 3, but under what circumstances will you allow patches rated 2 or 1 to be installed?

For more information about HP patch ratings, see “HP-UX Patch Ratings”.

Patch Management and Software Depots

Users with multiple systems generally find that, regardless of the type of patching strategy they choose to implement, patch management is best accomplished by managing patches in centralized software depots. You should maintain one depot for each set of similarly configured systems. You then use these depots as your patch source for all patch installations. In this way, you can maintain the same patch level on all the systems with less overall effort. Using depots also minimizes reboots when you install new patches. You should be able to install the entire content of a single depot with only a single reboot.

For more information about these SD-UX software depots, see Chapter 8: “Using Software Depots for Patch Management”.

Proactive Patching Strategy

The goal of a proactive patching strategy is problem prevention. Many patches that provide defect fixes are released long before you need them on your system. The crux of proactive patching is identifying these patches and applying them in a safe manner. By definition, your starting point for proactive patching should be a system you believe to be functioning normally. Most proactive patching can be scheduled and carefully controlled. This is one of the benefits of this approach. See Chapter 9: “Using HP-UX Software Assistant for Patch Management” and see Chapter 10: “Using Dynamic Root Disk for Patch Management”.

As compared with the reactive patching strategy (see the following section), proactive patching generally creates more system change and requires regularly scheduled patch installation maintenance windows. Although the system down time associated with patch installation is a disadvantage of proactive patching, HP highly recommends proactive patching as the strategy of choice.

The following benefits can be achieved by implementing a proactive patch management strategy:

Problem avoidance
Reduced risk
Reduced unplanned down time
Enhanced functionality and tools
Increased time for testing

Because proactive patching involves installation of patches before a problem occurs, this strategy allows more time to complete sufficient testing than does reactive patching.

Acquiring Patches for Proactive Patching

Although patching is not a one-size-fits-all process, the following generic recommended strategy embodies many of our customers' best practices:

Identify the patches to acquire. You can identify and track these on an ongoing basis, or you can engage in patch analysis that targets a specific proactive patching cycle.
Acquire the latest Quality Pack (QPK) patch bundle and, if you are planning any hardware changes, the latest Hardware Enablement (HWE) patch bundle.
Determine whether the patches included in the standard HP-UX patch bundles cover your entire list of identified patches. Use the ITRC Patch Database to acquire any missing patches.
Scan the patches for warnings, and run the HP-UX Software Assistant Tool.
Create one depot for the acquired patches and copy them into it. You can choose to copy the latest Operating Environment (OE) products to the depot.
Test the depot content.
Create a deployment plan and roll out the new depot within your maintenance window.

The following details apply to acquiring the latest QPK and HWE patch bundles:

The QPK patch bundle is an excellent vehicle for proactive patching and was created for this purpose. The HWE patch bundle contains patches required by new hardware products that HP has released. To enable or preenable support for new hardware, you should select this bundle. New HP-UX core enhancements are introduced as part of the Software Pack (SPK). If you want to install one of these new features, search for the Software Pack documentation on the HP Technical documentation Web site at http://docs.hp.com.
All the standard HP-UX patch bundles can be downloaded from the ITRC and are available on media from HP. For more information, see Chapter 5: “What Are Standard HP-UX Patch Bundles?”.
If you have a support contract at the Mission Critical level, you are entitled to a regular customer patch analysis from HP. This analysis results in the creation of custom patch bundles for your distinct computing environments.

Use the ITRC Patch Database to acquire any patches that you have not yet obtained. Compare the entire list of patches that you identified specifically for an environment with the content of the patch bundles.

If you are missing just a few patches, use the ITRC Patch Database to acquire them. For more information about using the ITRC, see Chapter 6: “Using the IT Resource Center”.
If you are missing numerous patches, you should use the SWA Tool to acquire them. See “Using HP-UX Software Assistant for Patch Management”.

The following details apply to patches with warnings, and security patches.

Although HP attempts to include only the highest-quality patches in the standard HP-UX patch bundles, occasionally a warning is issued for a patch in one of those bundles. You can review individual patch bundles for warnings using the ITRC Patch Bundles page.
You can acquire more up-to-date patches individually. Security patches are good examples of patches that you might obtain individually rather than as a part of a bundle. The Security Patch Check Tool can help you identify any security patches missing from a system. The ITRC should be your primary resource for downloading these individual patches.

Advanced Topic: HP-UX Software Assistant

HP provides the HP-UX Software Assistant (SWA) tool which you can access using the ITRC. Many HP-UX users find this tool to be especially well suited to acquiring patches for proactive patching. The SWA tool incorporates key functionality from Security Patch Check (SecPatchCk bundle name), ITRC patch assessment, and more. SWA can perform a number of checks including published security issues, installed patches with warnings, and missing patches with critical fixes. Once an analysis has been performed, you can use SWA to download any recommended patches or patch bundles and create a depot ready for installation

For information, see Chapter 9: “Using HP-UX Software Assistant for Patch Management”.

Reactive Patching Strategy

Reactive patching involves installing patches to restore system functionality after a problem occurs. The goal of reactive patching is to fix the problem as quickly as possible and with as little user disruption as possible.

Because reactive patching is so disruptive, typically only the most critical problems: panics, failures, and corruption are reactively patched. Your action depends on the software change management strategy you use. When you use a restrictive strategy (see “Recommendations for Software Change Management ”), the fewer critical problems you will need to reactively fix.

More granular changes are generally safer. While proactive patching usually involves the installation of many patches at one time, reactive patching involves installing only the patches believed to be necessary. Another difference between these two approaches is that reactive patching is likely to be performed under greater pressure and urgency than proactive patching. Even customers who typically use a proactive patch strategy may at times find it necessary to patch reactively.

The following are benefits of reactive patching:

Timely problem resolution
Controlled, minimal changes

Reactive patching has some important disadvantages as compared with proactive patching. The process of identifying a problem fix can be made more difficult as your system falls further behind the most recent patch levels available. In addition, the required patch will likely contain much more new content than if you had performed frequent proactive updates. You might also find it difficult to perform adequate testing in reactive patching situations, and this could lead to the introduction of additional problems.

Acquiring Patches for Reactive Patching

The easiest way to identify your required patch is to call the HP Response Center. This works only if you have the appropriate support contract. Alternatively, you can carefully research the problem using resources such as the ITRC. The ITRC's self-solve tools, such as the search technical knowledge base link, can help with that query. For more information, see Chapter 6: “Using the IT Resource Center”.

Next, using the ITRC Patch Database, you must identify the patches needed to resolve the problem. For reactive patch management, patch acquisition and installation should be strictly limited to the smallest set of patches believed to provide a solution to a current system problem. Do not use the unplanned down time as an opportunity to make unrelated changes. This is especially true for mission-critical systems.

Once you know what patches are needed to solve the problem, you must determine when to patch your system. In making this decision, you should consider the following factors:

Severity of the problem
Frequency of occurrence
Availability of system down time for patching

Follow these steps to patch your system reactively:

Isolate the problem and identify the patches with the highest HP rating that represent a potential fix.
Acquire the needed patches and any patches needed to satisfy dependencies.
If you have a patch depot, add these patches to it and use this as a test base.
Test the patch. In some cases the problem is so serious (such as a when a critical system is down) that you might need to omit the test step. This is especially true if it takes a long time to replicate the problem, or if the configuration is difficult to replicate. If you choose to omit testing, do so only with the knowledge of the risks you might incur.
Determine a suitable time to install the patches.
Install the patches.

If you have multiple, similarly configured systems and you need to patch one of them reactively, consider patching the remaining systems as soon as it is reasonably possible. This is because it is likely that your other systems will suffer the same problems at some future point. Additionally, there are benefits to maintain the same patch level on similar systems.

Advanced Topic: Security Patching Strategy

Security patching requires both urgency and a need to be proactive. It does not fit neatly into the proactive or reactive patching strategies. At times, you might need to apply security patches proactively prior to the next scheduled patch installation maintenance window.

When you use the ITRC to acquire patches, it is safe practice to obtain patches listed as recommended. Because of the urgency associated with security fixes, there are many instances when a security patch is too new to have this rating. However, many customers give a new security fix priority over an older patch recommended by the ITRC. Because most patches that fix a security problem fix only a single problem, this practice is not as risky as it may seem.

Advanced Topic: Scanning for Security Patches

You can use the SWA Tool to identify security patches for installation. This tool also identifies patches that have associated warnings. For more information about SWA, see Chapter 9: “Using HP-UX Software Assistant for Patch Management”.