IPSec UDP Negotiation

You can configure IPSec and IPFilter so that there is some overlap in the configurations. However, you must be sure the overlapping configurations do not block each other.

IPSec negotiates between two machines on a connection using the UDP protocol from port 500 and port 4500 if IPSec NAT traversal is used.

If the IPFilter configuration is so broad that it is blocking all UDP traffic, then IPSec cannot complete negotiations. When an IPSec negotiation is not completed, the encrypted packets are not received. If this happens, you will see an IPSec error on the initiating side of “MM negotiation timeout.”

To let IPSec complete negotiations, configure IPFilter to let the IPSec negotiation packets through.

Figure 8-3 Scenario Two

In Scenario Two, IPFilter is configured to block UDP traffic on
machine A, you want all TCP traffic to pass through, and, from
machine B on the network, you want all TCP traffic encrypted.
Machine A has IP address 10.10.10.10 and machine B has IP address 15.15.15.15.

As the TCP traffic with machine B must be encrypted, you configure IPSec on both machines using IPSec Manager. To do so, use the IP addresses to specify that the TCP traffic is to be encrypted.

When TCP traffic is initiated from A to B or from B to A, IPSec on both machines communicates through a UDP/500 connection. You must configure IPFilter on machine A to let this traffic through. To do so, add the following rules to your configuration:

pass in quick proto UDP from 15.15.15.15 port = 500 to 10.10.10.10 port = 500
pass out quick proto UDP from 10.10.10.10 port = 500 to 15.15.15.15 port = 500
block in proto UDP
block out proto UDP

These rules let IPSec traffic pass correctly.

	NOTE: You must configure IPFilter to pass traffic both in and out on UDP port 500 for IPSec to work properly. If IPFilter is used with IPSec requiring the NAT traversal function, UDP port 4500 must be set to pass for in and out traffic.