CP R81.10 ClusterXL AdminGuide

27 June 2021
CLUSTERXL
R81.10
Administration Guide
[Classification: Protected]
Check Point Copyright Notice
© 2021 Check Point Software Technologies Ltd.
All rights reserved. This product and related documentation are protected by copyright and distributed under
licensing restricting their use, copying, distribution, and decompilation. No part of this product or related
documentation may be reproduced in any form or by any means without prior written authorization of Check
Point. While every precaution has been taken in the preparation of this book, Check Point assumes no
responsibility for errors or omissions. This publication and features described herein are subject to change
without notice.
RESTRICTED RIGHTS LEGEND:

Use, duplication, or disclosure by the government is subject to restrictions as set forth in subparagraph (c)
(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.227-7013 and FAR
52.227-19.
TRADEMARKS:
Refer to the Copyright page for a list of our trademarks.
Refer to the Third Party copyright notices for a list of relevant copyrights and third-party licenses.
Important Information
Latest Software
We recommend that you install the most recent software release to stay up-to-date with the
latest functional improvements, stability fixes, security enhancements and protection against
new and evolving attacks.
Certifications
For third party independent certification of Check Point products, see the Check Point
Certifications page.
Check Point R81.10

For more about this release, see the R81.10 home page.
Latest Version of this Document in English

Open the latest version of this document in a Web browser.
Download the latest version of this document in PDF format.
Feedback
Check Point is engaged in a continuous effort to improve its documentation.
Please help us by sending your comments.
ClusterXL R81.10 Administration Guide | 3

Revision History
Date Description
27 June 2021 First release of this document

Table of Contents
Table of Contents
Glossary 12
Introduction to ClusterXL 36
The Need for Clusters 36
ClusterXL Solution 36
How ClusterXL Works 36
The Cluster Control Protocol 37
ClusterXL Requirements and Compatibility 38
Check Point Appliances and Open Servers 38
Supported Number of Cluster Members 39
Hardware Requirements for Cluster Members 39
Software Requirements for Cluster Members 39
VMAC Mode 40
Supported Topologies for Synchronization Network 42
Clock Synchronization in ClusterXL 45
IPv6 Support for ClusterXL 45
Synchronized Cluster Restrictions 45
High Availability and Load Sharing Modes in ClusterXL 47
Introduction to High Availability and Load Sharing modes 47
High Availability 47
Load Sharing 48
Example ClusterXL Topology 49
Example Diagram 49
Defining the Cluster Member IP Addresses 50
Defining the Cluster Virtual IP Addresses 50
Defining the Synchronization Network 50
Configuring Cluster Addresses on Different Subnets 51
ClusterXL Mode Considerations 52
Choosing High Availability, Load Sharing, or Active-Active mode 52
Considerations for the Load Sharing Mode 52
IP Address Migration 52
ClusterXL Modes 53
High Availability Mode 53
Load Sharing Modes 56

Table of Contents
Load Sharing Multicast Mode 56

Load Sharing Unicast Mode 57
ClusterXL Mode Comparison 59
Cluster Failover 60
What is Failover? 60
When Does a Failover Occur? 60
What Happens When a Cluster Member Recovers? 61
How a Recovered Cluster Member Obtains the Security Policy 61
General Failover Limitations 62
Active-Active Mode in ClusterXL 63
Introduction 63
Configuring Active-Active mode 64
Dynamic Routing Failover 69
Synchronizing Connections in the Cluster 71
The Synchronization Network 72
How State Synchronization Works 73
Configuring Services to Synchronize After a Delay 74
Configuring Services not to Synchronize 75
Sticky Connections 77
Introduction to Sticky Connections 77
VPN Tunnels with 3rd Party Peers and Load Sharing 77
Third-Party Gateways in Hub and Spoke VPN Deployments 78
Configuring a Third-Party Gateway in a Hub and Spoke VPN Deployment 79
Non-Sticky Connections 81
Non-Sticky Connection Example: TCP 3-Way Handshake 81
Synchronizing Non-Sticky Connections 82
Synchronizing Clusters on a Wide Area Network 83
Configuring ClusterXL 86
Installing Cluster Members 86
Configuring Routing for Client Computers 86
Configuring the Cluster Control Protocol (CCP) Settings 87
Configuring the Cluster Object and Members 88
Using the Wizard Mode in SmartConsole 88
Using the Classic Mode in SmartConsole 92

Table of Contents
Changing the Settings of Cluster Object in SmartConsole 100

Configuring General Properties 100
Working with Cluster Topology 100
Changing the Synchronization Interface 102
Adding Another Member to an Existing Cluster 103
Removing a Member from an Existing Cluster 103
Configuring a ClusterXL in Bridge Mode 104
Advanced Features and Procedures 105
Working with VPN in Cluster 106
Configuring VPN in Clusters 106
Defining VPN Peer Clusters with Separate Management Servers 107
Working with NAT in Cluster 108
Cluster Fold and Cluster Hide 108
Configuring NAT in Cluster 108
Configuring NAT on a Cluster Member 108
Working with VLANs in Cluster 110
Configuring Link Monitoring on the Cluster Interfaces 112
Working with Bond Interfaces in Cluster 115
Bond Interfaces (Link Aggregation) 116
Bond High Availability Mode in Cluster 119
Simple Redundant Topology 120
Fully Meshed Redundancy 121
Bond Failover in High Availability Mode 122
Configuring a Bond Interface in High Availability Mode 123
Making Sure the Bond Interface is Functioning Properly 124
Failover Support for VLANs 125
Sync Redundancy 126
Configuring Bond High Availability in VRRP Cluster 127
Bond Load Sharing Mode in Cluster 129
Bond Failover in Load Sharing Mode 130
Configuring a Bond Interface in High Availability Mode 131
Configuring Minimal Number of Required Slave Interfaces 132
Making Sure the Bond Interface is Functioning Properly 133
Group of Bonds 134
Introduction 134

Table of Contents
Creating a new Group of Bonds 136

Adding a Bond interface to the existing Group of Bonds 137
Removing a Bond interface from the existing Group of Bonds 138
Deleting a Group of Bonds 140
Monitoring 140
Logs 141
Limitations 141
Performance Guidelines for Bond Interfaces 143
Troubleshooting Issues with Bonded Interfaces 145
Troubleshooting Workflow 145
Connectivity Delays on Switches 145
Advanced Cluster Configuration 146
Controlling the Clustering and Synchronization Timers 147
Blocking New Connections Under Load 148
Defining Non-Monitored Interfaces 150
Configuring Policy Update Timeout 151
Enhanced 3-Way TCP Handshake Enforcement 152
Cluster IP Addresses on Different Subnets 154
Configuring Cluster Addresses on Different Subnets 155
Example of Cluster IP Addresses on Different Subnets 156
Limitations of Cluster Addresses on Different Subnets 158
Connectivity Between Cluster Members 158
Load Sharing Multicast Mode with "Semi-Supporting" Hardware 158
Manual Proxy ARP 158
Connecting to the Cluster Members from the Cluster Network 158
Configuring Anti-Spoofing 159
Adding Another Member to an Existing Cluster 160
Adding a New Cluster Member to the Cluster Object 160
Adding an Existing Security Gateway as a Cluster Member to the Cluster Object 163
Removing a Member from an Existing Cluster 166
ISP Redundancy on a Cluster 169
Introduction 169
ISP Redundancy Modes 171
Outgoing Connections 172
Incoming Connections 173

Table of Contents
Configuring ISP Redundancy on a Cluster 174

ISP Redundancy and VPN 179
Controlling ISP Redundancy from CLI 180
Force ISP Link State 180
The ISP Redundancy Script 180
Dynamic Routing Protocols in a Cluster Deployment 181
Router IP Address 181
Routing Table Synchronization 181
Wait for Clustering 181
Failure Recovery 182
ClusterXL Configuration Commands 183
Configuring the Cluster Member ID Mode in Local Logs 186
Registering a Critical Device 187
Unregistering a Critical Device 189
Reporting the State of a Critical Device 190
Registering Critical Devices Listed in a File 191
Unregistering All Critical Devices 193
Configuring the Cluster Control Protocol (CCP) Settings 194
Initiating Manual Cluster Failover 195
Configuring the Minimal Number of Required Slave Interfaces for Bond Load Sharing 199
Configuring the Multi-Version Cluster Mechanism 202
Monitoring and Troubleshooting Clusters 203
ClusterXL Monitoring Commands 204
Viewing Cluster State 209
Viewing Critical Devices 213
Viewing Cluster Interfaces 221
Viewing Bond Interfaces 225
Viewing Cluster Failover Statistics 229
Viewing Software Versions on Cluster Members 231
Viewing Delta Synchronization 232
Viewing IGMP Status 238
Viewing Cluster Delta Sync Statistics for Connections Table 239
Viewing Cluster IP Addresses 240
Viewing the Cluster Member ID Mode in Local Logs 241
Viewing Interfaces Monitored by RouteD 242

Table of Contents
Viewing Roles of RouteD Daemon on Cluster Members 243

Viewing Cluster Correction Statistics 244
Viewing the Cluster Control Protocol (CCP) Settings 246
Viewing Latency and Drop Rate of Interfaces 247
Viewing the State of the Multi-Version Cluster Mechanism 248
Viewing Full Connectivity Upgrade Statistics 249
Monitoring Cluster Status in SmartConsole 250
Background 250
ClusterXL Log Messages 250
General Logs 251
State Logs 251
Critical Device Logs 252
Interface Logs 252
Reason Strings 253
Working with SNMP Traps 255
How to Initiate Cluster Failover 256
Troubleshooting Issues with the Critical Device "routed" 257
Background 257
Standard Behavior of the Critical Device "routed" 257
Basic Troubleshooting Steps 258
ClusterXL Error Messages 259
Command Line Reference 260
ClusterXL Configuration Commands 261
ClusterXL Monitoring Commands 264
cpconfig 269
cphastart 272
cphastop 273
cp_conf fullha 274
cp_conf ha 275
fw hastat 277
fwboot ha_conf 278
ClusterXL Scripts 279
The clusterXL_admin Script 280
The clusterXL_monitor_ips Script 284
The clusterXL_monitor_process Script 288

Table of Contents
Cluster Management APIs 292

Introduction 292
List of APIs 292
API Examples 292
Known Limitations 306

Glossary
Glossary
3
3rd party Cluster

Cluster of Check Point Security Gateways that work together in a redundant
configuration. These Check Point Security Gateways are installed on X-Series XOS, or
IPSO OS. VRRP Cluster on Gaia OS is also considered a 3rd party cluster. The 3rd party
cluster handles the traffic, and Check Point Security Gateways perform only State
Synchronization.
Active
State of a Cluster Member that is fully operational: (1) In ClusterXL, this applies to the
state of the Security Gateway component (2) In 3rd party / OPSEC cluster, this applies to
the state of the cluster State Synchronization mechanism.
Active-Active
A cluster mode (in R80.40 and higher versions), where cluster members are located in
different geographical areas (different sites, different cloud availability zones). This mode
supports the configuration of IP addresses from different subnets on all cluster
interfaces, including the Sync interfaces. Each cluster member inspects all traffic routed
to it and synchronizes the recorded connections to its peer cluster members. The traffic
is not balanced between the cluster members.
Active Up
ClusterXL in High Availability mode that was configured as Maintain current active
Cluster Member in the cluster object in SmartConsole: (1) If the current Active member
fails for some reason, or is rebooted (for example, Member_A), then failover occurs
between Cluster Members - another Standby member will be promoted to be Active (for
example, Member_B). (2) When former Active member (Member_A) recovers from a
failure, or boots, the former Standby member (Member_B) will remain to be in Active
state (and Member_A will assume the Standby state).

Glossary
Active(!)
In ClusterXL, state of the Active Cluster Member that suffers from a failure. A problem
was detected, but the Cluster Member still forwards packets, because it is the only
member in the cluster, or because there are no other Active members in the cluster. In
any other situation, the state of the member is Down. Possible states: ACTIVE(!),
ACTIVE(!F) - Cluster Member is in the freeze state, ACTIVE(!P) - This is the Pivot
Cluster Member in Load Sharing Unicast mode, ACTIVE(!FP) - This is the Pivot Cluster
Member in Load Sharing Unicast mode and it is in the freeze state.
Active/Active
See "Load Sharing".
Active/Standby
See "High Availability".
Administrator
A user with permissions to manage Check Point security products and the network
environment.
API
In computer programming, an application programming interface (API) is a set of
subroutine definitions, protocols, and tools for building application software. In general
terms, it is a set of clearly defined methods of communication between various software
components.
Appliance
A physical computer manufactured and distributed by Check Point.
ARP Forwarding
Forwarding of ARP Request and ARP Reply packets between Cluster Members by
encapsulating them in Cluster Control Protocol (CCP) packets. Introduced in R80.10
version. For details, see sk111956.

Glossary
Backup
(1) In VRRP Cluster on Gaia OS - State of a Cluster Member that is ready to be promoted
to Master state (if Master member fails). (2) In VSX Cluster configured in Virtual System
Load Sharing mode with three or more Cluster Members - State of a Virtual System on a
third (and so on) VSX Cluster Member. (3) A Cluster Member or Virtual System in this
state does not process any traffic passing through cluster.
Blocking Mode
Cluster operation mode, in which Cluster Member does not forward any traffic (for
example, caused by a failure).
Bond
A virtual interface that contains (enslaves) two or more physical interfaces for
redundancy and load sharing. The physical interfaces share one IP address and one
MAC address. See "Link Aggregation".
Bonding
See "Link Aggregation".
Bridge Mode
A Security Gateway or Virtual System that works as a Layer 2 bridge device for easy
deployment in an existing topology.
CA
Certificate Authority. Issues certificates to gateways, users, or computers, to identify
itself to connecting entities with Distinguished Name, public key, and sometimes IP
address. After certificate validation, entities can send encrypted data using the public
keys in the certificates.
CCP
See "Cluster Control Protocol".

Glossary
Certificate
An electronic document that uses a digital signature to bind a cryptographic public key to
a specific identity. The identity can be an individual, organization, or software entity. The
certificate is used to authenticate one identity to another.
CGNAT
Carrier Grade NAT. Extending the traditional Hide NAT solution, CGNAT uses improved
port allocation techniques and a more efficient method for logging. A CGNAT rule
defines a range of original source IP addresses and a range of translated IP addresses.
Each IP address in the original range is automatically allocated a range of translated
source ports, based on the number of original IP addresses and the size of the translated
range. CGNAT port allocation is Stateless and is performed during policy installation.
See sk120296.
Cluster
Two or more Security Gateways that work together in a redundant configuration - High
Availability, or Load Sharing.
Cluster Control Protocol

Proprietary Check Point protocol that runs between Cluster Members on UDP port 8116,
and has the following roles: (1) State Synchronization (Delta Sync), (2) Health checks
(state of Cluster Members and of cluster interfaces): Health-status Reports, Cluster-
member Probing, State-change Commands, Querying for cluster membership. Note:
CCP is located between the Check Point Firewall kernel and the network interface
(therefore, only TCPdump should be used for capturing this traffic). Acronym: CCP.
Cluster Correction Layer

Proprietary Check Point mechanism that deals with asymmetric connections in Check
Point cluster. The CCL provides connections stickiness by "correcting" the packets to the
correct Cluster Member: In most cases, the CCL makes the correction from the CoreXL
SND; in some cases (like Dynamic Routing, or VPN), the CCL makes the correction from
the Firewall or SecureXL. Acronym: CCL.
Cluster Interface
An interface on a Cluster Member, whose Network Type was set as Cluster in
SmartConsole in cluster object. This interface is monitored by cluster, and failure on this
interface will cause cluster failover.
Cluster Member
A Security Gateway that is part of a cluster.

Glossary
Cluster Mode
Configuration of Cluster Members to work in these redundant modes: (1) One Cluster
Member processes all the traffic - High Availability or VRRP mode (2) All traffic is
processed in parallel by all Cluster Members - Load Sharing.
Cluster Topology
Set of interfaces on all members of a cluster and their settings (Network Objective, IP
address/Net Mask, Topology, Anti-Spoofing, and so on).
ClusterXL
Cluster of Check Point Security Gateways that work together in a redundant
configuration. The ClusterXL both handles the traffic and performs State
Synchronization. These Check Point Security Gateways are installed on Gaia OS: (1)
ClusterXL supports up to 5 Cluster Members, (2) VRRP Cluster supports up to 2 Cluster
Members, (3) VSX VSLS cluster supports up to 13 Cluster Members. Note: In ClusterXL
Load Sharing mode, configuring more than 4 Cluster Members significantly decreases
the cluster performance due to amount of Delta Sync traffic.
CoreXL
A performance-enhancing technology for Security Gateways on multi-core processing
platforms. Multiple Check Point Firewall instances are running in parallel on multiple
CPU cores.
CoreXL Firewall Instance

Also CoreXL FW Instance. On a Security Gateway with CoreXL enabled, the Firewall
kernel is copied multiple times. Each replicated copy, or firewall instance, runs on one
processing CPU core. These firewall instances handle traffic at the same time, and each
firewall instance is a complete and independent firewall inspection kernel.
CoreXL SND
Secure Network Distributer. Part of CoreXL that is responsible for: Processing incoming
traffic from the network interfaces; Securely accelerating authorized packets (if
SecureXL is enabled); Distributing non-accelerated packets between Firewall kernel
instances (SND maintains global dispatching table, which maps connections that were
assigned to CoreXL Firewall instances). Traffic distribution between CoreXL Firewall
instances is statically based on Source IP addresses, Destination IP addresses, and the
IP 'Protocol' type. The CoreXL SND does not really "touch" packets. The decision to stick
to a particular FWK daemon is done at the first packet of connection on a very high level,
before anything else. Depending on the SecureXL settings, and in most of the cases, the
SecureXL can be offloading decryption calculations. However, in some other cases,
such as with Route-Based VPN, it is done by FWK daemon.

Glossary
CPHA
General term in Check Point Cluster that stands for Check Point High Availability
(historic fact: the first release of ClusterXL supported only High Availability) that is used
only for internal references (for example, inside kernel debug) to designate ClusterXL
infrastructure.
CPUSE
Check Point Upgrade Service Engine for Gaia Operating System. With CPUSE, you can
automatically update Check Point products for the Gaia OS, and the Gaia OS itself. For
details, see sk92449.
Critical Device
Also known as a Problem Notification, or pnote. A special software device on each
Cluster Member, through which the critical aspects for cluster operation are monitored.
When the critical monitored component on a Cluster Member fails to report its state on
time, or when its state is reported as problematic, the state of that member is
immediately changed to Down. The complete list of the configured critical devices
(pnotes) is printed by the 'cphaprob -ia list' command or 'show cluster members pnotes
all' command.
DAIP Gateway
A Dynamically Assigned IP (DAIP) Security Gateway is a Security Gateway where the IP
address of the external interface is assigned dynamically by the ISP.
Data Type
A classification of data. The Firewall classifies incoming and outgoing traffic according to
Data Types, and enforces the Policy accordingly.
Database
The Check Point database includes all objects, including network objects, users,
services, servers, and protection profiles.
Dead
State reported by a Cluster Member when it goes out of the cluster (due to 'cphastop'
command (which is a part of 'cpstop'), or reboot).

Glossary
Decision Function
A special cluster algorithm applied by each Cluster Member on the incoming traffic in
order to decide, which Cluster Member should process the received packet. Each
Cluster Members maintains a table of hash values generated based on connections
tuple (source and destination IP addresses/Ports, and Protocol number).
Delta Sync
Synchronization of kernel tables between all working Cluster Members - exchange of
CCP packets that carry pieces of information about different connections and operations
that should be performed on these connections in relevant kernel tables. This Delta Sync
process is performed directly by Check Point kernel. While performing Full Sync, the
Delta Sync updates are not processed and saved in kernel memory. After Full Sync is
complete, the Delta Sync packets stored during the Full Sync phase are applied by order
of arrival.
Delta Sync Retransmission

It is possible that Delta Sync packets will be lost or corrupted during the Delta Sync
operations. In such cases, it is required to make sure the Delta Sync packet is re-sent.
The Cluster Member requests the sending Cluster Member to retransmit the
lost/corrupted Delta Sync packet. Each Delta Sync packet has a sequence number. The
sending member has a queue of sent Delta Sync packets. Each Cluster Member has a
queue of packets sent from each of the peer Cluster Members. If, for any reason, a Delta
Sync packet was not received by a Cluster Member, it can ask for a retransmission of
this packet from the sending member. The Delta Sync retransmission mechanism is
somewhat similar to a TCP Window and TCP retransmission mechanism. When a
member requests retransmission of Delta Sync packet, which no longer exists on the
sending member, the member prints a console messages that the sync is not complete.
Distributed Deployment
The Check Point Security Gateway and Security Management Server products are
deployed on different computers.
Domain
A network or a collection of networks related to an entity, such as a company, business
unit or geographical location.
Domain Log Server

A Log Server for a specified Domain, as part of a Multi-Domain Log Server. It stores and
processes logs from Security Gateways that are managed by the corresponding Domain
Management Server. Acronym: DLS.

Glossary
Down
State of a Cluster Member during a failure when one of the Critical Devices reports its
state as "problem": In ClusterXL, applies to the state of the Security Gateway
component; in 3rd party / OPSEC cluster, applies to the state of the State
Synchronization mechanism. A Cluster Member in this state does not process any traffic
passing through cluster.
Dying
State of a Cluster Member as assumed by peer members, if it did not report its state for
0.7 second.
Expert Mode
The name of the full command line shell that gives full system root permissions in the
Check Point Gaia operating system.
External Network
Computers and networks that are outside of the protected network.
External Users
Users defined on external servers. External users are not defined in the Security
Management Server database or on an LDAP server. External user profiles tell the
system how to identify and authenticate externally defined users.
Failback in Cluster
Also, Fallback. Recovery of a Cluster Member that suffered from a failure. The state of a
recovered Cluster Member is changed from Down to either Active, or Standby
(depending on Cluster Mode).
Failed Member
A Cluster Member that cannot send or accept traffic because of a hardware or software
problem.

Glossary
Failover
Also, Fail-over. Transferring of a control over traffic (packet filtering) from a Cluster
Member that suffered a failure to another Cluster Member (based on internal cluster
algorithms).
Failure
A hardware or software problem that causes a Security Gateway to be unable to serve
as a Cluster Member (for example, one of cluster interface has failed, or one of the
monitored daemon has crashed). Cluster Member that suffered from a failure is declared
as failed, and its state is changed to Down (a physical interface is considered Down only
if all configured VLANs on that physical interface are Down).
Firewall
The software and hardware that protects a computer network by analyzing the incoming
and outgoing network traffic (packets).
Flapping
Consequent changes in the state of either cluster interfaces (cluster interface flapping),
or Cluster Members (Cluster Member flapping). Such consequent changes in the state
are seen in the 'Logs & Monitor' > 'Logs' (if in SmartConsole > cluster object, the cluster
administrator set the 'Track changes in the status of cluster members' to 'Log').

Glossary
Flush and ACK

Also, FnA, F&A. Cluster Member forces the Delta Sync packet about the incoming
packet and waiting for acknowledgments from all other Active members and only then
allows the incoming packet to pass through. In some scenarios, it is required that some
information, written into the kernel tables, will be Sync-ed promptly, or else a race
condition can occur. The race condition may occur if a packet that caused a certain
change in kernel tables left Member_A toward its destination and then the return packet
tries to go through Member_B. In general, this kind of situation is called asymmetric
routing. What may happen in this scenario is that the return packet arrives at Member_B
before the changes induced by this packet were Sync-ed to this Member_B. Example of
such a case is when a SYN packet goes through Member_A, causing multiple changes
in the kernel tables and then leaves to a server. The SYN-ACK packet from a server
arrives at Member_B, but the connection itself was not Sync-ed yet. In this condition, the
Member_B will drop the packet as an Out-of-State packet (First packet isn't SYN). In
order to prevent such conditions, it is possible to‎ use ‎the‎"Flush ‎and ‎ACK" (F&A)
mechanism. This mechanism can send the Delta Sync packets with all the changes
accumulated so far in the Sync buffer to the other Cluster Members, hold the original
packet that induced these changes and wait for acknowledgment from all other (Active)
Cluster Members that they received the information in the Delta Sync packet. When all
acknowledgments arrived, the mechanism will release the held original packet. This
ensures that by the time the return packet arrived from a server at the cluster, all the
Cluster Members are aware of the connection. F&A is being operated at the end of the
Inbound chain and at the end of the Outbound chain (it is more common at the
Outbound).
Forwarding
Process of transferring of an incoming traffic from one Cluster Member to another
Cluster Member for processing. There are two types of forwarding the incoming traffic
between Cluster Members - Packet forwarding and Chain forwarding. Also see
"Forwarding Layer in Cluster" and "ARP Forwarding in Cluster".

Glossary
Forwarding Layer
The Forwarding Layer is a ClusterXL mechanism that allows a Cluster Member to pass
packets to peer Cluster Members, after they have been locally inspected by the firewall.
This feature allows connections to be opened from a Cluster Member to an external host.
Packets originated by Cluster Members are hidden behind the Cluster Virtual IP address.
Thus, a reply from an external host is sent to the cluster, and not directly to the source
Cluster Member. This can pose problems in the following situations: (1) The cluster is
working in High Availability mode, and the connection is opened from the Standby
Cluster Member. All packets from the external host are handled by the Active Cluster
Member, instead. (2) The cluster is working in a Load Sharing mode, and the decision
function has selected another Cluster Member to handle this connection. This can
happen since packets directed at a Cluster IP address are distributed between Cluster
Members as with any other connection. If a Cluster Member decides, upon the
completion of the firewall inspection process, that a packet is intended for another
Cluster Member, it can use the Forwarding Layer to hand the packet over to that Cluster
Member. In High Availability mode, packets are forwarded over a Synchronization
network directly to peer Cluster Members. It is important to use secured networks only,
as encrypted packets are decrypted during the inspection process, and are forwarded as
clear-text (unencrypted) data. In Load Sharing mode, packets are forwarded over a
regular traffic network. Packets that are sent on the Forwarding Layer use a special
source MAC address to inform the receiving Cluster Member that they have already
been inspected by another Cluster Member. Thus, the receiving Cluster Member can
safely hand over these packets to the local Operating System, without further inspection.
Full High Availability

Also, Full HA Cluster Mode. A special Cluster Mode (supported only on Check Point
appliances running Gaia OS or SecurePlatform OS, where each Cluster Member also
runs as a Security Management Server. This provides redundancy both between
Security Gateways (only High Availability is supported) and between Security
Management Servers (only High Availability is supported - see sk39345).

Glossary
Full Sync
Process of full synchronization of applicable kernel tables by a Cluster Member from the
working Cluster Member(s) when it tries to join the existing cluster. This process is meant
to fetch ‎a‎"snapshot" ‎of the applicable kernel tables of already Active Cluster Member(s).
Full Sync is performed during the initialization of Check Point software (during boot
process, the first time the Cluster Member runs policy installation, during 'cpstart', during
'cphastart'). Until the Full Sync process completes successfully, this Cluster Member
remains in the Down state, because until it is fully synchronized with other Cluster
Members, it cannot function as a Cluster Member. Meanwhile, the Delta Sync packets
continue to arrive, and the Cluster Member that tries to join the existing cluster, stores
them in the kernel memory until the Full Sync completes. The whole Full Sync process is
performed by fwd daemons on TCP port 256 over the Sync network (if it fails over the
Sync network, it tries the other cluster interfaces). The information is sent by fwd
daemons in chunks, while making sure they confirm getting the information before
sending the next chunk. Also see "Delta Sync".
Gaia
Check Point security operating system that combines the strengths of both
SecurePlatform and IPSO operating systems.
Gaia Clish
The name of the default command line shell in Check Point Gaia operating system. This
is a restrictive shell (role-based administration controls the number of commands
available in the shell).
Gaia gClish
The name of the global command line shell in Check Point Gaia operating system for
Security Appliances connected to Check Point Quantum Maestro Orchestrators and for
Security Gateway Modules on Scalable Chassis. Commands you run in this shell apply
to all Security Gateway Module / Security Appliances in the Security Group.
Gaia Portal
Web interface for Check Point Gaia operating system.

Glossary
Geo Cluster
A High Availability cluster mode (in R81 and higher versions), where cluster members
are located in different geographical areas (different sites, different cloud availability
zones). This mode supports the configuration of IP addresses from different subnets on
all cluster interfaces, including the Sync interfaces. The Active cluster member inspects
all traffic routed to the cluster and synchronizes the recorded connections to its peer
cluster members. The traffic is not balanced between the cluster members. See "High
Availability".
HA not started
Output of the 'cphaprob <flag>' command or 'show cluster <option>' command on the
Cluster Member. This output means that Check Point clustering software is not started
on this Security Gateway (for example, this machine is not a part of a cluster, or
'cphastop' command was run, or some failure occurred that prevented the ClusterXL
product from starting correctly).
High Availability
A redundant cluster mode, where only one Cluster Member (Active member) processes
all the traffic, while other Cluster Members (Standby members) are ready to be promoted
to Active state if the current Active member fails. In the High Availability mode, the
Cluster Virtual IP address (that represents the cluster on that network) is associated: (1)
With physical MAC Address of Active member (2) With virtual MAC Address (see
sk50840). Acronym: HA.
Hotfix
A piece of software installed on top of the current software in order to fix some wrong or
undesired behavior.
HTU
Stands for "HA Time Unit". All internal time in ClusterXL is measured in HTUs (the times
in cluster debug also appear in HTUs). Formula in the Check Point software: 1 HTU = 10
x fwha_timer_base_res = 10 x 10 milliseconds = 100 ms.

Glossary
Hybrid
Starting in R80.20, on Security Gateways with 40 or more CPU cores, Software Blades
run in the user space (as 'fwk' processes). The Hybrid Mode refers to the state when you
upgrade Cluster Members from R80.10 (or below) to R80.20 (or above). The Hybrid
Mode is the state, in which the upgraded Cluster Members already run their Software
Blades in the user space (as fwk processes), while other Cluster Members still run their
Software Blades in the kernel space (represented by the fw_worker processes). In the
Hybrid Mode, Cluster Members are able to synchronize the required information.
ICA
Internal Certificate Authority. A component on Check Point Management Server that
issues certificates for authentication.
Init
State of a Cluster Member in the phase after the boot and until the Full Sync completes.
A Cluster Member in this state does not process any traffic passing through cluster.
Internal Network
Computers and resources protected by the Firewall and accessed by authenticated
users.
IP Tracking
Collecting and saving of Source IP addresses and Source MAC addresses from
incoming IP packets during the probing. IP tracking is a useful for Cluster Members to
determine whether the network connectivity of the Cluster Member is acceptable.
IP Tracking Policy
Internal setting that controls, which IP addresses should be tracked during IP tracking:
(1) Only IP addresses from the subnet of cluster VIP, or from subnet of physical cluster
interface (this is the default) (2) All IP addresses, also outside the cluster subnet.
IPv4
Internet Protocol Version 4 (see RFC 791). A 32-bit number - 4 sets of numbers, each set
can be from 0 - 255. For example, 192.168.2.1.

Glossary
IPv6
Internet Protocol Version 6 (see RFC 2460 and RFC 3513). 128-bit number - 8 sets of
hexadecimal numbers, each set can be from 0 - ffff. For example,
FEDC:BA98:7654:3210:FEDC:BA98:7654:3210.
Jumbo Hotfix Accumulator

Collection of hotfixes combined into a single package. Acronyms: JHA, JHF.
Link Aggregation
Technology that joins (aggregates) multiple physical interfaces together into one virtual
interface, known as a bond interface. Also known as Interface Bonding, or Interface
Teaming. This increases throughput beyond what a single connection could sustain, and
to provides redundancy in case one of the links should fail.
Load Sharing
Also, Load Balancing mode. A redundant cluster mode, where all Cluster Members
process all incoming traffic in parallel. See "Load Sharing Multicast Mode" and "Load
Sharing Unicast Mode". Acronym: LS.
Load Sharing Multicast

Load Sharing Cluster Mode, where all Cluster Members process all traffic in parallel.
Each Cluster Member is assigned the equal load of [ 100% / number_of_members ]. The
Cluster Virtual IP address (that represents the cluster on that network) is associated with
Multicast MAC Address 01:00:5E:X:Y:Z (which is generated based on last 3 bytes of
cluster Virtual IP address on that network). A ClusterXL decision algorithm (Decision
Function) on all Cluster Members decides, which Cluster Member should process the
given packet.
Load Sharing Unicast

Load Sharing Cluster Mode, where one Cluster Member (called Pivot) accepts all traffic.
Then, the Pivot member decides to process this traffic, or to forward it to other non-Pivot
Cluster Members. The traffic load is assigned to Cluster Members based on the hard-
coded formula per the value of Pivot_overhead attribute (see sk34668). The Cluster
Virtual IP address (that represents the cluster on that network) is associated with: (1)
Physical MAC Address of Pivot member (2) Virtual MAC Address (see sk50840).

Glossary
Log
A record of an action that is done by a Software Blade.
Log Server
A dedicated Check Point computer that runs Check Point software to store and process
logs in Security Management Server or Multi-Domain Security Management
environment.
Management High Availability

Deployment and configuration mode of two Check Point Management Servers, in which
they automatically synchronize the management databases with each other. In this
mode, one Management Server is Active, and the other is Standby. Acronyms:
Management HA, MGMT HA.
Management Interface
Interface on Gaia computer, through which users connect to Portal or CLI. Interface on a
Gaia Security Gateway or Cluster member, through which Management Server connects
to the Security Gateway or Cluster member.
Management Server
A Check Point Security Management Server or a Multi-Domain Server.
Master
State of a Cluster Member that processes all traffic in cluster configured in VRRP mode.
Multi-Domain Log Server

A computer that runs Check Point software to store and process logs in Multi-Domain
Security Management environment. The Multi-Domain Log Server consists of Domain
Log Servers that store and process logs from Security Gateways that are managed by
the corresponding Domain Management Servers. Acronym: MDLS.
Multi-Domain Security Management

A centralized management solution for large-scale, distributed environments with many
different Domain networks.

Glossary
Multi-Domain Server
A computer that runs Check Point software to host virtual Security Management Servers
called Domain Management Servers. Acronym: MDS.
Multi-Version Cluster
The Multi-Version Cluster (MVC) mechanism lets you synchronize connections between
cluster members that run different versions. This lets you upgrade to a newer version
without a loss in connectivity and lets you test the new version on some of the cluster
members before you decide to upgrade the rest of the cluster members.
MVC
See "Multi-Version Cluster".
Network Object
Logical representation of every part of corporate topology (physical machine, software
component, IP Address range, service, and so on).
Network Objective
Defines how the cluster will configure and monitor an interface - Cluster, Sync,
Cluster+Sync, Monitored Private, Non-Monitored Private. Configured in SmartConsole >
cluster object > 'Topology' pane > 'Network Objective'.
Non-Blocking Mode
Cluster operation mode, in which Cluster Member keeps forwarding all traffic.
Non-Monitored Interface
An interface on a Cluster Member, whose Network Type was set as Private in
SmartConsole, in cluster object.
Non-Pivot
A Cluster Member in the Unicast Load Sharing cluster that receives all packets from the
Pivot Cluster Member.

Glossary
Non-Sticky Connection
A connection is called non-sticky, if the reply packet returns via a different Cluster
Member, than the original packet (for example, if network administrator has configured
asymmetric routing). In Load Sharing mode, all Cluster Members are Active, and in
Static NAT and encrypted connections, the Source and Destination IP addresses
change. Therefore, Static NAT and encrypted connections through a Load Sharing
cluster may be non-sticky.
Open Server
A physical computer manufactured and distributed by a company, other than Check
Point.
Packet Selection
Distinguishing between different kinds of packets coming from the network, and
selecting, which member should handle a specific packet (Decision Function
mechanism): CCP packet from another member of this cluster; CCP packet from another
cluster or from a Cluster; Member with another version (usually older version of CCP);
Packet is destined directly to this member; Packet is destined to another member of this
cluster; Packet is intended to pass through this Cluster Member; ARP packets.
Pingable Host
Some host (that is, some IP address) that Cluster Members can ping during probing
mechanism. Pinging hosts in an interface's subnet is one of the health checks that
ClusterXL mechanism performs. This pingable host will allow the Cluster Members to
determine with more precision what has failed (which interface on which member). On
Sync network, usually, there are no hosts. In such case, if switch supports this, an IP
address should be assigned on the switch (for example, in the relevant VLAN). The IP
address of such pingable host should be assigned per this formula: IP_of_pingable_host
= IP_of_physical_interface_on_member + ~10. Assigning the IP address to pingable
host that is higher than the IP addresses of physical interfaces on the Cluster Members
will give some time to Cluster Members to perform the default health checks. Example:
IP address of physical interface on a given subnet on Member_A is 10.20.30.41; IP
address of physical interface on a given subnet on Member_B is 10.20.30.42; IP address
of pingable host should be at least 10.20.30.5

Glossary
Pivot
A Cluster Member in the Unicast Load Sharing cluster that receives all packets. Cluster
Virtual IP addresses are associated with Physical MAC Addresses of this Cluster
Member. This Pivot Cluster Member distributes the traffic between other Non-Pivot
Cluster Members.
Pnote
See "Critical Device".
Preconfigured Mode
Cluster Mode, where cluster membership is enabled on all Cluster Members to be.
However, no policy had been yet installed on any of the Cluster Members - none of them
is actually configured to be primary, secondary, and so on. The cluster cannot function, if
one Cluster Member ‎fails.‎ In ‎this ‎scenario,‎the "preconfigured mode" takes place. The
preconfigured mode also comes into effect when no policy is yet installed, right after the
Cluster Members came up after boot, or when running the 'cphaconf init' command.
Primary Up
ClusterXL in High Availability mode that was configured as Switch to higher priority
Cluster Member in the cluster object in SmartConsole: (1) Each Cluster Member is given
a priority (SmartConsole > cluster object > 'Cluster Members' pane). Cluster Member
with the highest priority appears at the top of the table, and Cluster Member with the
lowest priority appears at the bottom of the table. (2) The Cluster Member with the
highest priority will assume the Active state. (3) If the current Active Cluster Member with
the highest priority (for example, Member_A), fails for some reason, or is rebooted, then
failover occurs between Cluster Members. The Cluster Member with the next highest
priority will be promoted to be Active (for example, Member_B). (4) When the Cluster
Member with the highest priority (Member_A) recovers from a failure, or boots, then
additional failover occurs between Cluster Members. The Cluster Member with the
highest priority (Member_A) will be promoted to Active state (and Member_B will return
to Standby state).
Private Interface
An interface on a Cluster Member, whose Network Type was set as 'Private' in
SmartConsole in cluster object. This interface is not monitored by cluster, and failure on
this interface will not cause any changes in Cluster Member's state.

Glossary
Probing
If a Cluster Member fails to receive status for another member (does not receive CCP
packets from that member) on a given segment, Cluster Member will probe that segment
in an attempt to illicit a response. The purpose of such probes is to detect the nature of
possible interface failures, and to determine which module has the problem. The
outcome of this probe will determine what action is taken next (change the state of an
interface, or of a Cluster Member).
Problem Notification
See "Critical Device".
Ready
State of a Cluster Member during after initialization and before promotion to the next
required state - Active / Standby / VRRP Master / VRRP Backup (depending on Cluster
Mode). A Cluster Member in this state does not process any traffic passing through
cluster. A member can be stuck in this state due to several reasons - see sk42096.
Rule
A set of traffic parameters and other conditions in a Rule Base that cause specified
actions to be taken for a communication session.
Rule Base
Also Rulebase. All rules configured in a given Security Policy.
SecureXL
Check Point product that accelerates IPv4 and IPv6 traffic. Installed on Security
Gateways for significant performance improvements.
Security Gateway
A computer that runs Check Point software to inspect traffic and enforces Security
Policies for connected network resources.
Security Management Server

A computer that runs Check Point software to manage the objects and policies in Check
Point environment.

Glossary
Security Policy
A collection of rules that control network traffic and enforce organization guidelines for
data protection and access to resources with packet inspection.
Selection
The packet selection mechanism is one of the central and most important components in
the ClusterXL product and State Synchronization infrastructure for 3rd party clustering
solutions. Its main purpose is to decide (to select) correctly what has to be done to the
incoming and outgoing traffic on the Cluster Member. (1) In ClusterXL, the packet is
selected by Cluster Member(s) depending on the cluster mode: In HA modes - by Active
member; In LS Unicast mode - by Pivot member; In LS Multicast mode - by all members.
Then the Cluster Member applies the Decision Function (and the Cluster Correction
Layer). (2) In 3rd party / OPSEC cluster, the 3rd party software selects the packet, and
Check Point software just inspects it (and performs State Synchronization).
SIC
Secure Internal Communication. The Check Point proprietary mechanism with which
Check Point computers that run Check Point software authenticate each other over SSL,
for secure communication. This authentication is based on the certificates issued by the
ICA on a Check Point Management Server.
Single Sign-On
A property of access control of multiple related, yet independent, software systems. With
this property, a user logs in with a single ID and password to gain access to a connected
system or systems without using different usernames or passwords, or in some
configurations seamlessly sign on at each system. This is typically accomplished using
the Lightweight Directory Access Protocol (LDAP) and stored LDAP databases on
(directory) servers. Acronym: SSO.
SmartConsole
A Check Point GUI application used to manage Security Policies, monitor products and
events, install updates, provision new devices and appliances, and manage a multi-
domain environment and each domain.
SmartDashboard
A legacy Check Point GUI client used to create and manage the security settings in
R77.30 and lower versions.
SmartUpdate
A legacy Check Point GUI client used to manage licenses and contracts.

Glossary
Software Blade
A software blade is a security solution based on specific business needs. Each blade is
independent, modular and centrally managed. To extend security, additional blades can
be quickly added.
SSO
See "Single Sign-On".
Standalone
A Check Point computer, on which both the Security Gateway and Security Management
Server products are installed and configured.
Standby
State of a Cluster Member that is ready to be promoted to Active state (if the current
Active Cluster Member fails). Applies only to ClusterXL High Availability Mode.
State Synchronization
Technology that synchronizes the relevant information about the current connections
(stored in various kernel tables on Check Point Security Gateways) among all Cluster
Members over Synchronization Network. Due to State Synchronization, the current
connections are not cut off during cluster failover.
Sticky Connection
A connection is called sticky, if all packets are handled by a single Cluster Member (in
High Availability mode, all packets reach the Active Cluster Member, so all connections
are sticky).
Subscribers
User Space processes that are made aware of the current state of the ClusterXL state
machine and other clustering configuration parameters. List of such subscribers can be
obtained by running the 'cphaconf debug_data' command (see sk31499).
Sync Interface
Also, Secured Interface, Trusted Interface. An interface on a Cluster Member, whose
Network Type was set as Sync or Cluster+Sync in SmartConsole in cluster object. This
interface is monitored by cluster, and failure on this interface will cause cluster failover.
This interface is used for State Synchronization between Cluster Members. The use of
more than one Sync Interfaces for redundancy is not supported because the CPU load
will increase significantly due to duplicate tasks performed by all configured
Synchronization Networks. See sk92804.

Glossary
Synchronization Network
Also, Sync Network, Secured Network, Trusted Network. A set of interfaces on Cluster
Members that were configured as interfaces, over which State Synchronization
information will be passed (as Delta Sync packets ). The use of more than one
Synchronization Network for redundancy is not supported because the CPU load will
increase significantly due to duplicate tasks performed by all configured Synchronization
Networks. See sk92804.
Traffic
Flow of data between network devices.
Users
Personnel authorized to use network resources and applications.
VLAN
Virtual Local Area Network. Open servers or appliances connected to a virtual network,
which are not physically connected to the same network.
VLAN Trunk
A connection between two switches that contains multiple VLANs.
VMAC
Virtual MAC address. When this feature is enabled on Cluster Members, all Cluster
Members in High Availability mode and Load Sharing Unicast mode associate the same
Virtual MAC address with Virtual IP address. This allows avoiding issues when
Gratuitous ARP packets sent by cluster during failover are not integrated into ARP cache
table on switches surrounding the cluster. See sk50840.
VSX
Virtual System Extension. Check Point virtual networking solution, hosted on a computer
or cluster with virtual abstractions of Check Point Security Gateways and other network
devices. These Virtual Devices provide the same functionality as their physical
counterparts.

Glossary
VSX Gateway
Physical server that hosts VSX virtual networks, including all Virtual Devices that provide
the functionality of physical network devices. It holds at least one Virtual System, which
is called VS0.

Introduction to ClusterXL
The Need for Clusters
Security Gateways and VPN connections are business critical devices. The failure of a Security Gateway or
VPN connection can result in the loss of active connections and access to critical data. The Security
Gateway between the organization and the world must remain open under all circumstances.
ClusterXL Solution
ClusterXL is a Check Point software-based cluster solution for Security Gateway redundancy and Load
Sharing. A ClusterXL Security Cluster contains identical Check Point Security Gateways.
n A High Availability Security Cluster ensures Security Gateway and VPN connection redundancy by
providing transparent failover to a backup Security Gateway in the event of failure.
n A Load Sharing Security Cluster provides reliability and also increases performance, as all members
are active.
Item Description
1 Internal network
2 Switch for internal network
3 Security Gateways with ClusterXL Software Blade
4 Switch for external networks
5 Internet
How ClusterXL Works

ClusterXL uses State Synchronization to keep active connections alive and prevent data loss when a
Cluster Member fails. With State Synchronization, each Cluster Member "knows" about connections that go
through other Cluster Members.

ClusterXL uses virtual IP addresses for the cluster itself and unique physical IP and MAC addresses for the
Cluster Members. Virtual IP addresses do not belong to physical interfaces.
Note - This guide contains information only for Security Gateway clusters. For additional
information about the use of ClusterXL with VSX, see the R81.10 VSX Administration
Guide.
The Cluster Control Protocol

The Cluster Control Protocol (CCP) packets are the glue that links together the members in the Security
Cluster.
CCP traffic is distinct from ordinary network traffic and can be viewed using any network sniffer.
CCP runs on UDP port 8116 between the Cluster Members, and has the following roles:
n It allows Cluster Members to report their own states and learn about the states of other members by
sending keep-alive packets (this only applies to ClusterXL clusters).
n State Synchronization (Delta Sync).
The Check Point CCP is used by all ClusterXL modes.
Important - There is no need to add an explicit rule to the Security Policy Rule Base that
accepts CCP packets.
For more information, see:

n "Configuring the Cluster Control Protocol (CCP) Settings" on page 194
n sk25977

ClusterXL Requirements and Compatibility
ClusterXL Requirements and

Compatibility
In This Section:
Check Point Appliances and Open Servers 38

Supported Number of Cluster Members 39
Hardware Requirements for Cluster Members 39
Software Requirements for Cluster Members 39
VMAC Mode 40
Supported Topologies for Synchronization Network 42
Clock Synchronization in ClusterXL 45
IPv6 Support for ClusterXL 45
Check Point Appliances and Open Servers

You can install ClusterXL on Check Point appliances in one of these configurations:
n A Distributed configuration - the Cluster Members and the Security Management Server are installed
on different computers.
n A Full High Availability configuration - the Cluster Members and the Security Management Servers
are installed on the same computers (each computer runs a Standalone configuration).
You can install ClusterXL on Open Servers only in a distributed configuration - the Cluster Members and the
Security Management Server are installed on different computers.
To see the ClusterXL supported platforms, see the R81.10 Release Notes.
For installation instructions, see the R81.10 Installation and Upgrade Guide.

Supported Number of Cluster Members

n ClusterXL in High Availability mode supports up to 5 Cluster Members.
n ClusterXL in Load Sharing mode supports up to 5 Cluster Members.
Note - Configuring more than 4 members significantly decreases cluster

performance due to amount of Delta Sync.
n VRRP Cluster on Gaia OS supports only 2 Cluster Members (see sk105170).

n Virtual System Load Sharing (VSLS) mode supports up to 13 Cluster Members.
Hardware Requirements for Cluster Members

ClusterXL operation completely relies on internal timers and calculation of internal timeouts, which are
based on hardware clock ticks.
Therefore, in order to avoid unexpected behavior, ClusterXL is supported only between machines with
identical CPU characteristics.
Best Practice - To avoid unexpected fail-overs due to issues with CCP packets on
cluster interfaces, we strongly recommend to pair only identical physical interfaces as
cluster interfaces - even when connecting the Cluster Members via a switch.
For example:
n Intel 82598EB on Member_A with Intel 82598EB on Member_B
n Broadcom NeXtreme on Member_A with Broadcom NeXtreme on Member_B
Note - There is no requirement for throughput of Sync interface to be identical to, or

larger than throughput of traffic interfaces (although, to prevent a possible bottle neck, a
good practice for throughput of Sync interface is to be at least identical to throughput of
traffic interfaces).
Software Requirements for Cluster Members

ClusterXL is supported only between identical operating systems - all Cluster Members must be installed on
the same operating system).
ClusterXL is supported only between identical Check Point software versions - all Cluster Members must be
installed with identical Check Point software, including OS build and hotfixes.
All Check Point software components must be the same on all Cluster Members. Meaning that the same
Software Blades and features must be enabled on all Cluster Members:

n SecureXL status on all Cluster Members must be the same (either enabled, or disabled)
n Number of CoreXL Firewall instances on all Cluster Members must be the same
Notes:
l A Cluster Member with a greater number of CoreXL Firewall instances
changes its state to DOWN.

l Fail-over from a Cluster Member to a peer Cluster Member with a greater
number of CoreXL Firewall instances keeps all connections.

l Fail-over from a Cluster Member to a peer Cluster Member with a smaller
number of CoreXL Firewall instances interrupts some connections. The

connections that are interrupted are those that pass through CoreXL
Firewall instances that do not exist on the peer Cluster Member.
n Advanced Dynamic Routing status and configuration on all Cluster Members must be the same
Otherwise, traffic might not be processed as expected and/or state of Cluster Members might change
unexpectedly. In addition, Full Sync will fail.
VMAC Mode
When ClusterXL is configured in High Availability mode or Load Sharing Unicast mode (not Multicast), a
single Cluster Member is associated with the Cluster Virtual IP address. In a High Availability environment,
the single member is the Active member. In a Load Sharing environment, the single member is the Pivot.
After fail-over, the new Active member (or Pivot member) broadcasts a series of Gratuitous ARP Requests
(GARPs). The GARPS associate the Virtual IP address of the cluster with the physical MAC address of the
new Active member or the new Pivot.
When this happens:
n A member with a large number of Static NAT entries can transmit too many GARPs
Switches may not integrate these GARP updates quickly enough into their ARP tables. Switches
continue to send traffic to the physical MAC address of the member that failed. This results in traffic
outage until the switches have fully updated ARP cache tables.
n Network components, such as VoIP phones, ignore GARPs
These components continue to send traffic to the MAC address of the failed member.
To minimize possible traffic outage during a fail-over, configure the cluster to use a virtual MAC address
(VMAC).
By enabling Virtual MAC in ClusterXL High Availability mode, or Load Sharing Unicast mode, all Cluster
Members associate the same Virtual MAC address with all Cluster Virtual Interfaces and the Virtual IP
address. In Virtual MAC mode, the VMAC that is advertised by the Cluster Members (through GARP
Requests) keeps the real MAC address of each member and adds a Virtual MAC address on top of it.
For local connections and sync connections, the real physical MAC address of each Cluster Member is still
associated with its real IP address.
Note - Traffic originating from the Cluster Members will be sent with the VMAC Source address.
You can enable VMAC in SmartConsole, or on the command line. See sk50840.

Failover time in a cluster with enabled VMAC mode is shorter than a failover in a cluster that uses a physical
MAC addresses.
Configuring VMAC Mode in SmartConsole
1. Double-click the Cluster object to open its Properties window.

2. On the ClusterXL and VRRP page, select Use Virtual MAC.
3. Click OK.
4. Install the Access Control Policy on this cluster object.
Configuring VMAC Mode on the Command Line
On each Cluster Member, set the same value for the global kernel parameter fwha_vmac_global_
param_enabled.
1. Connect to the command line on each Cluster Member.
2. Log in to the Expert mode.
3. Get the current value of this kernel parameter. Run:
fw ctl get int fwha_vmac_global_param_enabled
4. Set the new value for this kernel parameter temporarily (does not survive reboot). Run:
n To enable VMAC mode:
fw ctl set int fwha_vmac_global_param_enabled 1
n To disable VMAC mode:
fw ctl set int fwha_vmac_global_param_enabled 0
5. Make sure the state of the VMAC mode was changed. Run on each Cluster Member:
n In Gaia Clish:
show cluster members interfaces all
n In the Expert mode:
cphaprob -a if
When VMAC mode is enabled, output of this command shows the VMAC address of each virtual
cluster interface.
6. To set the new value for this kernel parameter permanently:
Follow the instructions in sk26202 to add this line to the $FWDIR/boot/modules/fwkern.conf
file:
fwha_vmac_global_param_enabled=<value>

Supported Topologies for Synchronization

Network
Note - The topology numbers below are for convenience only.
Topology 1
1. The Sync interface is a Bond of several physical slave interfaces.

To work with this topology, you can configure the Bond interface in High Availability or Load
Sharing mode.
2. All physical slave interfaces on all Cluster Members connect to the same switch.
3. In this topology, Cluster Members monitor only the link state of the slave interfaces.
Topology 2

To work with this topology, you can configure the Bond interface in High Availability or Load
Sharing mode.
2. On each Cluster Member, physical slave interfaces of the same Bond connect to different
switches.
3. The switches must connect to each other through a cable.
4. In this topology, Cluster Members monitor only the link state of the slave interfaces.

Topology 3 (Enhanced Active/Backup Bond)

To work with this topology:
a. You must configure the Bond interface in the Active-Backup mode.
b. You must not configure the primary slave explicitly.
2. On each Cluster Member, physical slave interfaces of the same Bond connect to different
switches.
3. The switches do not need to connect to each other through a cable.
4. In this topology, Cluster Members:
n Monitor the link state of the slave interfaces.
n Monitor the path between the Cluster Members, communicate about their connectivity, and
agree on which bond slave interface to use.
n Send Cluster Control Protocol (CCP) packets only on the active slave interface (there is no
CCP broadcast on all the slave interfaces).
n Fail over only if all slave interfaces are down.
Additional information
If a Cluster Member does not receive CCP packets from other Cluster Members (for
example, a cable is disconnected between switches), it sends a dedicated CCP packet to
all other Cluster Members.
This dedicated CCP packet includes a map of all available bond slave interfaces on that
Cluster Member.
When other Cluster Members receive this dedicated CCP packet, they:
a. Compare the received map of available bond slave interfaces to the state of their
own bond slave interfaces.
b. Perform bond internal failover accordingly.
n Fail over to a specific selected slave interface instead of iterating over all available slave
interfaces.

Topology 4 (Enhanced Active/Backup Bond)

To work with this topology:
a. You must configure the Bond interface in the Active-Backup mode.
b. You must not configure the primary slave explicitly.
2. On each Cluster Member, physical slave interfaces of the same Bond connect to different pairs of
switches.
3. These pairs of switches connect to each other (chained together).
4. In this topology, Cluster Members:
n Monitor the link state of the slave interfaces.
n Monitor the path between the Cluster Members, communicate about their connectivity, and
agree on which bond slave interface to use.
n Send Cluster Control Protocol (CCP) packets only on the active slave interface (there is no
CCP broadcast on all the slave interfaces).
n Fail over only if all slave interfaces are down.
Additional information
If a Cluster Member does not receive CCP packets from other Cluster Members (for
example, a cable is disconnected between switches), it sends a dedicated CCP packet to
all other Cluster Members.
This dedicated CCP packet includes a map of all available bond slave interfaces on that
Cluster Member.
When other Cluster Members receive this dedicated CCP packet, they:
a. Compare the received map of available bond slave interfaces to the state of their
own bond slave interfaces.
b. Perform bond internal failover accordingly.
n Fail over to a specific selected slave interface instead of iterating over all available slave
interfaces.

Clock Synchronization in ClusterXL

When using ClusterXL, make sure to synchronize the clocks of all of the Cluster Members. You can
synchronize the clocks manually, or using a protocol such as NTP. Features, such as VPN, only function
properly when the clocks of all of the Cluster Members are synchronized.
IPv6 Support for ClusterXL

R81.10 ClusterXL supports High Availability clusters for IPv6. IPv6 status information is synchronized and
the IPv6 clustering mechanism is activated during failover.
You can define IPv6 addresses for:

n Cluster virtual interfaces
n Member physical interfaces
Limitations:
n IPv6 is not supported for Load Sharing clusters.
n You cannot define IPv6 address for synchronization interfaces.
IPv6 in ClusterXL High Availability:

During failover, a cluster sends gratuitous ARP request packets to update hosts and routers connected to
cluster interfaces. It does this by advertising the new MAC address for the virtual cluster IPv4 addresses.
ClusterXL updates the IPv6 network during failovers. ClusterXL sends Neighbor Advertisement messages
to update the neighbor cache (which is equivalent to the ARP cache in IPv4) by advertising the new MAC
address for the virtual cluster IPv6 address. In addition, ClusterXL will reply to any Neighbor Solicitation with
a target address equal to the Virtual Cluster IPv6 address.
Note - ClusterXL failover event detection is based on IPv4 probing. During state
transition the IPv4 driver instructs the IPv6 driver to reestablish IPv6 network
connectivity to the HA cluster.
Synchronized Cluster Restrictions

These restrictions apply when you synchronize Cluster Members:
n The use of more than one dedicated physical interface for synchronization redundancy is not
supported.
You can use Bonding for synchronization interface redundancy (see "Sync Redundancy" on
page 126).
Synchronization interface redundancy is not supported for VRRP Clusters. See sk92804.
n All Cluster Members must run on identically configured hardware platforms.
n If a Cluster Member goes down, user-authenticated connections through that member are lost.

Other Cluster Members cannot restore the connection.

Cluster Members maintain client-authenticated or session-authenticated connections.
The reason for this restriction is that a user space process on Cluster Members maintains the user
authentication state.
Cluster Members cannot synchronize the user space information in the same way as they
synchronize the kernel space information.
Cluster Members save the states of Session Authentication and Client Authentication in kernel tables,
which they synchronize.
n Cluster Members cannot synchronize the connection statutes that use system resources. The reason
is the same as for the user-authenticated connections.
n Accounting information for connections is accumulated on each Cluster Member, sent to the
Management Server, and aggregated.
In the event of a cluster failover, the accounting information that is not yet sent to the Management
Server, is lost.
To minimize this risk, you can reduce the time interval when accounting information is sent.
To do this, in the cluster object > Logs > Additional Logging pane, set a lower value for the Update
Account Log every attribute.

High Availability and Load Sharing Modes in ClusterXL
High Availability and Load Sharing

Modes in ClusterXL
Introduction to High Availability and Load
Sharing modes
ClusterXL is a software-based High Availability and Load Sharing solution that distributes network traffic
between clusters of redundant Security Gateways.
ClusterXL has these High Availability features:
n Transparent failover in case of member failures
n Zero downtime for mission-critical environments (when using State Synchronization)
n Enhanced throughput (in Load Sharing modes)
n Transparent upgrades
All members in the cluster are aware of the connections passing through each of the other members. The
Cluster Members synchronize their connection and status information across a secure synchronization
network.
The glue that binds the members in a ClusterXL cluster is the Cluster Control Protocol (CCP), which is used
to pass synchronization and other information between the Cluster Members.
High Availability
In a High Availability cluster, only one member is active (Active/Standby operation). In the event that the
active Cluster Member becomes unavailable, all connections are re-directed to a designated standby
without interruption. In a synchronized cluster, the standby Cluster Members are updated with the state of
the connections of the Active Cluster Member.
In a High Availability cluster, each member is assigned a priority. The highest priority member serves as the
Security Gateway in normal circumstances. If this member fails, control is passed to the next highest priority
member. If that member fails, control is passed to the next member, and so on.
Upon Security Gateway recovery, you can maintain the current Active Security Gateway (Active Up), or to
change to the highest priority Security Gateway (Primary Up).
ClusterXL High Availability mode supports both IPv4 and IPv6.

High Availability and Load Sharing Modes in ClusterXL
Load Sharing
ClusterXL Load Sharing distributes traffic within a cluster so that the total throughput of multiple members is
increased. In Load Sharing configurations, all functioning members in the cluster are active, and handle
network traffic (Active/Active operation).
If any member in a cluster becomes unreachable, transparent failover occurs to the remaining operational
members in the cluster, thus providing High Availability. All connections are shared between the remaining
Security Gateways without interruption.
ClusterXL Load Sharing modes do not support IPv6.

Example ClusterXL Topology

ClusterXL uses unique physical IP and MAC addresses for each Cluster Member, and a virtual IP
addresses for the cluster itself.
Cluster interface virtual IP addresses do not belong to any real member interface.
Example Diagram
The following diagram illustrates a two-member ClusterXL cluster, showing the cluster Virtual IP addresses
and members physical IP addresses.
This sample deployment is used in many of the examples presented in this chapter.
Item Description
1 Internal network
2 Internal switch (internal cluster IP address 10.10.0.100)
3 Security Gateway - Cluster Member A
3a Virtual interface to the internal network (10.10.0.1)
3b Interface to the Cluster Sync network (10.0.10.1)
3c Virtual interface to the external network (192.168.10.1)
4 Security Gateway - Cluster Member B
4a Virtual interface to the internal network (10.10.0.2)
4b Interface to the Cluster Sync network (10.0.10.2)
4c Virtual interface to the external network (192.168.10.2)
5 External switch (external cluster IP address 192.168.10.100)
6 Internet
Each Cluster Member has three interfaces: one external interface, one internal interface, and one for
synchronization. Cluster Member interfaces facing in each direction are connected via a hub or switch.

All Cluster Member interfaces facing the same direction must be in the same network. For example, there
must not be a router between Cluster Members.
The Management Server can be located anywhere, and connection should be established to either the
internal or external cluster IP addresses.
These sections present ClusterXL configuration concepts shown in the example.
Note - In these examples, RFC 1918 private addresses in the range 192.168.0.0 to
192.168.255.255 are treated as public IP addresses.
Defining the Cluster Member IP Addresses

The guidelines for configuring each Cluster Member are as follows:
All members within the cluster must have at least three interfaces:
n An interface facing the external network (that, for example, faces the Internet).
n An interface facing the internal network.
n An interface used for synchronization.
All interfaces pointing in a certain direction must be on the same network.
For example, in the previous illustration, there are two Cluster Members, Member_A and Member_B. Each
has an interface with an IP address facing the Internet through a hub or a switch. This is the external
interface with IP address 192.168.10.1 on Member_A and IP address 192.168.10.2 on Member_B.
Note - This release presents an option to use only two interfaces per member, one
external and one internal, and to run synchronization over the internal interface. We do
not recommend this configuration. It should be used for backup only. (See
"Synchronizing Connections in the Cluster" on page 71.)
Defining the Cluster Virtual IP Addresses

In the previous illustration, the IP address of the cluster is 192.168.10.100.
The cluster has one external virtual IP address and one internal virtual IP address.
The external IP address is 192.168.10.100, and the internal IP address is 10.10.0.100.
Defining the Synchronization Network

The previous illustration shows a synchronization interface with a unique IP address on each Cluster
Member - IP 10.0.10.1 on Member_A and IP 10.0.10.2 on Member_B.

Configuring Cluster Addresses on Different Subnets

Only one public IP address is required in a ClusterXL cluster, for the virtual cluster interface that faces the
Internet. Physical IP addresses of all Cluster Members can be private.
Configuring different subnets for the cluster IP addresses and the members IP addresses (see "Cluster IP
Addresses on Different Subnets" on page 154) is useful to:
n Configure a cluster to replace one Security Gateway in a pre-configured network, without the need to
allocate new IP addresses to the Cluster Members.
n Allow organizations to use only one public IP address for the ClusterXL Cluster. This saves public IP
addresses.

ClusterXL Mode Considerations
ClusterXL Mode Considerations

Choosing High Availability, Load Sharing, or Active-Active
mode
Which cluster mode to choose depends on the need and requirements of the organization.
n A High Availability cluster mode ensures fail-safe connectivity for the organization.
n A Load Sharing cluster mode ensures fail-safe connectivity for the organization and provides the
additional benefit of increased performance.
n An Active-Active cluster mode supports deployment of Cluster Members in different geographical
areas (in different networks).
See "ClusterXL Mode Comparison" on page 59.
Considerations for the Load Sharing Mode

Load Sharing Multicast mode is an efficient way to handle a high traffic load, because the load is distributed
optimally between all Active Cluster Members.
However, not all switches can be used for Load Sharing Multicast mode. Load Sharing Multicast mode
associates a multicast Cluster MAC addresses with a unicast Cluster Virtual IP addresses. This ensures that
traffic destined for the cluster is received by all Cluster Members.
In response to ARP Request packets for Cluster Virtual IP address, Cluster Members send ARP Replies
that contain a unicast Cluster Virtual IP address and a multicast MAC address. Some switches do not accept
such ARP Replies. For some switches, adding a static ARP entry for the unicast Cluster Virtual IP address
and the multicast MAC address will solve the issue. Other switches do not accept this type of static ARP
entry.
Another consideration is whether your deployment includes networking devices with interfaces operating in
a promiscuous mode. If on the same network segment there exist two such networking devices, and a
ClusterXL in Load Sharing Multicast mode, traffic destined for the cluster that is generated by one of the
networking device could also be processed by the other networking device.
For these cases, use Load Sharing Unicast mode, which does not require the use of multicast MAC address
for the Cluster Virtual IP addresses.
In addition, see:
n sk101539 - ClusterXL Load Sharing mode limitations and important notes
n sk162637 - ClusterXL Load Sharing mode in R80.20 and higher
IP Address Migration
If you wish to provide High Availability or Load Sharing to an existing Security Gateways configuration, we
recommend taking the existing IP addresses from the Active Security Gateway, and make these the Cluster
Virtual IP addresses, when feasible. Doing so lets you avoid altering of current IPsec endpoint identities, as
well keep Hide NAT configurations the same in many cases.

ClusterXL Modes
ClusterXL Modes
ClusterXL has several working modes.
This section briefly describes each mode and its relative advantages and disadvantages.
n High Availability Mode
n Load Sharing Multicast Mode
n Load Sharing Unicast Mode
n Active-Active Mode (see "Active-Active Mode in ClusterXL" on page 63)
Note - Many examples in the section refer to the sample deployment shown in the
"Example ClusterXL Topology" on page 49.
High Availability Mode

The ClusterXL High Availability mode provides basic High Availability capabilities in a cluster environment.
This means that the cluster can provide Firewall services even when it encounters a problem, which on a
regular Security Gateway results in a complete loss of connectivity. When combined with Check Point State
Synchronization, ClusterXL High Availability can maintain connections through failover events, in a user-
transparent manner, allowing a flawless connectivity experience. As a result, High Availability provides a
backup mechanism, which organizations can use to reduce the risk of unexpected downtime, especially in a
mission-critical environment (such as money transactions).
To achieve this, ClusterXL High Availability mode designates one of the Cluster Members as the Active
member, while the other Cluster Members remain in Standby mode. The cluster associates the Virtual IP
addresses with the physical MAC addresses of the physical interfaces on the Active member (by matching
the cluster Virtual IP address with the unique MAC address of the appropriate physical interface). Therefore,
all traffic directed at the cluster Virtual IP addresses, is actually routed (and filtered) by the Active Cluster
Member.
The role of each Cluster Member is chosen according to its cluster state and Cluster Member priority.
Cluster Member priorities correspond to the order, in which they appear in the Cluster Members page of the
cluster object in SmartConsole. The top-most Cluster Member has the highest priority. You can modify this
ranking at any time (requires policy installation and causes a failover).
In addition to its role as a Firewall, the Active Cluster Member is also responsible for informing the Standby
Cluster Members of any changes to its cluster state and kernel tables. This keeps the peer Cluster Members
up-to-date with the current traffic passing through the cluster.
Whenever the Active Cluster Member detects a problem that is severe enough to prevent this Cluster
Member from working correctly, failover occurs in the cluster. One of the Standby Cluster Members (the
Cluster Member with the next highest priority) assumes the role of the Active Cluster Member. If State
Synchronization is enabled, the new Active Cluster Member recognizes any open connections and handles
them according to their last known state.
Upon the recovery of a failed former Active Cluster Member with a higher priority, the role of the Active
Cluster Member may or may not be switched back to that Cluster Member. This depends on the cluster
object configuration - Maintain current active Cluster Member, or Switch to higher priority Cluster.
It is important to note that Standby Cluster Members may encounter problems as well. In this case, in the
event of a cluster failover, the Standby Cluster Members are not considered for the role of a new Active
Cluster Member.

ClusterXL Modes
Example
This scenario describes a connection from a Client computer on the external network to a Web server
behind the Cluster (on the internal network).
The cluster of two members is configured in High Availability mode.
Example topology:
[Client on]
[external network]
{IP 192.168.10.78/24}
{DG 192.168.10.100}
|
|
{VIP 192.168.10.100/24}
/ \
| |
{IP 192.168.10.1/24} {IP 192.168.10.2/24}
| |
{Active} {Standby}
[Member1]-----sync-----[Member2]
| |
{IP 10.10.0.1/24} {IP 10.10.0.2/24}
| |
\ /
{VIP 10.10.0.100/24}
|
|
{DG 10.10.0.100}
{IP 10.10.0.34/24}
[Web server on]
[internal network]
Chain of events:
1. The user tries to connect from his Client computer 192.168.10.78 to the Web server 10.10.0.34.
2. The Default Gateway on Client computer is 192.168.10.100 (the cluster Virtual IP address).
3. The Client computer issues an ARP Request for IP 192.168.10.100.
4. The Active Cluster Member (Member1) handles the ARP Request for IP 192.168.10.100.
5. The Active Cluster Member sends the ARP Reply with the MAC address of the external interface,
on which the IP address 192.168.10.1 is configured.
6. The Client computer sends the HTTP request packet to the Active Cluster Member - to the VIP
address 192.168.10.100 and MAC address of the corresponding external interface.
7. The Active Cluster Member handles the HTTP request packet.
8. The Active Cluster Member sends the HTTP request packet to the Web server 10.10.0.34.
9. The Web server handles the HTTP request packet.
10. The Web server generates the HTTP response packet.
11. The Default Gateway on Web server computer is 10.10.0.100 (the cluster Virtual IP address).
12. The Web server issues an ARP Request for IP 10.10.0.100.

ClusterXL Modes
13. The Active Cluster Member handles the ARP Request for IP 10.10.0.100.
14. The Active Cluster Member sends the ARP Reply with the MAC address of the internal interface,
15. The Web server sends the HTTP response packet to the Active Cluster Member - to VIP address
10.10.0.100 and MAC address of the corresponding internal interface.
16. The Active Cluster Member handles the HTTP response packet.
17. The Active Cluster Member sends the HTTP response packet to the Client computer
192.168.10.78.
18. From now, all traffic between the Client computer and the Web server is routed through the Active
Cluster Member (Member1).
19. If a failure occurs on the current Active Cluster Member (Member1), the cluster fails over.
20. The Standby Cluster Member (Member2) assumes the role of the Active Cluster Member.
21. The Standby Cluster Member sends Gratuitous ARP Requests to both the 192.168.10.x and the
10.10.0.x networks.
These GARP Requests associate the Cluster Virtual IP addresses with the MAC addresses of the
physical interfaces on the new Active Cluster Member (former Standby Cluster Member):
n Cluster VIP address 192.168.10.100 and MAC address of the corresponding external
interface, on which the IP address 192.168.10.2 is configured.
n Cluster VIP address 10.10.0.100 and MAC address of the corresponding internal interface,
22. From now, all traffic between the Client computer and the Web server is routed through the new
Active Cluster Member (Member2).
23. The former Active member (Member1) is now considered to be "down". Upon the recovery of a
former Active Cluster Member, the role of the Active Cluster Member may or may not be switched
back to that Cluster Member, depending on the cluster object configuration.

ClusterXL Modes
Load Sharing Modes

See:
n sk101539 - ClusterXL Load Sharing mode limitations and important notes
n sk162637 - ClusterXL Load Sharing mode in R80.20 and higher
Load Sharing Multicast Mode

Load Sharing lets you distribute network traffic between Cluster Members. In contrast to High Availability,
where only a single member is active at any given time, all Cluster Members in a Load Sharing solution are
active. The Cluster Members decide, which Cluster Member handles which packet. The cluster decision
function performs this task - examining each packet going through the cluster, and determining which
Cluster Member should handle it. As a result, a Load Sharing cluster utilizes all Cluster Members, which
usually leads to an increase in its total throughput.
It is important to understand that ClusterXL Load Sharing, provides a full High Availability solution as well.
When all Cluster Members are Active, traffic is evenly distributed between the Cluster Members. In case of a
failover event, caused by a problem in one of the Cluster Members, the processing of all connections
handled by the faulty member is immediately taken over by the other Cluster Members.
ClusterXL offers two separate Load Sharing solutions: Multicast and Unicast. The two modes differ in the
way Cluster Members receive the packets sent to the cluster. This section describes the Load Sharing
Multicast mode.
The Multicast mechanism, which is provided by the Ethernet network layer, allows several interfaces to be
associated with a single physical (MAC) address. Unlike Broadcast, which binds all interfaces in the same
subnet to a single MAC address, Multicast enables grouping within networks. This means that it is possible
to select the interfaces within a single subnet that will receive packets sent to a given MAC address.
ClusterXL uses the Multicast mechanism to associate the cluster Virtual IP addresses with Multicast MAC
addresses. This makes sure that all packets sent to the cluster, acting as a default gateway on the network,
will reach all Cluster Members. Each Cluster Member then decides, whether it should process the packets
or not. This decision is the core of the Load Sharing mechanism: it has to assure that at least one Cluster
Member will process each packet (so that traffic is not blocked), and that no two Cluster Members will
handle the same packets (so that traffic is not duplicated).
An additional requirement of the decision function, is to route each connection through a Cluster Member, to
ensure that packets that belong to a single connection will be processed by the same Cluster Member.
Unfortunately, this requirement cannot always be enforced, and in some cases, packets of the same
connection will be handled by different Cluster Members. ClusterXL handles these situations using its State
Synchronization mechanism, which mirrors connections on all Cluster Members.
Example
This scenario describes a user logging from the Internet to a Web server behind the cluster that is
configured in Load Sharing Multicast mode.
1. The user requests a connection from 192.168.10.78 (his computer) to 10.10.0.34 (the Web
server).
2. A router on the 192.168.10.x network recognizes 192.168.10.100 (the cluster Virtual IP address)
as the default gateway to the 10.10.0.x network.
3. The router issues an ARP Request for IP 192.168.10.100.

ClusterXL Modes
4. One of the Active Cluster Members processes the ARP Request, and responds with the Multicast
MAC assigned to the cluster Virtual IP address of 192.168.10.100.
5. When the Web server responds to the user requests, it recognizes 10.10.0.100 as its default
gateway to the Internet.
7. One of the Active members processes the ARP Request, and responds with the Multicast MAC
address assigned to the cluster Virtual IP address of 10.10.0.100.
8. All packets sent between the user and the Web server reach every Cluster Member, which decide
whether to process each packet.
9. When a cluster failover occurs, one of the Cluster Members goes down. However, traffic still
reaches all of the Active Cluster Members. Therefore, there is no need to make changes in the
network ARP routing. The only thing that changes, is the cluster decision function, which takes into
account the new state of the Cluster Members.
Load Sharing Unicast Mode

Load Sharing Unicast mode provides a Load Sharing solution adapted to environments, where Multicast
Ethernet cannot operate. In this mode, a single Cluster Member, referred to as Pivot, is associated with the
cluster Virtual IP addresses. This Pivot Cluster Member is the only Cluster Member to receive all packets
sent to the cluster. The Pivot Cluster Member is then responsible for propagating the packets to other
Cluster Members, creating a Load Sharing mechanism. Distribution of packets is performed by applying a
decision function on each packet, the same way it is done in Load Sharing Multicast mode. The difference is
that only one Cluster Member performs this selection: any non-Pivot member that receives a forwarded
packet will handle it, without applying the decision function. Note that non-Pivot Cluster Members are still
active, because they perform routing and Firewall tasks on a share of the traffic (although they do not
perform decisions).
Even though the Pivot Cluster Member is responsible for the decision process, it still acts as a Security
Gateway that processes packets (for example, the decision it makes, can be to handle a packet by itself).
However, since its additional tasks can be time consuming, it is usually assigned a smaller share of the total
load.
When a failure occurs in a non-Pivot member, its handled connections are redistributed between active non-
Pivot Cluster Members, providing the same High Availability capabilities of High Availability and Load
Sharing Multicast. When the Pivot Cluster Member encounters a problem, a regular failover event occurs,
and, in addition, another active non-Pivot member assumes the role of the new Pivot. The Pivot member is
always the active member with the highest priority. This means that when a former Pivot recuperates, it will
retain its previous role.
Example
In this scenario, we use a Load Sharing Unicast cluster as the Security Gateway between the end user
computer and the Web server.
1. The user requests a connection from 192.168.10.78 (his computer) to 10.10.0.34 (the Web
server).
2. A router on the 192.168.10.x network recognizes 192.168.10.100 (the cluster Virtual IP address)
as the default gateway to the 10.10.0.x network.
3. The router issues an ARP Request for IP 192.168.10.100.

ClusterXL Modes
4. The Pivot Cluster Member handles the ARP Request, and responds with the MAC address that
corresponds to its own unique IP address of 192.168.10.1.
5. When the Web server responds to the user requests, it recognizes 10.10.0.100 as its default
gateway to the Internet.
7. The Pivot Cluster Member handles the ARP Request, and responds with the MAC address that
corresponds to its own unique IP address of 10.10.0.1.
8. The user request packet reaches the Pivot Cluster Member on interface 192.168.10.1.
9. The Pivot Cluster Member decides that the second non-Pivot Cluster Member should handle this
packet, and forwards it to 192.168.10.2.
10. The second Cluster Member recognizes the packet as a forwarded packet, and handles it.
11. Further packets are processed by either the Pivot Cluster Member, or forwarded and processed by
the non-Pivot Cluster Member.
12. When a failover occurs from the current Pivot Cluster Member, the second Cluster Member
assumes the role of Pivot.
13. The new Pivot member sends Gratuitous ARP Requests to both the 192.168.10.x and the
10.10.0.x networks. These GARP requests associate the cluster Virtual IP address of
192.168.10.100 with the MAC address that corresponds to the unique IP address of
192.168.10.2, and the cluster Virtual IP address of 10.10.0.100 with the MAC address that
correspond to the unique IP address of 10.10.0.2.
14. Traffic sent to the cluster is now received by the new Pivot Cluster Member, and processed by it
(as it is currently the only Active Cluster Member).
15. When the former Pivot Cluster Member recovers, it re-assumes the role of Pivot, by associating
the cluster Virtual IP addresses with its own unique MAC addresses.

ClusterXL Modes
ClusterXL Mode Comparison

This table summarizes the similarities and differences between the ClusterXL modes.
Load
High Load Sharing Active-
Feature Sharing
Availability Multicast Active
Unicast
High Availability Yes Yes Yes No
Load Sharing No Yes Yes No
Performance Good Excellent Very Good Good
State Synchronization Optional Mandatory Mandatory Optional
Hardware Support All routers Not all routers are All routers All routers
supported
Number of members that deal with 1 N N N

network traffic
Number of members that receive 1 N 1 N

packets from router
How cluster answers ARP Unicast Unicast Unicast N/A

requests for a MAC address
VLAN Tagging Support Yes Yes Yes Yes

Cluster Failover
Cluster Failover
What is Failover?
Failover is a cluster redundancy operation that automatically occurs if a Cluster Member is not functional.
When this occurs, other Cluster Members take over for the failed Cluster Member.
In the High Availability mode:
n If the Active Cluster Member detects that it cannot function as a Cluster Member, it notifies the peer
Standby Cluster Members that it must go down. One of the Standby Cluster Members (with the next
highest priority) will promote itself to the Active state.
n If one of the Standby Cluster Members stops receiving Cluster Control Protocol (CCP) packets from
the current Active Cluster Member, that Standby Cluster Member can assume that the current Active
Cluster Member failed. As a result, one of the Standby Cluster Members (with the next highest
priority) will promote itself to the Active state.
n If you do not use State Synchronization in the cluster, existing connections are interrupted when
cluster failover occurs.
In Load Sharing modes:
n If a Cluster Member detects that it cannot function as a Cluster Member, it notifies the peer Cluster
Members that it must go down. Traffic load will be redistributed between the working Cluster
Members.
n If the Cluster Members stop receiving Cluster Control Protocol (CCP) packets from one of their peer
Cluster Member, those working Cluster Members can assume that their peer Cluster Member failed.
As a result, traffic load will be redistributed between the working Cluster Members.
n Because by design, all Cluster Members are always synchronized, current connections are not
interrupted when cluster failover occurs.
To tell each Cluster Member that the other Cluster Members are alive and functioning, the ClusterXL Cluster
Control Protocol (CCP) maintains a heartbeat between Cluster Members. If after a predefined time, no CCP
packets are received from a Cluster Member, it is assumed that the Cluster Member is down. As a result,
cluster failover can occur.
Note that more than one Cluster Member may encounter a problem that will result in a cluster failover event.
In cases where all Cluster Members encounter such problems, ClusterXL will try to choose a single Cluster
Member to continue operating. The state of the chosen member will be reported as Active(!). This situation
lasts until another Cluster Member fully recovers. For example, if a cross cable connecting the sync
interfaces on Cluster Members malfunctions, both Cluster Members will detect an interface problem. One of
them will change to the Down state, and the other to Active (!) state.
When Does a Failover Occur?

A failover takes place when one of the following occurs in a cluster:
n Any Critical Device reports its state as "problem" (see "Viewing Critical Devices" on page 213).
For example, the "fwd" process failed, or Security Policy is uninstalled on a Cluster Member.
n A Cluster Member does not receive Cluster Control Protocol (CCP) packets from its peer Cluster
Member.
For more on failovers, see sk62570.

Cluster Failover
What Happens When a Cluster Member Recovers?

In the High Availability mode:
n If cluster object is configured as Maintain current active Cluster Member, it means any Cluster
Member that becomes Active, remains Active.
If the Cluster Member with highest priority fails, cluster failover occurs. A Cluster Member with the
next highest priority becomes Active.
If the Cluster Member with highest priority recovers, cluster failover does not occurs again, and that
Cluster Member becomes Standby.
n If cluster object is configured as Switch to higher priority Cluster Member, it means that Cluster
Member with the highest priority always has to be Active.
Cluster Member with the highest priority is the Cluster Member that appears at the top of the list in
Cluster object > Cluster Members pane.
If the Cluster Member with the highest priority fails, cluster failover occurs. A peer Cluster Member in
Standby state, with the next highest priority, becomes Active.
If the Cluster Member with the highest priority recovers, cluster failover occurs again. The Cluster
Member with the highest priority becomes Active again. The Cluster Member with the next highest
priority that was Active, returns to the Standby state.
In the Load Sharing modes:
n When the failed Cluster Member recovers, all connections are redistributed between all Active
Cluster Members.
How a Recovered Cluster Member Obtains the Security

Policy
The Administrator installs the Security Policy on the cluster object, rather than separately on individual
Cluster Members. The policy is automatically installed on all Cluster Members. The policy is sent to the IP
addresses defined in the General Properties page of the cluster member object.
When a failed cluster member recovers, first it tries to fetch a policy from one of the peer Active Cluster
Members. The assumption is that the other Cluster Members have a more up to date policy. If fetching a
policy from peer cluster member fails, the recovered cluster member compares its own local policy to the
policy on its Management Server. If the policy on the Management Server is more up to date than the one on
the recovered cluster member, the policy is fetched from the Management Server. If the cluster member
does not have a local policy, it retrieves one from the Management Server. This ensures that all Cluster
Members use the same policy at any given moment.

Cluster Failover
General Failover Limitations

Some connections may not survive cluster failover:
n Security Servers connections.
n Connections that are handled by the Check Point services, in which the option Synchronize
connections on cluster is disabled.
n Connections initiated by the Cluster Member itself.
n TCP connections handled by the Check Point Active Streaming (CPAS) or Passive Streaming Layer
(PSL) mechanism.
n Connections handled by Software Blades:
l If the IPS Software Blade in the cluster object R77.30 is configured to Prefer connectivity, and
the Cluster Member that owns the connections is Down, then the connection is accepted
without inspection.
Otherwise, the Cluster Members drop the connection.
l For all other Software Blades:
o If the destination Cluster Member is available, the connection is forwarded to the Cluster
Member that owns the connection.
o If the destination Cluster Member is not available, the Cluster Members drop the
connection.

Active-Active Mode in ClusterXL

Introduction
R80.40 introduced a new ClusterXL mode called Active-Active.
This mode is designed for a cluster, whose Cluster Members are located in different geographical areas
(different sites, different cloud availability zones).
The IP addresses of the interfaces on each Cluster Member are on different networks (including the Sync
interfaces).
Each Cluster Member inspects all traffic routed to it and synchronizes the recorded connections to its peer
Cluster Members.
The traffic is not balanced between the members.
Example Topology:
Important:
n You can configure the Active-Active mode only on Management Server R80.40
(and higher) and only on ClusterXL R80.40 (and higher).
n The Active-Active mode does not provide Load Sharing of the traffic.
Administrator must monitor the load on each Cluster Member (see "Monitoring
and Troubleshooting Clusters" on page 203).
n Cluster Control Protocol (CCP) Encryption must be enabled, which is the default
(see "Configuring the Cluster Control Protocol (CCP) Settings" on page 194).
n It is possible to configure the Cluster Control Protocol (CCP) to work on Layer 2.
n It is possible to enable Dynamic Routing on each Cluster Member, so they
become routers in the applicable Area or Autonomous System on the site.
In this case, you must enable the Bidirectional Forwarding Detection (BFD - ip-
reachability-detection) on each interface that is configured for dynamic routing.

Configuring Active-Active mode

Step Instructions
1 Install one ClusterXL Cluster Member on each site.

See the R81.10 Installation and Upgrade Guide.
Important:
On all Cluster Members in Active-Active mode, names of interfaces that belong to
the same "side" must be identical (Known Limitation PMTR-70256).
Examples:
n If you connected the interface eth1 to Switch #A on one Cluster Member,
then you must connect the interface eth1 to Switch #A on all other Cluster
Members.
n If you configured the interface eth2 as a Sync interface on one Cluster
Member, then you must configure the interface eth2 as a Sync on all other
Cluster Members.
2 Optional: On each Cluster Member, enable Dynamic Routing, so they become routers in the
applicable Area or Autonomous System on the site.
Important - In this case, you must enable the Bidirectional Forwarding Detection
(BFD - ip-reachability-detection) on each interface that is configured for dynamic
routing.
See the R81.10 Gaia Advanced Routing Administration Guide.
3 Connect with SmartConsole to the Management Server.
4 From the left navigation panel, click Gateways & Servers.
5 Create a new Cluster object in one of these ways:
n From the top toolbar, click the New ( ) > Cluster > Cluster.
n In the top left corner, click Objects menu > More object types > Network Object >
Gateways and Servers > Cluster > New Cluster.
n In the top right corner, click Objects Pane > New > More > Network Object >
Gateways and Servers > Cluster > Cluster.
In the Check Point Security Gateway Cluster Creation window, you must click Classic
Mode.
6 From the left tree, click General Properties.

a. In the IPv4 Address field, you must enter the 0.0.0.0 address.
b. On the Network Security tab, you must clear IPsec VPN.
c. For the list of supported Software Blades, see the Known Limitation PMTR-70255 in
sk170418.

Step Instructions
7 From the left tree, click Cluster Members.

a. Click Add > New Cluster Member.
The Cluster Member Properties window opens.
b. In the Name field, enter the applicable name for the first Cluster Member object.
c. Configure the main physical IP address(es) for this Cluster Member object.
In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
addresses that you configured on the Management Connection page of the Cluster
Member's First Time Configuration Wizard.
Make sure the Security Management Server or Multi-Domain Server can connect to
these IP addresses.
d. Click Communication.
e. In the One-time password and Confirm one-time password fields, enter the same
Activation Key you entered during the Cluster Member's First Time Configuration
Wizard.
f. Click Initialize.
g. Click Close.
h. Click OK.
i. Repeat Steps a-h to add the second Cluster Member.
If the Trust State field does not show Trust established, perform these steps:
a. Connect to the command line on the Cluster Member.
b. Make sure there is a physical connectivity between the Cluster Member and the
Management Server (for example, pings can pass).
c. Run:
cpconfig
d. Enter the number of this option:
Secure Internal Communication
e. Follow the instructions on the screen to change the Activation Key.
f. In SmartConsole, click Reset.
g. Enter the same Activation Key you entered in the cpconfig menu.
h. In SmartConsole, click Initialize.

Step Instructions
8 From the left tree, click ClusterXL and VRRP.

a. In the Select the cluster mode and configuration section, select Active-Active.
b. In the Tracking section, select the applicable option.
c. Optional: Select Use State Synchronization.
This configures the Cluster Members to synchronize the information about the
connections they inspect.
Best Practice - Enable this setting to prevent connection drops after a cluster
failover.
d. Optional: Select Start synchronizing [ ] seconds after connection initiation and enter
the applicable value.
This option is available only for clusters R80.20 and higher.
To prevent the synchronization of short-lived connections (which decreases the cluster
performance), you can configure the Cluster Members to start the synchronization of all
connections a number of seconds after they start.
Range: 2 - 60 seconds
Default: 3 seconds
Notes:
n This setting in the cluster object applies to all connections that pass
through the cluster.
You can override this global cluster synchronization delay in the
properties of applicable services - see "Configuring Services to
Synchronize After a Delay" on page 74.
n The greater this value, the fewer short-lived connections the Cluster
Members have to synchronize.
n The connections that the Cluster Members did not synchronize, do not
survive a cluster failover.
Best Practice - Enable and configure this setting to increase the cluster
performance.
9 Click OK to update the cluster object properties with the new cluster mode.
10 Open the cluster object and continue the configuration.

Step Instructions
11 From the left tree, click Network Management.

a. From the top, click the Get Interfaces > Get Interfaces With Topology.
Important - On all Cluster Members in Active-Active mode, names of
interfaces that belong to the same "side" must be identical (Known Limitation
PMTR-70256).
b. Select each interface and click Edit.
c. From the left tree, click the General page.
d. In the General section, in the Network Type field, select the applicable type:
n For cluster traffic interfaces, select Cluster.
In the IPv4 field, the dummy IP address 0.0.0.0 / 24 is configured automatically.
Note - SmartConsole requires an IP address configured for each

interface. Cluster Members do not get and do not use this IP address.
nFor cluster synchronization interfaces, select Sync.

Notes:
l Check Point cluster supports only one synchronization network.
If redundancy is required, configure a Bond interface.

l In the Network Type field, it is not supported to select
"Cluster+Sync" (Known Limitation PMTR-70259) when you

deploy a cluster in a cloud (for example: AWS, Azure).
n For interfaces that do not pass the traffic between the connected networks, select
Private.
e. In the Member IPs section, make sure the IPv4 address and its Net Mask are correct on
each Cluster Member.
f. In the Topology section:
Make sure the settings are correct in the Leads To and Security Zone fields.
Only these options are supported on cluster interfaces (Known Limitation PMTR-
70260):
n Override > Network defined by routes (this is the default).
n Override > Specific > select the applicable Network object or Network Group
object.
12 Click OK.
13 Publish the SmartConsole session.
14 Configure and install the applicable Access Control Policy.
15 Configure and install the applicable Threat Prevention Policy.

Step Instructions
16 Examine the cluster state.

See "Viewing Cluster State" on page 209.
Example
Member1> show cluster state
Cluster Mode: Active Active with IGMP Membership
ID Unique Address Assigned Load State Name
1 (local) 11.22.33.245 N/A ACTIVE Member1

2 11.22.33.246 N/A ACTIVE Member2
Active PNOTEs: None
... ... ...
Member1>

Dynamic Routing Failover

By design, a Cluster Member changes its state to DOWN in these cases:
n If there is an issue with the Sync interface (interface state or interface link).
In this case, the Critical Device Interfaces Active Check on the Cluster Member reports its state as
"problem".
n If you run the command "clusterXL_admin down" (see "The clusterXL_admin Script" on
page 280).
In this case, the Critical Device admin_down on the Cluster Member reports its state as "problem".
When the cluster state of a Cluster Member is DOWN, it stops processing the dynamic routing traffic to force
the next hop router to update its routing tables. As a result, there may be a network outage, because it takes
time for dynamic routing protocols to update their routing tables and propagate the changes.

Note - If it is necessary that Cluster Members change their cluster state because of
other Critical Devices, you must manually configure this behavior.
Procedure
1. Connect to the command line on the Cluster Member.

3. Get the list of all Critical Devices:
cphaprob -l list
4. Copy the names of the applicable Critical Devices (case sensitive)
5. Back up the current $FWDIR/boot/modules/fwkern.conf file:
cp -v $FWDIR/boot/modules/fwkern.conf{,_BKP}
6. Edit the current $FWDIR/boot/modules/fwkern.conf file:
vi $FWDIR/boot/modules/fwkern.conf
7. Add this line:
fwha_disable_member_state_
pnotelist="<
NameOfCriticalDevice1>,<NameOfCriticalDevice2>,..."
Example:
fwha_disable_member_state_pnotelist="ted,Interface
Active Check,cvpnd"
Important:
n This configuration files do not support space characters,
tabulation characters, and comments (lines that contain the #
character).
n You must write the names of Critical Devices exactly as they
appear in the output of the "cphaprob -l list" command (in
the line "Device Name:").
n Names of Critical Devices must be separated by comma (without
spaces). Do not enclose the names of Critical Devices in quotes.
8. Save the changes in the file and exit the editor.
9. Reboot the Cluster Member.
10. If you cannot reboot at this time, then configure this behavior temporarily with this
command:
fw ctl set str fwha_disable_member_state_pnotelist
"<NameOfCriticalDevice1>,<NameOfCriticalDevice2>,..."
Example:
fw ctl set str fwha_disable_member_state_pnotelist
"ted,Interface Active Check,cvpnd"
Important:
n You must write the names of Critical Devices exactly as they
appear in the output of the "cphaprob -l list" command (in
the line "Device Name:").
n Names of Critical Devices must be separated by comma (without
spaces).
11. Make sure the Cluster Member applied the new configuration:
fw ctl get str fwha_disable_member_state_pnotelist

Synchronizing Connections in the Cluster
Synchronizing Connections in the

Cluster
A failure of a Firewall results in an immediate loss of active connections in and out of the organization. Many
of these connections, such as financial transactions, may be mission critical, and losing them will result in
the loss of critical data. ClusterXL supplies an infrastructure that ensures that no data is lost in case of a
failure, by making sure each Cluster Member is aware of the connections going through the other members.
Passing information about connections and other Security Gateway states between the Cluster Members is
called State Synchronization.
Every IP-based service (including TCP and UDP) recognized by the Security Gateway, is synchronized.
Members of a ClusterXL in Load Sharing mode must be synchronized.
Members of a ClusterXL in High Availability mode do not have to be synchronized. Although, if they are not
synchronized, current connections are interrupted during cluster failover.

The Synchronization Network
The Synchronization Network

The Synchronization Network is used to transfer synchronization information about connections and other
Security Gateway states between Cluster Members.
The synchronization network carries the most sensitive Security Policy information in the organization.
Therefore, it is critical that you protect it against both malicious and unintentional threats.
We recommend that you secure the synchronization interfaces using one of these strategies:
n Enable the CCP Encryption (this is the default) on the Cluster Members (see "Configuring the Cluster
Control Protocol (CCP) Settings" on page 194).
n Use a dedicated synchronization network.
n Connecting the physical network interfaces of the Cluster Members directly using a cross-cable. In a
cluster with three or more members, use a dedicated hub or switch.
Notes:
n See "Supported Topologies for Synchronization Network" on page 42.
n You can synchronize members across a WAN. See "Synchronizing Clusters on a
Wide Area Network" on page 83.
n In ClusterXL, the synchronization network is supported on the lowest VLAN tag of
a VLAN interface.
For example, if three VLANs with tags 10, 20 and 30 are configured on interface
eth1, only interface eth1.10 may be used for synchronization.

How State Synchronization Works
How State Synchronization Works

Synchronization works in two modes:
n Full Sync transfers all Security Gateway kernel table information from one Cluster Member to
another.
The fwd daemon handles the Full Sync using an encrypted TCP connection on port 256.
n Delta Sync transfers the changes in the kernel tables between Cluster Members.
The Security Gateway kernel handles the Delta Sync using UDP connections on port 8116.
Full Sync is used for initial transfers of state information, when a Cluster Member joins the cluster. If a
Cluster Member is brought up after being down, it performs the Full Sync with the Active peer Cluster
Member(s). After all Cluster Members are synchronized, only updates are transferred using the Delta Sync,
because the Delta Sync is quicker than the Full Sync.
State Synchronization traffic typically makes up around 90% of all Cluster Control Protocol (CCP) traffic.
Cluster Members distinguish the State Synchronization packets from the rest of CCP traffic based on the
opcode in the UDP data header.

Configuring Services to Synchronize After a Delay
Configuring Services to Synchronize After a

Delay
Some TCP services (for example, HTTP) are characterized by connections with a very short duration. There
is no point to synchronize these connections, because every synchronized connection consumes resources
on Cluster Members, and the connection is likely to have finished by the time a cluster failover occurs.
For short-lived services, you can use the Delayed Notifications feature to delay telling the Cluster Member
about a connection, so that the connection is only synchronized, if it still exists X seconds after the
connection was initiated. The Delayed Notifications feature requires SecureXL to be enabled on all Cluster
Members (this is the default).
Procedure:
1. In SmartConsole, click Objects > Object Explorer.
2. In the left tree, click the small arrow on the left of the Services to expand this category
3. In the left tree, select TCP.
4. Search for the applicable TCP service.
5. Double-click the applicable TCP service.
6. In the TCP service properties window, click Advanced page.
7. At the top, select Override default settings.
On Domain Management Server: select Override global domain settings.
8. At the bottom, in the Cluster and synchronization section, select Start synchronizing and enter the
applicable value.
Important - This change applies to all policies that use this service.
9. Click OK.
10. Close the Object Explorer.
11. Publish the SmartConsole session.
12. Install the Access Control Policy on the cluster object.
Note - The Delayed Notifications setting in the service object is ignored, if Connection
Templates are not offloaded by the Firewall to SecureXL. For additional information
about the Connection Templates, see the R81.10 Performance Tuning Administration
Guide.

Configuring Services not to Synchronize

Synchronization of connections incurs a performance cost. Not all connections that go through a cluster
must be synchronized:
n Protocols that run solely between Cluster Members need not be synchronized. Although, you can
synchronize them, you go not gain any benefit. This synchronization information does not help during
a cluster failover.
n You can decide not to synchronize TCP, UDP and other service types. By default, Cluster Members
synchronize all these services.
n The VRRP and the IGMP protocols are not synchronized by default (but you can choose to turn on
synchronization for these protocols).
n Broadcast and multicast connections are not, and cannot be, synchronized.
You may choose not to synchronize a service if these conditions are true:
n A significant amount of traffic goes through the cluster. Not synchronizing the service reduces the
amount of synchronization traffic, and so enhances cluster performance.
n The service typically opens short connections, whose loss may not be noticed. DNS (over UDP) and
HTTP are typically responsible for most connections, frequently have short life, and inherent
recoverability in the application level. Services that open long connections, such as FTP, should
always be synchronized.
n Configurations that ensure bi-directional stickiness for all connections, do not require synchronization
to operate (only to maintain High Availability). Such configurations include:
l Any cluster in High Availability mode (for example, ClusterXL High Availability, or VRRP on
Gaia).
l ClusterXL in a Load Sharing mode with clear connections (no VPN, or Static NAT).
l VPN and Static NAT connections passing through a ClusterXL cluster in a Load Sharing mode
(either multicast, or unicast) may not maintain bi-directional stickiness. State Synchronization
must be turned on for such environments.
You can have a synchronized service and a non-synchronized definition of a service, and use them
selectively in the Rule Base. For more information, see the R81.10 Security Management Administration
Guide.

To configure a service not to synchronize in a cluster:

1. In SmartConsole, click Objects > Object Explorer.
2. In the left tree, select Services.
3. Double-click the applicable existing synchronized service, for which you need to create a non-
synchronized counterpart service.
4. Write down all the settings from both the General and Advanced pages.
5. Click OK.
6. Click New > Service > > select the applicable service type.
7. Enter the applicable name that distinguishes the new non-synchronized counterpart service from the
existing synchronized service.
8. On the General page, configure the same settings as in the existing synchronized service.
9. On the Advanced page:
a. Configure the same settings as in the existing synchronized service.
b. In the Cluster and synchronization section, clear Synchronize connections if State
Synchronization is enabled on the cluster.
Important - This change applies to all policies that use this service.
10. Click OK.

11. Close the Object Explorer.
12. Use the synchronized service and the non-synchronized counterpart service in the applicable rules in
the applicable Access Control Policies.

Sticky Connections
Sticky Connections
In This Section:
Introduction to Sticky Connections 77

VPN Tunnels with 3rd Party Peers and Load Sharing 77
Third-Party Gateways in Hub and Spoke VPN Deployments 78
Configuring a Third-Party Gateway in a Hub and Spoke VPN Deployment 79
Introduction to Sticky Connections

A connection is considered sticky, when all of its packets are handled, in either direction, by a single Cluster
Member.
This is the case in High Availability mode, where all connections are routed through the same Cluster
Member, and hence, are sticky.
This is also the case in Load Sharing mode, when there are no VPN peers, Static NAT rules, or SIP traffic.
In Load Sharing mode, there are cases, where it is necessary to ensure that a connection that starts on a
specific Cluster Member will continue to be processed by the same Cluster Member in both directions.
Certain connections can be made sticky by enabling the Sticky Decision Function in the cluster object in
SmartConsole.
The Check Point Cluster Correction Layer (CCL) deals with asymmetric connections in Check Point cluster.
VPN Tunnels with 3rd Party Peers and Load Sharing

Check Point provides interoperability with third-party vendor gateways by enabling them to peer with Check
Point Security Gateways. A special case occurs when certain third-party peers (for example, Microsoft
LT2P, Cisco gateways) attempt to establish VPN tunnels with ClusterXL in Load Sharing mode. These VPN
peers are limited in their ability to store SAs, which means that a VPN session that begins on one Cluster
Member and, due to Load Sharing, is routed on the return trip through another Cluster Member, is
unrecognized and dropped.
Item Description
1 Internal network
2 Switch for internal network

Sticky Connections
Item Description
3 ClusterXL in Load Sharing mode - Cluster Members "A" and "B"
5 Internet
6 3rd party peer VPN Gateway
7 3rd party peer laptop with VPN client
In this scenario:
n A third-party peer (gateway or client) attempts to create a VPN tunnel.
n Cluster Members "A" and "B" belong to a ClusterXL in Load Sharing mode.
The third-party peers, lacking the ability to store more than one set of SAs, cannot negotiate a VPN tunnel
with multiple Cluster Members, and therefore the Cluster Member cannot complete the routing transaction.
This issue is resolved for certain third-party peers or gateways that can save only one set of SAs by making
the connection sticky. The Cluster Correction Layer (CCL) makes sure that a single Cluster Member
processes all VPN sessions, initiated by the same third-party gateway.
Third-Party Gateways in Hub and Spoke VPN Deployments

Another case, where Load Sharing mode requires the connection stickiness, which the Cluster Correction
Layer (CCL) provides, is when integrating certain third-party gateways into a Hub and Spoke deployment.
Without the ability to store more than one set of SAs, a third-party gateway must maintain its VPN tunnels on
a single Cluster Member in order to avoid duplicate SAs.
Item Description
4 Internet
5 Gateway - Spoke A

Sticky Connections
Item Description
6 Gateway - Spoke B
In this sample deployment:

n The intent of this deployment is to enable hosts that reside behind Spoke A to communicate with
hosts behind Spoke B.
n The ClusterXL in Load Sharing mode, is composed of Cluster Members "A" and "B", and serves as a
VPN Hub.
n Spoke A is a third-party gateway, and is connected by a VPN tunnel that passes through the VPN Hub
to Spoke B.
n Spoke B can be either another third-party gateway, or a Check Point Security Gateway.
Spokes A and B must be set to always communicate using the same Cluster Member. The Cluster
Correction Layer (CCL) solves half of this problem, in that a single Cluster Member processes all VPN
sessions initiated by either third-party gateway.
To make sure that all communications between Spokes A and B are always using the same Cluster
Member, you must make some changes to the applicable user.def file (see sk98239). This second step
ensures that both third-party gateways always connect to the same Cluster Member.
Configuring a Third-Party Gateway in a Hub and Spoke VPN

Deployment
To configure a third-party gateway as a spoke in a Hub and Spoke VPN deployment, perform the following
on the Management Server:
1. Create a Tunnel Group to handle traffic from specific peers. Edit the applicable user.def file (see
sk98239), and add a line similar to the following:
all@{member1,member2} vpn_sticky_gws = {<10.10.10.1;1>,

<20.20.20.1;1>};
The elements of this configuration are as follows:

n all - All cluster interfaces
n member1,member2 - Names of Cluster Members in SmartConsole
n vpn_sticky_gws - Table name
n 10.10.10.1 - IP address of Spoke A
n 20.20.20.1 - IP address of Spoke B
n ;1 - Tunnel Group Identifier, which indicates that the traffic from these IP addresses should be
handled by the same Cluster Member
2. Other VPN peers can be added to the Tunnel Group by including their IP addresses in the same
format as shown above. To continue with the example above, adding Spoke C would look like this:
all@{member1,member2} vpn_sticky_gws = {<10.10.10.1;1>, <20.20.20.1;1>,

<30.30.30.1;1>};

Sticky Connections
The Tunnel Group Identifier ;1 stays the same, which means that the listed peers will always connect
through the same Cluster Member.
Note - More tunnel groups than Cluster Members may be defined.
This procedure turns off Load Sharing for the affected connections. If the implementation is to
connect multiple sets of third-party gateways one to another, a form of Load Sharing can be
accomplished by setting Security Gateway pairs to work in tandem with specific Cluster Members.
For instance, to set up a connection between two other spokes (C and D), simply add their IP
addresses to the line and replace the Tunnel Group Identifier ;1 with ;2. The line would then look
something like this:
all@{member1,member2} vpn_sticky_gws = {<10.10.10.1;1>, <20.20.20.1;1>,

<192.168.15.5;2>, <192.168.1.4;2>,};
Note that there are now two peer identifiers: ;1 and ;2. Spokes A and B will now connect through one
Cluster Member, and Spokes C and D through another.
Note - The tunnel groups are shared between active Cluster Members. In case of
a change in cluster state (for example, failover or Cluster Member attach/detach),
the reassignment is performed according to the new state.

Non-Sticky Connections
In This Section:
Non-Sticky Connection Example: TCP 3-Way Handshake 81

Synchronizing Non-Sticky Connections 82
A connection is called sticky if a single Cluster Member handles all packets of that connection. In a non-
sticky connection, the response packet of a connection returns through a different Cluster Member than the
original request packet.
The cluster synchronization mechanism knows how to handle non-sticky connections properly. In a non-
sticky connection, a Cluster Member can receive an out-of-state packet, which Firewall normally drops
because it poses a security risk.
In Load Sharing configurations, all Cluster Members are active. In Static NAT and encrypted connections,
the source and destination IP addresses change. Therefore, Static NAT and encrypted connections through
a Load Sharing cluster may be non-sticky. Non-stickiness may also occur with Hide NAT, but ClusterXL has
a mechanism to make it sticky.
In High Availability configurations, all packets reach the Active member, so all connections are sticky. If
failover occurs during connection establishment, the connection is lost, but synchronization can be
performed later.
If the other members do not know about a non-sticky connection, the packet will be out-of-state, and the
connection will be dropped for security reasons. However, the Synchronization mechanism knows how to
inform other members of the connection. The Synchronization mechanism thereby prevents out-of-state
packets in valid, but non-sticky connections, so that these non-sticky connections are allowed.
Non-sticky connections will also occur if the network Administrator has configured asymmetric routing,
where a reply packet returns through a different Security Gateway than the original packet.
Non-Sticky Connection Example: TCP 3-Way Handshake

The 3-way handshake that initiates all TCP connections can very commonly lead to a non-sticky (often
called asymmetric routing) connection. This diagram shows a sample scenario:
Item Description
1 Client

Item Description
4 Server
The client initiates a connection by sending a SYN packet to the server. The SYN passes through Cluster
Member A, but the SYN-ACK reply returns through Cluster Member B. This is a non-sticky connection,
because the reply packet returns through a different Security Gateway than the original packet.
The synchronization network notifies Cluster Member B. If Cluster Member B is updated before the SYN-
ACK packet sent by the server reaches it, the connection is handled normally. If, however, synchronization
is delayed, and the SYN-ACK packet is received on Cluster Member B before the SYN flag has been
updated, then the Security Gateway treats the SYN-ACK packet as out-of-state, and drops the connection.
You can configure enhanced 3-Way TCP Handshake enforcement to address this issue (see "Enhanced 3-
Way TCP Handshake Enforcement" on page 152).
Synchronizing Non-Sticky Connections

The synchronization mechanism prevents out-of-state packets in valid, but non-sticky connections. The way
it does this is best illustrated with reference to the 3-way handshake that initiates all TCP data connections.
The 3-way handshake proceeds as follows:
1. SYN (client to server)
2. SYN-ACK (server to client)
3. ACK (client to server)
4. Data (client to server)
To prevent out-of-state packets, the following sequence (called "Flush and Ack") occurs
1. Cluster Member receives first packet (SYN) of a connection.
2. Suspects that it is non-sticky.
3. Hold the SYN packet.
4. Send the pending synchronization updates to all Cluster Members (including all changes relating to
this packet).
5. Wait for all the other Cluster Members to acknowledge the information in the sync packet.
6. Release held SYN packet.
7. All Cluster Members are ready for the SYN-ACK.

Synchronizing Clusters on a Wide Area Network

Organizations sometimes need to locate Cluster Members in geographical locations that are distant from
each other.
A typical example is a replicated Data Center, whose locations are widely separated for disaster recovery
purposes.
In such a configuration, it is clearly impractical to use a cross cable for the synchronization network.
The synchronization network can be spread over remote sites, which makes it easier to deploy
geographically distributed clustering.
There are two limitations to this capability:
1. The synchronization network must guarantee no more than 100ms latency and no more than 5%
packet loss.
2. The synchronization network may include only Layer 2 networking devices - switches and hubs.
No Layer 3 routers are allowed on the synchronization network, because routers drop Cluster Control
Protocol (CCP) packets.
Note - The Active-Active mode also supports Layer 3 (see "Active-Active Mode in
ClusterXL" on page 63).
You can monitor and troubleshoot geographically distributed clusters using the command line interface.

supported.
page 126).

Server, is lost.


supported.
page 126).
Server, is lost.

Configuring ClusterXL
Configuring ClusterXL
This procedure describes how to configure the Load Sharing Multicast, Load Sharing Unicast, and High
Availability modes from scratch.
Their configuration is identical, apart from the mode selection in SmartConsole Cluster object or Cluster
creation wizard.
Installing Cluster Members

Step Instructions
1 See "ClusterXL Requirements and Compatibility" on page 38.
2 See R81.10 Installation and Upgrade Guide.
Configuring Routing for Client Computers

Example topology:
[internal network 10.10.2.0/24] --- (VIP 10.10.2.100/24) [Cluster] (VIP 192.168.2.100/24) --- [external
network 192.168.2.0/24]
To configure routing for client computers:

1. Computers on the internal network 10.10.2.0/24 must be configured with Default Gateway IP
10.10.2.100
2. Computers on the external network 192.168.2.0/24 must be configured with Default Gateway IP
192.168.2.100
3. For Proxy ARP configuration, see sk30197.
4. In addition, see "Cluster IP Addresses on Different Subnets" on page 154.

Configuring the Cluster Control Protocol (CCP) Settings
Configuring the Cluster Control Protocol (CCP)

Settings
Important - In a Cluster, you must configure all the Cluster Members in the same way.
Description
Cluster Members configure the Cluster Control Protocol (CCP) mode automatically.
Important - In R81.10, the CCP always runs in the unicast mode.
You can configure the Cluster Control Protocol (CCP) Encryption on the Cluster Members.
See "Viewing the Cluster Control Protocol (CCP) Settings" on page 246.
Syntax for configuring the Cluster Control Protocol (CCP) Encryption
Shell Command
Gaia Clish set cluster member ccpenc {off | on}
Expert mode cphaconf ccp_encrypt {off | on}

cphaconf ccp_encrypt_key <Key String>

Configuring the Cluster Object and Members

You can use one of these procedures to define a cluster object and its members:
n Simple Mode (Wizard) - Lets you quickly create a new cluster and configure some basic cluster
properties:
l Cluster properties and Virtual IP addresses
l Properties and Topology of Cluster Members
l Synchronization interfaces and IP addresses
See "Using the Wizard Mode in SmartConsole" below.
n Classic Mode - Opens the Cluster Gateway Properties window, where you manually create a cluster
and configure its properties.
See "Using the Classic Mode in SmartConsole" on page 92.
The Cluster Gateway Properties window lets you:
n Enable and configure Software Blades for the cluster
n Configure other cluster properties that you cannot configure with the wizards
n Change the properties of an existing cluster
See "Changing the Settings of Cluster Object in SmartConsole" on page 100.
Using the Wizard Mode in SmartConsole

This version includes two wizards:
n Check Point Appliances and Open Servers
n Check Point Small Office Appliances
Wizard for Check Point Appliances or Open Servers
The Cluster Wizard is recommended for all Check Point Appliances (except models lower than 3000
series), and for Open Server platforms.
To create a new cluster using Wizard Mode:
Step Instructions
1 In SmartConsole, click Objects menu > More object types > Network Object > Gateways
and Servers > Cluster > New Cluster.
2 In Check Point Security Gateway Cluster Creation window, click Wizard Mode.

Step Instructions
3 In the Cluster General Properties window:

a. In the Cluster Name field, enter unique name for the cluster object.
b. In the Cluster IPv4 Address, enter the unique Cluster Virtual IPv4 addresses for this
cluster.
This is the main IPv4 address of the cluster object.
c. In the Cluster IPv6 Address, enter the unique Cluster Virtual IPv6 addresses for this
cluster.
Important - You must define a corresponding IPv4 address for every IPv6
address. This release does not support pure IPv6 addresses.
d. In the Choose the Cluster's Solution field, select the applicable option and click
Next:
n Check Point ClusterXL and then select High Availability or Load Sharing
n Gaia VRRP
4 In the Cluster member's properties window perform these steps for each Cluster Member
and click Next.
We assume you create a new cluster object from the scratch.
a. Click Add > New Cluster Member to configure each Cluster Member.
b. In the Cluster Name field, enter unique name for the Cluster Member object.
c. In the Cluster IPv4 Address, enter the unique Cluster Virtual IPv4 addresses for this
Cluster Member.
This is the main IPv4 address of the Cluster Member object.
d. In the Cluster IPv6 Address, enter the unique Cluster Virtual IPv6 addresses for this
Cluster Member.
This is the main IPv6 address of the Cluster Member object.
e. In the Activation Key and Confirm Activation Key fields, enter a one-time password
that you entered in First Time Configuration Wizard during the installation of this
Cluster Member.
Management Server will try to establish SIC with each Cluster Member.
The Trust State field should show Trust established.
g. Click OK.

Step Instructions
5 In the Cluster Topology window, define a network type (network role) for each cluster
interface and define the Cluster Virtual IP addresses. Click Next.
The wizard automatically calculates the subnet for each cluster network and assigns it to
the applicable interface on each Cluster Member. The calculated subnet shows in the
upper section of the window.
The available network objectives are:
Network
Description
Objective
Cluster A cluster interface that connects to an internal or external network.

Interface Enter the Cluster Virtual IP addresses for each network (internal or
external).
In addition, see "Cluster IP Addresses on Different Subnets" on
page 154.
Cluster Sync A cluster Synchronization interface.

Interface In Load Sharing mode, you must define a synchronization interface.
n For Check Point Appliances or Open Servers:
The Synchronization Network is supported only on the lowest
VLAN tag of a VLAN interface.
n For Small Office appliances (models lower than 3000 series):
You can only select 1st sync and only for the LAN2/SYNC
interface.
You cannot configure VLANs on the Synchronization
interface.
Important - Make sure that you do not define IPv6
address for Sync interfaces. The wizard does not let you
define an interface with an IPv6 address as a Sync
interface.
Private An interface that is not part of the cluster.

Cluster does not monitor this interface.
Cluster failover does not occur if a fault occurs on this interface.
This option is recommended for the dedicated management
interface.
6 In the Cluster Definition Wizard Complete window, click Finish.
After you complete the wizard, we recommend that you open the cluster object and complete the
configuration:
n Define Anti-Spoofing properties for each interface
n Change the Topology settings for each interface, if necessary
n Define the Network Type
n Configure other Software Blades, features and properties as necessary

Wizard for Small Office Appliances
The Small Office Cluster wizard is recommended for Centrally Managed Check Point appliances -
models lower than 3000 series.
To create a new Small Office cluster using Wizard Mode:
Step Instructions
1 In SmartConsole, click Objects menu > More object types > Network Object > Gateways
and Servers > Cluster > New Small Office Cluster.
2 In Check Point Security Gateway Cluster Creation window, click Wizard Mode.
3 In the Cluster General Properties window:

a. Enter a unique name for the cluster object.
b. Select the correct hardware type.
c. Click Next.
4 In the Cluster Members window:

a. Enter the member name and IPv4 addresses for each Cluster Member.
b. Enter the one-time password for SIC trust.
c. Click Next.
d. Management Server will try to establish SIC with the Primary Cluster Member.
5 In the Configure WAN Interface page, configure the Cluster Virtual IPv4 address.
6 Define the Cluster Virtual IPv4 addresses for the other cluster interfaces.
7 Click Next, and then Finish to complete the wizard.
After you complete the wizard, we recommend that you open the cluster object and complete the
configuration:
n Define Anti-Spoofing properties for each interface
n Change the Topology settings for each interface, if necessary
n Define the Network Type
n Configure other Software Blades, features and properties as necessary

Using the Classic Mode in SmartConsole

Step Instructions
1 Connect with SmartConsole to the Security Management Server or Domain Management

Server that should manage this ClusterXL.
3 Create a new Cluster object in one of these ways:
n From the top toolbar, click the New ( ) > Cluster > Cluster.
Gateways and Servers > Cluster > New Cluster.
Gateways and Servers > Cluster > Cluster.
4 In the Check Point Security Gateway Creation window, click Classic Mode.
The Gateway Cluster Properties window opens.
5 On the General Properties page > Machine section:

a. In the Name field, make sure you see the configured applicable name for this ClusterXL
object.
b. In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
these IP addresses.
6 On the General Properties page > Platform section, select the correct options:
a. In the Hardware field:
If you install the Cluster Members on Check Point Appliances, select the correct
appliances series.
If you install the Cluster Members on Open Servers, select Open server.
b. In the Version field, select R81.10.
c. In the OS field, select Gaia.
7 On the General Properties page:

a. On the Network Security tab, make sure the ClusterXL Software Blade is selected.
b. Enable the additional applicable Software Blades on the Network Security tab and on
the Threat Prevention tab.

Step Instructions
8 On the Cluster Members page:

a. Click Add > New Cluster Member.
b. In the Name field, enter the applicable name for this Cluster Member object.
c. Configure the main physical IP address(es) for this Cluster Member object.
In the IPv4 Address and IPv6 Address fields, configure the same IPv4 and IPv6
these IP addresses.
Note - You can configure the Cluster Virtual IP address to be on a different
network than the physical IP addresses of the Cluster Members. In this case,
you must configure the required static routes on the Cluster Members.
d. Click Communication.
e. In the One-time password and Confirm one-time password fields, enter the same
Activation Key you entered during the Cluster Member's First Time Configuration
Wizard.
g. Click Close.
h. Click OK.
i. Repeat Steps a-h to add the second Cluster Member, and so on.
If the Trust State field does not show Trust established, perform these steps:
a. Connect to the command line on the Cluster Member.
b. Make sure there is a physical connectivity between the Cluster Member and the
Management Server (for example, pings can pass).
c. Run:
cpconfig
d. Enter the number of this option:
Secure Internal Communication
e. Follow the instructions on the screen to change the Activation Key.
f. In SmartConsole, click Reset.
g. Enter the same Activation Key you entered in the cpconfig menu.
h. In SmartConsole, click Initialize.
9 On the ClusterXL and VRRP page:

a. In the Select the cluster mode and configuration section, select the applicable mode:
n High Availability and ClusterXL
n Load Sharing and Multicast or Unicast
n Active-Active (follow "Active-Active Mode in ClusterXL" on page 63)
b. In the Tracking section, select the applicable option.
c. In the Advanced Settings section:

Step Instructions
n If you selected the High Availability mode, then:

i. Optional: Select Use State Synchronization.
This configures the Cluster Members to synchronize the information about
the connections they inspect.
Best Practice - Enable this setting to prevent connection drops

after a cluster failover.
ii. Optional: Select Start synchronizing [ ] seconds after connection

initiation and enter the applicable value.
To prevent the synchronization of short-lived connections (which decreases
the cluster performance), you can configure the Cluster Members to start
the synchronization of all connections a number of seconds after they start.
Default: 3 seconds
Notes:
l This setting in the cluster object applies to all connections
that pass through the cluster.

You can override this global cluster synchronization delay
in the properties of applicable services - see "Configuring
Services to Synchronize After a Delay" on page 74.
l The greater this value, the fewer short-lived connections
the Cluster Members have to synchronize.

l The connections that the Cluster Members did not
synchronize, do not survive a cluster failover.
Best Practice - Enable and configure this setting to increase the

cluster performance.
iii. Optional: Select Use Virtual MAC.

This configure all Cluster Members to associate the same virtual MAC
address with the Virtual IP address on the applicable interfaces (each
Virtual IP address has its unique Virtual MAC address).
For more information, see sk50840.
iv. Select the Cluster Member recovery method - which Cluster Member to
select as Active during a fallback (return to normal operation after a cluster
failover):
l Maintain current active Cluster Member
a. The Cluster Member that is currently in the Active state,

remains in this state.
b. Other Cluster Members that return to normal operation, remain
the Standby state.
l Switch to higher priority Cluster Member
a. The Cluster Member that has the highest priority (appears at

the top of the list on the Cluster Members page of the cluster
object) becomes the new Active.
b. The state of the previously Active Cluster Member changes to
Standby.
c. Other Cluster Members that return to normal operation remain
the Standby state.

Step Instructions
n If you selected the Load Sharing > Multicast mode, then:

i. Optional: Select Use Sticky Decision Function.
This option is available only for clusters R80.10 and lower.
For more information, click the (?) button in the top right corner.
Default: 3 seconds
Notes:




Step Instructions
iii. Select the connection sharing method between the Cluster Members:
l IPs, Ports, SPIs
Configures each Cluster Member to inspect all connections with the

same Source and Destination IP address, the same Source and
Destination ports, and the same IPsec SPI numbers.
This is the least "sticky" sharing configuration that provides the best
sharing distribution between Cluster Members.
This method decreases the probability that a certain connection
passes through the same Cluster Member in both inbound and
outbound directions
We recommend this method.
l IPs, Ports

same Source and Destination IP address and the same Source and
Destination ports, regardless of the IPsec SPI numbers.
Use this method only if there are problems when distributing IPsec
packets between Cluster Members.
l IPs

same Source and Destination IP address, regardless of the Source
and Destination ports and IPsec SPI numbers.
This is the most "sticky" sharing configuration that provides the worst
This method increases the probability that a certain connection
outbound directions
Use this method only if there are problems when distributing packets
with different port numbers or distributing IPsec packets between
Cluster Members.

Step Instructions
n If you selected the Load Sharing > Unicast mode, then:

i. Optional: Select Use Sticky Decision Function.
This option is available only for clusters R80.10 and lower.
For more information, click the (?) button in the top right corner.
Default: 3 seconds
Notes:



iii. Optional: Select Use Virtual MAC.

This configure all Cluster Members to associate the same virtual MAC
address with the Virtual IP address on the applicable interfaces (each
Virtual IP address has its unique Virtual MAC address).

Step Instructions
iv. Select the connection sharing method between the Cluster Members:
l IPs, Ports, SPIs

same Source and Destination IP address, the same Source and
Destination ports, and the same IPsec SPI numbers.
This is the least "sticky" sharing configuration that provides the best
This method decreases the probability that a certain connection
outbound directions
We recommend this method.
l IPs, Ports

same Source and Destination IP address and the same Source and
Destination ports, regardless of the IPsec SPI numbers.
Use this method only if there are problems when distributing IPsec
packets between Cluster Members.
l IPs

same Source and Destination IP address, regardless of the Source
and Destination ports and IPsec SPI numbers.
This is the most "sticky" sharing configuration that provides the worst
This method increases the probability that a certain connection
outbound directions
Use this method only if there are problems when distributing packets
with different port numbers or distributing IPsec packets between
Cluster Members.

Step Instructions
10 On the Network Management page:

a. Select each interface and click Edit. The Network: <Name of Interface> window
opens.
b. From the left tree, click the General page.
c. In the General section, in the Network Type field, select the applicable type:
n For cluster traffic interfaces, select Cluster.
Make sure the Cluster Virtual IPv4 address and its Net Mask are correct.
n For cluster synchronization interfaces, select Sync or Cluster+Sync.
Notes:
l We do not recommend the configuration Cluster+Sync.
l Check Point cluster supports only these settings:
o One Sync interface.

o One Cluster+Sync interface.
o One Sync interface and one Cluster+Sync interface.
l For Check Point Appliances or Open Servers:
The Synchronization Network is supported only on the lowest

VLAN tag of a VLAN interface.
n For interfaces that do not pass the traffic between the connected networks, select
Private.
d. In the Member IPs section, make sure the IPv4 address and its Net Mask are correct on
each Cluster Member.
Notes:
n For a ClusterXL in High Availability mode that is deployed in a Cloud
environment (Geo Cluster):
You can configure IP addresses that belong to different networks on
cluster synchronization interfaces and on cluster traffic interfaces.
n For cluster traffic interfaces, you can configure the Cluster Virtual IP
address to be on a different network than the physical IP addresses of
the Cluster Members. In this case, you must configure the required
static routes on the Cluster Members. See "Configuring Cluster
Addresses on Different Subnets" on page 155.
e. In the Topology section:
n Make sure the settings are correct in the Leads To and Security Zone fields.
n Make sure to enable the Anti-Spoofing.
11 Click OK.
12 Publish the SmartConsole session.

Changing the Settings of Cluster Object in SmartConsole

The Cluster Gateway Properties window in a cluster object contains many different ClusterXL properties,
as well as other properties related to Security Gateway and Software Blades functionality.
This section includes only the properties and procedures directly related to ClusterXL.
Configuring General Properties

1. In the Name field, enter a unique name for this cluster object.
2. In the IPv4 Address field, enter the unique Cluster Virtual IPv4 addresses for this cluster. This is the
main IPv4 address of the cluster object.
3. In the Cluster IPv6 Address field, enter the unique Cluster Virtual IPv6 addresses for this cluster.
4. In the Hardware field, select the correct hardware platform.

5. In the Version field, select the correct Check Point version.
6. In the OS field, select the correct operating system.
7. Configure the applicable cluster type:
n To work with ClusterXL or with VRRP on Gaia, select ClusterXL.
Go to the ClusterXL and VRRP pane and configure the applicable settings.
n To work with any other cluster mode, clear ClusterXL.
Go to the 3rd Party Configuration pane and configure the applicable settings.
8. On the Network Security tab, enable other Software Blades as necessary.
9. Click OK.
Working with Cluster Topology
IPv6 Considerations
To activate IPv6 functionality for an interface, define an IPv6 address for the applicable interface on each
Cluster Member and in the cluster object. All interfaces configured with an IPv6 address must also have a
corresponding IPv4 address. If an interface does not require IPv6, only the IPv4 definition address is
necessary.
Note - You must configure synchronization interfaces with IPv4 addresses only. This is
because the synchronization mechanism works using IPv4 only. All IPv6 information
and states are synchronized using this interface.
1. Connect with SmartConsole to the Security Management Server or Domain Management Server that
manages this cluster.

2. From the left navigation panel, click Gateways & Servers.

3. Open the cluster object.
4. From the left tree, click the Network Management page.
5. Select a cluster interface and click Edit.
6. From the left navigation tree, click General page:
a. In the General section, configure these settings for Cluster Virtual Interface:
n Network Type - one of these: Cluster, Sync, Cluster + Sync, Private
The available network types (network objectives) are:
Network
Description
Type
Cluster An interface that connects to an internal or external network.
Cluster + A cluster interface that also works as a Synchronization interface.

Sync We do not recommend this configuration because it adds the Delta
Sync traffic to the interface.
Sync An interface used exclusively for cluster state synchronization.
Private An interface that is not part of the cluster. ClusterXL does not monitor
the state of this interface. As a result, there is no cluster failover if a
fault occurs with this interface. This option is recommended for the
management interface.
n Virtual IPv4 - Virtual IPv4 address assigned to this Cluster Virtual Interface
n Virtual IPv6 - Virtual IPv6 address assigned to this Cluster Virtual Interface
address. This release does not support the configuration of only IPv6
addresses.
7. In the Member IPs section, click Modify and configure these settings:
n Physical IPv4 address and Mask Length assigned to the applicable physical interface on each
Cluster Member
n Physical IPv6 address and Mask Length assigned to the applicable physical interface on each
Cluster Member
address. This release does not support the configuration of only IPv6 addresses.
In addition, see "Configuring Cluster Addresses on Different Subnets" on page 155.

8. In the Topology section, click Modify and configure these settings:

n Leads To - one of these: Internet (External), This Network (Internal)

n Security Zone - one of these: User defined, According to topology (ExternalZone,
InternalZone)
n Anti-Spoofing - whether to perform the Anti-Spoofing, and how to do it (Detect, Prevent)
9. From the left navigation tree, click QoS page:
a. In the Bandwidth section, configure these settings:
n Inbound Active - rate limit for inbound traffic
n Outbound Active - rate limit for outbound traffic
b. In the DiffServ and Low Latency classes section, configure the applicable classes.
10. From the left navigation tree, click Advanced page:
a. In the Multicast Restrictions section, configure the applicable settings for dropping multicast
packets
b. In the Interfaces Names section, configure the names of applicable interfaces
11. Click OK.
Changing the Synchronization Interface
Important - Schedule a maintenance window, because changing the synchronization

interface can impact the traffic.
To change the IPv4 address on the synchronization interface on Cluster Members:

1. On each Cluster Member, change the IPv4 address on the Sync interface.
Use Gaia Portal, or Gaia Clish.
See the R81.10 Gaia Administration Guide.
5. In the Gateway Cluster Properties window, click Network Management page.
6. Click Get Interfaces > Get Interfaces With Topology.
7. Make sure the settings are correct.
8. Select the Sync interface and click Edit.
9. From the left navigation tree, click General page.
10. In the General section, in the Network Type field, select Sync.
11. Click OK.


To change the synchronization interface on Cluster Members to a new interface:

1. On each Cluster Member:
a. Configure a new interface that you will use as a new Sync interface.
b. Delete the IPv4 address from the old Sync interface.
Use Gaia Portal, or Gaia Clish.
See the R81.10 Gaia Administration Guide.
5. In the Gateway Cluster Properties window, click Network Management page.
6. Click Get Interfaces > Get Interfaces With Topology.
7. Make sure the settings are correct.
8. Right-click on the old Sync interface and click Delete Interface.
9. Select the new interface and click Edit.
10. From the left navigation tree, click General page.
12. Click OK.
13. In SmartConsole, install the Access Control Policy on this cluster object.
Adding Another Member to an Existing Cluster

See "Adding Another Member to an Existing Cluster" on page 160.
Removing a Member from an Existing Cluster

See "Removing a Member from an Existing Cluster" on page 166.

Configuring a ClusterXL in Bridge Mode
Configuring a ClusterXL in Bridge Mode

See the R81.10 Installation and Upgrade Guide - Chapter Special Scenarios for Security Gateways -
Section Deploying a Security Gateway or a ClusterXL in Bridge Mode.

Advanced Features and Procedures
Advanced Features and Procedures

This section describes advanced configuration in cluster.

Working with VPN in Cluster

This section describes the configuration of VPN in cluster.
Configuring VPN in Clusters

Configuring a cluster using SmartConsole is very similar to configuring a single Security Gateway.
All attributes of the VPN are defined in the Cluster object, except for two attributes that are defined for each
Cluster Member object.
1. In SmartConsole, open the cluster object.
2. In the left navigation tree, go to Cluster Members page.
3. Select each Cluster Member and click Edit.
4. Go the VPN tab:
n In the Office Mode for Remote access section:
If you wish to use Office Mode for Remote Access, select Offer Manual Office Mode and
define the IP pool allocated to each Cluster Member.
n In the Certificate List with keys stored on the Security Gateway section:
If your Cluster Member supports hardware storage for IKE certificates, define the certificate
properties.
In that case, Management Server directs the Cluster Member to create the keys and supply
only the required material for creation of the certificate request.
The certificate is downloaded to the Cluster Member during policy installation.
5. Click OK to close the Cluster Member Properties window.
6. In the left navigation tree, go to ClusterXL and VRRP page.
7. Make sure to select Use State Synchronization.
This is required to synchronize IKE keys.
8. In the left navigation tree, go to Network Management > VPN Domain page.
9. Define the encryption domain of the cluster.
Select one of the two possible settings:
n All IP addresses behind Cluster Members based on Topology information. This is the
default option.
n Manually defined. Use this option if the cluster IP address is not on the member network, in
other words, if the cluster virtual IP address is on a different subnet than the Cluster Member
interfaces. In that case, select a network or group of networks, which must include the virtual IP
address of the cluster, and the network or group of networks behind the cluster.
10. Click OK to close the Gateway Cluster Properties window.
11. Install the Access Control Policy on the cluster.

Defining VPN Peer Clusters with Separate Management

Servers
When working with a VPN peer that is a Check Point Cluster, and the VPN peer is managed by a different
Management Server, do NOT define another cluster object. Instead, do the following:
1. In SmartConsole, go to Objects menu > More object types > Network Object > Gateways and
Servers > More > New Externally Managed VPN Gateway.
The Externally Managed Check Point Gateway window opens.
2. In the General Properties page, configure the name and the IP address.
3. In the Topology page, click New to add the external and internal cluster interfaces on the VPN peer.
4. In the VPN Domain section of the Topology page, define the encryption domain of the externally
managed Security Gateway to be behind the internal Virtual IP address of the Security Gateway.
If the encryption domain is just one subnet, select All IP addresses behind Gateway based on
Topology information.
If the encryption domain includes more than one subnet, select Manually defined.
5. Click OK.

Working with NAT in Cluster

This section describes the configuration of NAT in cluster.
Cluster Fold and Cluster Hide

Network Address Translation (NAT) is a fundamental aspect of the way ClusterXL works.
n When a Cluster Member establishes an outgoing connection towards the Internet, the source address
in the outgoing packets, is the physical IP address of the Cluster Member interface.
The source IP address is changed using NAT to that of the external Virtual IP address of the cluster.
This address translation is called "Cluster Hide".
l When working with VRRP on Gaia cluster, this corresponds to the default setting in the
ClusterXL and VRRP page of the cluster object of Hide Cluster Members outgoing traffic
behind the Cluster IP address being selected.
l When working with VRRP on IPSO cluster, this corresponds to the default setting in the 3rd
Party Configuration page of the cluster object of Hide Cluster Members' outgoing traffic
behind the Cluster's IP address being selected.
n When a client establishes an incoming connection to external (virtual) address of the cluster,
ClusterXL changes the destination IP address using NAT to that of the physical external address of
one of the Cluster Members. This address translation is called "Cluster Fold".
l When working with VRRP on Gaia cluster, this corresponds to the default setting in the
ClusterXL and VRRP page of the cluster object of Forward Cluster incoming traffic to
Cluster Members IP address being selected.
l When working with IPSO IP Clustering cluster, this corresponds to the default setting in the
3rd Party Configuration page of the cluster object of Forward Cluster incoming traffic to
Cluster Members' IP addresses being selected.
Configuring NAT in Cluster

Network Address Translation (NAT) can be performed on a Cluster, in the same way as it is performed on a
Security Gateway.
This NAT is in addition to the automatic "Cluster Fold" and "Cluster Hide" address translations.
To configure NAT, edit the Cluster object, and in the Cluster Properties window, click the NAT page. Do
NOT configure the NAT tab of the Cluster Member object.
Configuring NAT on a Cluster Member

It is possible to perform Network Address Translation (NAT) on a non-cluster interface of a Cluster Member.
A possible scenario for this is if the non-Cluster interface of the Cluster Member is connected to another
(non-cluster) internal Security Gateway, and you wish to hide the address of the non-Cluster interface of the
Cluster Member.
Performing this NAT means that when a packet originates behind or on the non-Cluster interface of the
Cluster Member, and is sent to a host on the other side of the internal Security Gateway, the source address
of the packet will be translated.

To configure NAT on a non-cluster interface of a Cluster Member:

1. Edit the Cluster object.
2. In the Cluster Member page, edit the Cluster Member object.
3. In the Cluster Member Properties window, click the NAT tab.
4. Configure Static or Hide NAT as applicable.

Working with VLANs in Cluster

A VLAN switch tags packets that originate in a VLAN with a four-byte header that specifies, which switch
port it came from
No packet is allowed to go from a switch port in one VLAN to a switch port in another VLAN, apart from ports
("global" ports) that are defined so that they belong to all the VLANs.
The Cluster Member is connected to the global port of the VLAN switch, and this logically divides a single
physical port into many VLAN ports each associated with a VLAN tagged interface (VLAN interface) on the
Cluster Member.
When defining VLAN tags on an interface, cluster IP addresses can be defined only on the VLAN interfaces
(the tagged interfaces).
Defining a cluster IP address on a physical interface that has VLANs is not supported.
This physical interface has to be defined with the Network Type Private.
ClusterXL (including VSX) supports the Synchronization Network (CCP packets that carry Delta Sync
information) only on the lowest VLAN ID (VLAN tag).
For example, if three VLANs with IDs 10, 20 and 30 are configured on interface eth1, then you can use
only the VLAN interface eth1.10 for the State Synchronization.
This is the default interface monitoring in Check Point cluster:
Monitoring in
Interface
ClusterXL (non- Monitoring in VSX Cluster
type
VSX)
Physical Monitors all cluster Monitors all cluster interfaces.

interfaces interfaces.
VLAN Monitors only lowest VSX High Availability (non-VSLS):

interfaces VLAN ID configured
on a physical n Monitors only lowest and highest VLAN IDs configured on
interface. a physical interface.
n Monitors only lowest VLAN ID, if both VLAN IDs reside on
the same Virtual System.
Monitors only lowest Virtual System Load Sharing:

and highest VLAN
IDs configured on a n Monitors all VLAN IDs configured on a physical interface
physical interface. on each Virtual System.
n When a Virtual System is connected to a Virtual Switch
with the same physical interface and a lower VLAN ID,
the wrp interface that leads to the Virtual Switch is
considered the lowest VLAN ID for the physical interface.
You can customize the default monitoring of VLAN IDs:

Need to Monitoring in ClusterXL (non-

Monitoring in VSX Cluster
monitor VLAN VSX)
Only the lowest Enabled by default. Must disable the monitoring of all VLAN IDs - set
VLAN ID the value of the kernel parameter fwha_
monitor_all_vlan to 0.
See sk92826.
Only the lowest Enabled by default. VSX High Availability (non-VSLS): Enabled by
and highest Controlled by the kernel default.
VLAN IDs parameter fwha_monitor_ Controlled by the kernel parameter fwha_
low_high_vlans. monitor_low_high_vlans.
See sk92826. See sk92826.
All VLAN IDs Disabled by default. Virtual System Load Sharing: Disabled by default.
Controlled by the kernel Controlled by the kernel parameter fwha_
parameter fwha_monitor_ monitor_all_vlan.
all_vlan. See sk92826.
See sk92826.
Only specific Disabled by default. Disabled by default.

VLAN IDs Controlled by the kernel Controlled by the kernel parameter fwha_
parameter fwha_monitor_ monitor_specific_vlan.
specific_vlan. See sk92784.
See sk92784.

Configuring Link Monitoring on the Cluster Interfaces
Configuring Link Monitoring on the Cluster

Interfaces
Description
This procedure configures the Cluster Member to monitor only the physical link on the cluster interfaces
(instead of monitoring the Cluster Control Protocol (CCP) packets):
n If a link disappears on the configured interface, the Cluster Member changes the interface's state to
DOWN.
This causes the Cluster Member to change its state to DOWN.
n If a link appears again on the configured interface, the Cluster Member changes the interface's state
back to UP.
This causes the Cluster Member to change its state back to ACTIVE or STANDBY.
See "Viewing Cluster State" on page 209.

Procedure
Step Instructions
1 Connect to the command line on the Cluster Member.
2 Log in to the Expert mode.
3 See if the $FWDIR/conf/cpha_link_monitoring.conf file already exists:

stat $FWDIR/conf/cpha_link_monitoring.conf
4 If the $FWDIR/conf/cpha_link_monitoring.conf file already exists, create a backup

copy:
cp -v $FWDIR/conf/cpha_link_monitoring.conf{,_BKP}
If the $FWDIR/conf/cpha_link_monitoring.conf file does not exist, create it:
touch $FWDIR/conf/cpha_link_monitoring.conf
5 Edit the $FWDIR/conf/cpha_link_monitoring.conf file:

vi $FWDIR/conf/cpha_link_monitoring.conf
6 n To monitor the link only on specific interfaces:

Enter the names of the applicable interfaces - each name on a new separate line.
Example:
eth2
eth4
n To monitor the link on all interfaces:

Enter only this word:
all
7 Save the changes in the file and exit the editor.

Step Instructions
8 Reboot the Cluster Member.
Important - This can cause a failover.
Best Practices:
n In High Availability cluster
1. Perform the configuration steps on all Cluster Members
2. Reboot all the Standby Cluster Members
3. Initiate a manual failover on the Active Cluster Member
4. Reboot the former Active Cluster Member
n In Load Sharing Unicast cluster
2. Reboot all the non-Pivot Cluster Members
3. Initiate a manual failover on the Pivot Cluster Member
4. Reboot the former Pivot Cluster Member
n In Load Sharing Multicast cluster
2. Reboot all Cluster Members except one
3. Initiate a manual failover on the remaining Cluster Member
4. Reboot the remaining Cluster Member
Note - See "Initiating Manual Cluster Failover" on page 195.

Working with Bond Interfaces in Cluster
Working with Bond Interfaces in Cluster

This section describes the configuration of Bond Interfaces and Group of Bonds in cluster.
Bonding (Link Aggregation) Terminology

n Link Aggregation (Interface Bonding): Networking technology that binds multiple physical interfaces
together into one virtual interface.
n Bond: A group of physical interfaces that operate together as one virtual interface and share an IP
address and MAC address. A bond is identified by the cluster by its Bond ID (for example: bond0).
n Bond Interface: The logical representation of the bond.
n Slave (enslaved interface): A physical interface that is a member of a bond. Slaves do not have an IP
Address and in some cases share the same MAC address.

Bond Interfaces (Link Aggregation)

Check Point security devices support Link Aggregation, a technology that joins multiple physical interfaces
into one virtual interface, known as a bond interface.
The bond interface share the load among many interfaces, which gives fault tolerance and increases
throughput. Check Point devices support the IEEE 802.3ad Link Aggregation Control Protocol (LACP) for
dynamic link aggregation.
Item Description
1 Security Gateway
1A Interface 1
1B Interface 2
2 Bond Interface
3 Router
A bond interface (also known as a bonding group or bond) is identified by its Bond ID (for example: bond1)
and is assigned an IP address. The physical interfaces included in the bond are called slaves and do not
have IP addresses.

You can configure a bond interface to use one of these functional strategies:
High Availability (Active/Backup)
Gives redundancy when there is an interface or a link failure. This strategy also supports switch
redundancy.
Bond High Availability works in Active/Backup mode - interface Active/Standby mode. When an Active
slave interface is down, the connection automatically fails over to the primary slave interface. If the
primary slave interface is not available, the connection fails over to a different slave interface.
Load Sharing (Active/Active)
All slave interfaces in the UP state are used simultaneously.

Traffic is distributed among the slave interfaces to maximize throughput. Bond Load Sharing does not
support switch redundancy.
Note - Bonding Load Sharing mode requires SecureXL to be enabled on Security

Gateway or each Cluster Member.
You can configure Bond Load Sharing to use one of these modes:
Mode Description
Round Selects the Active slave interfaces sequentially.

Robin
Note - Scalable Chassis 60000 / 40000 do not support this mode (Known
Limitation MBS-4080).
802.3ad Dynamically uses Active slave interfaces to share the traffic load.
This mode uses the LACP protocol, which fully monitors the interface link between the
Check Point Security Gateway and a switch.
XOR All slave interfaces in the UP state are Active for Load Sharing.
Traffic is assigned to Active slave interfaces based on one of these transmit hash
policies:
n Layer 2 information (XOR of hardware MAC addresses)
n Layer 3+4 information (IP addresses and Ports)
ABXOR Slave interfaces in the UP state are assigned to sub-groups called bundles.
Only one bundle is Active at a time.
All slave interfaces in the Active bundle share the traffic load.
The system assigns traffic to all interfaces in the Active bundle based on the defined
transmit hash policy.
Note - Scalable Chassis 60000 / 40000 do not support this mode (Known
Limitation MBS-1520).

For Bonding High Availability mode and for Bonding Load Sharing mode:
n The number of bond interfaces that can be defined is limited by the maximal number of interfaces
supported by each platform.
See the R81.10 Release Notes.
n Up to 8 physical slave interfaces can be configured in a single bond interface.

Bond High Availability Mode in Cluster
Bond High Availability Mode in Cluster

When dealing with mission-critical applications, an enterprise requires its network to be highly available.
Clustering provides redundancy at the Security Gateway level.
Without Bonding, redundancy of Network Interface Cards (NICs) or of the switches on either side of the
Security Gateway are only possible in a cluster, and only by failover from one Cluster Member to another
Cluster Member.

Simple Redundant Topology
Simple Redundant Topology

You can have redundancy of clustering without Bonding.
If a switch or Cluster Member fails, a High Availability cluster solution provides system redundancy.
For example, you can have a redundant system with two synchronized Cluster Members deployed in a
simple redundant topology.
Item Description
1 Cluster Member GW1 with interfaces connected to the external switches (5 and 6)
3 Interconnecting network C1
4 Interconnecting network C2
5 Switch S1
6 Switch S2
If Cluster Member GW1 (1), its NIC, or switch S1 (5) fails, Cluster Member GW2 (2) becomes the only Active
member, connecting to switch S2 (6) over network C2 (4).
If any component fails (Cluster Member, NIC, or switch), the result of the failover is that no further
redundancy exists.
A further failure of any active component completely stops network traffic through this cluster.

Fully Meshed Redundancy
Fully Meshed Redundancy

The Bonding High Availability mode, when deployed with ClusterXL, enables a higher level of reliability by
providing granular redundancy in the network. This granular redundancy is achieved by using a fully
meshed topology, which provides for independent backups for both NICs and switches.
A fully meshed topology further enhances the redundancy in the system by providing a backup to both the
interface and the switch, essentially backing up the cable. Each Cluster Member has two external interfaces,
one connected to each switch.
Item Description
3 Interconnecting network
4 Switch S1
5 Switch S2
In this scenario:
n GW1 and GW2 are Cluster Members in the High Availability mode, each connected to the two
external switches
n S1 and S2 are external switches
n Item 3 are the network connections
If any of the interfaces on a Cluster Member that connect to an external switch fails, the other interface
continues to provide the connectivity.
If any Cluster Member, its NIC, or switch fails, the other Cluster Member, connecting to switch S2 over
network C2. If any component fails (Cluster Member , NIC, or switch), the result of the failover is that no
further redundancy exists. A further failure of any active component completely stops network traffic.
Bonding provides High Availability of NICs. If one fails, the other can function in its place.

Bond Failover in High Availability Mode
Bond Failover in High Availability Mode

In Bond High Availability mode, configured on Cluster Members, bond internal failover can occur in one of
these cases:
n An active interface detects a failure in the link state, in a monitored interface.
n ClusterXL detects a failure in sending or receiving Cluster Control Protocol (CCP) packets.
Either of these failures causes a failover within the interface bond, or between Cluster Members, depending
on the circumstances.
When a failure is detected, a log is recorded:
1. Connect with SmartConsole to the Management Server.
2. From the left navigation panel, click Logs & Monitor > Logs.

Configuring a Bond Interface in High Availability Mode

On each Cluster Member, follow the instructions in the R81.10 Gaia Administration Guide - Chapter Network
Management - Section Network Interfaces - Section Bond Interfaces (Link Aggregation).

Making Sure the Bond Interface is Functioning Properly

Examine the bond information to make sure the bond interface is UP:
Step Instructions
1 Connect to the command line on each Cluster Member.
2 Examine the cluster interfaces in one of these ways:

n In Gaia Clish:
show cluster members interfaces
all
cphaprob -am if
See "Viewing Cluster Interfaces" on page 221.
3 Examine the bond interfaces in one of these ways:

n In Gaia Clish:
show cluster bond {all | name
<Name of Bond>}
show bonding groups

cphaprob show_bond <Name of
Bond>
See "Viewing Bond Interfaces" on page 225.

Failover Support for VLANs
Failover Support for VLANs

In Bonding High Availability mode, ClusterXL monitors VLAN IDs for connectivity failure or
miscommunication, and initiate a failover when a failure is detected.
In a VLAN-enabled switched environment, ClusterXL monitors the VLAN with the lowest ID number. The
monitoring is conducted by sending ClusterXL Control Protocol (CCP) packets on round-trip paths at a set
interval.
The lowest VLAN ID indicates the status of the physical connection. This VLAN ID is always monitored, and
a connectivity failure causes ClusterXL to initiate a failover.
ClusterXL does not detect a VLAN configuration problems on switches.
For more information, see "Working with VLANs in Cluster" on page 110.

Sync Redundancy
Sync Redundancy
The use of more than one physical synchronization interface (1st sync, 2nd sync, 3rd sync) for
synchronization redundancy is not supported. For synchronization redundancy, you can use bond
interfaces.
Requirements and Limitations:

n The bond slave interfaces on each Cluster Member must connect to the same switch or VLAN (for
example, physical interface eth1 on all Cluster Members must connect to the same switch).
n We recommend that interfaces and other network hardware support the IEEE 802.3 bond mode.
n If you use a Bond in High Availability mode, you must add slave interfaces to the bonding group in the
same order on all Cluster Members.
Important - See "Supported Topologies for Synchronization Network" on page 42.
To configure bond interfaces for Sync High Availability:

1. Configure a bond interface on each Cluster Member with unused slave interfaces. See "Configuring a
Bond Interface in High Availability Mode" on page 131.
5. From the left tree, click Network Management.
6. At the top, click Get Interfaces > Get Interfaces With Topology.
7. Select the applicable interface and click Edit.
9. Click OK.
11. On each Cluster Member, make sure that the Sync interfaces are in the bond.
Examine the cluster interfaces in one of these ways:
n In Gaia Clish:
show cluster members interfaces all
cphaprob -am if

Configuring Bond High Availability in VRRP Cluster

The R80.20 version introduced an improved Active/Backup Bond mechanism (Enhanced Bond) when
working in ClusterXL.
If you work with ClusterXL, the Enhanced Bond feature is enabled by default, and no additional
configuration is required.
If you change your cluster configuration from ClusterXL to VRRP (MCVR & VRRP), or configure the VRRP
(MCVR & VRRP) cluster from scratch, the Enhanced Bond feature is disabled by default.
If you change your cluster configuration from VRRP to ClusterXL, you must manually enable the Enhanced
Bond feature.
To enable the Enhanced Bond feature in VRRP Cluster, set the value of the kernel parameter fwha_bond_
enhanced_enable to 1 on each VRRP Cluster Member. You can set the value of the kernel parameter
temporarily, or permanently.
Setting the value of the kernel parameter temporarily
Important - This change does not survive reboot.
Step Instructions
1 Connect to the command line on each VRRP Cluster Member.
3 Set the value of the kernel parameter fwha_bond_enhanced_enable to 1:

fw ctl set int fwha_bond_enhanced_enable 1
4 Make sure the value of the kernel parameter fwha_bond_enhanced_enable was set to
1:
fw ctl get int fwha_bond_enhanced_enable

Setting the value of the kernel parameter permanently
Step Instructions
3 Back up the current $FWDIR/boot/modules/fwkern.conf file:

4 Edit the current $FWDIR/boot/modules/fwkern.conf file:

5 Add this line to the file (spaces and comments are not allowed):
fwha_bond_enhanced_enable=1
6 Save the changes in the file and exit the editor.
7 Reboot the Cluster Member.
8 Make sure the value of the kernel parameter fwha_bond_enhanced_enable was set to
1:
fw ctl get int fwha_bond_enhanced_enable
Important - If you change your cluster configuration from VRRP to ClusterXL, you must
remove the kernel parameter configuration from each Cluster Member.

Bond Load Sharing Mode in Cluster
Bond Load Sharing Mode in Cluster

In Bond Load Sharing mode:
n All slave interfaces are active, and connections are balanced between the bond slave interfaces,
similar to the way ClusterXL Load Sharing balances connections between Cluster Members.
n Each connection is assigned to a specific slave interface. For the individual connection, only one
slave interface is active. On failure of that interface, the bond fails over the connection to one of the
other slave interfaces, which adds the failed interface connection to the connections it is already
handling.
n All the slave interfaces of a bond must be connected to the same switch. The switch itself must
support and be configured for Bonding, by the same standard (for example, 802.3ad, or XOR) as the
Security Gateway bond.
Important - Bond Load Sharing mode requires SecureXL to be enabled on each Cluster
Member (this is the default).

Bond Failover in Load Sharing Mode
Bond Failover in Load Sharing Mode

In Bond Load Sharing mode, configured on Cluster Members, bond internal failover can occur in one of
these cases:
n An active interface detects a failure in the link state, in a monitored interface.
n ClusterXL detects a failure in sending or receiving Cluster Control Protocol (CCP) packets.
Either of these failures will induce a failover within the interface bond, or between Cluster Members,
depending on the circumstances.
When a failure is detected, a log is recorded:
2. From the left navigation panel, click Logs & Monitor > Logs.


On each Cluster Member, follow the instructions in the R81.10 Gaia Administration Guide - Chapter Network
Management - Section Network Interfaces - Section Bond Interfaces (Link Aggregation).

Configuring Minimal Number of Required Slave Interfaces
Configuring Minimal Number of Required Slave Interfaces

ClusterXL considers a bond in Load Sharing mode to be in the "down" state when fewer than a minimal
number of required slave interfaces stay in the "up" state.
By default, the minimal number of required slave interfaces, which must stay in the "up" state in a bond of n
slave interfaces is n-1.
If one more slave interface fails (when n-2 slave interfaces stay in the "up" state), ClusterXL considers the
bond interface to be in the "down" state, even if the bond contains more than two slave interfaces.
If a smaller number of slave interfaces can pass the expected traffic, you can configure explicitly the minimal
number of required slave interfaces.
Follow "Configuring the Minimal Number of Required Slave Interfaces for Bond Load Sharing" on page 199.


Examine the bond information to make sure the bond interface is UP:
Step Instructions
2 Examine the cluster interfaces in one of these ways:

n In Gaia Clish:
show cluster members interfaces
all
cphaprob -am if
3 Examine the bond interfaces in one of these ways:

n In Gaia Clish:
show cluster bond {all | name
<Name of Bond>}
show bonding groups

cphaprob show_bond <Name of
Bond>

Group of Bonds
Group of Bonds
In This Section:
Introduction
Group of Bonds, which is a logical group of existing Bond interfaces, provides additional link redundancy.
Example topology with Group of Bonds
n There is one router - R

n There are two switches that connect to the router R: SW-1 and SW-2
n There are two Cluster Members GW-A (Active) and GW-B (Standby)
n There are two Bond interfaces on each Cluster Member: Bond-1 and Bond-2
n On the Cluster Member GW-A:
l Bond-1 interface connects to the switch SW-1
n On the Cluster Member GW-B:

Group of Bonds
Chain of events without Group of Bonds
1. The Cluster Member GW-A is the Active and the Cluster Member GW-B is the Standby.
2. On the Cluster Member GW-A, the Bond-1 interface fails.
3. On the Cluster Member GW-A, the Critical Device Interface Active Check reports its state as
"problem".
4. The Cluster Member GW-A changes its cluster state from Active to Down.
5. The cluster fails over - the Cluster Member GW-B changes its cluster state from Standby to Active.
This is not the desired behavior, because the Cluster Member GW-A connects not only to the switch SW-
1, but also to the switch SW-2. In our example topology, there is no actual reason to fail-over from the
Cluster Member GW-A to the Cluster Member GW-B.
In order to overcome this problem, Cluster Members use the Group of Bonds consisting of Bond-1 and
Bond-2. The Group of Bonds fails only when both Bond interfaces fail on the Cluster Member. Only then
the cluster fails over.
Chain of events with configured Group of Bonds
1. The Cluster Member GW-A is the Active and the Cluster Member GW-B is the Standby.
2. On the Cluster Member GW-A, the Bond-1 interface fails.

Group of Bonds
3. On the Cluster Member GW-A, the Critical Device Interface Active Check reports its state as
"problem".
4. The Cluster Member GW-A does not change its cluster state from Active to Down.
5. On the Cluster Member GW-A, the Bond-2 interface fails as well.
6. The Cluster Member GW-A changes its cluster state from Active to Down.
7. The cluster fails over - the Cluster Member GW-B changes its cluster state from Standby to Active.
Creating a new Group of Bonds

This procedure lets you create a new Group of Bonds.
Procedure

3. In VSX Cluster, switch to the context of the applicable Virtual System:
vsenv <VSID>
4. Modify the current $FWDIR/boot/modules/fwkern.conf file:

a. Backup the current file:
b. Edit the current file:
c. Add these two lines at the bottom of the file (spaces or comments are not allowed):
fwha_group_of_bonds_str=<Name for Group of Bonds>:<List of all

Bonds in this Group separated by comma>
fwha_arp_probe_method=1
Example:
fwha_group_of_bonds_str=GoB0:bond0,bond1;GoB1:bond2,bond3
Note - The kernel parameter "fwha_arp_probe_method" configures

the Cluster Member to use the Virtual IP address as the Source IP
address in the ARP Requests during the probing of the local network.
d. Save the changes in the file and exit the editor.

Group of Bonds
5. Change the value of the kernel parameter fwha_group_of_bonds_str to add the Group of
Bonds on-the-fly:
fw ctl set str fwha_group_of_bonds_str '<Name for Group of

Bonds>:<List of all Bonds in this Group separated by comma>'
Example:
fw ctl set str fwha_group_of_bonds_str

'GoB0:bond0,bond1;GoB1:bond2,bond3'
Notes:
n The apostrophe characters are mandatory part of the syntax.
n Spaces are not allowed in the value of the kernel parameter fwha_
group_of_bonds_str.
6. Change the value of the kernel parameter fwha_arp_probe_method on-the-fly:
fw ctl set int fwha_arp_probe_method 1
7. Make sure the Cluster Member accepted the new configuration:
fw ctl get str fwha_group_of_bonds_str

fw ctl get int fwha_arp_probe_method
8. In SmartConsole, install the Access Control Policy on the cluster object.
Adding a Bond interface to the existing Group of Bonds

This procedure lets you add an additional bond interface to the existing Group of Bonds.
Procedure

vsenv <VSID>


Group of Bonds
c. Edit the value of the kernel parameter fwha_group_of_bonds_str to add the Bond
interface to the existing Group of Bonds.
Example:
fwha_group_of_bonds_str=GoB0:bond0,bond1;GoB1:bond2,bond3,bond4

5. Get the current value of the kernel parameter fwha_group_of_bonds_str and copy it:
6. Reset the current value of the kernel parameter fwha_group_of_bonds_str:
fw ctl set str fwha_group_of_bonds_str ''
7. Make sure the Cluster Member reset the value of the kernel parameter fwha_group_of_bonds_
str:
8. Change the value of the kernel parameter fwha_group_of_bonds_str to add the Bond
interface to the existing Group of Bonds on-the-fly:

Example:

'GoB0:bond0,bond1;GoB1:bond2,bond3,bond4'
Notes:
group_of_bonds_str.
Removing a Bond interface from the existing Group of Bonds

This procedure lets you remove a bond interface from an existing Group of Bonds.
Procedure

Group of Bonds

vsenv <VSID>

c. Edit the value of the kernel parameter fwha_group_of_bonds_str to remove the Bond
interface from the existing Group of Bonds.
Example:
fwha_group_of_bonds_str=GoB0:bond0,bond1;GoB1:bond2,bond3

5. Get the current value of the kernel parameter fwha_group_of_bonds_str and copy it:
str:
8. Change the value of the kernel parameter fwha_group_of_bonds_str to remove the Bond
interface from the existing Group of Bonds on-the-fly:

Example:

'GoB0:bond0,bond1;GoB1:bond2,bond3
Notes:
group_of_bonds_str.

Group of Bonds
Deleting a Group of Bonds

This procedure lets you delete an existing Group of Bonds.
Procedure

vsenv <VSID>

c. Delete these two lines in the file:
fwha_group_of_bonds_str=<Name for Group of Bonds>:<List of all

Bonds in this Group separated by comma>

str:
Monitoring
To see the configured Groups of Bonds, run the "cphaprob show_bond_groups" command. See
"Viewing Bond Interfaces" on page 225.

Group of Bonds
Logs
Cluster Members generate some applicable logs.
Applicable log files
1. User Space logs:

n In non-VSX Cluster:
In the /var/log/messages files.
Output of the dmesg command.
n In VSX Cluster:
In the $FWDIR/log/fwk.elg file in the context of the applicable Virtual System.
2. Kernel Space logs:
n In the kernel module fw, enable the debug flags error and ioctl
n Kernel module cluster, enable the debug flag if
See the R81.10 Quantum Security Gateway Guide - Chapter Kernel Debug on Security Gateway.
The kernel debug shows:
n A physical slave interface goes down/up.
n A Bond interface goes down/up (regardless of a change in the Cluster Member's state).
n A Group of Bonds goes down/up.
Note - These logs are generated only once per event.
Limitations
Specific limitations apply to a Group of Bonds.
List of Limitations
n The maximal length on the text string "<Name for Group of Bonds>" is 16 characters.
n The maximal length on this text string is 1024 characters:
<Name for Group of Bonds>:<List of all Bonds in this Group separated

by comma>
n You can configure the maximum of five Groups of Bonds on a Cluster Member or Virtual System.
n You can configure the maximum of five Bond interfaces in each Groups of Bonds.
n Group of Bonds feature does support Virtual Switches and Virtual Routers. Meaning, do not
configure Groups of Bonds in the context of these Virtual Devices.
n Group of Bonds feature supports only Bond interfaces that belong to the same Virtual System.

Group of Bonds
You cannot configure bonds that belong to different Virtual Systems into the same Group of
Bonds.
You must perform all configuration in the context of the applicable Virtual System.
n Group of Bonds feature does support Sync interfaces (an interface on a Cluster Member, whose
Network Type was set as Sync or Cluster+Sync in SmartConsole in cluster object).
n Group of Bonds feature does support Bridge interfaces.
n If a Bond interface goes down on one Cluster Member, the "cphaprob show_bond_groups"
command (see "Viewing Bond Interfaces" on page 225) on the peer Cluster Members also shows
the same Bond interface as DOWN.
This is because the peer Cluster Members stop receiving the CCP packets on that Bond interface
and cannot probe the local network to determine that their Bond interface is really working.
n After you add a Bond interface to the existing Group of Bonds, you must install the Access Control
Policy on the cluster object.
n After you remove a Bond interface from the existing Group of Bonds, you must install the Access
Control Policy on the cluster object.

Performance Guidelines for Bond Interfaces

For optimal performance, follow these guidelines:
1. Configure static affinities of bond slave interfaces to CPU Cores.
2. Whenever possible, dedicate one processing core to each interface.
3. If there are more physical interfaces than CPU cores, then some CPU cores handle two or more
interfaces.
Use pairs of slave interface of the same position with internal and external bonds.
a. To view positions of slave interface in a bond, run in the Expert mode:
cat /proc/net/bonding/<Name of Bond Interface>
b. Note the sequence of the interfaces in the output.

Compare this sequence for the two bonds (external bond and its respective internal bond).
Slave interfaces that appear in the same position in the two bonds are interface pairs.
Set these pairs to be handled by one processing CPU core.
Example configuration
An appliance has:
n Four processing CPU cores:
core 0, core 1, core 2, and core 3
n Two bond interfaces:
bond0 with slave interfaces eth0, eth1, and eth2
bond1 with slave interfaces eth3, eth4, and eth5
In such case, two of the CPU cores need to handle two slave interfaces each.
An optimal configuration can be:
CPU core bond0 bond1
0 eth0 eth3
1 eth1 eth4
2 eth2
3 eth5
For more information, see the R81.10 Performance Tuning Administration Guide:
n Chapter SecureXL > Section SecureXL Commands and Debug > Section 'sim' and 'sim6' > Section
sim affinity.

n Chapter CoreXL > Section Configuring Affinity Settings.

n Chapter CoreXL > Section Affinity Settings for 16000 and 26000 Appliances.

Troubleshooting Issues with Bonded Interfaces
Troubleshooting Issues with Bonded Interfaces

Troubleshooting Workflow
1. View the logs from this cluster in SmartConsole > Logs & Monitor > Logs.
2. On the Cluster Members, examine the status of the bond interface in one of these ways:
n In Gaia Clish:
show cluster bond name <Name of Bond>
cphaprob show_bond <Name of Bond>

3. If there is a problem, see if the physical link is down:
a. Look for a slave interface that reports the status of the link as "no".
b. Examine the cable connections and other hardware.
c. Examine the port configuration on the switch, to which this slave interface connects.
On a VSX Cluster Member, reboot is needed after these actions on a bond interface:
1. Changing a bond mode.
2. Adding a slave interface into an existing bond.
Note - Removing a slave interface from an existing bond, does not require a reboot.
Connectivity Delays on Switches

Connectivity delays may occur in switches during some internal bond failovers. With the various features
that are now included on some switches, it can take close to a minute for a switch to begin servicing a newly
connected interface.
These are suggestions for reducing the startup time after link failure.
1. Disable auto-negotiation on the relevant interface.
2. On Cisco switches, enable the PortFast feature (see the applicable Cisco documentation).
Warning - The PortFast feature should never be used on ports that connect to
switches or hubs. It is important that the Spanning Tree complete the initialization
procedure in these situations. Otherwise, these connections may cause physical
loops where packets are continuously forwarded (or even multiply) in such a way
that can cause the network to fail.
3. Disable STP on the switch ports (see the applicable switch vendor documentation).

Advanced Cluster Configuration
Advanced Cluster Configuration

A number of synchronization and ClusterXL capabilities are controlled by kernel parameters.
Important: - In a Cluster, you must configure all the Cluster Members in the same way.
See the R81.10 Quantum Security Gateway Guide - Chapter Working with Kernel Parameters on Security
Gateway.

Controlling the Clustering and Synchronization Timers
Controlling the Clustering and Synchronization Timers
Best Practice - Do not change the default values.
These kernel parameters control the clustering and synchronization timers.
Parameter Description Default Value
fwha_timer_sync_res The frequency of sync flush operations on the cluster. 1

Operations occur every number of milliseconds:
10 x (fwha_timer_sync_res) x (fwha_
timer_base_res)
fwha_timer_base_res Must be divisible by 10 with no remainders. 10

Blocking New Connections Under Load
Important - This section applies only to ClusterXL Load Sharing modes.
The reason for blocking new connections is that new connections are the main source of new Delta
Synchronization traffic. Delta Synchronization may be at risk, if new traffic continues to be processed at high
rate.
A related error message in cluster logs and in the /var/log/messages file is:
State synchronization is in risk
Reducing the amount of traffic passing through the Cluster Member protects the Delta Synchronization
mechanism. See sk43896: Blocking New Connections Under Load in ClusterXL.
These kernel parameters let you control how Cluster Member behave:
Kernel
Description
Parameter
fw_sync_ Controls how Cluster Member detect heavy loads and whether they start blocking new
block_new_ connections.
conns Load is considered heavy when the synchronization transmit queue of the Cluster
Member starts to fill beyond the value of the kernel parameter "fw_sync_buffer_
threshold".
n To enable blocking new connections under load, set the value of the "fw_
sync_block_new_conns" to 0.
n To disable blocking new connections under load, set the value of the "fw_
sync_block_new_conns" to -1 (must use the hex value 0xFFFFFFFF). This
is the default.
Note - Blocking new connections when sync is busy is only recommended
for ClusterXL Load Sharing deployments. While it is possible to block new
connections in ClusterXL High Availability mode, doing so does not solve
inconsistencies in sync, because the High Availability mode prevents that
from happening.
fw_sync_ Configures the maximum percentage of the buffer that may be filled before new
buffer_ connections are blocked (see the parameter "fw_sync_block_new_conns" above).
threshold The default percentage value is 80, with a buffer size of 512.
By default, if more than 410 consecutive packets are sent without getting an ACK on
any one of them, new connections are dropped.

Kernel
Description
Parameter
fw_sync_ Determines the type of connections that can be opened while the system is in a
allowed_ blocking state.
protocols Thus, the user can have better control over the system behavior in cases of unusual
load.
The value of this kernel parameter is a combination of flags, each specifying a different
type of connection. The required value is the result of adding the separate values of
these flags.
Summary table:
Flag Value
ICMP_CONN_ALLOWED 1
TCP_CONN_ALLOWED 2 (except for data connections)
UDP_CONN_ALLOWED 4 (except for data connections)
TCP_DATA_CONN_ 8 (the control connection should be established or

ALLOWED allowed)
UDP_DATA_CONN_ 16 (the control connection should be established or

ALLOWED allowed)
The default value is 24, which is the sum of "TCP_DATA_CONN_ALLOWED" (value 8)
and UDP_DATA_CONN_ALLOWED (value 16). This means that the default allows only
TCP and UDP data connections to be opened under load.

Defining Non-Monitored Interfaces
Defining Non-Monitored Interfaces

Non-Monitored interfaces are Cluster Member interfaces that are not monitored by the ClusterXL
mechanism.
You may wish to define an interface as non-monitored, if the interface is down for a long time, and you wish
the Cluster Member to continue to be Active.
5. Select the interface, which is down for a long time and click Edit.
6. In the Network Type section, select Private.
7. Click OK.

Configuring Policy Update Timeout
Configuring Policy Update Timeout

When policy is installed on a cluster, the Cluster Members undertake a negotiation process to make sure all
of them have received the same policy before they actually apply it.
This negotiation process has a timeout mechanism, which makes sure a Cluster Member does not wait
indefinitely for responses from other Cluster Members, which is useful in cases when another Cluster
Member goes down when policy is being installed (for example).
In configurations, on which policy installation takes a long time (usually caused by a policy with a large
number of rules), a cluster with more than two members, and slow members, this timeout mechanism may
expire prematurely.
It is possible to tune the timeout with the kernel parameter fwha_policy_update_timeout_factor.
The default value of this kernel parameter is 1, which should be sufficient for most configurations.
For configurations where the situation described above occurs, setting the value of this kernel parameter to
2 should be sufficient.
Warning - Do not set the value of this kernel parameter to a number larger than 3.
See the R81.10 Quantum Security Gateway Guide - Chapter Working with Kernel Parameters on Security
Gateway.

Enhanced 3-Way TCP Handshake Enforcement

The standard enforcement for a 3-way handshake that initiates a TCP connection provides adequate
security by guaranteeing one-directional stickiness.
This means that it ensures that the SYN-ACK will always arrive after the SYN. However, it does not
guarantee that the ACK will always arrive after the SYN-ACK, or that the first data packet will arrive after the
ACK.
If you wish to have stricter policy that denies all out-of-state packets, you can configure the synchronization
mechanism so that all the TCP connection initiation packets arrive in the right sequence (SYN, SYN-ACK,
ACK, followed by the data).
Warning - The price for this extra security is a considerable delay in TCP connection
establishment.
Procedure to enable the enhanced TCP Handshake enforcement
1. Close all SmartConsole windows connected to the Management Server.

2. Connect with GuiDBedit Tool (see sk13009) to the Security Management Server or Domain
Management Server that manages this cluster.
3. In the left upper pane, go to Table > Network Objects > network_objects.
4. In the right upper pane, select the cluster object (the Class Name column shows gateway_
cluster).
5. Press the CTRL+F keys (or go to Search menu > Find).
6. In the Find window, paste this string and click Find Next:
sync_tcp_handshake_mode
7. In the lower pane, right-click on the sync_tcp_handshake_mode property and select Edit.
8. Choose complete_sync and click OK.
For more information, see the section "Synchronization modes for TCP 3-way handshake" below.
9. To save the changes, from the File menu select Save All.
10. Close GuiDBedit Tool.
11. Connect with SmartConsole to the Security Management Server or Domain Management Server
that manages this cluster.
12. In SmartConsole, install the Access Control Policy onto the Cluster object.
Synchronization modes for TCP 3-way handshake
Mode Instructions
Minimal This is the default 3-way handshake synchronization mode.

sync The 3-way handshake is not enforced.
This mode offers the best connectivity for users who are willing to compromise on
security is this case.

Mode Instructions
Complete All 3-way handshake packets are Sync-and-ACK'ed, and the 3-way handshake is
sync enforced.
This mode slows down connection establishment considerably.
It may be used when there is no way to know where the next packet goes (for
example, in 3rd party clusters).
Smart sync In most cases, we can assume that if SYN and SYN-ACK were encountered by the
same cluster member, then the connection is “sticky”.
ClusterXL uses one additional flag in Connections Table record that says, “If this
member encounters a 3-way handshake packet, it should sync all other cluster
members”.
When a SYN packet arrives, the member that encountered it, records the connection
and turns off its flag. All other members are synchronized, and by using a post-sync
handler, their flag is turned on (in their Connections Tables).
If the same member encounters the SYN-ACK packet, the connection is sticky, thus
other cluster members are not informed.
Otherwise, the relevant member will inform all other member (since its flag is turned
on).
The original member (that encountered the SYN) will now turn on its flag, thus all
members will have their flag on.
In this case, the third packet of the 3-way handshake is also synchronized.
If for some reason, our previous assumption is not true (i.e., one cluster member
encountered both SYN and SYN-ACK packets, and other members encountered the
third ACK), then the “third” ACK will be dropped by the other cluster members, and we
rely on the periodic sync and TCP retransmission scheme to complete the 3-way
handshake.
This 3-way handshake synchronization mode is a good solution for ClusterXL Load
Sharing users that want to enforce 3-way handshake verification with the minimal
performance cost.
This 3-way handshake synchronization mode is also recommended for ClusterXL
High Availability.

Cluster IP Addresses on Different Subnets
Cluster IP Addresses on Different Subnets

You can configure cluster Virtual IP addresses in different subnets than the physical IP addresses of the
Cluster Members.
The network "sees" the cluster as one Security Gateway that operates as a network router. The network is
not aware of the internal cluster structure and physical IP addresses of Cluster Members.
Advantages of using different subnets:

n You can create a cluster in an existing subnet that has a shortage of available IP addresses.
n You use only one Virtual IP address for the cluster. All other IP addresses can be on other subnets.
n You can "hide" physical Cluster Members' IP addresses behind the cluster Virtual IP address. This
security practice is almost the same as NAT.
Note - This capability is available only for ClusterXL clusters.
Traffic sent from Cluster Members to internal or external networks is hidden behind the cluster Virtual IP
addresses and cluster MAC addresses. The cluster MAC address assigned to cluster interfaces is:
Cluster Mode MAC Address
High Availability MAC address of the Active Cluster Member's interface
Load Sharing Multicast Multicast MAC address of the cluster Virtual IP Address
Load Sharing Unicast MAC address of the Pivot Cluster Member's interface
The use of different subnets with cluster objects has some limitations - see "Limitations of Cluster
Addresses on Different Subnets" on page 158.


When using a cluster, in which the Cluster Virtual IP address and physical IP addresses of Cluster Members
are on different subnets, it is necessary to define the settings manually (see "Example of Cluster IP
Addresses on Different Subnets" on page 156).
1. Define these static routes for each cluster Virtual IP address
The required static routes must use applicable local member's interface as the next hop gateway
for network of cluster Virtual IP address.
If you do not define the static routes correctly, Cluster Members are not able to pass traffic.
Note - In VSX Cluster, you must configure all routes only in SmartConsole in
the VSX Cluster object.
For configuration instructions, see the R81.10 Gaia Administration Guide - Chapter Network
Management - Sections IPv4 Static Routes and IPv6 Static Routes.
2. Define the Cluster Members Network
a. In SmartConsole, use the Classic Mode to manually create a new cluster.

b. Define the Cluster Members and their physical interfaces.
c. From the left tree, click Network Management.
d. Select each cluster interface and click Edit.
e. In the General section, in the Virtual IPv4 field, enter the IPv4 address.
f. In the Member IPs, make sure the IP addresses are correct.
g. Click OK.
h. Install the Access Control Policy on this cluster object.
For more details, see the chapter "Configuring the Cluster Object and Members" on page 88.

Example of Cluster IP Addresses on Different Subnets

In this example, a cluster separates the network 172.16.6.0 / 24 (Side "A") from the network 172.16.4.0 / 24
(Side "B").
The Cluster Members use these IP addresses:
n 192.168.1.x / 24 for Side "A"
n 172.16.6.100 / 24 for Cluster Virtual IP Address for Side "A"
n 192.168.2.x / 24 for Side "B"
n 172.16.4.100 / 24 for Cluster Virtual IP Address for Side "B"
n 192.168.3.x / 24 for the Synchronization network
Procedure:
1. Configure static routes on the Cluster Members
On each Cluster Member, configure these static routes:

n Next hop gateway for network 172.16.6.0 is the local interface with the IP address
192.168.1.x
n Next hop gateway for network 172.16.4.0 is the local interface with the IP address
192.168.2.x
See the R81.10 Gaia Administration Guide > Chapter Network Management > Section IPv4 Static
Routes.
2. Configure Cluster IP Addresses in SmartConsole
a. Connect with SmartConsole to the Management Server.

b. From the left navigation panel, click Gateways & Servers.
c. Open the cluster object.
d. From the left tree, click Network Management.
e. Select each cluster interface and click Edit.

f. Configure these settings:
IP address of IP address of
Interface Properties
Cluster Interface "A" Cluster Interface "B
The General section - 172.16.6.100 / 24 172.16.4.100 / 24

the Virtual IPv4 field
The Member IPs section 192.168.1.1 / 24 192.168.2.1 / 24

192.168.1.2 / 24 192.168.2.2 / 24
g. Click OK.
h. Install the Access Control Policy on this cluster object.

Limitations of Cluster Addresses on Different Subnets

This new feature does not yet support all the capabilities of ClusterXL.
Some features require additional configuration to work properly, while others are not supported.
Connectivity Between Cluster Members

Since ARP requests issued by Cluster Members are hidden behind the cluster IP and MAC addresses,
requests sent by one Cluster Member to the other may be ignored by the destination computer.
To allow Cluster Members to communicate with each other, a static ARP should be configured for each
Cluster Member, stating the MAC addresses of all other Cluster Members. IP packets sent between Cluster
Members are not altered, and therefore no changes should be made to the routing table.
Note - Static ARP is not required in order for the Cluster Members to work properly as a
cluster, since the cluster synchronization protocol does not rely on ARP.
Load Sharing Multicast Mode with "Semi-Supporting" Hardware

Although not all types of network hardware work with multicast MAC addresses, some routers can pass
such packets, even though they are unable to handle ARP Replies containing a multicast MAC address.
Where a router semi-supports Load Sharing Multicast mode, it is possible to configure the cluster MAC
address as a static ARP entry in the router internal tables, and thus allow it to communicate with the cluster.
When different subnets are used for the cluster IP addresses, static ARP entries containing the router MAC
address need to be configured on each Cluster Member. This is done because this kind of router will not
respond to ARP Requests containing a multicast source MAC address. These special procedures are not
required when using routers that fully support multicast MAC addresses.
Manual Proxy ARP

When using Static NAT, the cluster can be configured to automatically recognize the hosts hidden behind it,
and issue ARP replies with the cluster MAC address, on their behalf. This process is known as Automatic
Proxy ARP.
However, if you use the ClusterXL VMAC mode or different subnets for the cluster IP addresses, this
mechanism will not work, and you must configure the proxy ARP manually. To do so, in SmartConsole, click
Menu > Global properties > NAT Network Address Translation, and disable Automatic ARP
Configuration. Then create the $FWDIR/conf/local.arp file.
For instructions, see sk30197.
Connecting to the Cluster Members from the Cluster Network

Because the unique IP addresses may be chosen arbitrarily, there is no guarantee that these addresses are
accessible from the subnet of the cluster IP address.
To access the Cluster Members through their unique IP addresses, you must configure routes on the
accessing Cluster Member, such that the cluster IP is the Default Gateway for the subnet of the unique IP
addresses.

Configuring Anti-Spoofing
3. Create a Group object, which contains the objects of both the external network and the internal
network.
In the "Example of Cluster IP Addresses on Different Subnets" on page 156, suppose Side "A" is the
external network, and Side "B" is the internal network.
You must configure the Group object to contain both the network 172.16.4.0 / 24 and the network
192.168.2.0 / 24.
6. Select the cluster interface and click Edit.
7. On the General page, in the Topology section, click Modify.
8. Select Override.
9. Select This Network (Internal).
10. Select Specific
11. Select the Group object that contains the objects of both the external network and the internal
network.
12. Click OK.

Important - Schedule a full maintenance window to perform this procedure.
Best Practice - Before you change the current configuration, export a complete
management database with "migrate_server" command. See the R81.10 CLI
Reference Guide.
Adding a New Cluster Member to the Cluster Object

1. Install a new Cluster Member
Install a new Cluster Member you plan to add to the existing cluster.
See the R81.10 Installation and Upgrade Guide > Chapter Installing a ClusterXL, VSX Cluster,
VRRP Cluster.
Follow only the step "Install the Cluster Members".
Important - The new Cluster Member must run the same version with the
same Hotfixes as the existing Cluster Members.
2. Configure the new Cluster Member
On the new Cluster Member you plan to add to the existing cluster:
a. Configure or change the IP addresses on the applicable interfaces to match the current
cluster topology.
Use Gaia Portal or Gaia Clish.
See R81.10 Gaia Administration Guide.
b. Configure or change the applicable static routes to match the current cluster topology.
c. Connect to the command line.
d. Log in to Gaia Clish or the Expert mode.
e. Start the Check Point Configuration Tool. Run:
cpconfig
f. Select the option Enable cluster membership for this gateway and enter y to confirm.
g. Reboot the new Cluster Member.
3. Configure the cluster object in SmartConsole
a. Connect with SmartConsole to the Security Management Server or Domain Management

Server that manages this cluster.


c. Open the existing cluster object.
d. In the Cluster Members page, click Add > New Cluster Member.
The Cluster Members Properties window opens.
Follow the instructions on the screen to configure this new Cluster Member.
e. Click the General tab.
f. In the Name field, enter a Cluster Member name.
g. In the IPv4 Address field, enter a physical IPv4 addresses.
The Management Server must be able to connect to the Cluster Member at this IPv4
address.
This IPv4 address can be an internal, or external. You can use a dedicated management
interface on the Cluster Member.
Important - You must define a corresponding IPv4 address for every

IPv6 address. This release does not support the configuration of only
IPv6 addresses.
h. In the IPv6 Address field, enter a physical IPv6 address, if you need to use IPv6.
The Management Server must be able to connect to the Cluster Member at this IPv6
address. This IPv6 address can be an internal, or external. You can use a dedicated
management interface on the Cluster Member.
Important - You must define a corresponding IPv4 address for every

IPv6 address. This release does not support the configuration of only
IPv6 addresses.
i. Click Communication, and initialize Secure Internal Communication (SIC) trust.

Enter the same key you entered during First Time Configuration Wizard on the new Cluster
Member.
j. Click the NAT tab to configure the applicable NAT settings.
k. Click the VPN tab to configure the applicable VPN settings.
l. Click OK.
m. From the left tree, click Network Management.
n Make sure all interfaces are defined correctly.
n Make sure all IP addresses are defined correctly.
n. Click OK.
o. Publish the SmartConsole session.
p. Install the Access Control Policy on this cluster object.
Policy installation must succeed on all Cluster Members.

q. Install the Threat Prevention Policy on this cluster object.

4. Examine the cluster state
On each Cluster Member (existing and the newly added):

a. Connect to the command line.
b. Log in to Gaia Clish or the Expert mode.
c. Make sure all Cluster Members detect each other and agree on their cluster states. Run:
Shell Command
Gaia Clish i. set virtual-system

<VSID>
ii. show cluster state
Expert mode cphaprob [-vs <VSID>] state
For more information, see "Viewing Cluster State" on page 209.

If Cluster Members do not detect each other, or do not agree on their cluster states, then restart
the clustering.
Procedure
a. Connect to the command line on each Cluster Member (existing and the newly added).
c. Restart the clustering on each Cluster Member.
Run:
cphastop
cphastart
Important - This temporarily causes the Cluster Member not to be a

part of the cluster. As a result, cluster failover can occur.
d. Make sure all Cluster Members detect each other and agree on their cluster states. Run:
Shell Command

<VSID>

Adding an Existing Security Gateway as a Cluster Member

to the Cluster Object
Important - The existing Security Gateway must run the same version with the same
Hotfixes as the existing Cluster Members.
1. Configure the existing Security Gateway
On the existing Security Gateway you plan to add to the existing cluster:
a. Configure or change the IP addresses on the applicable interfaces to match the current
cluster topology.
See R81.10 Gaia Administration Guide.
b. Configure or change the applicable static routes to match the current cluster topology.
c. Connect to the command line.
d. Log in to Gaia Clish or the Expert mode.
e. Start the Check Point Configuration Tool. Run:
cpconfig
f. Select the option Enable cluster membership for this gateway and enter y to confirm.
g. Reboot the Security Gateway.

d. From the left tree, click Cluster Members.

e. Click Add > Add Existing Gateway.

Follow the instructions on the screen to configure this new Cluster Member.
Read the warning and click Yes:
If you add <Name_of_Security_Gateway_object> to the cluster, it

will be converted to a cluster member. Some settings will be
lost.
The following settings will still remain:
-SIC
-VPN
-NAT (except for IP Pools)
In order to revert the conversion, session must be discarded.
Are you sure you want to continue?
f. In the list of Cluster Members, select the new Cluster Member and click Edit.
g. Click the NAT tab and configure the applicable NAT settings.
h. Click the VPN tab and configure the applicable VPN settings.
i. From the left tree, click Network Management.
j. Click OK.
k. Install the Access Control Policy on this cluster object.
l. Install the Threat Prevention Policy on this cluster object.
3. Examine the cluster state
On each Cluster Member (existing and the newly added):

c. Make sure all Cluster Members detect each other and agree on their cluster states. Run:
Shell Command

<VSID>
For more information, see "Viewing Cluster State" on page 209.

If Cluster Members do not detect each other, or do not agree on their cluster states, then restart
the clustering.

Procedure
a. Connect to the command line on each Cluster Member (existing and the newly added).
c. Restart the clustering on each Cluster Member.
Run:
cphastop
cphastart
Important - This temporarily causes the Cluster Member not to be a

part of the cluster. As a result, cluster failover can occur.
Shell Command

<VSID>

Important - Schedule a full maintenance window to perform this procedure.
Best Practice - Before you change the current configuration, export a complete
management database with "migrate_server" command. See the R81.10 CLI
Reference Guide.

d. From the left tree, click Cluster Members page.
e. Click Remove > Delete Cluster Member.
Confirm when prompted.
Important:
n This operation deletes the object.
n There must be at least two Cluster Members in the cluster
object.
f. From the left tree, click Network Management.
g. Click OK.
h. Install the Access Control Policy on the cluster object.
2. Restart the clustering and examine the cluster state
On each existing Cluster Member:


c. Restart the clustering.

Run:
cphastop
cphastart
Important - This temporarily causes the Cluster Member not to be a part

of the cluster. As a result, cluster failover can occur.
Shell Command

<VSID>
3. Configure the removed Cluster Member
On the Security Gateway you removed from the existing cluster:

c. Start the Check Point Configuration Tool. Run:
cpconfig
d. Select the option Disable cluster membership for this gateway and enter y to confirm.
e. Select the option Secure Internal Communication > enter y to confirm > enter the new
Activation Key. Make sure to write it down.
f. Exit from the cpconfig menu.
g. Reboot the Security Gateway.
4. Establish SIC with the removed Security Gateway
If you need to use the Security Gateway you removed from the existing cluster, then establish
Secure Internal Communication (SIC) with it.
Server that manages this Security Gateway.

c. Create a new Security Gateway object in one of these ways:
n From the top toolbar, click the New ( ) > Gateway.

Gateways and Servers > New Gateway.
Gateways and Servers > Gateway.
d. Follow the instructions on the screen.
Enter the same Activation Key you entered earlier in the cpconfig menu.
e. Click OK.
f. Publish the SmartConsole session.
g. Install the Access Control Policy on the Security Gateway object.
h. Install the Threat Prevention Policy on the Security Gateway object.

ISP Redundancy on a Cluster

In This Section:
Introduction 169
ISP Redundancy Modes 171
Outgoing Connections 172
Incoming Connections 173
Important - ISP Redundancy is not supported if Dynamic Routing is configured (Known

Limitation PMTR-68991).
Note - For information about ISP Redundancy on a Security Gateway, see the R81.10
Quantum Security Gateway Guide.
Introduction
ISP Redundancy connects Cluster Members to the Internet through redundant Internet Service Provider
(ISP) links.
ISP Redundancy monitors the ISP links and chooses the best current link.
Notes:
n ISP Redundancy requires a minimum of two external interfaces and supports up
to a maximum of ten.
To configure more than two ISP links, the Management Server and the Cluster
must run the version R81.10 and higher.
n ISP Redundancy is intended to traffic that originates on your internal networks
and goes to the Internet.
Important:
n You must connect each Cluster Member with a dedicated physical interface to
each of the ISPs.
n The IP addresses assigned to physical interfaces on each Cluster Member must
be on the same subnet as the Cluster Virtual IP address.

IP addresses in the table below are only examples.
Item Description
1 Internal network
2 Switches
3 Cluster Member A
3a Cluster interface connected to the internal network (IP address 10.10.10.0/24)

n Interface IP address 10.10.10.11
n Virtual IP address 10.10.10.1
3b Cluster interface (IP address 20.20.20.11) connected to the Sync network (IP address
20.20.20.0/24)
3c Cluster interface connected to a switch that connects to ISP A

3d Cluster interface connected to a switch that connects to ISP B

4 Cluster Member B
4a Cluster interface connected to the internal network (IP address 10.10.10.0/24)

4b Cluster interface (IP address 20.20.20.22) connected to the Sync network (IP address
20.20.20.0/24)
4c Cluster interface connected to a switch that connects to ISP B


Item Description
4d Cluster interface connected to a switch that connects to ISP A

5 ISP B
6 ISP A
7 Internet
ISP Redundancy Modes

ISP Redundancy configuration modes control the behavior of outgoing connections from internal clients to
the Internet:
Mode Description
Load Sharing Uses all links to distribute the load of connections.

The incoming connections are alternated.
You can configure best relative loads for the links (set a faster link to handle more
load).
New connections are randomly assigned to a link.
If one link fails, the other link takes the load.
In this mode, incoming connections can reach the application servers through any
of ISP links because the Cluster can answer DNS requests for the IP address of
internal servers with IP addresses from both ISPs by alternating their order.
Primary/Backup Uses one link for connections.

It switches to the Backup link, if the Primary link fails.
When the Primary link is restored, new connections are assigned to it.
Existing connections continue on the Backup link until they are complete.
In this mode, incoming connections (from the Internet to application servers in the
DMZ or internal networks) also benefit, because the Cluster returns packets using
the same ISP Link, through which the connection was initiated.
Best Practice:
n If all ISP links are basically the same, use the Load Sharing mode to ensure that
you are making the best use of all ISP links.
n You may prefer to use one of your ISP links that is more cost-effective in terms of
price and reliability.
In that case, use the Primary/Backup mode and set the more cost-effective ISP
as the Primary ISP link.

Outgoing Connections
Mode Description
Load Sharing Outgoing traffic that exits the Cluster on its way to the Internet is distributed
between the ISP Links.
You can set a relative weight for how much you want each of the ISP Links to be
used.
For example, if one link is faster, it can be configured to route more traffic across
that ISP link than the other links.
Primary/Backup Outgoing traffic uses an active primary link.

Hide NAT is used to change the source address of outgoing packets to the
address of the interface, through which the packet leaves the Cluster.
This allows return packets to be automatically routed through the same ISP link,
because their destination address is the address of the correct link.
Administrator configures the Hide NAT settings.

Incoming Connections
For external users to make incoming connections, the administrator must:
1. Give each application server one routable IP address for each ISP.
2. Configure Static NAT to translate the routable addresses to the real server address.
If the servers handle different services (for example, HTTP and FTP), you can use NAT to employ only
routable IP addresses for all the publicly available servers.
External clients use one of the assigned IP addresses. In order to connect, the clients must be able to
resolve the DNS name of the server to the correct IP address.
Example
Note - The example below is for two ISP links. The subnets 172.16.0.0/24 and
192.168.0.0/24 represent public routable IP addresses.
The Web server www.example.com is assigned an IP address from each ISP:

n 192.168.1.2 from ISP A
n 172.16.2.2 from ISP B
If the ISP Link A is down, then IP address 192.168.1.2 becomes unavailable, and the clients must be
able to resolve the URL www.example.com to the IP address 172.16.2.2.
An incoming connection is established, based on this example, in the following sequence:
1. When an external client on the Internet contacts www.example.com, the client sends a DNS
query for the IP address of this URL.
The DNS query reaches the Cluster.
The Cluster has a built-in mini-DNS server that can be configured to intercept DNS queries (of
Type A) for servers in its domain.
2. A DNS query arriving at an interface that belongs to one of the ISP links, is intercepted by the
Cluster.
3. If the Cluster recognizes the name of the host, it sends one of the following replies:
n In ISP Redundancy Primary/Backup mode, the Cluster replies only with the IP addresses
associated with the Primary ISP link, as long as the Primary ISP link is active.
n In ISP Redundancy Load Sharing mode, the Cluster replies with two IP addresses,
alternating their order.
4. If the Cluster is unable to handle DNS requests (for example, it may not recognize the host name),
it passes the DNS query to its original destination or the DNS server of the domain example.com.
5. When the external client receives the reply to its DNS query, it opens a connection. Once the
packets reach the Cluster, the Cluster uses Static NAT to translate the destination IP address
192.168.1.2 or 172.16.2.2 to the real server IP address 10.0.0.2.
6. The Cluster routes the reply packets from the server to the client through the same ISP link that
was used to initiate the connection.

Configuring ISP Redundancy on a Cluster

manages this Cluster.
3. Open the applicable Cluster object.
4. Click Other > ISP Redundancy.
5. Select Support ISP Redundancy.
6. Select the redundancy mode - Load Sharing or Primary/Backup.
7. Configure the ISP Links (at least two, at maximum ten).
To configure more than two ISP links, the Management Server and the Cluster must run the version
R81.10 and higher.
Procedure
Make sure you have the ISP data - the speed of the link and next hop IP address.
Automatic vs Manual configuration:
n If the Cluster object has at least two interfaces with the Topology "External" in the Network
Management page, you can configure the ISP links automatically.
Configuring ISP links automatically
a. Click Other > ISP Redundancy.

b. Click Set initial configuration.
The ISP Links are added automatically.
c. For Primary/Backup mode, make sure the Primary interface is first in the list. Use
the arrows on the right to change the order.
d. Click OK.

n If the Cluster object has only one interface with the Topology "External" in the Network
Management page, you must configure the ISP links manually.
Configuring ISP links manually

b. In the IPS Links section, click Add.
The ISP Link window opens.
c. Click the General tab.
d. In the Name field, enter a name of this link (desired text).
The name you enter here is used in the ISP Redundancy commands (see
"Controlling ISP Redundancy from CLI" on page 180).
e. Select the Interface of the Cluster for this ISP link.
l If the Cluster object has at least two interfaces with the Topology "External"
in the Network Management page, set each ISP link to a different interface.
If one of the ISP links is the connection to a backup ISP, configure the ISP
Redundancy Script (see "Controlling ISP Redundancy from CLI" on
page 180).
l If the Cluster object has only one interface with the Topology "External" in
the Network Management page, set each ISP link to connect to this
interface.
f. Configure the Next Hop IP Address.
l If the Cluster object has at least two interfaces with the Topology "External"
in the Network Management page, leave this field empty and click Get from
routing table. The next hop is the default gateway.
l If the Cluster object has only one interface with the Topology "External" in
the Network Management page, set each ISP link to a different next hop
router.
g. For ISP Redundancy in Load Sharing mode, enter the Weight value.
For equal traffic distribution between the IPS link, enter the applicable ratio in each
ISP link (100% / Number of ISP Links):
l For two ISP links, enter 50 in each
l For three ISP links, enter 33 in each
l For four ISP links, enter 25 in each
l and so on
If one ISP link is faster, increase this value and decrease it for the other ISP links,
so that the sum of these values is always equal 100.
h. Click the Advanced tab.
i. Define hosts to be monitored, to make sure the link is working.
Add the applicable objects to the Selected hosts section.
j. Click OK.

8. Configure the Cluster to be the DNS server.

Procedure
The Cluster, or a DNS server behind it, must respond to DNS queries.
It resolves IP addresses of servers in the DMZ (or another internal network).
Get a public IP address from each ISP. If public IP addresses are not available, register the
domain to make the DNS server accessible from the Internet.
The Cluster intercepts DNS queries "Type A" for the web servers in its domain that come from
external hosts.
n If the Cluster recognizes the external host, it replies:
l In ISP Redundancy Load Sharing mode, the Cluster replies with IP addresses of all
ISP links, alternating their order.
l In ISP Redundancy Primary/Backup mode, the Cluster replies with the IP addresses
of the active ISP link.
n If the Cluster does not recognize the host, it passes the DNS query on to the original
destination, or to the domain DNS server.
To enable the DNS server:

b. Select Enable DNS Proxy.
c. Click Configure.
d. Add your DMZ or Web servers.
Configure each server with a public IP address from each ISP.
e. In the DNS TTL, enter a number of seconds.
This sets a Time To Live for each DNS reply.
DNS servers in the Internet cannot cache your DNS data in the reply for longer than the
TTL.
f. Click OK.
g. Configure Static NAT to translate the public IP addresses to the real server's IP address.
External clients use one of the configured IP addresses.
Note - If the servers use different services (for example, HTTP and
FTP), you can use NAT for only the configured public IP addresses.

h. Define an Access Control Policy rule:
Services &
Destinatio Install
Name Source VPN Application Action Track
n On
s
DNS Proxy Applicable Applicable DNS Any domain_udp Accept None Policy
sources Servers Targets
To register the domain and get IP addresses:

a. Register your domain with each ISP.
b. Tell the ISP the configured IP addresses of the DNS server that respond to DNS queries for
the domain.
c. For each server in the DMZ, get a public IP address from each ISP.
d. In SmartConsole, click Menu > Global properties.
e. From the left tree, click NAT - Network Address Translation.
f. In the Manual NAT rules section, select Translate destination on client side.
g. Click OK.
9. Configure the Access Control Policy for ISP Redundancy.

Procedure
The Access Control Policy must allow connections through the ISP links, with Automatic Hide NAT
on network objects that start outgoing connections.
a. In the properties of the object for an internal network, select NAT > Add Automatic Address
Translation Rules.
b. Select Hide behind the gateway.
c. Click OK.

d. Define rules for publicly reachable servers (Web servers, DNS servers, DMZ servers).
n If you have one public IP address from each ISP for the Cluster, define Static NAT.
Allow specific services for specific servers.
For example, configure NAT rules, so that incoming HTTP connections from your
ISPs reach a Web server, and DNS connections from your ISPs reach the DNS
server.
Example: Manual Static Rules for a Web Server and a DNS Server
Origi Origin Translat Transla

Original Transla Inst
nal al ed ted Comm
Destinat ted all
Sourc Servic Destinat Service ent
ion Source On
e es ion s
Any Host object http = S 50.50.50.2 = Polic Incoming

with IP Original Original y Web - ISP
address of Targe A
Web Server ts
Any Host object http = S 60.60.60.2 = Polic Incoming

with IP Original Original y Web - ISP
address of Targe B
Web Server ts
Any Host object domain_ = S 50.50.50.3 = Polic Incoming

with IP udp Original Original y DNS - ISP
address of Targe A
DNS Server ts
Any Host object domain_ = S 60.60.60.3 = Polic Incoming

with IP udp Original Original y DNS - ISP
address of Targe B
DNS Server ts
n If you have a public IP address from each ISP for each publicly reachable server (in
addition to the Cluster), configure the applicable NAT rules:
i. Give each server a private IP address.
ii. Use the public IP addresses in the Original Destination.
iii. Use the private IP address in the Translated Destination.
iv. Select Any as the Original Service.
Note - If you use Manual NAT, then automatic ARP does not work for the IP
addresses behind NAT. You must configure the local.arp file as described
in sk30197.
10. Install the Access Control Policy on this Cluster object.

ISP Redundancy and VPN
ISP Redundancy and VPN
Note - ISP Redundancy settings override the VPN Link Selection settings.
When ISP Redundancy is enabled, VPN encrypted connections survive a failure of an ISP link.
The settings in the ISP Redundancy page override settings in the IPsec VPN > Link Selection page.
Configuring ISP Redundancy for VPN with a Check Point peer
Step Instructions
1 Connect with SmartConsole to the Security Management Server or Domain Management

Server that manages this Cluster.
3 Open the Cluster object.
4 In the left navigation tree, go to Other > ISP Redundancy.
5 Select Apply settings to VPN traffic.
6 In the left navigation tree, go to IPsec VPN > Link Selection.
7 Make sure that Use ongoing probing. Link redundancy mode shows the mode of the ISP
Redundancy:
High Availability (for Primary/Backup) or Load Sharing.
The VPN Link Selection now only probes the ISP configured in ISP Redundancy.
Configuring ISP Redundancy for VPN with a third-party peer
If the VPN peer is not a Check Point Security Gateway, the VPN may fail, or the third-party device may
continue to encrypt traffic to a failed ISP link.
n Make sure the third-party VPN peer recognizes encrypted traffic from the secondary ISP link as
coming from the Check Point cluster.
n Change the configuration of ISP Redundancy to not use these Check Point technologies:
l Use Probing - Makes sure that Link Selection uses another option.
l The options Load Sharing, Service Based Link Selection, and Route based probing work
only on Check Point Security Gateways/ Clusters / Security Groups.
If used, the Security Gateway / Cluster Members / Security Group uses one link to connect
to the third-party VPN peer.
The link with the highest prefix length and lowest metric is used.

Controlling ISP Redundancy from CLI
Controlling ISP Redundancy from CLI

You can control the ISP Redundancy behavior from CLI.
Force ISP Link State

Use the "fw isp_link" command to force the ISP link state to Up or Down.
Use this to test installation and deployment, or to force the Cluster Members to recognize the true link state if
it cannot (the ISP link is down but the gateway sees it as up).
n You can run this command on the Cluster Members:
fw isp_link <Name of ISP Link in SmartConsole> {up | down}
n You can run this command on the Security Management Server:
fw isp_link <Name of Cluster Member Object> <Name of ISP Link in

SmartConsole> {up | down}
For more information, see the R81.10 CLI Reference Guide > Chapter Security Gateway Commands -
Section fw - Section fw isp_link.
The ISP Redundancy Script

When the Cluster Member starts, or an ISP link state changes, the $FWDIR/bin/cpisp_update script
runs on the Cluster Member.
This script changes the default route of the Cluster Member.
For example, you can force the Cluster Member to change the state of a dialup interface to match that state
of its ISP link.
Edit this script to enable a dialup connection for one of the ISP links.
To configure a dialup connection:

1. In the script on the Cluster Member, enter the command to change the dialup interface state:
n If the ISP link goes down:
fw isp_link <Name of ISP Link in SmartConsole> down
n If the ISP link goes up:
fw isp_link <Name of ISP Link in SmartConsole> up
2. If you use PPPoE or PPTP xDSL modems, in the PPPoE or PPTP configuration, the Use Peer as
Default Gateway option must be cleared.

Dynamic Routing Protocols in a Cluster Deployment
Dynamic Routing Protocols in a Cluster

Deployment
ClusterXL supports Dynamic Routing (Unicast and Multicast) protocols as an integral part of Gaia Operating
System.
As the network infrastructure views the clustered Security Gateway as a single logical entity, failure of a
Cluster Member will be transparent to the network infrastructure and will not result in a ripple effect.
Router IP Address
All Cluster Members use the Cluster Virtual IP address(es) as Router IP address(es).
Routing Table Synchronization
Routing information is synchronized among the Cluster Members using the Forwarding Information Base
(FIB) Manager process.
This is done to prevent traffic interruption in case of failover, and used for Load Sharing and High Availability
modes.
The FIB Manager is the responsible for the routing information.
The FIB Manager is registered as a Critical Device called "FIB". If the slave goes out of sync, this Critical
Device reports its state as "problem". As a result, the slave member changes its state to "DOWN" until the
FIB Manager is synchronized.
Wait for Clustering
n When Dynamic Routing protocols and/or DHCP Relay are configured on cluster, the "Wait for
Clustering" option must be enabled in these cluster modes:
l ClusterXL High Availability
l ClusterXL Load Sharing Unicast
l ClusterXL Load Sharing Multicast
l VSX High Availability
l VSX Load Sharing (VSLS)
n When Dynamic Routing protocols and/or DHCP Relay are configured on cluster, the "Wait for
Clustering" must be disabled in these cluster modes:
l VRRP Cluster on Gaia OS

Dynamic Routing Protocols in a Cluster Deployment
Failure Recovery
Dynamic Routing on ClusterXL avoids creating a ripple effect upon failover by informing the neighboring
routers that the router has exited a maintenance mode.
The neighboring routers then reestablish their relationships to the cluster, without informing the other routers
in the network.
These restart protocols are widely adopted by all major networking vendors.
This table lists the RFC and drafts compliant with Check Point Dynamic Routing:
Protocol RFC or Draft
OSPF Graceful restart RFC 3623
OSPF LLS RFC 5613
BGP Graceful restart RFC 4724

ClusterXL Configuration Commands

Description
These commands let you configure internal behavior of the Clustering Mechanism.
Important:
n We do not recommend that you run these commands. These commands must be
run automatically only by the Security Gateway or the Check Point Support.
n In a Cluster, you must configure all the Cluster Members in the same way.
Syntax
Notes:
n In Gaia Clish:
Enter the set cluster<ESC><ESC> to see all the available commands.
n In Expert mode:
Run the cphaconf command see all the available commands.
You can run the cphaconf commands only from the Expert mode.
n Syntax legend:
1. Curly brackets or braces { }:
Enclose a list of available commands or parameters, separated by the
vertical bar |, from which user can enter only one.
2. Angle brackets < >:
Enclose a variable - a supported value user needs to specify explicitly.
3. Square brackets or brackets [ ]:
Enclose an optional command or parameter, which user can also enter.
n You can include these commands in scripts to run them automatically.
The meaning of each command is explained in the next sections.
Table: ClusterXL Configuration Commands
Description Command in Command in
of Command Gaia Clish Expert Mode
Configure how to show the Cluster Member in set cluster cphaconf mem_id_mode {id
local ClusterXL logs - by its Member ID or its member | name}
Member Name (see "Configuring the Cluster idmode {id |
Member ID Mode in Local Logs" on page 186) name}
Register a single Critical Device (Pnote) on the N / A cphaconf set_pnote -d

Cluster Member (see "Registering a Critical <Name of Device> -t
Device" on page 187) <Timeout in Sec> -s
{ok|init|problem} [-p]
[-g] register
Unregister a single Critical Device (Pnote) on N / A cphaconf set_pnote -d

the Cluster Member (see "Unregistering a <Name of Device> [-p] [-
Critical Device" on page 189) g] unregister

Table: ClusterXL Configuration Commands (continued)

Report (change) a state in a single Critical N / A cphaconf set_pnote -d

Device (Pnote) on the Cluster Member (see <Name of Device> -s
"Reporting the State of a Critical Device" on {ok|init|problem} [-g]
page 190) report
Register several Critical Devices (Pnotes) from N / A cphaconf set_pnote -f

a file on the Cluster Member (see "Registering <Name of File> [-g]
Critical Devices Listed in a File" on page 191) register
Unregister all Critical Devices (Pnotes) on the N / A cphaconf set_pnote -a [-

Cluster Member (see "Unregistering All Critical g] unregister
Devices" on page 193)
Configure the Cluster Control Protocol (CCP) set cluster cphaconf ccp_encrypt
Encryption on the Cluster Member (see member {off | on}
"Configuring the Cluster Control Protocol (CCP) ccpenc {off cphaconf ccp_encrypt_key
Settings" on page 194) | on} <Key String>
Configure the Cluster Forwarding Layer on the set cluster cphaconf forward {off |
Cluster Member (controls the forwarding of member on}
traffic between Cluster Members) forwarding
Note - For Check Point use only. {off | on}
Print the current cluster configuration as loaded N / A cphaconf debug_data

in the kernel on the Cluster Member (for details,
see sk93306)
Start internal failover between slave interfaces N / A cphaconf failover_bond

of specified bond interface - only in Bond High <bond_name>
Availability mode (for details, see sk93306)
Configure what happens during a failover after a N / A cphaconf enable_bond_

Bond already failed over internally (for details, failover <bond_name>
see sk93306)
Initiate manual cluster failover (see "Initiating set cluster clusterXL_admin {down |
Manual Cluster Failover" on page 195) member admin up}
{down | up}
Configure the minimal number of required N / A cphaconf bond_ls {set

slaves interfaces for Bond Load Sharing (see <Bond Name> <Value> |
"Configuring the Minimal Number of Required remove <Bond Name>}
Slave Interfaces for Bond Load Sharing" on
page 199)
Configuring Link Monitoring on the Cluster N / A N / A

Interfaces (see "Configuring Link Monitoring on
the Cluster Interfaces" on page 112)


Configuring the Multi-Version Cluster N / A cphaconf mvc {off | on}

Mechanism (see "Configuring the Multi-Version
Cluster Mechanism" on page 202)
List of the Gaia Clish set cluster member commands
set cluster member admin {down | up} [permanent]

set cluster member ccpenc {off | on}
set cluster member forwarding {off | on}
set cluster member idmode {id | name}
set cluster member mvc {off | on}
List of the cphaconf commands
Note - Some commands are not applicable to 3rd party clusters.
cphaconf [-D] <options> start

cphaconf stop
cphaconf [-t <Sync IF 1>...] [-d <Non-Monitored IF 1>...] add
cphaconf clear-secured
cphaconf clear-non-monitored
cphaconf debug_data
cphaconf delete_link_local [-vs <VSID>] <IF name>
cphaconf set_link_local [-vs <VSID>] <IF name> <Cluster IP>
cphaconf mem_id_mode {id | name}
cphaconf failover_bond <bond_name>
cphaconf [-s] {set | unset | get} var <Kernel Parameter Name> [<Value>]
cphaconf bond_ls {set <Bond Name> <Value> | remove <Bond Name>}
cphaconf set_pnote -d <Device> -t <Timeout in sec> -s {ok | init | problem}
[-p] [-g] register
cphaconf set_pnote -f <File> [-g] register
cphaconf set_pnote -d <Device> [-p] [-g] unregister
cphaconf set_pnote -a [-g] unregister
cphaconf set_pnote -d <Device> -s {ok | init | problem} [-g] report
cphaconf ccp_encrypt {off | on}

Configuring the Cluster Member ID Mode in Local Logs
Configuring the Cluster Member ID Mode in

Local Logs
Description
This command configures how to show the Cluster Member in the local ClusterXL logs - by its Member ID
(default), or its Member Name.
This configuration applies to these local logs:
n /var/log/messages
n dmesg
n $FWDIR/log/fwd.elg
See "Viewing the Cluster Member ID Mode in Local Logs" on page 241.
Syntax
Shell Command
Gaia Clish set cluster member idmode {id | name}
Expert mode cphaconf mem_id_mode {id | name}
Example
[Expert@Member1:0]# cphaprob names
Current member print mode in local logs is set to: ID
[Expert@Member1:0]#
[Expert@Member1:0]# cphaconf mem_id_mode name
Member print mode in local logs: NAME
[Expert@Member1:0]#
Current member print mode in local logs is set to: NAME
[Expert@Member1:0]#

Registering a Critical Device
Description
You can add a user-defined critical device to the default list of critical devices. Use this command to register
<device> as a critical process, and add it to the list of devices that must run for the Cluster Member to be
considered active. If <device> fails, then the Cluster Member is seen as failed.
If a Critical Device fails to report its state to the Cluster Member in the defined timeout, the Critical Device,
and by design the Cluster Member, are seen as failed.
Define the status of the Critical Device that is reported to ClusterXL upon registration.
This initial status can be one of these:
n ok - Critical Device is alive.
n init - Critical Device is initializing. The Cluster Member is Down. In this state, the Cluster Member
cannot become Active.
n problem - Critical Device failed. If this state is reported to ClusterXL, the Cluster Member immediately
goes Down. This causes a failover.
Syntax
Shell Command
Gaia N/A
Clish
Expert cphaconf set_pnote -d <Name of Critical Device> -t <Timeout in Sec>

mode -s {ok | init | problem} [-p] [-g] register
Notes:
n The "-t" flags specifies how frequently to expect the periodic reports from this Critical
Device.
If no periodic reports should be expected, then enter the value 0 (zero).
n The "-p" flag makes these changes permanent (survive reboot).
n The "-g" flag applies the command to all configured Virtual Systems.
Restrictions
n Total number of critical devices (pnotes) on Cluster Member is limited to 16.
n Name of any critical device (pnote) on Cluster Member is limited to 15 characters, and must not
include white spaces.

Related topics
n "Viewing Critical Devices" on page 213
n "Reporting the State of a Critical Device" on page 190
n "Registering Critical Devices Listed in a File" on page 191
n "Unregistering a Critical Device" on page 189
n "Unregistering All Critical Devices" on page 193

Unregistering a Critical Device
Unregistering a Critical Device
Description
This command unregisters a user-defined Critical Device (Pnote). This means that this device is no longer
considered critical.
If a Critical Device was registered with a state "problem", before you ran this command, then after you run
this command, the status of the Cluster Member depends only on the states of the remaining Critical
Devices.
Syntax
Shell Command
Gaia Clish N/A
Expert cphaconf set_pnote -d <Name of Critical Device> [-p] [-g]

mode unregister
Notes:
n The "-p" flag makes these changes permanent.
This means that after you reboot, these Critical Devices remain
unregistered.
Related topics
n "Registering a Critical Device" on page 187

Reporting the State of a Critical Device
Reporting the State of a Critical Device
Description
This command manually reports (changes) the state of a Critical Device to ClusterXL.
The reported state can be one of these:
n init - Critical Device is initializing. The Cluster Member is Down. In this state, the Cluster Member
cannot become Active.
n problem - Critical Device failed. If this state is reported to ClusterXL, the Cluster Member immediately
goes Down. This causes a failover.
If a Critical Device fails to report its state to the Cluster Member within the defined timeout, the Critical
Device, and by design the Cluster Member, are seen as failed. This is true only for Critical Devices with
timeouts. If a Critical Device is registered with the "-t 0" parameter, there is no timeout. Until the Critical
Device reports otherwise, the state of the Critical Device is considered to be the last reported state.
Syntax
Shell Command
Gaia N/A
Clish
Expert cphaconf set_pnote -d <Name of Critical Device> -s {ok | init |

mode problem} [-g] report
Notes:
n If the "<Name of Critical Device>" reports its state as "problem", then the
Cluster Member reports its state as failed.
Related topics

Registering Critical Devices Listed in a File
Description
This command registers all the user-defined Critical Devices listed in the specified file.
This file must be a plain-text ASCII file, with each Critical Device defined on a separate line.
Each definition must contain three parameters, which must be separated by a space or a tab character:
<Name of Device> <Timeout> <Status>
Where:
Parameter Description
<Name of The name of the Critical Device.

Device>
n Maximal name length is 15 characters
n The name must not include white spaces (space or tab characters).
<Timeout> If the Critical Device <Name of Device> fails to report its state to the Cluster Member
within this specified number of seconds, the Critical Device (and by design the Cluster
Member), are seen as failed.
For no timeout, use the value 0 (zero).
<Status> The Critical Device <Name of Device> reports one of these statuses to the Cluster
Member:
n init- Critical Device is initializing. The Cluster Member is Down. In this state,
the Cluster Member cannot become Active.
n problem - Critical Device failed. If this state is reported to ClusterXL, the Cluster
Member immediately goes Down. This causes a failover.
Syntax
Shell Command
Gaia Clish N/A
Expert mode cphaconf set_pnote -f /<Path>/<Name of File> [-g] register
Note - The "-g" flag applies the command to all configured Virtual Systems.

Related topics

Unregistering All Critical Devices
Unregistering All Critical Devices
Description
This command unregisters all critical devices from the Cluster Member.
Syntax
Shell Command
Gaia Clish N/A
Expert mode cphaconf set_pnote -a [-g] unregister
Notes:
n The "-a" flag specifies that all Pnotes must be unregistered
n The "-g" flag applies the command to all configured Virtual
Systems
Related topics

Configuring the Cluster Control Protocol (CCP) Settings
Configuring the Cluster Control Protocol (CCP)

Settings
Description
Cluster Members configure the Cluster Control Protocol (CCP) mode automatically.
You can configure the Cluster Control Protocol (CCP) Encryption on the Cluster Members.
See "Viewing the Cluster Control Protocol (CCP) Settings" on page 246.
Syntax for configuring the Cluster Control Protocol (CCP) Encryption
Shell Command
Gaia Clish set cluster member ccpenc {off | on}
Expert mode cphaconf ccp_encrypt {off | on}


Initiating Manual Cluster Failover

Description
This command initiates a manual cluster failover (see sk55081 and "How to Initiate Cluster Failover" on
page 256).
Syntax
Shell Command
Gaia Clish set cluster member admin {down | up}
Expert mode clusterXL_admin {down | up}

Example

[Expert@Member1:0]# cphaprob state
Cluster Mode: High Availability (Active Up) with IGMP Membership
1 (local) 11.22.33.245 100% ACTIVE Member1

2 11.22.33.246 0% STANDBY Member2
Active PNOTEs: None
... ...
[Expert@Member1:0]#
[Expert@Member1:0]# clusterXL_admin down

This command does not survive reboot. To make the change permanent, please run 'set cluster member admin
down/up permanent' in clish or add '-p' at the end of the command in expert mode
Setting member to administratively down state ...
Member current state is DOWN
[Expert@Member1:0]#
1 (local) 11.22.33.245 0% DOWN Member1

2 11.22.33.246 100% ACTIVE Member2
Active PNOTEs: ADMIN
Last member state change event:

Event Code: CLUS-111400
State change: ACTIVE -> DOWN
Reason for state change: ADMIN_DOWN PNOTE
Event time: Sun Sep 8 19:35:06 2019
Last cluster failover event:

Transition to new ACTIVE: Member 1 -> Member 2
Reason: ADMIN_DOWN PNOTE
Cluster failover count:

Failover counter: 2
Time of counter reset: Sun Sep 8 16:08:34 2019 (reboot)
[Expert@Member1:0]#
[Expert@Member1:0]# clusterXL_admin up
This command does not survive reboot. To make the change permanent, please run 'set cluster member admin
down/up permanent' in clish or add '-p' at the end of the command in expert mode
Setting member to normal operation ...
Member current state is STANDBY
[Expert@Member1:0]#
1 (local) 11.22.33.245 0% STANDBY Member1

2 11.22.33.246 100% ACTIVE Member2
Active PNOTEs: None

State change: DOWN -> STANDBY
Reason for state change: There is already an ACTIVE member in the cluster (member 2)



Failover counter: 2
[Expert@Member1:0]#

Configuring the Minimal Number of Required Slave Interfaces for Bond Load Sharing
Configuring the Minimal Number of Required

Slave Interfaces for Bond Load Sharing
Description
ClusterXL considers a bond in Load Sharing mode to be in the "down" state when fewer than a minimal
number of required slave interfaces stay in the "up" state.
By default, the minimal number of required slave interfaces, which must stay in the "up" state in a bond of n
slave interfaces is n-1.
If one more slave interface fails (when n-2 slave interfaces stay in the "up" state), ClusterXL considers the
bond interface to be in the "down" state, even if the bond contains more than two slave interfaces.
If a smaller number of slave interfaces can pass the expected traffic, you can configure explicitly the minimal
number of required slave interfaces.
Divide your maximal expected traffic speed by the speed of your slave interfaces and round up the result to
find an applicable minimal number of required slave interfaces.
Notes:
n Cluster Members save the configuration in the $FWDIR/conf/cpha_bond_ls_
config.conf file.
n The commands below save the changes in this file.
n Each line in the file has this syntax:
<Name of Bond Interface> <Minimal Number of Required
Slave Interfaces>
In addition, see "Viewing Bond Interfaces" on page 225.
Syntax to add the minimal number of required slave interfaces for a specific Bond interface
Shell Command
Gaia N/A
Clish
Expert cphaconf bond_ls set <Name of Bond Interface> <Minimal Number of

mode Required Slave Interfaces>

Syntax to remove the configured minimal number of required slave interfaces for a specific Bond
interface
Shell Command
Gaia Clish N/A
Expert mode cphaconf bond_ls remove <Name of Bond Interface>
Syntax to see the current configuration of the minimal number of required slave interfaces
Shell Command
Gaia Clish N/A
Expert mode cat $FWDIR/conf/cpha_bond_ls_config.conf
Procedure
Step Instructions
3 Add or remove the minimal number of required slave interfaces for a specific Bond interface:
cphaconf bond_ls set <Bond> <Minimal Number of Slaves>
cphaconf bond_ls remove <Bond>
4 Examine the configuration:

cat $FWDIR/conf/cpha_bond_ls_config.conf
5 In SmartConsole, install the Access Control policy on this cluster object.

Example
[Expert@Member1:0]# cat $FWDIR/conf/cpha_bond_ls_config.conf

# ... (truncated for brevity) ...
# Example:
# bond0 2
[Expert@Member1:0]#
[Expert@Member1:0]# cphaconf bond_ls set bond1 2

Set operation succeeded

# Example:
# bond0 2
bond1 2
[Expert@Member1:0]#
[Expert@Member1:0]# cphaconf bond_ls remove bond1

Remove operation succeeded
[Expert@Member1:0]#

# Example:
# bond0 2
[Expert@Member1:0]#

Configuring the Multi-Version Cluster Mechanism
Configuring the Multi-Version Cluster

Mechanism
Description
This command changes the state of the Multi-Version Cluster (MVC) Mechanism - enable or disable it.
Important:
n The MVC Mechanism is disabled by default.
n For limitations of the MVC Mechanism, see the R81.10 Installation and Upgrade
Guide > Chapter Upgrading Gateways and Clusters > Section Upgrading
ClusterXL, VSX Cluster, VRRP Cluster > Section Multi-Version Cluster Upgrade.
Syntax
Shell Command
Gaia Clish set cluster member mvc {off | on}
Expert mode cphaconf mvc {off | on}
Parameters
off Disables the MVC Mechanism on this Cluster Member.
on Enables the MVC Mechanism on this Cluster Member.
Notes:
n This command does not provide an output. To view the current state of the MVC
Mechanism, see "Viewing the State of the Multi-Version Cluster Mechanism" on
page 248.
n The change made with this command survives reboot.
n If a specific scenario requires you to disable the MVC Mechanism before the first
start of an R81.10 Cluster Member (for example, immediately after an upgrade to
R81.10), then disable it before the first policy installation on this Cluster Member.

Monitoring and Troubleshooting Clusters
Monitoring and Troubleshooting

Clusters
This section describes how to monitor and troubleshooting clusters.

ClusterXL Monitoring Commands

Description
Use the monitoring commands to make sure that the cluster and the Cluster Members work properly, and to
define Critical Devices. A Critical Device (also known as a Problem Notification, or pnote) is a special
software device on each Cluster Member, through which the critical aspects for cluster operation are
monitored. When the critical monitored component on a Cluster Member fails to report its state on time, or
when its state is reported as problematic, the state of that member is immediately changed to 'Down'.
Syntax
Notes:
n In Gaia Clish:
Enter the show cluster<ESC><ESC> to see all the available commands.
n In Expert mode:
Run the cphaprob command see all the available commands.
You can run the cphaprob commands from Gaia Clish as well.
n Syntax legend:
Table: ClusterXL Monitoring Commands

Show states of Cluster Members and their names (see show cluster cphaprob [-vs
"Viewing Cluster State" on page 209) state <VSID>] state
Show Critical Devices (Pnotes) and their states on the show cluster cphaprob [-l]
Cluster Member (see "Viewing Critical Devices" on members pnotes [-ia] [-e]
page 213) {all | problem} list
Show cluster interfaces on the cluster member (see show cluster cphaprob [-vs
"Viewing Cluster Interfaces" on page 221) members all] [-a] [-
interfaces {all m] if
| secured |
virtual | vlans}
Show cluster bond configuration on the Cluster Member show cluster cphaprob
(see "Viewing Bond Interfaces" on page 225) bond {all | name show_bond
<bond_name>} [<bond_name>]

Table: ClusterXL Monitoring Commands (continued)

Show groups of bonds on the Cluster Member (see N / A cphaprob

"Viewing Bond Interfaces" on page 225) show_bond_
groups
Show (and reset) cluster failover statistics on the Cluster show cluster cphaprob [-
Member (see "Viewing Cluster Failover Statistics" on failover [reset reset {-c | -
page 229) {count | h}] [-l
history}] <count>]
show_failover
Show information about the software version (including show cluster cphaprob
hotfixes) on the local Cluster Member and its release release
matches/mismatches with other Cluster Members (see
"Viewing Software Versions on Cluster Members" on
page 231)
Show Delta Sync statistics on the Cluster Member (see show cluster cphaprob [-
"Viewing Delta Synchronization" on page 232) statistics sync reset]
[reset] syncstat
Show Delta Sync statistics for the Connections table on show cluster cphaprob [-
the Cluster Member (see "Viewing Cluster Delta Sync statistics reset] ldstat
Statistics for Connections Table" on page 239) transport
[reset]
Show the Cluster Control Protocol (CCP) mode on the show cluster cphaprob [-vs
Cluster Member (see "Viewing Cluster Interfaces" on members all] -a if
page 221) interfaces
virtual
Show the IGMP membership of the Cluster Member (see show cluster cphaprob igmp
"Viewing IGMP Status" on page 238) members igmp
Show cluster unique IP's table on the Cluster Member show cluster cphaprob
(see "Viewing Cluster IP Addresses" on page 240) members ips tablestat
show cluster cphaprob -m
members tablestat
monitored
Show the Cluster Member ID Mode in local logs - by show cluster cphaprob
Member ID (default) or Member Name (see "Viewing the members idmode names
Cluster Member ID Mode in Local Logs" on page 241)
Show interfaces, which the RouteD monitors on the show ospf cphaprob
Cluster Member when you configure OSPF (see "Viewing interfaces routedifcs
Interfaces Monitored by RouteD" on page 242) [detailed]
Show roles of RouteD daemon on Cluster Members (see show cluster cphaprob
"Viewing Roles of RouteD Daemon on Cluster Members" roles roles
on page 243)


Show Cluster Correction Statistics (see "Viewing Cluster N / A cphaprob [{-d

Correction Statistics" on page 244) | -f | -s}]
corr
Show the Cluster Control Protocol (CCP) mode (see show cluster cphaprob -a
"Viewing the Cluster Control Protocol (CCP) Settings" on members if
virtual
Show the Cluster Control Protocol (CCP) Encryption show cluster cphaprob ccp_
settings (see "Viewing the Cluster Control Protocol (CCP) members ccpenc encrypt
Settings" on page 246)
Shows the state of the Multi-Version Cluster (see show cluster N / A

"Viewing the State of the Multi-Version Cluster members mvc
Mechanism" on page 248)
Shows the latency and the drop rate of each interface N / A N / A

(see "Viewing Latency and Drop Rate of Interfaces" on
page 247)
Show Full Connectivity Upgrade statistics (see "Viewing N / A cphaprob

Full Connectivity Upgrade Statistics" on page 249) fcustat

List of the Gaia Clish show cluster commands
show cluster
bond
all
name <Name of Bond>
failover
members
ccpenc
idmode
igmp
interfaces
all
secured
virtual
vlans
ips
monitored
mvc
pnotes
all
problem
release
roles
state
statistics
sync [reset]
transport [reset]

List of the cphaprob commands
cphaprob [-vs <VSID>] state

cphaprob [-reset {-c | -h}] [-l <count>] show_failover
cphaprob names
cphaprob [-reset] [-a] syncstat
cphaprob [-reset] ldstat
cphaprob [-l] [-i[a]] [-e] list
cphaprob [-vs all] [-a] [-m] if
cphaprob latency
cphaprob show_bond [<bond_name>]
cphaprob show_bond_groups
cphaprob igmp
cphaprob fcustat
cphaprob [-m] tablestat
cphaprob routedifcs
cphaprob roles
cphaprob release
cphaprob ccp_encrypt
cphaprob [{-d | -f | -s}] corr

Viewing Cluster State

Description
This command monitors the cluster status (after you set up the cluster).
Syntax
Shell Command
Gaia Clish 1. set virtual-system

<VSID>
2. show cluster state
Example
Member1> show cluster state
1 (local) 11.22.33.245 100% ACTIVE(!) Member1

2 11.22.33.246 0% DOWN Member2
Active PNOTEs: COREXL

State change: INIT -> ACTIVE(!)
Reason for state change: All other machines are dead (timeout), FULLSYNC PNOTE
v Cluster failover count:
Failover counter: 0
Member1>
Description of the "cphaprob state" command output fields:

Table: Description of the output fields
Field Description
Cluster Mode Can be one of these:

n Load Sharing (Multicast).
n Load Sharing (Unicast).
n High Availability (Primary Up).
n High Availability (Active Up).
n Virtual System Load Sharing
n For third-party clustering products: Service, refer to Clustering Definitions and
Terms, for more information.

Table: Description of the output fields (continued)

Field Description
ID n In the High Availability mode - indicates the Cluster Member priority, as

configured in the cluster object in SmartConsole.
n In Load Sharing mode - indicates the Cluster Member ID, as configured in the
cluster object in SmartConsole.
Unique Usually, shows the IP addresses of the Sync interfaces.

Address In some cases, can show IP addresses of other cluster interfaces.
Assigned n In the ClusterXL High Availability mode - shows the Active Cluster Member with
Load 100% load, and all other Standby Cluster Members with 0% load.
n In ClusterXL Load Sharing modes (Unicast and Multicast) - shows all Active
Cluster Members with 100% load.
State n In the ClusterXL High Availability mode, only one Cluster Member in a fully-
functioning cluster must be ACTIVE, and the other Cluster Members must be in
the STANDBY state.
n In the ClusterXL Load Sharing modes (Unicast and Multicast), all Cluster
Members in a fully-functioning cluster must be ACTIVE.
n In 3rd-party clustering configuration, all Cluster Members in a fully-functioning
cluster must be ACTIVE. This is because this command only reports the status of
the Full Synchronization process.
See the summary table below.
Name Shows the names of Cluster Members' objects as configured in SmartConsole.
Active Shows the Critical Devices that report theirs states as "problem" (see "Viewing Critical
PNOTEs Devices" on page 213).
Last member Shows information about the last time this Cluster Member changed its cluster state.
state change
event
Event Code Shows an event code.

For information, see sk125152.
State change Shows the previous cluster state and the new cluster state of this Cluster Member.
Reason for Shows the reason why this Cluster Member changed its cluster state.
state change
Event time Shows the date and the time when this Cluster Member changed its cluster state.
Last cluster Shows information about the last time a cluster failover occurred.
failover event
Transition to Shows which Cluster Member became the new Active.

new ACTIVE
Reason Shows the reason for the last cluster failover.
Event time Shows the date and the time of the last cluster failover.


Field Description
Cluster Shows information about the cluster failovers.

failover count
Failover Shows the number of cluster failovers since the boot.

counter Notes:
n This value survives reboot.
n This counter is synchronized between Cluster Members.
Time of Shows the date and the time of the last counter reset, and the reset initiator.
counter reset
When you examine the state of the Cluster Member, consider whether it forwards packets, and whether it
has a problem that prevents it from forwarding packets. Each state reflects the result of a test on critical
devices. This table shows the possible cluster states, and whether or not they represent a problem.
Table: Description of the cluster states
Is this
Cluster Forwarding
Description state a
State packets?
problem?
ACTIVE Everything is OK. Yes No
ACTIVE(!) A problem was detected, but the Cluster Member still Yes Yes
ACTIVE(!F) forwards packets, because it is the only member in
ACTIVE(!P) the cluster, or because there are no other Active
ACTIVE(!FP) members in the cluster. In any other situation, the
state of the member is Down.
n ACTIVE(!) - See above.
n ACTIVE(!F) - See above. Cluster Member is
in the freeze state.
n ACTIVE(!P) - See above. This is the Pivot
Cluster Member in Load Sharing Unicast
mode.
n ACTIVE(!FP) - See above. This is the Pivot
Cluster Member in Load Sharing Unicast mode
and it is in the freeze state.
DOWN One of the Critical Devices reports its state as No Yes

"problem" (see "Viewing Critical Devices" on
page 213).
LOST The peer Cluster Member lost connectivity to this No Yes

local Cluster Member (for example, while the peer
Cluster Member is rebooted).

Table: Description of the cluster states (continued)

Is this
Cluster Forwarding
Description state a
State packets?
problem?
READY State Ready means that the Cluster Member No No

recognizes itself as a part of the cluster and is literally
ready to go into action, but, by design, something
prevents it from taking action. Possible reasons that
the Cluster Member is not yet Active include:
n Not all required software components were
loaded and initialized yet and/or not all
configuration steps finished successfully yet.
Before a Cluster Member becomes Active, it
sends a message to the rest of the Cluster
Members, to check if it can become Active. In
High Availability mode it checks if there is
already an Active member and in Load Sharing
Unicast mode it checks if there is a Pivot
member already. The member remains in the
Ready state until it receives the response from
the rest of the Cluster Members and decides
which, which state to choose next (Active,
Standby, Pivot, or non-Pivot).
n Software installed on this Cluster Member has
a higher version than all the other Cluster
Members. For example, when a cluster is
upgraded from one version of Check Point
Security Gateway to another, and the Cluster
Members have different versions of Check
Point Security Gateway, the Cluster Members
with the new version have the Ready state, and
the Cluster Members with the previous version
have the Active/Active Attention state.
This applies only when the Multi-Version
Cluster Mechanism is disabled (see "Viewing
the State of the Multi-Version Cluster
Mechanism" on page 248).
See sk42096 for a solution.
STANDBY Applies only to a High Availability mode. Means that No No

the Cluster Member waits for an Active Cluster
Member to fail in order to start packet forwarding.
BACKUP Applies only to a VSX Cluster in Virtual System Load No No

Sharing mode with three or more Cluster Members
configured.
State of a Virtual System on a third (and so on) VSX
Cluster Member.
INIT The Cluster Member is in the phase after the boot and No No
until the Full Sync completes.

Viewing Critical Devices

Description
There are a number of built-in Critical Devices, and the Administrator can define additional Critical Devices.
When a Critical Device reports its state as a "problem", the Cluster Member reports its state as "DOWN".
To see the list of Critical Devices on a Cluster Member, and of all the other Cluster Members, run the
commands listed below on the Cluster Member.
Table: Built-in Critical Devices
Meaning of
Meaning of the "problem"
Critical Device Description the "OK"
state
state
Problem Monitors all the Critical Devices. None of the At least one of the Critical
Notification Critical Devices on this Cluster
Devices on Member reports its state as
this Cluster problem.
Member
report its state
as problem.
Init Monitors if "HA module" was This Cluster

initialized successfully. See Member
sk36372. receives
cluster state
information
from peer
Cluster
Members.
Interface Monitors the state of cluster All cluster At least one of the cluster
Active Check interfaces. interfaces on interfaces on this Cluster
this Cluster Member is down (CCP
Member are packets are not sent and/or
up (CCP received on time).
packets are
sent and
received on
all cluster
interfaces).
Load Pnote is currently not used (see

Balancing sk36373).
Configuration
Recovery Monitors the state of a Virtual State of a State of a Virtual System

Delay System (see sk92353). Virtual cannot be changed yet on
System can this Cluster Member.
be changed
on this Cluster
Member.

Table: Built-in Critical Devices (continued)

Meaning of
state
state
CoreXL Monitors CoreXL configuration Number of Number of configured

Configuration for inconsistencies on all Cluster configured CoreXL Firewall instances on
Members. CoreXL this Cluster Member is
Firewall different from peer Cluster
instances on Members.
this Cluster Important - A Cluster
Member is the Member with a greater
same as on all number of CoreXL Firewall
peer Cluster instances changes its state
Members. to DOWN.
Fullsync Monitors if Full Sync on this This Cluster This Cluster Member was not
Cluster Member completed Member able to complete Full Sync.
successfully. completed
Full Sync
successfully.
Policy Monitors if the Security Policy is This Cluster Security Policy is not
installed. Member currently installed on this
successfully Cluster Member.
installed
Security
Policy.
fwd Monitors the Security Gateway fwd daemon fwd daemon on this Cluster
process called fwd. on this Cluster Member did not report its
Member state on time.
reported its
state on time.
cphad Monitors the ClusterXL process cphamcset cphamcset daemon on this

called cphamcset. daemon on Cluster Member did not
also see the this Cluster report its state on time.
$FWDIR/log/cphamcset.el Member
g file. reported its
state on time.
routed Monitors the Gaia process routed routed daemon on this

called routed. daemon on Cluster Member did not
this Cluster report its state on time.
Member
reported its
state on time.


Meaning of
state
state
cvpnd Monitors the Mobile Access cvpnd cvpnd daemon on this

back-end process called cvpnd. daemon on Cluster Member did not
This pnote appears if Mobile this Cluster report its state on time.
Access Software Blade is Member
enabled. reported its
state on time.
ted Monitors the Threat Emulation ted daemon ted daemon on this Cluster
process called ted. on this Cluster Member did not report its
Member state on time.
reported its
state on time.
VSX Monitors all Virtual Systems in On VS0, Minimum of blocking states

VSX Cluster. means that of all Virtual Systems is not
states of all "active" (the VSIDs will be
Virtual printed on the line
Systems are Problematic VSIDs:) on
not Down on this Cluster Member.
this Cluster
Member.
On other
Virtual
Systems,
means that
VS0 is alive
on this Cluster
Member.
Instances This pnote appears in VSX HA The number There is a mismatch between
mode (not VSLS) cluster. of CoreXL the number of CoreXL
Firewall Firewall instances in the
instances in received CCP packet and the
the received number of loaded CoreXL
CCP packet Firewall instances on this
matches the VSX Cluster Member or this
number of Virtual System (see
loaded sk106912).
CoreXL
Firewall
instances on
this VSX
Cluster
Member or
this Virtual
System.


Meaning of
state
state
Hibernating This pnote appears in VSX This Virtual

VSLS mode cluster with 3 and System is in
more Cluster Members. This "Backup"
pnote shows if this Virtual (hibernated)
System is in "Backup" state on this
(hibernated) state. Also see Cluster
sk114557. Member.
admin_down Monitors the Critical Device User ran the clusterXL_

admin_down. admin down command on
this Cluster Member.
See "The clusterXL_admin
Script" on page 280.
host_monitor Monitors the Critical Device All monitored At least one of the monitored
host_monitor. IP addresses IP addresses on this Cluster
User executed the on this Cluster Member did not reply to at
$FWDIR/bin/clusterXL_ Member least one ping.
monitor_ips script. replied to
See "The clusterXL_monitor_ips pings.
Script" on page 284.
A name of a user User executed the All monitored At least one of the monitored
space process $FWDIR/bin/clusterXL_ user space user space on this Cluster
(except fwd, monitor_process script. processes on Member processes is not
routed, cvpnd, See "The clusterXL_monitor_ this Cluster running.
ted) process Script" on page 288. Member are
running.


Meaning of
state
state
Local Probing Monitors the probing CCP packets At least one of the cluster
mechanism on the cluster are received interfaces on this Cluster
interfaces (see the term Probing on all cluster Member does not receive
in the "Glossary" on page 12). interfaces. CCP packets for 5 seconds.
The probing started for the
network connected to the
affected interface.
Important:
n The state of
this Critical
Device does
not affect
the cluster
state of a
Cluster
Member.
This Critical
Device is
only an
indicator for
the probing
mechanism
(instead of
running a
cluster
debug).
n If there is a
real issue
with a
cluster
interface,
the Critical
Device
"
Interface
Active
Check"
reports its
state as
"problem".
Syntax
Shell Command
Gaia Clish show cluster members pnotes {all | problem}
Expert mode cphaprob [-l] [-ia] [-e] list

Where:
Command Description
show cluster Shows cluster full list of Critical Devices

members pnotes
all
show cluster Prints the list of all the "Built-in Devices" and the "Registered
members pnotes Devices"
problem
cphaprob -l Prints the list of all the "Built-in Devices" and the "Registered
Devices"
cphaprob -i list When there are no issues on the Cluster Member, shows:
There are no pnotes in problem state
When a Critical Device reports a problem, prints only the Critical Device that
reports its state as "problem".
cphaprob -ia list When there are no issues on the Cluster Member, shows:
When a Critical Device reports a problem, prints the Critical Device "Problem
Notification" and the Critical Device that reports its state as "problem"
cphaprob -e list When there are no issues on the Cluster Member, shows:
When a Critical Device reports a problem, prints only the Critical Device that
reports its state as "problem"
Related topics

Examples
Example 1 - Critical Device 'fwd'
Critical Device fwd reports its state as problem because the fwd process is down.
[Expert@Member1:0]# cphaprob -l list
Built-in Devices:
Device Name: Interface Active Check

Current state: OK
Device Name: Recovery Delay

Current state: OK
Device Name: CoreXL Configuration

Current state: OK
Registered Devices:
Device Name: Fullsync

Registration number: 0
Timeout: none
Current state: OK
Time since last report: 1753.7 sec
Device Name: Policy

Timeout: none
Current state: OK
Device Name: routed

Timeout: none
Current state: OK
Device Name: fwd

Timeout: 30 sec
Current state: problem
Process Status: DOWN
Device Name: cphad

Timeout: 30 sec
Current state: OK
Process Status: UP
Device Name: VSX

Timeout: none
Current state: OK
Device Name: Init

Timeout: none
Current state: OK
[Expert@Member1:0]#

Example 2 - Critical Device 'CoreXL Configuration'
Critical Device CoreXL Configuration reports its state as problem because the numbers of CoreXL
Firewall instances do not match between the Cluster Members.
[Expert@Member1:0]# cphaprob -l list
Built-in Devices:
Device Name: Interface Active Check

Current state: OK
Device Name: Recovery Delay

Current state: OK
Device Name: CoreXL Configuration

Current state: problem (non-blocking)
Registered Devices:
Device Name: Fullsync

Timeout: none
Current state: OK
Device Name: Policy

Timeout: none
Current state: OK
Device Name: routed

Timeout: none
Current state: OK
Device Name: fwd

Timeout: 30 sec
Current state: OK
Process Status: UP
Device Name: cphad

Timeout: 30 sec
Current state: OK
Process Status: UP
Device Name: VSX

Timeout: none
Current state: OK
Device Name: Init

Timeout: none
Current state: OK
[Expert@Member1:0]#

Viewing Cluster Interfaces

Description
This command shows the state of the Cluster Member interfaces and the virtual cluster interfaces.
ClusterXL treats the interfaces as Critical Devices. ClusterXL makes sure that interfaces can send and
receive CCP packets.
ClusterXL also sets the required minimal number of functional interfaces to the largest number of functional
interfaces ClusterXL detected since the last reboot. If the number of functional interfaces is less than the
required number, ClusterXL declares the Cluster Member as failed and starts a failover. The same applies
to the synchronization interfaces, where only good synchronization interfaces are counted.
When an interface is DOWN, it means that the interface cannot receive or send CCP packets, or both. An
interface may also be able to receive, but not send CCP packets. The time you see in the command's output
is the number of seconds that elapsed since the interface was last able to receive or send a CCP packet.
Syntax
Shell Command
Gaia Clish 1. set virtual-system <VSID>

2. show cluster members interfaces {all | secured | virtual |
vlans}
Expert cphaprob [-vs all] [-a] [-m] if

mode

Where:
Command Description
show cluster members interfaces Shows full list of all cluster interfaces:
all
n including the number of required interfaces
n including Network Objective
n including VLAN monitoring mode, or list of
monitored VLAN interfaces
show cluster members interfaces Shows only cluster interfaces (Cluster and Sync) and
secured their states:
n without Network Objective
n without VLAN monitoring mode
n without monitored VLAN interfaces
show cluster members interfaces Shows full list of cluster virtual interfaces and their states:
virtual
show cluster members interfaces Shows only monitored VLAN interfaces

vlans
cphaprob if Shows only cluster interfaces (Cluster and Sync) and

their states:
n without Network Objective
cphaprob -a if Shows full list of cluster interfaces and their states:

cphaprob -a -m if Shows full list of all cluster interfaces and their states:
n including VLAN monitoring mode, or list of
monitored VLAN interfaces
Output
The output of these commands must be identical to the configuration in the cluster object's Network
Management page in SmartConsole.

Example
[Expert@Member1:0]# cphaprob -a -m if
CCP mode: Manual (Unicast)

Required interfaces: 4
Required secured interfaces: 1
Interface Name: Status:
eth0 UP
eth1 (S) UP
eth2 (LM) UP
bond1 (LS) UP
S - sync, LM - link monitor, HA/LS - bond type
Virtual cluster interfaces: 3
eth0 192.168.3.247
eth2 44.55.66.247
bond1 77.88.99.247
No VLANs are monitored on the member
[Expert@Member1:0]#
Description of the "cphaprob -a -m if" command output fields:

Field, or Text Description
CCP mode Shows the CCP mode.

The default mode is Unicast.
Important - In R81.10, the CCP always runs in the unicast

mode.
Required interfaces Shows the total number of monitored cluster interfaces, including the
Sync interface.
This number is based on the configuration of the cluster object >
Network Management page.
Required secured interfaces Shows the total number of the required Sync interfaces.
Non-Monitored This means that Cluster Member does not monitor the state of this
interface.
In SmartConsole, in the cluster object > Network Management page,
administrator configured the Network Type Private for this interface.
UP This means that Cluster Member monitors the state of this interface.
The current cluster state of this interface is UP, which means this
interface can send and receive CCP packets.
administrator configured one of these Network Types for this
interface: Cluster, Sync, or Cluster + Sync.


Field, or Text Description
DOWN This means that Cluster Members monitors the state of this interface.
The current cluster state of this interface is DOWN, which means this
interface cannot send CCP packets, receive CCP packets, or both.
interface: Cluster, Sync, or Cluster + Sync.
(S) This interface is a Sync interface.

interface: Sync, or Cluster + Sync.
(LM) This interface is configured in the $FWDIR/conf/cpha_link_

monitoring.conf file.
Cluster Member monitors only the link on this interface (does not
monitor the received or sent CCP packets).
See "Configuring Link Monitoring on the Cluster Interfaces" on
page 112.
(HA) This interface is a Bond interface in High Availability mode.
(LS) This interface is a Bond interface in Load Sharing mode.
Virtual cluster interfaces Shows the total number of the configured virtual cluster interfaces.
No VLANs are monitored on Shows the VLAN monitoring mode - there are no VLAN interfaces
the member configured on the cluster interfaces.
Monitoring mode is Monitor all Shows the VLAN monitoring mode - there are some VLAN interfaces
VLANs: All VLANs are configured on the cluster interfaces, and Cluster Member monitors all
monitored VLAN IDs.
Monitoring mode is Monitor Shows the VLAN monitoring mode - there are some VLAN interfaces
specific VLAN: Only specified configured on the cluster interfaces, and Cluster Member monitors
VLANs are monitored only specific VLAN IDs.

Viewing Bond Interfaces

Description
This command shows the configuration of bond interfaces and their slave interfaces.
Syntax
Shell Command
Gaia Clish 1. show cluster bond {all | name <bond_

name>}
2. show bonding groups
Expert mode cphaprob show_bond [<bond_name>]

Where:
Command Description
show cluster bond all Shows configuration of all configured bond interfaces
show bonding groups
cphaprob show_bond
show cluster bond name <bond_ Shows configuration of the specified bond interface
name>
cphaprob show_bond <bond_name>
cphaprob show_bond_groups Shows the configured Groups of Bonds and their

settings.

Examples
Example 1 - 'cphaprob show_bond'
[Expert@Member2:0]# cphaprob show_bond
|Slaves |Slaves |Slaves

Bond name |Mode |State |configured |link up |required
-----------+-------------------+------+-----------+--------+--------
bond1 | High Availability | UP | 2 | 2 | 1
Legend:
-------
UP! - Bond interface state is UP, yet attention is required
Slaves configured - number of slave interfaces configured on the bond
Slaves link up - number of operational slaves
Slaves required - minimal number of operational slaves required for bond to be UP
[Expert@Member2:0]#
Member2> show bonding groups

Bonding Interface: 1
Bond Configuration
xmit-hash-policy Not configured
down-delay 200
primary Not configured
lacp-rate Not configured
mode active-backup
up-delay 200
mii-interval 100
Bond Interfaces
eth3
eth4
Member2>
Description of the output fields for the "cphaprob show_bond" and "show cluster bond all"
commands:
Field Description
Bond name Name of the Gaia bonding group.
Mode Bonding mode of this Gaia bonding group.

One of these:
n High Availability
n Load Sharing
State State of the Gaia bonding group:

n UP - Bond interface is fully operational
n UP! - Bond interface state is UP, yet attention is required
n DOWN - Bond interface failed
Slaves Total number of physical slave interfaces configured in this Gaia bonding group.
configured
Slaves link Number of operational physical slave interfaces in this Gaia bonding group.
up
Slaves Minimal number of operational physical slave interfaces required for the state of this
required Gaia bonding group to be UP.

Example 2 - 'cphaprob show_bond <bond_name>'

[Expert@Member2:0]# cphaprob show_bond bond1
Bond name: bond1

Bond mode: High Availability
Bond status: UP
Configured slave interfaces: 2

In use slave interfaces: 2
Required slave interfaces: 1
Slave name | Status | Link

----------------+-----------------+-------
eth4 | Active | Yes
eth3 | Backup | Yes
[Expert@Member2:0]#
Description of the output fields for the "cphaprob show_bond <bond_name>" and "show cluster
bond name <bond_name>" commands:
Field Description
Bond name Name of the Gaia bonding group.
Bond mode Bonding mode of this Gaia bonding group. One of these:
n High Availability
n Load Sharing
Bond status Status of the Gaia bonding group. One of these:

n UP! - Bond interface state is UP, yet attention is required
Configured Total number of physical slave interfaces configured in this Gaia bonding group.
slave
interfaces
In use slave Number of operational physical slave interfaces in this Gaia bonding group.
interfaces
Required Minimal number of operational physical slave interfaces required for the state of this
slave Gaia bonding group to be UP.
interfaces
Slave name Names of physical slave interfaces configured in this Gaia bonding group.


Field Description
Status Status of physical slave interfaces in this Gaia bonding group.

One of these:
n Active - In High Availability or Load Sharing bonding mode. This slave
interface is currently handling traffic.
n Backup - In High Availability bonding mode only. This slave interface is ready
and can support internal bond failover.
n Not Available - In High Availability or Load Sharing bonding mode. The
physical link on this slave interface is lost, or this Cluster Member is in status
Down. The bond cannot failover internally in this state.
Link State of the physical link on the physical slave interfaces in this Gaia bonding group.
One of these:
n Yes - Link is present
n No - Link is lost
Example 3 - 'cphaprob show_bond_groups'

[Expert@Member2:0]# cphaprob show_bond_groups
| Required | Bonds | Bonds

Group of bonds name | State | active bonds | in group | status
--------------------+-----------+--------------+----------+--------+
GoB0 | UP | 1 | |
| | | bond1 | UP
| | | bond2 | UP
Legend:
---------
Bonds in group - a list of the bonds in the bond group
Required active bonds - number of required active bonds
[Expert@Member2:0]#
Description of the output fields for the "cphaprob show_bond_groups" command:

Field Description
Group of bonds name Name of the Group of Bonds.
State State of the Group of Bonds. One of these:

n UP - Group of Bonds is fully operational
n DOWN - Group of Bonds failed
Required active bonds Number of required active bonds in this Group of Bonds.
Bonds in group Names of the Gaia bond interfaces configured in this Group of Bonds.
Bonds status State of the Gaia bond interface. One of these:


Viewing Cluster Failover Statistics

Description
This command shows the cluster failover statistics on the Cluster Member:
n Number of failovers that happened
n Failover reason
n The time of the last failover event
Syntax to show the statistics
Shell Command
Gaia Clish show cluster failover
Expert mode cphaprob [-l <number>] show_failover
Syntax to reset the statistics
Shell Command
Gaia Clish show cluster failover reset {count | history}
Expert mode cphaprob -reset {-c | -h} show_failover
Parameters
-l <number> Specifies how many of last failover events to show (between 1 and 50)
count Resets the counter of failover events

-c
history Resets the history of failover events

-h

Example
[Expert@Member1:0]# cphaprob show_failover


Failover counter: 1
Cluster failover history (last 20 failovers since reboot/reset on Sun Sep 8 16:08:34 2019):
No. Time: Transition: CPU: Reason:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - -
1 Sun Sep 8 18:21:44 2019 Member 2 -> Member 1 01 ADMIN_DOWN PNOTE
[Expert@Member1:0]#

Viewing Software Versions on Cluster Members
Viewing Software Versions on Cluster Members

Description
This command shows information about the software version (including private hotfixes) on the local Cluster
Member and its matches / mismatches with other Cluster Members.
Syntax
Shell Command
Gaia Clish show cluster release
Expert mode cphaprob release
Example
[Expert@Member1:0]# cphaprob release
Release: R80.40 T136
Kernel build: 994000117

FW1 build: 994000116
FW1 private fixes: None
ID SW release
1 (local) R80.40 T136

2 R80.40 T136
[Expert@Member1:0]#

Viewing Delta Synchronization

Heavily loaded clusters and clusters with geographically separated members pose special challenges.
High connection rates, and large distances between the members can lead to delays that affect the
operation of the cluster.
Monitor the operation of the State Synchronization mechanism in highly loaded and distributed clusters.
Perform these troubleshooting steps:

1. Examine the Delta Sync statistics counters:
Shell Command
Gaia Clish show cluster statistics sync
Expert mode cphaprob syncstat
2. Change the values of the applicable synchronization global configuration parameters.

3. Reset the Delta Sync statistics counters:
Shell Command
Gaia Clish show cluster statistics sync reset
Expert mode cphaprob -reset syncstat
4. Examine the Delta Sync statistics to see if the problem is solved.

5. Solve any identified problem.

Example output of the "show cluster statistics sync" and "cphaprob syncstat" commands
from a Cluster Member:
Delta Sync Statistics
Sync status: OK
Drops:
Lost updates................................. 0
Lost bulk update events...................... 0
Oversized updates not sent................... 0
Sync at risk:
Sent reject notifications.................... 0
Received reject notifications................ 0
Sent messages:
Total generated sync messages................ 26079
Sent retransmission requests................. 0
Sent retransmission updates.................. 0
Peak fragments per update.................... 1
Received messages:
Total received updates....................... 3710
Received retransmission requests............. 0
Sync Interface:
Name......................................... eth1
Link speed................................... 1000Mb/s
Rate......................................... 46000 [Bps]
Peak rate.................................... 46000 [Bps]
Link usage................................... 0%
Total........................................ 376827[KB]
Queue sizes (num of updates):

Sending queue size........................... 512
Receiving queue size......................... 256
Fragments queue size......................... 50
Timers:
Delta Sync interval (ms)..................... 100
Reset on Sun Sep 8 16:09:15 2019 (triggered by fullsync).
Each section of the output is described below.

The "Sync status:" section
This section shows the status of the Delta Sync mechanism. One of these:
n Sync status: OK
n Sync status: Off - Full-sync failure
n Sync status: Off - Policy installation failure
n Sync status: Off - Cluster module not started
n Sync status: Off - SIC failure
n Sync status: Off - Full-sync checksum error
n Sync status: Off - Full-sync received queue is full
n Sync status: Off - Release version mismatch
n Sync status: Off - Connection to remote member timed-out
n Sync status: Off - Connection terminated by remote member

n Sync status: Off - Could not start a connection to remote member

n Sync status: Off - cpstart
n Sync status: Off - cpstop
n Sync status: Off - Manually disabled sync
n Sync status: Off - Was not able to start for more than X second
n Sync status: Off - Boot
n Sync status: Off - Connectivity Upgrade (CU)
n Sync status: Off - cphastop
n Sync status: Off - Policy unloaded
n Sync status: Off - Hibernation
n Sync status: Off - OSU deactivated
n Sync status: Off - Sync interface down
n Sync status: Fullsync in progress
n Sync status: Problem (Able to send sync packets, unable to receive sync
packets)
n Sync status: Problem (Able to send sync packets, saving incoming sync
packets)
n Sync status: Problem (Able to send sync packets, able to receive sync
packets)
n Sync status: Problem (Unable to send sync packets, unable to receive
sync packets)
n Sync status: Problem (Unable to send sync packets, saving incoming sync
packets)
n Sync status: Problem (Unable to send sync packets, able to receive sync
packets)
The "Drops:" section
This section shows statistics for drops on the Delta Sync network.
Field Description
Lost updates Shows how many Delta Sync updates this Cluster Member considers as lost (based
on sequence numbers in CCP packets).
If this counter shows a value greater than 0, this Cluster Member lost Delta Sync
updates.
Possible mitigation:
Increase the size of the Sending Queue and the size of the Receiving Queue:
n Increase the size of the Sending Queue, if the counter Received reject
notification is increasing.
n Increase the size of the Receiving Queue, if the counter Received reject
notification is not increasing.


Field Description
Lost bulk Shows how many times this Cluster Member missed Delta Sync updates.
update (bulk update = twice the size of the local receiving queue)
events This counter increases when this Cluster Member receives a Delta Sync update with
a sequence number much greater than expected. This probably indicates some
networking issues that cause massive packet drops.
This counter increases when the amount of missed Delta Sync updates is more than
twice the local Receiving Queue Size.
n If the counter's value is steady, this might indicate a one-time synchronization
problem that can be resolved by running manual Full Sync. See sk37029.
n If the counter's value keeps increasing, probable there are some networking
issues. Increase the sizes of both the Receiving Queue and Sending Queue.
Oversized Shows how many oversized Delta Sync updates were discarded before sending
updates not them.
sent This counter increases when Delta Sync update is larger than the local Fragments
Queue Size.
n If the counter's value is steady, increase the size of the Sending Queue.
n If the counter's value keeps increasing, contact Check Point Support.
The "Sync at risk:" section
This section shows statistics that the Sending Queue is at full capacity and rejects Delta Sync
retransmission requests.
Field Description
Sent reject Shows how many times this Cluster Member rejected Delta Sync retransmission
notifications requests from its peer Cluster Members, because this Cluster Member does not
hold the requested Delta Sync update anymore.
Received Shows how many reject notifications this Cluster Member received from its peer
reject Cluster Members.
notification
The "Sent updates:" section
This section shows statistics for Delta Sync updates sent by this Cluster Member to its peer Cluster
Members.
Field Description
Total generated Shows how many Delta Sync updates were generated.
sync messages This counts the Delta Sync updates, Retransmission Requests, Retransmission
Acknowledgments, and so on.


Field Description
Sent Shows how many times this Cluster Member asked its peer Cluster Members to
retransmission retransmit specific Delta Sync update(s).
requests Retransmission requests are sent when certain Delta Sync updates (with a
specified sequence number) are missing, while the sending Cluster Member
already received Delta Sync updates with advanced sequences.
Note - Compare the number of Sent retransmission requests to the Total
generated sync messages of the other Cluster Members.
A large counter's value can imply connectivity problems. If the counter's value is
unreasonably high (more than 30% of the Total generated sync messages of
other Cluster Members), contact Check Point Support equipped with the entire
output and a detailed description of the network topology and configuration.
Sent Shows how many times this Cluster Member retransmitted specific Delta Sync
retransmission update(s) at the requests from its peer Cluster Members.
updates
Peak fragments Shows the peak amount of fragments in the Fragments Queue on this Cluster
per update Member (usually, should be 1).
The "Received updates:" section
This section shows statistics for Delta Sync updates that were received by this Cluster Member from its
peer Cluster Members.
Field Description
Total received Shows the total number of Delta Sync updates this Cluster Member received
updates from its peer Cluster Members.
This counts only Delta Sync updates (not Retransmission Requests,
Retransmission Acknowledgments, and others).
Received Shows how many retransmission requests this Cluster Member received from
retransmission its peer Cluster Members.
requests A large counter's value can imply connectivity problems. If the counter's value is
unreasonably high (more than 30% of the Total generated sync messages on
this Cluster Member), contact Check Point Support equipped with the entire
output and a detailed description of the network topology and configuration.
The "Queue sizes (num of updates):" section
This section shows the sizes of the Delta Sync queues.


Field Description
Sending Shows the size of the cyclic queue, which buffers all the Delta Sync updates that
queue size were already sent until it receives an acknowledgment from the peer Cluster
Members.
This queue is needed for retransmitting the requested Delta Sync updates.
Each Cluster Member has one Sending Queue.
Default: 512 Delta Sync updates, which is also the minimal value.
Receiving Shows the size of the cyclic queue, which buffers the received Delta Sync updates in
queue size two cases:
n When Delta Sync updates are missing, this queue is used to hold the
remaining received Delta Sync updates until the lost Delta Sync updates are
retransmitted (Cluster Members must keep the order, in which they save the
Delta Sync updates in the kernel tables).
n This queue is used to re-assemble a fragmented Delta Sync update.
Each Cluster Member has one Receiving Queue.
Fragments Shows the size of the queue, which is used to prepare a Delta Sync update before
queue size moving it to the Sending Queue.
Notes:
n This queue must be smaller than the Sending Queue.
n This queue must be significantly smaller than the Receiving Queue.
The "Timers:" section
This section shows the Delta Sync timers.
Field Description
Delta Sync Shows the interval at which this Cluster Member sends the Delta Sync updates
interval (ms) from its Sending Queue.
The base time unit is 100ms (or 1 tick).
Default: 100 ms, which is also the minimum value.
See Increasing the Sync Timer.
The "Reset on XXX (triggered XXX)" section
Shows the date and the time of last statistics reset.

In parentheses, it shows how the last statistics was triggered - "manually", or "by fullsync".

Viewing IGMP Status
Viewing IGMP Status

Description
This command shows the IGMP membership status.
Syntax
Shell Command
Gaia Clish show cluster members igmp
Expert mode cphaprob igmp
Example
[Expert@Member1:0]# cphaprob igmp
IGMP Membership: Enabled

Supported Version: 2
Report Interval [sec]: 60
IGMP queries are replied only by Operating System
Interface Host Group Multicast Address Last ver. Last Query[sec]

------------------------------------------------------------------------------
eth0 224.168.3.247 01:00:5e:28:03:f7 N/A N/A
eth1 224.22.33.250 01:00:5e:16:21:fa N/A N/A
eth2 224.55.66.247 01:00:5e:37:42:f7 N/A N/A
[Expert@Member1:0]#

Viewing Cluster Delta Sync Statistics for Connections Table
Viewing Cluster Delta Sync Statistics for Connections Table

Description
This command shows Delta Sync statistics about the operations performed in the Connections Kernel Table
(id 8158).
The output shows operations such as creating a new connection (SET), updating a connection (REFRESH),
deleting a connection (DELETE), and so on.
Syntax
Shell Command
Gaia Clish show cluster statistics transport [reset]
Expert mode cphaprob [-reset] ldstat
The "reset" flag resets the kernel statistics, which were collected since the last reboot or reset.
Example
[Expert@Member1:0]# cphaprob ldstat
Operand Calls Bytes Average Ratio %

----------------------------------------------------------
ERROR 0 0 0 0
SET 354 51404 145 33
RENAME 0 0 0 0
REFRESH 1359 70668 52 46
DELETE 290 10440 36 6
SLINK 193 12352 64 8
UNLINK 0 0 0 0
MODIFYFIELDS 91 7280 80 4
RECORD DATA CONN 0 0 0 0
COMPLETE DATA CONN 0 0 0 0
Total bytes sent: 161292 (0 MB) in 1797 packets. Average 89
[Expert@Member1:0]#

Viewing Cluster IP Addresses
Viewing Cluster IP Addresses

Description
This command shows the IP addresses and interfaces of the Cluster Members.
Syntax to see all interfaces
Shell Command
Gaia Clish show cluster members ips
Expert mode cphaprob tablestat
Syntax to see only the monitored interfaces
Shell Command
Gaia Clish show cluster members monitored
Expert mode cphaprob -m tablestat
Example
Note - To see name of interfaces that correspond to numbers in the "Interface" column,
run the fw ctl iflist command.
[Expert@Member1:0]# cphaprob tablestat
---- Unique IP's Table ----
Member Interface IP-Address

------------------------------------------
(Local)
0 1 192.168.3.245
0 2 11.22.33.245
0 3 44.55.66.245
1 1 192.168.3.246
1 2 11.22.33.246
1 3 44.55.66.246
------------------------------------------
[Expert@Member1:0]#
[Expert@Member1:0]# fw ctl iflist
1 : eth0
2 : eth1
3 : eth2
[Expert@Member1:0]#

Viewing the Cluster Member ID Mode in Local Logs
Viewing the Cluster Member ID Mode in Local Logs

Description
This command shows how the local ClusterXL logs show the Cluster Member - by its Member ID (default),
or its Member Name.
See "Configuring the Cluster Member ID Mode in Local Logs" on page 186.
Syntax
Shell Command
Gaia Clish show cluster members idmode
Expert mode cphaprob names
Example
Current member print mode in local logs is set to: ID
[Expert@Member1:0]#

Viewing Interfaces Monitored by RouteD
Viewing Interfaces Monitored by RouteD

Description
This command shows the interfaces, which the RouteD daemon monitors on the Cluster Member when you
configure OSPF.
The idea is that if you configure OSPF, Cluster Member monitors these interfaces and does not bring up the
Cluster Member unless RouteD daemon says it is OK to bring up the Cluster Member. This is used mainly in
ClusterXL High Availability Primary Up configuration to avoid premature failbacks.
Syntax
Shell Command
Gaia Clish show ospf interfaces [detailed]
Expert mode cphaprob routedifcs
Example 1
[Expert@Member1:0]# cphaprob routedifcs
No interfaces are registered.
[Expert@Member1:0]#
Example 2
[Expert@Member1:0]# cphaprob routedifcs
Monitored interfaces registered by routed:
eth0
[Expert@Member1:0]#

Viewing Roles of RouteD Daemon on Cluster Members
Viewing Roles of RouteD Daemon on Cluster Members

Description
This command shows on which Cluster Member the RouteD daemon runs as a Master.
Notes:
n In ClusterXL High Availability, the RouteD daemon must run as a Master only on
the Active Cluster Member.
n In ClusterXL Load Sharing, the RouteD daemon must run as a Master only on
one of the Active Cluster Members and as a Non-Master on all other Cluster
Members.
n In VRRP Cluster, the RouteD daemon must run as a Master only on the VRRP
Master Cluster Member.
Syntax
Shell Command
Gaia Clish show cluster role
Expert mode cphaprob roles
Example
[Expert@Member1:0]# cphaprob roles
ID Role
1 (local) Master
2 Non-Master
[Expert@Member1:0]#

Viewing Cluster Correction Statistics

Description
This command shows the Cluster Correction Statistics on each Cluster Member.
The Cluster Correction Layer (CCL) is a mechanism that deals with asymmetric connections.
The CCL provides connections stickiness by "correcting" the packets to the correct Cluster Member:
n In most cases, the CCL makes the correction from the CoreXL SND.
n In some cases (like Dynamic Routing, or VPN), the CCL makes the correction from the Firewall or
SecureXL.
In some cases, ClusterXL needs to send some data along with the corrected packet (currently, only in VPN).
For such packets, the output shows "with metadata".
Note - For more information about CoreXL, see the R81.10 Performance Tuning
Administration Guide.
Syntax
Shell Command
Gaia Clish N/A
Expert mode cphaprob [{-d | -f | -s}] corr
Where:
Command Description
cphaprob corr Shows Cluster Correction Statistics for all traffic.
cphaprob -d corr Shows Cluster Correction Statistics for CoreXL SND only.
cphaprob -f corr Shows Cluster Correction Statistics for CoreXL Firewall instances only.
cphaprob -s corr Shows Cluster Correction Statistics for SecureXL only.
Example 1 - For all traffic
[Expert@Member1:0]# cphaprob corr
Getting stats for SXL device 0, may take a few seconds...
Cluster Correction Stats (All Traffic):

------------------------------------------------------
Sent packets: 156 (0 with metadata)
Sent bytes: 34,568
Received packets: 0 (0 with metadata)
Received bytes: 0
Send errors: 0
Receive errors: 0
Local asymmetric conns: 0
[Expert@Member1:0]#

Example 2 - For CoreXL SND only
[Expert@Member1:0]# cphaprob -d corr
Cluster Correction Stats (Dispatcher Corrections only):

------------------------------------------------------
Sent bytes: 0
Received bytes: 0
Send errors: 0
Receive errors: 0
[Expert@Member1:0]#
Example 3 - For CoreXL Firewall instances only
[Expert@Member1:0]# cphaprob -f corr
Cluster Correction Stats (Firewall instances only):

------------------------------------------------------
Sent bytes: 34,568
Received bytes: 0
Send errors: 0
Receive errors: 0
[Expert@Member1:0]#
Example 4 - For SecureXL only
[Expert@Member1:0]# cphaprob -s corr
Getting stats for SXL device 0, may take a few seconds...
Cluster Correction Stats (SXL Devices only):

------------------------------------------------------
Sent bytes: 0
Received bytes: 0
Send errors: 0
Receive errors: 0
[Expert@Member1:0]#

Viewing the Cluster Control Protocol (CCP) Settings
Viewing the Cluster Control Protocol (CCP) Settings

Description
n You can view the Cluster Control Protocol (CCP) mode on the Cluster Members.
n You can view the Cluster Control Protocol (CCP) Encryption on the Cluster Members - enabled or
disabled (and the encryption key).
See "Configuring the Cluster Control Protocol (CCP) Settings" on page 194
Syntax for viewing the Cluster Control Protocol (CCP) mode
Shell Command
Gaia Clish show cluster members interfaces virtual
Expert mode cphaprob -a if
Syntax for viewing the Cluster Control Protocol (CCP) Encryption
Shell Command
Gaia Clish show cluster members ccpenc
Expert mode cphaprob ccp_encrypt

cphaprob ccp_encrypt_key

Viewing Latency and Drop Rate of Interfaces
Viewing Latency and Drop Rate of Interfaces

Description
This command shows the latency and the drop rate of each interface.
Syntax
Shell Command
Gaia Clish N/A
Expert mode cphaprob latency
Example
[Expert@Member1:0]# cphaprob latency
id 2
Latency | Drop
[msec] | rate
eth0 0.000 0%
eth1 0.000 0%
eth2 0.000 0%
[Expert@Member1:0]#

Viewing the State of the Multi-Version Cluster Mechanism
Viewing the State of the Multi-Version Cluster Mechanism

Description
This command shows the state of the Multi-Version Cluster (MVC) Mechanism - enabled (ON) or disabled
(OFF).
See "Configuring the Multi-Version Cluster Mechanism" on page 202.
Syntax
Shell Command
Gaia Clish show cluster members mvc
Expert mode cphaprob mvc
Example
Member1> show cluster members mvc
ON
Member1>

Viewing Full Connectivity Upgrade Statistics
Viewing Full Connectivity Upgrade Statistics

Description
This command shows the Full Connectivity Upgrade statistics when you upgrade between minor versions.
Syntax
Shell Command
Gaia Clish N/A
Expert mode cphaprob fcustat
Example
[Expert@Member1:0]# cphaprob fcustat
During FCU....................... no
Connection module map............ none
Table id map (remote->local)..... none
Table handlers ..................

8151 --> 0x0x7f97c421d860 (sip_state)
8158 --> 0x0x7f97c43d8e30 (connections)
LD handlers......................
ok - 0
failed - 0
Global handlers ................. none
[Expert@Member1:0]#

Monitoring Cluster Status in SmartConsole

Background
To see the applicable logs in SmartConsole:
n From the left navigation panel, click Logs & Monitor > Logs.
To get logs about changes in the state of Cluster Members:
3. From the left tree, click ClusterXL and VRRP.
5. In the Tracking field, select Log.
6. Click OK.
ClusterXL Log Messages

This section uses these conventions:
1. Square brackets are used to indicate place holders, which are substituted by relevant data when an
actual log message is issued (for example, "[NUMBER]" is replaced by a numeric value).
2. Angle brackets are used to indicate alternatives, one of which is used in actual log messages.
The different alternatives are separated with a vertical line (for example, "<up|down>" indicates that
either "up" or "down" is used).
3. These placeholders are frequently used:
n ID: A unique Cluster Member identifier, starting from "1". This corresponds to the order, in
which Cluster Members are sorted in the cluster object on the Cluster Members page.
n IP: Any unique IP address that belongs to the Cluster Member.
n MODE: The cluster mode (for example, New HA, LS Multicast, and so on).
n STATE: The state of the member (for example, active, down, standby).
n DEVICE: The name of a Critical Device (for example, Interface Active Check, fwd).

General Logs
Log Description
Starting <ClusterXL|State Indicates that ClusterXL (or State

Synchronization>. Synchronization, for 3rd party clusters) was
successfully started on the reporting Cluster
Member.
This message is usually issued after a
member boots, or after an explicit call to
cphastart.
Stopping <ClusterXL|State Informs that ClusterXL (or State

Synchronization>. Synchronization) was deactivated on this
Cluster Member.
The Cluster Member is no longer a part of the
cluster (even if configured to be so), until
ClusterXL is restarted.
Unconfigured cluster Computers changed This message is usually issued when a

their MAC Addresses. Please reboot the Cluster Member is shut down, or after an
cluster so that the changes take affect. explicit call to the cphastop.
State Logs
Log Description
Mode inconsistency detected: This message should rarely happen.

member [ID] ([IP]) will change It indicates that another Cluster Member has reported a
its mode to [MODE]. Please re- different cluster mode than is known to the local Cluster
install the security policy on Member.
the cluster. This is usually the result of a failure to install the Access
Control Policy on all Cluster Members.
To correct this problem, install the Access Control
Policy again.
Note - The cluster continues to operate after
a mode inconsistency is detected by altering
the mode of the reporting Cluster Member to
match the other Cluster Members. However,
we highly recommend to re-install the policy
as soon as possible.

Log Description
State change of member [ID] When a Cluster Member needs to change its state (for
([IP]) from [STATE] to [STATE] example, when an Active Cluster Member encounters a
was cancelled, since all other problem and needs to change its state to "Down"), it
members are down. Member remains first queries the other Cluster Members for their state.
[STATE]. If all other Cluster Members are down, this Cluster
Member cannot change its state to a non-active one
(otherwise the cluster cannot function).
Thus, the reporting Cluster Member continues to
function, despite its problem (and usually reports its
state as "Active(!)").
member [ID] ([IP]) <is active|is This message is issued whenever a Cluster Member
down|is stand-by|is initializing> changes its state.
([REASON]). The log text specifies the new state of the Cluster
Member.
Critical Device Logs
Log Description
[DEVICE] on member [ID] The Critical Device is working normally.

([IP]) status OK ([REASON])
[DEVICE] on member [ID] Either an error was detected by the Critical Device, or the
([IP]) detected a problem Critical Device has not reported its state for a number of
([REASON]). seconds (as set by the "timeout" option of the Critical Device)
[DEVICE] on member [ID] Indicates that the Critical Device has registered itself with the
([IP]) is initializing Critical Device mechanism, but has not yet determined its state.
([REASON]).
[DEVICE] on member [ID] This message should not normally appear.

([IP]) is in an unknown Contact Check Point Support.
state ([STATE ID])
([REASON]).
Interface Logs
Log Description
interface [INTERFACE NAME] of Indicates that this interface is working normally - it is

member [ID] ([IP]) is up. able to receive and transmit Cluster Control Protocol
(CCP) packets on the expected subnet.

Log Description
interface [INTERFACE NAME] of This message is issued whenever an interface

member [ID] ([IP]) is down encounters a problem, either in receiving or
(receive <up|down>, transmit transmitting Cluster Control Protocol (CCP) packets.
<up|down>). Note that in this case the interface may still be working
properly, as far as the OS is concerned, but is unable
to communicate with other Cluster Members.
interface [INTERFACE NAME] of Indicates that a new interface was registered with the
member [ID] ([IP]) was added. Cluster Member (meaning that Cluster Control
Protocol (CCP) packets arriving on this interface).
Usually, this message is the result of activating an
interface (such as issuing the "ifconfig up"
command).
The interface is now included in the ClusterXL reports
(in the output of the applicable CLI commands.
Note that the interface may still be reported as
"Disconnected", in case it was configured as such
for ClusterXL.
interface [INTERFACE NAME] of Indicates that an interface was detached from the
member [ID] ([IP}) was removed. Cluster Member, and is therefore no longer monitored
by ClusterXL.
Reason Strings
Log Description
member [ID] ([IP]) reports This text can be included in a Critical Device log message
more interfaces up. describing the reasons for a problem report: another Cluster
Member has more interfaces reported to be working, than the
local Cluster Member does.
Usually, this means that the local Cluster Member has a faulty
interface, and that its peer Cluster Member can do a better job
as a Cluster Member. The local Cluster Member changes it
state to "Down", leaving the peer Cluster Member specified in
the message to handle traffic.
member [ID] ([IP]) has more This message is issued when Cluster Members in the same
interfaces - check your cluster have a different number of interfaces.
disconnected interfaces A Cluster Member with fewer interfaces than the maximal
configuration in the number in the cluster (the reporting Cluster Member) may not
<discntd.if file|registry> be working properly, as it is missing an interface required to
operate against a cluster IP address, or a synchronization
network.
If some of the interfaces on the other Cluster Member are
redundant, and should not be monitored by ClusterXL, they
should be explicitly designated as "Non-Monitored". See
"Defining Non-Monitored Interfaces" on page 150.

Log Description
[NUMBER] interfaces ClusterXL has detected a problem with one or more of the
required, only [NUMBER] up. monitored interfaces.
This does not necessarily mean that the Cluster Member
changes its state to "Down", as the other Cluster Members may
have less operational interfaces.
In such a condition, the Cluster Member with the largest number
of operational interfaces will remain up, while the others go
down.

Working with SNMP Traps
Working with SNMP Traps

You can configure and see SNMP traps for ClusterXL High Availability.
To configure an SNMP trap:

1. Connect to the command line on the Management Server.
3. On a Multi-Domain Server, go to the context of the applicable Domain Management Server:
mdsenv <IP Address or Name of Domain Management Server>
4. Run:
threshold_config
For more information, see the R81.10 CLI Reference Guide > Chapter Security Management Server
Commands > Section threshold_config.
5. From the Threshold Engine Configuration Options menu, select (9) Configure Thresholds.
6. From the Threshold Categories menu, select (2) High Availability.
7. Select the applicable traps.
8. Select and configure these actions for the specified trap:
n Enable/Disable Threshold
n Set Severity
n Set Repetitions
n Configure Alert Destinations
9. From the Threshold Engine Configuration Options menu, select (7) Configure alert destinations.
10. Configure your alert destinations.
11. From the Threshold Engine Configuration Options menu, select (3) Save policy.
You can optionally save the policy to a file.
12. In SmartConsole, install the Access Control Policy on this cluster object.
Note - You can download the most recent Check Point MIB files from sk90470.

How to Initiate Cluster Failover
How to Initiate Cluster Failover

For more information on initiating manual cluster failovers, see sk55081.
Method 1 (recommended)
Command
Change in the State Command
in the Expert Notes
of the Cluster Member in Gaia Clish
Mode
Change the state of the Cluster set cluster member clusterXL_ Does not disable
Member to DOWN admin down admin down Delta Sync.
Change the state of the Cluster set cluster member clusterXL_ Does not initiate
Member to UP admin up admin up Full Sync.
See:
n "Initiating Manual Cluster Failover" on page 195
n "The clusterXL_admin Script" on page 280
Method 2 (not recommended)
Change in the State

Command in the Expert mode Notes
of the Cluster Member
Change the state of the 1. cphaconf set_pnote -d <Name of Does not

Cluster Member to DOWN Critical Device> -t 0 -s ok disable Delta
register Sync.
2. cphaconf set_pnote -d <Name of
Critical Device> -s problem
report
Change the state of the 1. cphaconf set_pnote -d <Name of Does not

Cluster Member to UP Critical Device> -s ok report initiate Full
2. cphaconf set_pnote -d <Name of Sync.
Critical Device> unregister
See:
Notes:
n In Load Sharing mode, the cluster distributes the traffic load between the
remaining Active members.
n In High Availability mode, the cluster fails over to a Standby Cluster Member with
the highest priority.

Troubleshooting Issues with the Critical Device "routed"
Troubleshooting Issues with the Critical Device

"routed"
Background
The Critical Device called routed monitors the state of the Dynamic Routing on the Cluster Member (see
"Viewing Critical Devices" on page 213).
This Critical Device makes sure that traffic is not assigned to a Cluster Member before it is ready to handle
the traffic.
The Gaia OS RouteD daemon handles all routing (static and dynamic) operations.
There can be an issue with Dynamic Routing with one or more of these symptoms:
n Cluster IP address connectivity problems
n Unexpected cluster failovers
n The state of the Critical Device routed is problem.
Example:
Device Name: routed

Timeout: none
Current state: problem
Time since last report: 10 sec
These are some of the common causes of this issue:

n Cluster misconfiguration
n Traffic on TCP port 2010 between Cluster Members is blocked
n The RouteD daemon did not get all of its routes
n The RouteD daemon did not start correctly
Standard Behavior of the Critical Device "routed"

Typically, the Critical Device routed reports its current state as problem when:
n A Cluster Member fails over
n A Cluster Member reboots
n There is an inconsistency in the Dynamic Routing configuration on Cluster Members
the Critical Device routed reports its current state as OK when
n A ClusterXL member tells the RouteD daemon that it is a Master
n The RouteD daemon gets the entire routing state from the Master

Troubleshooting Issues with the Critical Device "routed"
Basic Troubleshooting Steps

1. Examine the cluster interfaces to make sure they are configured correctly.
2. In ClusterXL High Availability mode, make sure that the RouteD daemon is running on the Active
Cluster Member.
3. Make sure traffic on TCP port 2010 between the Cluster Members is not blocked.
4. Generate RouteD cluster messages.
Run this command in the Expert mode on the cluster member:
dbset routed:instance:default:traceoptions:traceoptions:Cluster
Examine the /var/log/routed/log file.

5. In ClusterXL High Availability mode with the OSPF configuration, make sure the OSPF interface is up
on the Standby Cluster Member.
6. In the OSPF configuration, look for a "router-id" mismatch.
For advanced troubleshooting procedures and more information, see sk92787.
For troubleshooting OSPF and the RouteD daemon, see sk84520.

ClusterXL Error Messages
ClusterXL Error Messages

Each important cluster event that affects the Cluster Members has a unique code that appears in the
/var/log/messages file and dmesg.
Each cluster event message starts with the prefix CLUS-XXXXXX-Y.
Example: CLUS-214802-1
Part of the
Description
message
CLUS- Constant string.
XXXXXX A six-digit error code.

These error codes specify the events:
n Events related to Critical Devices
n Events related to the states of Cluster Members
n Events related to cluster synchronization
n Events related to policy installation
The first digit shows which Cluster Member generated the event:
n 1 - local member
n 2 - remote member
The second digit represents the cluster type event
The meaning of the other digits depends on the cluster event type.
Y Shows the ID or the NAME of the local Cluster Member that generated this log
message.
See "Configuring the Cluster Member ID Mode in Local Logs" on page 186.

Command Line Reference
Command Line Reference

See the R81.10 CLI Reference Guide.


Description
These commands let you configure internal behavior of the Clustering Mechanism.
Important:
n We do not recommend that you run these commands. These commands must be
run automatically only by the Security Gateway or the Check Point Support.
Syntax
Notes:
n In Gaia Clish:
Enter the set cluster<ESC><ESC> to see all the available commands.
n In Expert mode:
Run the cphaconf command see all the available commands.
You can run the cphaconf commands only from the Expert mode.
n Syntax legend:
Table: ClusterXL Configuration Commands
Configure how to show the Cluster Member in set cluster cphaconf mem_id_mode {id
local ClusterXL logs - by its Member ID or its member | name}
Member Name (see "Configuring the Cluster idmode {id |
Member ID Mode in Local Logs" on page 186) name}
Register a single Critical Device (Pnote) on the N / A cphaconf set_pnote -d

Cluster Member (see "Registering a Critical <Name of Device> -t
Device" on page 187) <Timeout in Sec> -s
{ok|init|problem} [-p]
[-g] register
Unregister a single Critical Device (Pnote) on N / A cphaconf set_pnote -d

the Cluster Member (see "Unregistering a <Name of Device> [-p] [-
Critical Device" on page 189) g] unregister


Report (change) a state in a single Critical N / A cphaconf set_pnote -d

Device (Pnote) on the Cluster Member (see <Name of Device> -s
"Reporting the State of a Critical Device" on {ok|init|problem} [-g]
page 190) report
Register several Critical Devices (Pnotes) from N / A cphaconf set_pnote -f

a file on the Cluster Member (see "Registering <Name of File> [-g]
Critical Devices Listed in a File" on page 191) register
Unregister all Critical Devices (Pnotes) on the N / A cphaconf set_pnote -a [-

Cluster Member (see "Unregistering All Critical g] unregister
Devices" on page 193)
Configure the Cluster Control Protocol (CCP) set cluster cphaconf ccp_encrypt
Encryption on the Cluster Member (see member {off | on}
"Configuring the Cluster Control Protocol (CCP) ccpenc {off cphaconf ccp_encrypt_key
Settings" on page 194) | on} <Key String>
Configure the Cluster Forwarding Layer on the set cluster cphaconf forward {off |
Cluster Member (controls the forwarding of member on}
traffic between Cluster Members) forwarding
Note - For Check Point use only. {off | on}
Print the current cluster configuration as loaded N / A cphaconf debug_data

in the kernel on the Cluster Member (for details,
see sk93306)
Start internal failover between slave interfaces N / A cphaconf failover_bond

of specified bond interface - only in Bond High <bond_name>
Availability mode (for details, see sk93306)
Configure what happens during a failover after a N / A cphaconf enable_bond_

Bond already failed over internally (for details, failover <bond_name>
see sk93306)
Initiate manual cluster failover (see "Initiating set cluster clusterXL_admin {down |
Manual Cluster Failover" on page 195) member admin up}
{down | up}
Configure the minimal number of required N / A cphaconf bond_ls {set

slaves interfaces for Bond Load Sharing (see <Bond Name> <Value> |
"Configuring the Minimal Number of Required remove <Bond Name>}
Slave Interfaces for Bond Load Sharing" on
page 199)
Configuring Link Monitoring on the Cluster N / A N / A

Interfaces (see "Configuring Link Monitoring on
the Cluster Interfaces" on page 112)


Configuring the Multi-Version Cluster N / A cphaconf mvc {off | on}

Mechanism (see "Configuring the Multi-Version
Cluster Mechanism" on page 202)
List of the Gaia Clish set cluster member commands
set cluster member admin {down | up} [permanent]

set cluster member ccpenc {off | on}
set cluster member forwarding {off | on}
set cluster member idmode {id | name}
set cluster member mvc {off | on}
List of the cphaconf commands
cphaconf [-D] <options> start

cphaconf stop
cphaconf [-t <Sync IF 1>...] [-d <Non-Monitored IF 1>...] add
cphaconf clear-secured
cphaconf clear-non-monitored
cphaconf debug_data
cphaconf delete_link_local [-vs <VSID>] <IF name>
cphaconf set_link_local [-vs <VSID>] <IF name> <Cluster IP>
cphaconf mem_id_mode {id | name}
cphaconf failover_bond <bond_name>
cphaconf [-s] {set | unset | get} var <Kernel Parameter Name> [<Value>]
cphaconf bond_ls {set <Bond Name> <Value> | remove <Bond Name>}
cphaconf set_pnote -d <Device> -t <Timeout in sec> -s {ok | init | problem}
[-p] [-g] register
cphaconf set_pnote -f <File> [-g] register
cphaconf set_pnote -d <Device> [-p] [-g] unregister
cphaconf set_pnote -a [-g] unregister
cphaconf set_pnote -d <Device> -s {ok | init | problem} [-g] report
cphaconf ccp_encrypt {off | on}


Description
Use the monitoring commands to make sure that the cluster and the Cluster Members work properly, and to
define Critical Devices. A Critical Device (also known as a Problem Notification, or pnote) is a special
software device on each Cluster Member, through which the critical aspects for cluster operation are
monitored. When the critical monitored component on a Cluster Member fails to report its state on time, or
when its state is reported as problematic, the state of that member is immediately changed to 'Down'.
Syntax
Notes:
n In Gaia Clish:
Enter the show cluster<ESC><ESC> to see all the available commands.
n In Expert mode:
Run the cphaprob command see all the available commands.
You can run the cphaprob commands from Gaia Clish as well.
n Syntax legend:
Table: ClusterXL Monitoring Commands

Show states of Cluster Members and their names (see show cluster cphaprob [-vs
"Viewing Cluster State" on page 209) state <VSID>] state
Show Critical Devices (Pnotes) and their states on the show cluster cphaprob [-l]
Cluster Member (see "Viewing Critical Devices" on members pnotes [-ia] [-e]
page 213) {all | problem} list
Show cluster interfaces on the cluster member (see show cluster cphaprob [-vs
"Viewing Cluster Interfaces" on page 221) members all] [-a] [-
interfaces {all m] if
| secured |
virtual | vlans}
Show cluster bond configuration on the Cluster Member show cluster cphaprob
(see "Viewing Bond Interfaces" on page 225) bond {all | name show_bond
<bond_name>} [<bond_name>]


Show groups of bonds on the Cluster Member (see N / A cphaprob

"Viewing Bond Interfaces" on page 225) show_bond_
groups
Show (and reset) cluster failover statistics on the Cluster show cluster cphaprob [-
Member (see "Viewing Cluster Failover Statistics" on failover [reset reset {-c | -
page 229) {count | h}] [-l
history}] <count>]
show_failover
Show information about the software version (including show cluster cphaprob
hotfixes) on the local Cluster Member and its release release
matches/mismatches with other Cluster Members (see
"Viewing Software Versions on Cluster Members" on
page 231)
Show Delta Sync statistics on the Cluster Member (see show cluster cphaprob [-
"Viewing Delta Synchronization" on page 232) statistics sync reset]
[reset] syncstat
Show Delta Sync statistics for the Connections table on show cluster cphaprob [-
the Cluster Member (see "Viewing Cluster Delta Sync statistics reset] ldstat
Statistics for Connections Table" on page 239) transport
[reset]
Show the Cluster Control Protocol (CCP) mode on the show cluster cphaprob [-vs
Cluster Member (see "Viewing Cluster Interfaces" on members all] -a if
virtual
Show the IGMP membership of the Cluster Member (see show cluster cphaprob igmp
"Viewing IGMP Status" on page 238) members igmp
Show cluster unique IP's table on the Cluster Member show cluster cphaprob
(see "Viewing Cluster IP Addresses" on page 240) members ips tablestat
show cluster cphaprob -m
members tablestat
monitored
Show the Cluster Member ID Mode in local logs - by show cluster cphaprob
Member ID (default) or Member Name (see "Viewing the members idmode names
Cluster Member ID Mode in Local Logs" on page 241)
Show interfaces, which the RouteD monitors on the show ospf cphaprob
Cluster Member when you configure OSPF (see "Viewing interfaces routedifcs
Interfaces Monitored by RouteD" on page 242) [detailed]
Show roles of RouteD daemon on Cluster Members (see show cluster cphaprob
"Viewing Roles of RouteD Daemon on Cluster Members" roles roles
on page 243)


Show Cluster Correction Statistics (see "Viewing Cluster N / A cphaprob [{-d

Correction Statistics" on page 244) | -f | -s}]
corr
Show the Cluster Control Protocol (CCP) mode (see show cluster cphaprob -a
"Viewing the Cluster Control Protocol (CCP) Settings" on members if
virtual
Show the Cluster Control Protocol (CCP) Encryption show cluster cphaprob ccp_
settings (see "Viewing the Cluster Control Protocol (CCP) members ccpenc encrypt
Settings" on page 246)
Shows the state of the Multi-Version Cluster (see show cluster N / A

"Viewing the State of the Multi-Version Cluster members mvc
Mechanism" on page 248)
Shows the latency and the drop rate of each interface N / A N / A

(see "Viewing Latency and Drop Rate of Interfaces" on
page 247)
Show Full Connectivity Upgrade statistics (see "Viewing N / A cphaprob

Full Connectivity Upgrade Statistics" on page 249) fcustat

List of the Gaia Clish show cluster commands
show cluster
bond
all
name <Name of Bond>
failover
members
ccpenc
idmode
igmp
interfaces
all
secured
virtual
vlans
ips
monitored
mvc
pnotes
all
problem
release
roles
state
statistics
sync [reset]
transport [reset]

List of the cphaprob commands
cphaprob [-vs <VSID>] state

cphaprob [-reset {-c | -h}] [-l <count>] show_failover
cphaprob names
cphaprob [-reset] [-a] syncstat
cphaprob [-reset] ldstat
cphaprob [-l] [-i[a]] [-e] list
cphaprob [-vs all] [-a] [-m] if
cphaprob latency
cphaprob show_bond [<bond_name>]
cphaprob igmp
cphaprob fcustat
cphaprob [-m] tablestat
cphaprob routedifcs
cphaprob roles
cphaprob release
cphaprob ccp_encrypt
cphaprob [{-d | -f | -s}] corr

cpconfig
cpconfig
Description
This command starts the Check Point Configuration Tool.
This tool configures specific settings for the installed Check Point products.
Important:
n On Scalable Platforms (Maestro and Chassis), you must connect to the
applicable Security Group.
Syntax on a Security Gateway / Cluster Member in Gaia Clish or the Expert mode
cpconfig
Syntax on a Scalable Platform Security Group in Gaia gClish or the Expert mode
cpconfig
Menu Options
Note - The options shown depend on the configuration and installed products.
Menu Option Description
Licenses and contracts Manages Check Point licenses and contracts on this Security
Gateway or Cluster Member.
SNMP Extension Obsolete. Do not use this option anymore.

To configure SNMP, see the R81.10 Gaia Administration Guide -
Chapter System Management - Section SNMP.
PKCS#11 Token Register a cryptographic token, for use by Gaia Operating System.
See details of the token, and test its functionality.
Random Pool Configures the RSA keys, to be used by Gaia Operating System.
Secure Internal Communication Manages SIC on the Security Gateway or Cluster Member.
This change requires a restart of Check Point services on the
Security Gateway or Cluster Member.
For more information, see:
n The R81.10 Security Management Administration Guide.
n sk65764: How to reset SIC.

cpconfig
Menu Option Description
Enable cluster membership for Enables the cluster membership on the Security Gateway.
this gateway This change requires a reboot of the Security Gateway.
For more information, see the R81.10 Installation and Upgrade
Guide.
Note - This section does not apply to Scalable Platforms (Maestro
and Chassis).
Disable cluster membership for Disables the cluster membership on the Security Gateway.
this gateway This change requires a reboot of the Security Gateway.
Guide.
and Chassis).
Enable Check Point Per Virtual Enables Virtual System Load Sharing on the VSX Cluster Member.
System State For more information, see the R81.10 VSX Administration Guide.
and Chassis).
Disable Check Point Per Virtual Disables Virtual System Load Sharing on the VSX Cluster Member.
System State For more information, see the R81.10 VSX Administration Guide.
and Chassis).
Enable Check Point ClusterXL Enables Check Point ClusterXL for Bridge mode.
for Bridge Active/Standby This change requires a reboot of the Cluster Member.
Guide.
and Chassis).
Disable Check Point ClusterXL Disables Check Point ClusterXL for Bridge mode.
for Bridge Active/Standby This change requires a reboot of the Cluster Member.
Guide.
and Chassis).
Check Point CoreXL Manages CoreXL on the Security Gateway / Cluster Member /
Scalable Platform Security Group.
After all changes in CoreXL configuration, you must reboot the
Security Gateway / Cluster Member / Security Group.
For more information, see the R81.10 Performance Tuning
Administration Guide.
Automatic start of Check Point Shows and controls which of the installed Check Point products
Products start automatically during boot.
Exit Exits from the Check Point Configuration Tool.

cpconfig
Example 1 - Menu on a single Security Gateway
[Expert@MySingleGW:0]# cpconfig
This program will let you re-configure
your Check Point products configuration.
Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Enable cluster membership for this gateway
(7) Check Point CoreXL
(8) Automatic start of Check Point Products
(9) Exit
Enter your choice (1-9) :
Example 2 - Menu on a Cluster Member
[Expert@MyClusterMember:0]# cpconfig
This program will let you re-configure
your Check Point products configuration.
Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Enable Check Point ClusterXL for Bridge Active/Standby
(9) Check Point CoreXL
(10) Automatic start of Check Point Products
(11) Exit
Enter your choice (1-11) :

cphastart
cphastart
Description
Starts the cluster configuration on a Cluster Member after it was stopped with the "cphastop" on page 273
command.
Best Practice - To start a Cluster Member, use the "cpstart" command.
Note - This command does not initiate a Full Synchronization on the Cluster Member.
Syntax
cphastart
[-h]
[-d]
Parameters
-h Shows the applicable built-in usage.
-d Runs the command in debug mode.

Use only if you troubleshoot the command itself.
Best Practice - If you use this parameter, then redirect the output to a file, or
use the script command to save the entire CLI session.
Refer to:
n These lines in the output file:
prepare_command_args: -D ... start
/opt/CPsuite-R81.10/fw1/bin/cphaconf clear-secured
/opt/CPsuite-R81.10/fw1/bin/cphaconf -D ...(truncated here
for brevity)... start
n The $FWDIR/log/cphastart.elg log file.

cphastop
cphastop
Description
Stops the cluster software on a Cluster Member.
Best Practice - To stop a Cluster Member, use the "cpstop" command.
Notes:
n This command stops the Cluster Member from passing traffic.
n This command stops the State Synchronization between this Cluster Member and
its peer Cluster Members.
n After you run this command, you can still open connections directly to this Cluster
Member.
n To start the cluster software, run the "cphastart" on page 272 command.
Syntax
cphastop

cp_conf fullha
cp_conf fullha
Important - This command does not apply to Scalable Platforms (Maestro and Chassis).
Description
Manages the state of the Full High Availability Cluster:
n Enables the Full High Availability Cluster
n Disables the Full High Availability Cluster
n Deletes the Full High Availability peer
n Shows the Full High Availability state
Important - To configure a Full High Availability cluster, follow the R81.10 Installation
and Upgrade Guide.
Syntax
cp_conf fullha
enable
del_peer
disable
state
Parameters
enable Enables the Full High Availability on this computer.
del_peer Deletes the Full High Availability peer from the configuration.
disable Disables the Full High Availability on this computer.
state Shows the Full High Availability state on this computer.
Example
[Expert@Cluster_Member:0]# cp_conf fullha state

FullHA is currently enabled
[Expert@Cluster_Member:0]#

cp_conf ha
cp_conf ha
Important - This command does not apply to Scalable Platforms (Maestro and Chassis).
Description
Enables or disables cluster membership on this Security Gateway.
Important - This command is for Check Point use only. To configure cluster
membership, you must use the "cpconfig" command.
Syntax
cp_conf ha {enable | disable} [norestart]
Parameters
enable Enables cluster membership on this Security Gateway.

This command is equivalent to the option Enable cluster membership for this
gateway in the "cpconfig" menu.
disable Disables cluster membership on this Security Gateway.

This command is equivalent to the option Disable cluster membership for this
gateway in the "cpconfig" menu.
norestart Optional: Specifies to apply the configuration change without the restart of Check
Point services. The new configuration takes effect only after reboot.
Example 1 - Enable the cluster membership without restart of Check Point services
[Expert@MyGW:0]# cp_conf ha enable norestart
Cluster membership for this gateway was enabled successfully

Important: This change will take effect after reboot.
[Expert@MyGW:0]#

cp_conf ha
Example 2 - Disable the cluster membership without restart of Check Point services
[Expert@MyGW:0]# cp_conf ha disable norestart

cpwd_admin:
Process CPHAMCSET process has been already terminated
Cluster membership for this gateway was disabled successfully

Important: This change will take effect after reboot.
[Expert@MyGW:0]#

fw hastat
fw hastat
Description
Shows information about Check Point computers in High Availability configuration and their states.
Note - This command is outdated. On cluster members, run the Gaia Clish command
"show cluster state", or the Expert mode command "cphaprob state". See
"Viewing Cluster State" on page 209.
Syntax
fw hastat [<Target1>] [<Target2>] ... [<TargetN>]
Parameters
<Target1> Specifies the Check Point computers to query.

<Target2> ... If you run this command on the Management Server, you can enter the applicable IP
<TargetN> address, or the resolvable HostName of the managed Security Gateway or Cluster
Member.
If you do not specify the target, the command queries the local computer.
Example - Querying the local Cluster Member
[Expert@Member1:0]# fw hastat
HOST NUMBER HIGH AVAILABILITY STATE MACHINE STATUS
192.168.3.52 1 active OK
[Expert@Member1:0]#

fwboot ha_conf
fwboot ha_conf
Description
Configures the cluster mechanism during boot.
Important - This command is for Check Point use only.
Notes:
n You must run this command from the Expert mode.
n To install a cluster, see the R81.10 Installation and Upgrade
Guide.
Syntax
[Expert@HostName:0]# $FWDIR/boot/fwboot ha_conf

ClusterXL Scripts
ClusterXL Scripts
You can use special scripts to change the state of Cluster Members.

The clusterXL_admin Script

Description
You can use the clusterXL_admin script to initiate a manual fail-over from a Cluster Member.
Location of this script on your Cluster Members is:
$FWDIR/bin/clusterXL_admin
Script Workflow
This shell script does one of these:
n Registers a Critical Device called "admin_down" and reports the state of that Critical Device as
"problem".
This gracefully changes the state of the Cluster Member to "DOWN".
n Reports the state of the registered Critical Device "admin_down" as "ok".
This gracefully changes the state of the Cluster Member to "UP".
Then, the script unregisters the Critical Device "admin_down".

Example

#! /bin/csh -f
#
# The script will cause the machine to get into down state, thus the member will not filter packets.
# It will supply a simple way to initiate a failover by registering a new device in problem state when
# a failover is required and will unregister the device when wanting to return to normal operation.
# USAGE:
# clusterXL_admin <up|down>
set PERSISTENT = ""
# checking number of arguments

if ( $#argv > 2 || $#argv < 1 ) then
echo "clusterXL_admin : Invalid Argument Count"
echo "Usage: clusterXL_admin <up|down> [-p]"
exit 1
else if ( "$1" != "up" && "$1" != "down" ) then
echo "clusterXL_admin : Invalid Argument ($1)"
exit 1
else if ( $#argv == 2 ) then
if ( "$2" != "-p" ) then
echo "clusterXL_admin : Invalid Argument ($2)"
exit 1
endif
set PERSISTENT = "-p"
endif
#checking if cpha is started

$FWDIR/bin/cphaprob stat | grep "Cluster" > /dev/null
if ($status) then
echo "HA is not started"
exit 1
endif
# Inform the user that the command can run with persistent mode.
if ("$PERSISTENT" != "-p") then
echo "This command does not survive reboot. To make the change permanent, please run 'set cluster
member admin down/up permanent' in clish or add '-p' at the end of the command in expert mode"
endif
if ( $1 == "up" ) then

echo "Setting member to normal operation ..."
$FWDIR/bin/cphaconf set_pnote -d admin_down $PERSISTENT unregister > & /dev/null
if ( ùname` == 'IPSO' ) then
sleep 5
else
sleep 1
endif
set stateArr = `$FWDIR/bin/cphaprob stat | grep "local"`
$FWDIR/bin/cphaprob stat | egrep "Sync only|Bridge Mode" > /dev/null

#If it's third party or bridge mode, use column 4 , otherwise 5
if ($status) then
set state = $stateArr[5]
else
endif
echo "Member current state is $state"

if (($state != "Active" && $state != "Standby") && ($state != "ACTIVE" && $state != "STANDBY" &&
$state != "ACTIVE(!)")) then
echo "Operation failed: member is still down, please run 'show cluster members pnotes
problem' in clish or 'cphaprob list' in expert mode for further details"
endif
exit 0
endif
if ( $1 == "down" ) then

echo "Setting member to administratively down state ..."
$FWDIR/bin/cphaconf set_pnote -d admin_down -t 0 -s problem $PERSISTENT register > & /dev/null
sleep 1
set stateArr = `$FWDIR/bin/cphaprob stat | grep "local"`
$FWDIR/bin/cphaprob stat | egrep "Sync only|Bridge Mode" > /dev/null

#If it's third party or bridge mode, use column 4 , otherwise 5

if ($status) then
else
endif
echo "Member current state is $state"

if ( $state == "Active attention" || $state == "ACTIVE(!)" ) then
echo "All the members within the cluster have problem/s and the local member was chosen to
become active"
else
if ( $state != "Down" && $state != "DOWN" ) then
echo "Operation failed: member is still down, please run 'show cluster members pnotes
problem' in clish or 'cphaprob list' in expert mode for further details"
endif
endif
exit 0
else
echo "clusterXL_admin : Invalid Option ($1)"
exit 1
endif

The clusterXL_monitor_ips Script

Description
You can use the clusterXL_monitor_ips script to ping a list of predefined IP addresses and change the state
of the Cluster Member to DOWN or UP based on the replies to these pings. For this script to work, you must
write the IP addresses in the $FWDIR/conf/cpha_hosts file - each IP address on a separate line. This
file does not support comments or spaces.
$FWDIR/bin/clusterXL_monitor_ips
Script Workflow
1. Registers a Critical Device called "host_monitor" with the status "ok".
2. Starts to send pings to the list of predefined IP addresses in the $FWDIR/conf/cpha_hosts file.
3. While the script receives responses to its pings, it does not change the status of that Critical Device.
4. If the script does not receive a response to even one ping, it reports the state of that Critical Device as
"problem".
This gracefully changes the state of the Cluster Member to DOWN.
If the script receives responses to its pings again, it changes the status of that Critical Device to "ok"
again.
Important - You must do these changes on all Cluster Members.

Example

#!/bin/sh
#
# The script tries to ping the hosts written in the file $FWDIR/conf/cpha_hosts. The names (must be
resolveable) ot the IPs of the hosrs must be written in seperate lines.
# the file must not contain anything else.
# We ping the given hosts every number of seconds given as parameter to the script.
# USAGE:
# cpha_monitor_ips X silent
# where X is the number of seconds between loops over the IPs.
# if silent is set to 1, no messages will appear on the console
#
# We initially register a pnote named "host_monitor" in the problem notification mechanism
# when we detect that a host is not responding we report the pnote to be in "problem" state.
# when ping succeeds again - we report the pnote is OK.
silent=0
if [ -n "$2" ]; then

if [ $2 -le 1 ]; then
silent=$2
fi
fi
hostfile=$FWDIR/conf/cpha_hosts
arch=ùname -s`
if [ $arch = "Linux" ]
then
#system is linux
ping="ping -c 1 -w 1"
else
ping="ping"
fi
$FWDIR/bin/cphaconf set_pnote -d host_monitor -t 0 -s ok register
TRUE=1
while [ "$TRUE" ]
do
result=1
for hosts in `cat $hostfile`
do
if [ $silent = 0 ]
then
echo "pinging $hosts using command $ping $hosts"
fi
if [ $arch = "Linux" ]
then
$ping $hosts > /dev/null 2>&1
else
$ping $hosts $1 > /dev/null 2>&1
fi
status=$?
if [ $status = 0 ]
then
if [ $silent = 0 ]
then
echo " $hosts is alive"
fi
else
if [ $silent = 0 ]
then
echo " $hosts is not responding "
fi
result=0
fi
done
if [ $silent = 0 ]
then
echo "done pinging"
fi
if [ $result = 0 ]
then
if [ $silent = 0 ]
then
echo " Cluster member should be down!"
fi
$FWDIR/bin/cphaconf set_pnote -d host_monitor -s problem report
else
if [ $silent = 0 ]
then
echo " Cluster member seems fine!"

fi
$FWDIR/bin/cphaconf set_pnote -d host_monitor -s ok report
fi
if [ "$silent" = 0 ]
then
echo "sleeping"
fi
sleep $1
echo "sleep $1"
done

The clusterXL_monitor_process Script

Description
You can use the clusterXL_monitor_process script to monitor if the specified user space processes run,
and cause cluster fail-over if these processes do not run. For this script to work, you must write the correct
case-sensitive names of the monitored processes in the $FWDIR/conf/cpha_proc_list file - each
process name on a separate line. This file does not support comments or spaces.
$FWDIR/bin/clusterXL_monitor_process
Script Workflow
1. Registers Critical Devices (with the status "ok") called as the names of the processes you specified in
the $FWDIR/conf/cpha_proc_list file.
2. While the script detects that the specified process runs, it does not change the status of the
corresponding Critical Device.
3. If the script detects that the specified process do not run anymore, it reports the state of the
corresponding Critical Device as "problem".
This gracefully changes the state of the Cluster Member to "DOWN".
If the script detects that the specified process runs again, it changes the status of the corresponding
Critical Device to "ok" again.
Important - You must do these changes on all Cluster Members.

Example

#!/bin/sh
#
# This script monitors the existance of processes in the system. The process names should be written
# in the $FWDIR/conf/cpha_proc_list file one every line.
#
# USAGE :
# cpha_monitor_process X silent
# where X is the number of seconds between process probings.
# if silent is set to 1, no messages will appear on the console.
#
#
# We initially register a pnote for each of the monitored processes
# (process name must be up to 15 charachters) in the problem notification mechanism.
# when we detect that a process is missing we report the pnote to be in "problem" state.
# when the process is up again - we report the pnote is OK.
if [ "$2" -le 1 ]
then
silent=$2
else
silent=0
fi
if [ -f $FWDIR/conf/cpha_proc_list ]
then
procfile=$FWDIR/conf/cpha_proc_list
else
echo "No process file in $FWDIR/conf/cpha_proc_list "
exit 0
fi
arch=ùname -s`
for process in `cat $procfile`

do
$FWDIR/bin/cphaconf set_pnote -d $process -t 0 -s ok -p register > /dev/null 2>&1
done
while [ 1 ]
do
result=1
for process in `cat $procfile`

do
ps -ef | grep $process | grep -v grep > /dev/null 2>&1
status=$?
if [ $status = 0 ]
then
if [ $silent = 0 ]
then
echo " $process is alive"
fi
# echo "3, $FWDIR/bin/cphaconf set_pnote -d $process -s ok report"
$FWDIR/bin/cphaconf set_pnote -d $process -s ok report
else
if [ $silent = 0 ]
then
echo " $process is down"
fi
$FWDIR/bin/cphaconf set_pnote -d $process -s problem report

result=0
fi
done
if [ $result = 0 ]
then
if [ $silent = 0 ]
then
echo " One of the monitored processes is down!"
fi
else
if [ $silent = 0 ]
then
echo " All monitored processes are up "

fi
fi
if [ "$silent" = 0 ]
then
echo "sleeping"
fi
sleep $1
done

Cluster Management APIs

Introduction
The purpose of Cluster APIs is to provide automation / orchestration of Check Point cluster in a way similar
to simple-gateway APIs.
These Cluster APIs support common cluster operations - such as creating a new cluster object, modifying
an existing cluster object (for example, adding or removing cluster members, manipulation of interfaces).
These Cluster APIs are called "simple" because they do not support all cluster object features.
For operations on cluster objects that are not provided by these APIs, use SmartConsole.
List of APIs
API Category API Description
Asynchronous add simple- Creates a new simple cluster object from scratch
cluster
set simple- Modifies an existing simple cluster object

cluster
Synchronous show simple- Shows an existing simple cluster object specified by its
cluster Name or UID
show simple- Shows all existing simple cluster objects

clusters
delete simple- Deletes an existing simple cluster object

cluster
API Examples
Example 1 - Adding a simple cluster object
API command:
Use this API to add a simple cluster object.
add simple-cluster
Once the API command finishes, and the session is published, a new cluster object appears in
SmartConsole.

Prerequisite:
1. All Cluster Members must already be installed.
2. The applicable interfaces on each Cluster Member must already be configured.
Example description:
n A simple ClusterXL in High Availability mode called cluster1
n With two cluster members called member1 and member2
n With three interfaces: eth0 (external), eth1 (sync), and eth2 (internal)
n Only the Firewall Software Blade is enabled (the IPsec VPN blade is disabled)
n Cluster software version is R80.20
Example cluster object topology:
Interface Cluster Member1 Member2
eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 /

255.255.255.0 255.255.255.0 255.255.255.0
eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 /

255.255.255.0 255.255.255.0
eth2 (Internal) 192.168.1.254 / 192.168.1.1 / 192.168.1.2 /

255.255.255.0 255.255.255.0 255.255.255.0
API example:
Important - In the API command you must use the same One-Time Password you
used on Cluster Members during the First Time Configuration Wizard.

{
"name" : "cluster1",
"color" : "yellow",
"version" : "R80.20",
"ip-address" : "172.23.5.254",
"os-name" : "Gaia",
"cluster-mode" : "cluster-xl-ha",
"firewall" : true,
"vpn" : false,
"interfaces" : [
{
"name" : "eth0",
"ip-address" : "172.23.5.254",
"network-mask" : "255.255.255.0",
"interface-type" : "cluster",
"topology" : "EXTERNAL",
"anti-spoofing" : "true"
},
{
"name" : "eth1",
"interface-type" : "sync",
"topology" : "INTERNAL",
"topology-settings": {
"ip-address-behind-this-interface": "network defined by the interface ip and net mask",
"interface-leads-to-dmz": false
}
},
{
"name" : "eth2",
"ip-address" : "192.168.1.254",
"network-mask" : "255.255.255.0",
"anti-spoofing" : "true",
"ip-address-behind-this-interface": "network defined by the interface ip and net mask",
"interface-leads-to-dmz": false
}
}
],
"members" : [ {
"name" : "member1",
"one-time-password" : "abcd",
"ip-address" : "172.23.5.1",
"interfaces" : [
{
"name" : "eth0",
"ip-address" : "172.23.5.1",
"network-mask" : "255.255.255.0"
},
{
"name" : "eth1",
"ip-address" : "1.1.1.1",
"network-mask" : "255.255.255.0"
},
{
"name" : "eth2",
"ip-address" : "192.168.1.1",
"network-mask" : "255.255.255.0"
}
]
},
{
"name" : "member2",
"one-time-password" : "abcd",
"ip-address" : "172.23.5.2",
"interfaces" : [
{
"name" : "eth0",
"ip-address" : "172.23.5.2",
"network-mask" : "255.255.255.0"

},
{
"name" : "eth1",
"ip-address" : "1.1.1.2",
"network-mask" : "255.255.255.0"
},
{
"name" : "eth2",
"ip-address" : "192.168.1.2",
"network-mask" : "255.255.255.0"
}
]
}
]
}
Example 2 - Modifying an existing cluster object - adding a cluster member
API command:
Use this API to add (scale up) Cluster Members.
set simple-cluster
Adding a Cluster Member called member3.
Interface Cluster Member1 Member2 Member3
eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 / 172.23.5.3 /

255.255.255.0 255.255.255.0 255.255.255.0 255.255.255.0
eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 / 1.1.1.3 /

255.255.255.0 255.255.255.0 255.255.255.0
eth2 (Internal) 192.168.1.254 / 192.168.1.1 / 192.168.1.2 / 192.168.1.3 /

255.255.255.0 255.255.255.0 255.255.255.0 255.255.255.0
API example:

{
"members" : { "add" :
{
"name" : "member3",
"ipv4-address" : "172.23.5.3",
"one-time-password" : "aaaa",
"interfaces" : [
{
"name" : "eth0",
"ip-address" : "172.23.5.3",
"network-mask" : "255.255.255.0"
},
{
"name" : "eth1",
"ip-address" : "1.1.1.3",
"network-mask" : "255.255.255.0"
},
{
"name" : "eth2",
"ip-address" : "192.168.1.3",
"network-mask" : "255.255.255.0"
}
]
}
}
}
Example 3 - Modifying an existing cluster object - removing a cluster member
API command:
Use this API to remove (scale down) Cluster Members.
set simple-cluster
Removing a Cluster Member called member3.
Interface Cluster Member1 Member2 Member3
eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 / 172.23.5.3 /

255.255.255.0 255.255.255.0 255.255.255.0 255.255.255.0
eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 / 1.1.1.3 /

255.255.255.0 255.255.255.0 255.255.255.0
eth2 (Internal) 192.168.1.254 / 192.168.1.1 / 192.168.1.2 / 192.168.1.3 /

255.255.255.0 255.255.255.0 255.255.255.0 255.255.255.0
API example:
{
"members" : { "remove" : "member3" }
}

Example 4 - Modifying an existing cluster object - adding a cluster interface
API command:
Use this API to add a cluster interface.
set simple-cluster
Adding a cluster interface called eth3.
eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 /

255.255.255.0 255.255.255.0 255.255.255.0
eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 /

255.255.255.0 255.255.255.0
eth2 (Internal) 192.168.1.254 / 192.168.1.1 / 192.168.1.2 /

255.255.255.0 255.255.255.0 255.255.255.0
eth3 (Internal) 10.10.10.254 / 10.10.10.1 / 10.10.10.2 /

255.255.255.0 255.255.255.0 255.255.255.0
API example:

{
"interfaces" : { "add" :
{
"name" : "eth3",
"ip-address" : "10.10.10.254",
"ipv4-mask-length" : "24",
}
},
"members" : { "update" :
[{
"name" : "member1" ,
"interfaces" :
{"name" : "eth3",
"ipv4-address" : "10.10.10.1",
"ipv4-network-mask" : "255.255.255.0"}
},
{
"interfaces" :
{"name" : "eth3",
"ipv4-address" : "10.10.10.2",
"ipv4-network-mask" : "255.255.255.0"}
}
]}
}
Example 5 - Modifying an existing cluster object - removing a cluster interface
API command:
Use this API to remove a cluster interface.
set simple-cluster
Removing a cluster interface called eth3.
eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 /

255.255.255.0 255.255.255.0 255.255.255.0
eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 /

255.255.255.0 255.255.255.0
eth2 (Internal) 192.168.1.254 / 192.168.1.1 / 192.168.1.2 /

255.255.255.0 255.255.255.0 255.255.255.0
eth3 (Internal) 10.10.10.254 / 10.10.10.1 / 10.10.10.2 /

255.255.255.0 255.255.255.0 255.255.255.0

API example:
{
"interfaces" : { "remove" : "eth3" }
}
Example 6 - Modifying an existing cluster object - changing settings of a cluster interface
API command:
Use this API to change settings of a cluster interface.
set simple-cluster
Changing the IP address of the cluster interfaces called eth2 from 192.168.x.254 / 255.255.255.0 to
172.30.1.x / 255.255.255.0
eth0 (External) 172.23.5.254 / 172.23.5.1 / 172.23.5.2 /

255.255.255.0 255.255.255.0 255.255.255.0
eth1 (Sync) N/A 1.1.1.1 / 1.1.1.2 /

255.255.255.0 255.255.255.0
eth2 (Internal) From: From: From:

192.168.1.254 / 192.168.1.1 / 192.168.1.2 /
255.255.255.0 255.255.255.0 255.255.255.0
To: To: To:
172.30.1.254 / 172.30.1.1 / 172.30.1.2 /
255.255.255.0 255.255.255.0 255.255.255.0
API example:

{
"interfaces" : { "update" :
{
"name" : "eth2",
"ip-address" : "172.30.1.254",
"ipv4-mask-length" : "24",
}
},
"members" : { "update" : [
{
"interfaces" :
{"name" : "eth2",
"ipv4-address" : "172.30.1.1",
"ipv4-mask-length" : "24"}
},
{
"interfaces" :
{"name" : "eth2",
"ipv4-address" : "172.30.1.2",
"ipv4-mask-length" : "24"}
}
]}
}
Example 7 - Modifying an existing cluster object - reestablishing SIC
API command:
Use this API to reestablish SIC with Cluster Members.
set simple-cluster
Prerequisite:
SIC must already be reset on the Cluster Members.
API example:
Important - In the API command you must use the same One-Time Password you
used on Cluster Members during the SIC reset.
{
"members" : { "update" :
[
{
"name" : "member1",
"one-time-password" : "aaaa"
},
{
"name" : "member2",
"one-time-password" : "aaaa"
}
]
}
}

Example 8 - Modifying an existing cluster object - enabling / disabling blades
API command:
Use this API to enable and disable Software Blades on Cluster Members.
set simple-cluster
Notes:
n To enable a Software Blade, set its value to true in the
API command.
n To disable a Software Blade, set its value to false in the
API command.
API example:
To enable all Software Blades supported by the Cluster API:
{
"vpn" : true,
"application-control" : true,
"url-filtering" : true,
"ips" : true,
"content-awareness" : true,
"anti-bot" : true,
"anti-virus" : true,
"threat-emulation" : true
}
To disable all Software Blades supported by the Cluster API:
{
"vpn" : false,
"application-control" : false,
"url-filtering" : false,
"ips" : false,
"content-awareness" : false,
"anti-bot" : false,
"anti-virus" : false,
"threat-emulation" : false
}
Example 9 - Viewing an existing cluster object
API command:
Use this API to view a specific existing cluster object.
show simple-cluster
Note - By default, the output shows up to 50 configured cluster interfaces.

API example - request:
{
"limit-interfaces" : "10",
"name" : "cluster1"
}
API example - response:

{
"uid": "e0ce560b-8a0a-4468-baa9-5f8eb2658b96",
"name": "cluster1",
"type": "simple-cluster",
"domain": {
"uid": "41e821a0-3720-11e3-aa6e-0800200c9fde",
"name": "SMC User",
"domain-type": "domain"
},
"meta-info": {
"lock": "unlocked",
"validation-state": "ok",
"last-modify-time": {
"posix": 1567417185885,
"iso-8601": "2019-09-02T12:39+0300"
},
"last-modifier": "aa",
"creation-time": {
"posix": 1567417140278,
"iso-8601": "2019-09-02T12:39+0300"
},
"creator": "aa"
},
"tags": [],
"read-only": false,
"comments": "",
"color": "yellow",
"icon": "NetworkObjects/cluster",
"groups": [],
"ipv4-address": "172.23.5.254",
"dynamic-ip": false,
"version": "R80.20",
"os-name": "Gaia",
"hardware": "Open server",
"firewall": true,
"firewall-settings": {
"auto-maximum-limit-for-concurrent-connections": true,
"maximum-limit-for-concurrent-connections": 25000,
"auto-calculate-connections-hash-table-size-and-memory-pool": true,
"connections-hash-size": 131072,
"memory-pool-size": 6,
"maximum-memory-pool-size": 30
},
"vpn": false,
"application-control": false,
"url-filtering": false,
"content-awareness": false,
"ips": false,
"anti-bot": false,
"anti-virus": false,
"threat-emulation": false,
"save-logs-locally": false,
"send-alerts-to-server": [
"harry-main-take-96"
],
"send-logs-to-server": [
"harry-main-take-96"
],
"send-logs-to-backup-server": [],

"logs-settings": {
"rotate-log-by-file-size": false,
"rotate-log-file-size-threshold": 1000,
"rotate-log-on-schedule": false,
"alert-when-free-disk-space-below-metrics": "mbytes",
"alert-when-free-disk-space-below": true,
"alert-when-free-disk-space-below-threshold": 3000,
"alert-when-free-disk-space-below-type": "popup alert",
"delete-when-free-disk-space-below-metrics": "mbytes",
"delete-when-free-disk-space-below": true,
"delete-when-free-disk-space-below-threshold": 5000,
"before-delete-keep-logs-from-the-last-days": false,
"before-delete-keep-logs-from-the-last-days-threshold": 0,
"before-delete-run-script": false,
"before-delete-run-script-command": "",
"stop-logging-when-free-disk-space-below-metrics": "mbytes",
"stop-logging-when-free-disk-space-below": true,
"stop-logging-when-free-disk-space-below-threshold": 100,
"reject-connections-when-free-disk-space-below-threshold": false,
"reserve-for-packet-capture-metrics": "mbytes",
"reserve-for-packet-capture-threshold": 500,
"delete-index-files-when-index-size-above-metrics": "mbytes",
"delete-index-files-when-index-size-above": false,
"delete-index-files-when-index-size-above-threshold": 100000,
"delete-index-files-older-than-days": false,
"delete-index-files-older-than-days-threshold": 14,
"forward-logs-to-log-server": false,
"perform-log-rotate-before-log-forwarding": false,
"update-account-log-every": 3600,
"detect-new-citrix-ica-application-names": false,
"turn-on-qos-logging": true
},
"interfaces": {
"total": 3,
"from": 1,
"to": 3,
"objects": [
{
"name": "eth0",
"ipv4-address": "172.23.5.254",
"ipv4-network-mask": "255.255.255.0",
"ipv4-mask-length": 24,
"ipv6-address": "",
"topology": "external",
"anti-spoofing": true,
"anti-spoofing-settings": {
"action": "prevent"
},
"security-zone": false,
"comments": "",
"color": "black",
"icon": "NetworkObjects/network",
"interface-type": "cluster"
},
{
"name": "eth1",
"ipv4-address": "1.1.1.0",
"ipv4-network-mask": "255.255.255.0",
"ipv6-address": "",
"topology": "internal",
"ip-address-behind-this-interface": "network defined by the interface ip and net
mask",

"anti-spoofing": true,
"anti-spoofing-settings": {
"action": "prevent"
},
"security-zone": false,
"comments": "",
"color": "black",
"icon": "NetworkObjects/network",
"interface-type": "cluster"
}
]
},
"cluster-mode": "cluster-xl-ha",
"cluster-members": [
{
"name": "member1",
"sic-state": "initialized",
"sic-message": "Initialized but trust not established",
"ip-address": "172.23.5.1",
"interfaces": [
{
"name": "eth0",
"ipv4-address": "172.23.5.1",
"ipv4-network-mask": "255.255.255.0",
"ipv6-address": "",
"ipv6-network-mask": "::",
"ipv6-mask-length": 0
},
{
"name": "eth1",
"ipv4-address": "1.1.1.1",
"ipv4-network-mask": "255.255.255.0",
"ipv6-address": "",
},
{
"name": "eth2",
"ipv4-address": "192.168.1.1",
"ipv4-network-mask": "255.255.255.0",
"ipv6-address": "",
}
]
},
{
"name": "member2",
"sic-state": "initialized",
"sic-message": "Initialized but trust not established",
"ip-address": "172.23.5.3",
"interfaces": [
{
"name": "eth0",
"ipv4-address": "172.23.5.2",
"ipv4-network-mask": "255.255.255.0",
"ipv6-address": "",
},
{
"name": "eth1",
"ipv4-address": "1.1.1.2",
"ipv4-network-mask": "255.255.255.0",
"ipv6-address": "",

Example 10 - Viewing all existing cluster objects
API command:
Use this API to view all existing cluster objects.
show simple-clusters
Example 11 - Deleting an existing cluster object
API command:
Use this API to delete a specific cluster object.
delete simple-cluster
API example:
{
"name" : "cluster1"
}
Known Limitations
n These Cluster APIs support only subset of cluster operations.
n These Cluster APIs support only basic configuration of Software Blades (similar to "simple-
gateway" APIs - see the Check Point Management API Reference).
n These Cluster APIs support only ClusterXL High Availability, ClusterXL Load Sharing, and
CloudGuard OPSEC clusters.
n These Cluster APIs do not support the configuration of a Cluster Virtual IP address on a different
subnet than the IP addresses of the Cluster Members.
For such configuration, use SmartConsole.
n These Cluster APIs do not support VRRP Clusters (either on Gaia OS or IPSO OS).
n These Cluster APIs support a limited subset of interface settings.
To change interface settings such as Topology, Anti-Spoofing and Security Zone, you must replace
the interface.

CP R81.10 ClusterXL AdminGuide

Uploaded by

Copyright:

Available Formats

CP R81.10 ClusterXL AdminGuide

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CP R81.10 ClusterXL AdminGuide

Uploaded by

Copyright:

Available Formats

27 June 2021

RESTRICTED RIGHTS LEGEND:

Check Point R81.10

Latest Version of this Document in English

ClusterXL R81.10 Administration Guide | 3

27 June 2021 First release of this document

ClusterXL R81.10 Administration Guide | 4

ClusterXL R81.10 Administration Guide | 5

Load Sharing Multicast Mode 56

ClusterXL R81.10 Administration Guide | 6

Changing the Settings of Cluster Object in SmartConsole 100

ClusterXL R81.10 Administration Guide | 7

Creating a new Group of Bonds 136

ClusterXL R81.10 Administration Guide | 8

Configuring ISP Redundancy on a Cluster 174

ClusterXL R81.10 Administration Guide | 9

Viewing Roles of RouteD Daemon on Cluster Members 243

ClusterXL R81.10 Administration Guide | 10

Cluster Management APIs 292

ClusterXL R81.10 Administration Guide | 11

3rd party Cluster

ClusterXL R81.10 Administration Guide | 12

ClusterXL R81.10 Administration Guide | 13

ClusterXL R81.10 Administration Guide | 14

Cluster Control Protocol

Cluster Correction Layer

ClusterXL R81.10 Administration Guide | 15

CoreXL Firewall Instance

ClusterXL R81.10 Administration Guide | 16

ClusterXL R81.10 Administration Guide | 17

Delta Sync Retransmission

Domain Log Server

ClusterXL R81.10 Administration Guide | 18

ClusterXL R81.10 Administration Guide | 19

ClusterXL R81.10 Administration Guide | 20

Flush and ACK

ClusterXL R81.10 Administration Guide | 21

Full High Availability

ClusterXL R81.10 Administration Guide | 22

ClusterXL R81.10 Administration Guide | 23

ClusterXL R81.10 Administration Guide | 24

ClusterXL R81.10 Administration Guide | 25

Jumbo Hotfix Accumulator

Load Sharing Multicast

Load Sharing Unicast

ClusterXL R81.10 Administration Guide | 26

Management High Availability

Multi-Domain Log Server

Multi-Domain Security Management

ClusterXL R81.10 Administration Guide | 27

ClusterXL R81.10 Administration Guide | 28

ClusterXL R81.10 Administration Guide | 29

ClusterXL R81.10 Administration Guide | 30

Security Management Server

ClusterXL R81.10 Administration Guide | 31

ClusterXL R81.10 Administration Guide | 32

ClusterXL R81.10 Administration Guide | 33

ClusterXL R81.10 Administration Guide | 34

ClusterXL R81.10 Administration Guide | 35

2 Switch for internal network