What Is a Web Logic Server Cluster?

A WebLogic Server cluster consists of multiple WebLogic Server server instances running simultaneously and working together to provide increased scalability and reliability. A cluster appears to clients to be a single WebLogic Server instance. The server instances that constitute a cluster can run on the same machine, or be located on different machines. You can increase a cluster's capacity by adding additional server instances to the cluster on an existing machine, or you can add machines to the cluster to host the incremental server instances. Each server instance in a cluster must run the same version of WebLogic Server. Benefits of Clustering Scalability High-Availability

What Types of Objects Can Be Clustered? Servlets JSPs EJBs Remote Method Invocation (RMI) objects Java Messaging Service (JMS) destinations Java Database Connectivity (JDBC) connections

What Types of Objects Cannot Be Clustered? File services including file shares Time services

WebLogic Server Communication In a Cluster WebLogic Server instances in a cluster communicate with one another using two basic network technologies: IP sockets, which are the conduits for peer-to-peer communication between clustered server instances. IP unicast or multicast, which server instances use to broadcast availability of services and heartbeats that indicate continued availability.

Note: When creating a cluster using the Configuration Wizard, the default cluster messaging mode is unicast. When creating a cluster using WLST, the default cluster messaging mode is multicast. Using IP Multicast for Backward Compatibility IP multicast is a simple broadcast technology that enables multiple applications to "subscribe" to a given IP address and port number and listen for messages. Note : When creating a new cluster, Oracle recommends that you use unicast for messaging within a cluster. For WebLogic Server versions 9.2 and earlier, you must use multicast for communications between clusters.

IP multicast broadcasts messages to applications, but it does not guarantee that messages are actually received. If an application's local multicast buffer is full, new multicast messages cannot be written to the buffer and the application is not notified when messages are "dropped." Because of this limitation, WebLogic Server instances allow for the possibility that they may occasionally miss messages that were broadcast over IP multicast. Note: A multicast address is an IP address in the range from 224.0.0.0 to 239.255.255.255. The default multicast value used by WebLogic Server is 239.192.0.0. You should not use any multicast address within the range x.0.0.1.

Introduction Traditional network computing applications involve communication between two computers. However, some important emerging applications such as LAN TV, desktop conferencing, corporate broadcasts, and collaborative computing require simultaneous communication between groups of computers. This process is known generically as multipoint communications. Design Considerations There are three ways to design multipoint networking applications: unicast, broadcast, and multicast. Unicast. With a unicast design, applications can send one copy of each packet to each member of the multicast group. This technique is simple to implement, but it has significant scaling restrictions if the group is large. In addition, it requires extra bandwidth, because the same information has to be carried multiple timeseven on shared links. Broadcast. In a broadcast design, applications can send one copy of each packet and address it to a broadcast address. This technique is even simpler than unicast for the application to implement. However, if this technique is used, the network must either stop broadcasts at the LAN boundary (a technique that is frequently used to prevent broadcast storms) or send the broadcast everywhere. Sending the broadcast everywhere is a significant usage of network resources if only a small group actually needed to see the packets. Multicast. With a multicast design, applications can send one copy of each packet and address it to the group of computers that want to receive it. This technique addresses packets to a group of receivers rather than to a single receiver, and it depends on the network to forward the packets to only the networks that need to receive them. Multicast can be implemented at both the data-link layer and the network layer. Ethernet and FDDI, for example, support unicast, multicast, and broadcast addresses. An individual computer can listen to a unicast address, several multicast addresses, and the broadcast address. Token Ring also supports the concept of multicast addressing but uses a different technique. Token Rings have functional addresses that can be used to address groups of receivers.

If the scope of an application is limited to a single LAN, using a data-link layer multicast technique is sufficient. However, many multipoint applications are valuable precisely because they are not limited to a single LAN. When a multipoint application is extended to an internet consisting of different media types, such as Ethernet, Token Ring, FDDI, ATM, Frame Relay, SMDS, and other networking technologies, it is best to implement multicast at the network layer. There are several parameters that the network layer must define in order to support multicast communications: Addressing. There must be a network-layer address that is used to communicate with a group of receivers rather than a single receiver. In addition, there must be a mechanism for mapping this address onto data-link layer multicast addresses where they exist. Dynamic registration. There must be a mechanism for the computer to communicate to the network that it is a member of a particular group. Without this ability, the network cannot know which networks need to receive traffic for each group. Multicast routing. The network must be able to build packet distribution trees that allow sources to send packets to all receivers. A primary goal of these packet distribution trees is to ensure that each packet exists only one time on any given network (that is, if there are multiple receivers on a given branch, there should only be one copy of the packets on that branch). IP Multicast The Internet Engineering Task Force has been developing standards that address each of the issues described above. Addressing. The IP address space is divided into four pieces: Class A, Class B, Class C, and Class D. Classes A, B, and C are used for unicast traffic. Class D is reserved for multicast traffic. Class D addresses are allocated dynamically. Dynamic registration. RFC 1112 defines the Internet Group Membership Protocol (IGMP). IGMP specifies how the host should inform the network that it is a member of a particular multicast group. Multicast routing. There are several standards available for routing IP Multicast traffic

WebLogic IP Multicast : A primer In Oracle WebLogic (formerly BEA WebLogic) versions prior to version 10.0, WebLogic Servers relied on IP multicast to ensure cluster membership (versions 10.0 and later provide the alternative of Unicast which is preferred over Multicast). This article will pertain to IP multicast used by WebLogic. What is IP multicast? IP multicast is a technology used to broadcast data (or datagrams) across a network using IP. For IP multicasting, certain special IP addresses called multicast addresses are defined. According to the

Internet Assigned Numbers Authority (IANA), its RFC3171 guidelines specify that addresses 224.0.0.0 to 239.255.255.255 are designated as multicast addresses. A multicast address is associated with a group of receivers. When a sender wishes to send a datagram to a group of receivers using IP multicast, it will send the datagram to the multicast address/port associated with that group of receivers. When routers or switches on the network receive the datagram, they know which servers (receivers) are associated with the multicast address (using IGMP) and so they make copies of the datagram and send a copy to every registered receiver. This is illustrated in the figure below:

Why does WebLogic use IP multicast? A WebLogic Cluster is a group of WebLogic servers which provides similar services, with resilience (if a server crashes and exits the cluster, it can rejoin the cluster later), high availability (if a server in the cluster crashes, other servers in the cluster can continue the provision of services) and load balancing (the load on all servers in a cluster can be uniformly distributed) for an application deployed on a WebLogic cluster. WebLogic makes these beneficial clustering features possible by using IP Multicast for the following: (1) Cluster heartbeats: All servers in a WebLogic cluster must always know which servers are part of the cluster. To make this possible, each server in the cluster uses IP multicast to broadcast regular "heartbeat" messages that advertise its availability.

(2) Cluster-wide JNDI updates: Each WebLogic Server in a cluster uses IP multicast to announce the availability of clustered objects that are deployed or removed locally. How does WebLogic use IP multicast? All servers in a WebLogic Cluster send out multicast fragments (heartbeat messages) from their interface addresses to the multicast IP address and port configured for the WebLogic Cluster. All servers in the cluster are registered with the multicast address and port and so every server in the cluster receives fragments from all other servers in the cluster as well as the fragment it sent out. So, since every server in the cluster sends out fragments every 10 seconds, based on the fragments it receives, it can determine which servers are still part of the cluster. If a server (say Server A) does not receive a fragment from another server in the cluster within 30 seconds (3 multicast heartbeat failures), then it will remove that server from its cluster membership list. When fragments from the removed server start arriving at Server A, then Server A will add the removed server back to its cluster membership list. This way, every server in a WebLogic cluster maintains its own cluster membership list. Regarding cluster-wide JNDI updates, each server instance in the cluster monitors these announcements and updates its local JNDI tree to reflect current deployments of clustered objects. Note: Clustered server instances also monitor IP sockets as a more immediate method of determining when a server instance has failed. The figure below illustrates how a WebLogic cluster uses IP multicast.

How do you configure and test multicast for a WebLogic Cluster? Configuring IP Multicast for a WebLogic Cluster is simple. The steps required are given below: STEP 1: If your WebLogic cluster is part of a network containing other clusters, obtain a multicast address and port for it, from your Network Admins. Understandably, a multicast address and port combination is unique for every WebLogic cluster. Several WebLogic clusters may share the same multicast address if and only if they use different multicast ports. Typically, in organizations, network admins allocate multicast addresses and ports for various projects to ensure there are no conflicts across the network. By default, WebLogic uses a multicast address of 237.0.0.1 and the listen port of the Administration server as the multicast port. STEP 2: Having obtained a multicast address and port for your WebLogic cluster, you must test them before starting your WebLogic cluster to ensure that there are no network glitches and conflicts with other WebLogic clusters. You may do so with the MulticastTest utility provided with the WebLogic installation (part of weblogic.jar). An example test for a cluster containing 2 WebLogic servers on UNIX hosts and using multicast address/port of 237.0.0.1/30000 is given below:

# Command to run on both server hosts (any one of the following within the WebLogic domain directory) # to set the WebLogic domain environment . ./setDomainEnv.sh . ./setEnv.sh

# Command to run on server 1 (within any directory) ${JAVA_HOME}/bin/java utils.MulticastTest -N server1 -A 237.0.0.1 -P 30000

# Command to run on server 2 (within any directory) ${JAVA_HOME}/bin/java utils.MulticastTest -N server2 -A 237.0.0.1 -P 30000

# NOTE: Both java commands must be run on both WebLogic server hosts concurrently.

STEP 3: After successfully testing the multicast address and port, you may use the WebLogic Administration Console to configure multicast for the cluster. Descriptions of various multicast parameters are available on the console itself. The three most important parameters are (1) Multicast Address, (2) Multicast Port and (3) Interface Address. The Interface Address may be left blank if the WebLogic servers use their hosts default interface. On multi-NIC machines or in WebLogic clusters with Network channels, you may have to configure an Interface Address. Given below is a screenshot from a WLS 8.1 SP6 Administration Console indicating the various multicast parameters that may be configured for a cluster. Note that the interface address is on a different screen as it is associated with each server in the cluster, rather than the cluster itself.

After configuring Multicast for a WebLogic cluster, you can monitor the health of the cluster and exchange of multicast fragments among the servers in the cluster by using the WebLogic Administration console. A screenshot of such a monitoring screen with WLS 8.1 SP6 is given below:

Note that the screenshot above indicates that: (1) All servers are participating in the cluster ("Servers" column). (2) Every server in the cluster is aware of every other server in the cluster. The "Known Servers" column is especially useful for large clusters to know exactly which servers are not participating in the cluster.

(3) The total number of fragments received by each server (34) is equal to the sum of all the fragments sent by all the servers in the cluster (17 + 17). Note that the "Fragments Sent" and "Fragments Received" columns on the console need not always indicate a correct relationship even if multicast works fine. Thats because these stats on the console are reset to 0 when servers are restarted. Troubleshooting WebLogics Multicast configuration When you encounter a problem with WebLogic multicast (or any problem for that matter), it is important to confirm the problem by executing as many tests as possible and gather as much data as possible when the problem occurs. For WebLogic multicast, you may confirm the problem by using the MulticastTest utility or checking the Administration console as described above. To troubleshoot WebLogic multicast, refer to the Oracle documentation. Also, check the section below to determine if the problem youve encountered is similar to one of the problems described, to provide you with a quick resolution. WebLogic Multicast eureka! Given below are WebLogic multicast problems which Ive encountered and investigated, along with solutions that worked: PROBLEM 1: SYMPTOMS: All WebLogic servers could not see any other server in the cluster. Tests using the MulticastTest utility failed indicating that all servers could only receive the multicast fragments which they sent out. ANALYSIS: The MulticastTest utility was tried with the correct multicast address, multicast port and interface address. No conflict with any other cluster was observed, but no messages were received from other servers. Assuming that all servers in the cluster are not hung, the symptoms indicate a problem with the underlying network or the multicast configuration on the network. SOLUTIONS: Solution 1: The Network Admin just gave us another multicast address/port pair and multicast tests worked. The multicast address/port pair which failed was not registered correctly on the network. Solution 2: The Network Admin informed us that more than one switch was used on the cluster network and this configuration did not ensure that multicast fragments sent by one server in the cluster were copied and transmitted to other servers in the cluster. Refer to this CISCO document for details regarding this problem and its solutions. As a tactical solution, the Network Admin configured static multicast MAC entries on the switches (Solution 4 in the CISCO document). This tactical solution requires the Network Admin to maintain those static entries, but since there werent too many WebLogic clusters using multicast on the network, this solution was chosen. Solution 3: The two managed servers in a cluster were in geographically separated data centres and several hops across the network were required for the servers to receive each others multicast

fragments. Increasing the multicast TTL solved this problem and both the MulticastTest utility and the WebLogic servers successfully multicasted. PROBLEM 2: SYMPTOMS: The following errors were seen in the WebLogic managed server logs and the managed servers did not start. did not start. Exception:weblogic.server.ServerLifecycleException: Failed to listen on multicast address weblogic.server.ServerLifecycleException: Failed to listen on multicast address at weblogic.cluster.ClusterCommunicationService.initialize()V(ClusterCommunicationService.java:48) at weblogic.t3.srvr.T3Srvr.initializeHere()V(T3Srvr.java:923) at weblogic.t3.srvr.T3Srvr.initialize()V(T3Srvr.java:669) at weblogic.t3.srvr.T3Srvr.run([Ljava/lang/String;)I(T3Srvr.java:343) at weblogic.Server.main([Ljava/lang/String;)V(Server.java:32) Caused by: java.net.BindException: Cannot assign requested address at jrockit.net.SocketNativeIO.setMulticastAddress(ILjava/net/InetAddress;)V(Unknown Source) at jrockit.net.SocketNativeIO.setMulticastAddress(Ljava/io/FileDescriptor;Ljava/net/InetAddress;)V(U nknown Source)

ANALYSIS: The errors occured irrespective of whichever multicast address/pair was used. The error indicates that the WebLogic server could not bind to an address to send datagrams to the multicast address. i.e. it could not bind to its Interface Address SOLUTION: The WebLogic server host was a multi-NIC machine and another interface had to be specified for communication with the multicast address/port. Specifying the correct interface address solved the problem.

PROBLEM 3: SYMPTOMS: The following errors were seen in the WebLogic managed server logs. The managed servers were running, but clustering features (like JNDI replication) were not working.

####<May 20, 2008 4:00:58 AM BST> <Error> <Cluster> <kips1host> <kips1_managed1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1211252458100> <BEA-000170> <Server kips1_managed1 did not receive the multicast packets that were sent by itself> ####<May 20, 2008 4:00:58 AM BST> <Critical> <Health> <kips1host> <kips1_managed1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1211252458100> <BEA-310006> <Critical Subsystem Cluster has failed. Setting server state to FAILED. Reason: Unable to receive self generated multicast messages>

ANALYSIS: The errors above indicate that the WebLogic server kips_managed1 could not receive its own multicast fragments from the multicast address/port. Probably, the servers multicast fragments did not reach the multicast address/port in the first place and this points to an issue with the configuration of the interface address or route from interface address to multicast address/port or multicast address/port (most likely the interface address or route as if the multicast address/port were wrong, the server would not have received multicast fragments from any other server, as in PROBLEM 1). SOLUTION: The WebLogic server kips_managed1 used -Dhttps.nonProxyHosts and -Dhttp.nonProxyHosts JVM parameters in its server startup command and these parameter values did not contain the name of the host which hosted kips_managed1. After including the relevant hostname in these parameter values, the errors stopped occurring. I am not sure how these HTTP proxy parameters affected selfgenerated multicast messages (will try to investigate this). PROBLEM 4: SYMPTOMS: All WebLogic servers were often (not always) part of a cluster and intermittently, servers in the cluster were removed and added and the LostMulticastMessageCount was increasing for some servers in the cluster. However, tests using the MulticastTest utility (when the cluster was stopped) were successful. ANALYSIS: The problem occurred intermittently when the WebLogic servers were running, but never occurred when using the MulticastTest utility. This indicates that the underlying IP multicast works fine and something is preventing the servers in the cluster from IP multicasting properly. Further analysis revealed that the servers had issues with JVM Garbage Collection with long stop-the-world pauses (> 30 secs) during which the JVM did absolutely nothing else other than garbage collection. Also, the times of occurrences of these long GC pauses correlated with the times of increases in LostMulticastMessageCount and removal of servers from the cluster. SOLUTION: The JVMs hosting the WebLogic servers were tuned to minimize stop-the-world GC pauses to allow the servers to multicast properly. For the specific GC problem I encountered, you may refer to the tuning details here.

Viewer Comments: I recommend the following: (1) If you do get a maintenance window, then use the multicast test utility for all managed servers for at least 10 minutes and check if every server can receive packets from every other server as well as its own packets. (2) In order to get more information in the server logs, you could enable the following debug flags: Set these flags from the command line during server startup by adding the following options: * -Dweblogic.debug.DebugCluster=true * -Dweblogic.debug.DebugClusterHeartBeats=true * -Dweblogic.debug.DebugClusterFragments=true

MulticastTest

The MulticastTest utility helps you debug multicast problems when configuring a WebLogic Cluster. The utility sends out multicast packets and returns information about how effectively multicast is working on your network. Specifically, MulticastTest displays the following types of information via standard output: A confirmation and sequence ID for each message sent out by the current server. The sequence and sender ID of each message received from any clustered server, including the current server. A missed-sequenced warning when a message is received out of sequence. A missed-message warning when an expected message is not received.

To use MulticastTest, start one copy of the utility on each node on which you want to test multicast traffic. WARNING: Do NOT run the MulticastTest utility by specifying the same multicast address (the -a parameter) as that of a currently running WebLogic Cluster. The utility is intended to verify that multicast is functioning properly before starting your clustered WebLogic Servers.

For information about setting up multicast, see the configuration documentation for the operating system and hardware of the WebLogic Server host machine. For more information about configuring a cluster Syntax $ java utils.MulticastTest -n name -a address [-p portnumber] [-t timeout] [-s send]

MulticastTest Arguments Argument -n name Definition Required. A name that identifies the sender of the sequenced messages. Use a different name for each test process you start.

-a address

The multicast address on which: (a) the sequenced messages should be broadcast; and (b) the servers in the clusters are communicating with each other. (The default is 237.0.0.1.)

-p portnumber

Optional. The multicast port on which all the servers in the cluster are communicating. (The multicast port is the same as the listen port set for WebLogic Server, which defaults to 7001 if unset.)

-t timeout

Optional. Idle timeout, in seconds, if no multicast messages are received. If unset, the default is 600 seconds (10 minutes). If a timeout is exceeded, a positive confirmation of the timeout is sent to stdout.

-s send

Optional. Interval, in seconds, between sends. If unset, the default is 2 seconds. A positive confirmation of each message sent out is sent to stdout.

Open 2 windows and run the same command in both windows but give diff names Example $ e:\Bea\user_projects\domains\Gui_domain>java utils.MulticastTest -N Server1 -a 239.192.0.1 ***** WARNING ***** WARNING ***** WARNING ***** Do NOT use the same multicast address as a running WLS cluster. Starting test. Hit any key to abort Using multicast address 239.192.0.1:7001 Will send messages under the name Server1 every 2 seconds Will print warning every 600 seconds if no messages are received I (Server1) sent message num 1 New Neighbor Server2 found on message number 1

Received message 2 from Server1 I (Server1) sent message num 2 Received message 2 from Server2 Received message 3 from Server1 I (Server1) sent message num 3 Received message 3 from Server2 Received message 4 from Server1 I (Server1) sent message num 4 Received message 4 from Server2 Received message 5 from Server1 I (Server1) sent message num 5 e:\Bea\user_projects\domains\Gui_domain>java utils.MulticastTest -N Server2 -a 239.192.0.1 ***** WARNING ***** WARNING ***** WARNING ***** Do NOT use the same multicast address as a running WLS cluster. Starting test. Hit any key to abort Using multicast address 239.192.0.1:7001 Will send messages under the name Server2 every 2 seconds Will print warning every 600 seconds if no messages are received New Neighbor Server1 found on message number 1 I (Server2) sent message num 1 Received message 2 from Server1 Received message 2 from Server2 I (Server2) sent message num 2 Received message 3 from Server1 Received message 3 from Server2 I (Server2) sent message num 3 Received message 4 from Server1

Received message 4 from Server2 I (Server2) sent message num 4 Received message 5 from Server1 Received message 5 from Server2 I (Server2) sent message num 5

MulticastMonitor The MulticastMontior utility monitors multicast traffic of a cluster in a given domain. It listens on the multicast address and port of the cluster and prints all the multicast messages sent on it.

Syntax $ java weblogic.cluster.MulticastMonitor <multicast_address> <multicast_port> <domain_name> <cluster_name> [<domaindir>]

MulticastMonitor Arguments Argument multicast_address Description The multicast address on which the servers in the clusters are communicating with each other.

multicast_port

The multicast port on which all the servers in the cluster are communicating.

domain_name

The name of the domain to which the monitored cluster belongs.

cluster_name

The name of the cluster to monitor.

domaindir

Optional. The name of the domain root directory. This is needed if the multicast messages are encrypted.

Please make both server are running before you test the below command to monitor Multicast Example:

e:\Bea\user_projects\domains\Gui_domain>java weblogic.cluster.MulticastMonitor 239.192.0.0 9001 Gui_domain Cluster1 Received heartbeat message of size 224 from Server1 @ Mon Jul 18 17:58:52 IST 2011 messageVersion:0,0,0 seqNum:54 fragment # 0 Received heartbeat message of size 224 from Server2 @ Mon Jul 18 17:58:57 IST 2011 messageVersion:0,0,0 seqNum:48 fragment # 0 Received heartbeat message of size 224 from Server1 @ Mon Jul 18 17:59:01 IST 2011 messageVersion:0,0,0 seqNum:55 fragment # 0 Received heartbeat message of size 224 from Server2 @ Mon Jul 18 17:59:07 IST 2011 messageVersion:0,0,0 seqNum:49 fragment # 0 Received heartbeat message of size 224 from Server1 @ Mon Jul 18 17:59:11 IST 2011 messageVersion:0,0,0 seqNum:56 fragment # 0 Received heartbeat message of size 224 from Server2 @ Mon Jul 18 17:59:17 IST 2011 messageVersion:0,0,0 seqNum:50 fragment # 0 Received heartbeat message of size 224 from Server1 @ Mon Jul 18 17:59:21 IST 2011 messageVersion:0,0,0 seqNum:57 fragment # 0 Received heartbeat message of size 224 from Server2 @ Mon Jul 18 17:59:27 IST 2011 messageVersion:0,0,0 seqNum:51 fragment # 0