XYZ Corporation

XYZ Directory Operational Runbook August 24, 2006

Table of Contents
1. DOCUMENT CONTROL.............................................................................................................5 2. INTRODUCTION/EXECUTIVE OVERVIEW................................................................................6 3. SCOPE........................................................................................................................................6 4. REFERENCES............................................................................................................................6 5. APPLICATION DESCRIPTION...................................................................................................7 5.1. BUSINESS OVERVIEW..................................................................................................................................7 5.2. TECHNICAL OVERVIEW...............................................................................................................................7 6. APPLICATION SYSTEM ARCHITECTURE................................................................................8 6.1. HARDWARE...............................................................................................................................................9 6.2. SOFTWARE..............................................................................................................................................10 6.3. DATABASE..............................................................................................................................................10 6.4. NETWORKING..........................................................................................................................................10 7. ON-LINE OPERATIONS............................................................................................................11 7.1. STARTUP SEQUENCE FOR WEBLOGIC SERVER..............................................................................................11 7.1.1. Starting Server from Windows Service Console............................................................11 7.2. WEBLOGIC ADMINISTRATION CONSOLE......................................................................................................13 7.3. SHUTDOWN SEQUENCE FOR WEBLOGIC SERVER...........................................................................................15 7.4. SHUTTING DOWN FROM THE CONTROL PANEL..............................................................................................15 7.5. SHUTTING DOWN FROM THE WEBLOGIC ADMINISTRATION CONSOLE...............................................................16 7.6. SETTING UP NEW USERS FOR THE XYZDIR APPLICATION............................................................................17 7.7. ADDING A WEBLOGIC USER TO THE XYZDIRUSERS GROUP..........................................................................21 8. TROUBLE SHOOTING PROCEDURES....................................................................................25 9. SANITY TEST...........................................................................................................................25 10. BATCH PROCESSING............................................................................................................26 11. CHANGE MANAGEMENT......................................................................................................27 12. PROBLEM MANAGEMENT....................................................................................................27 13. BUSINESS SYSTEM MONITORING.......................................................................................27 13.1. MONITORING XYZDIR SYSTEM VIA WEBLOGIC LOGGING COMPONENT ......................................................27 14. SECURITY.............................................................................................................................37 15. BACKUP AND RECOVERY....................................................................................................37 16. DISASTER RECOVERY..........................................................................................................37 17. SERVICE LEVEL AGREEMENTS...........................................................................................37

XYZ and ABC Confidential

XYZ Directory Operational Runbook

Table of Figures
Table of Contents.................................................................................................................2 Table of Figures....................................................................................................................3 List of Tables........................................................................................................................4 1. DOCUMENT CONTROL.............................................................................................................5 Document Location........................................................................................................5 2. INTRODUCTION/EXECUTIVE OVERVIEW................................................................................6 3. SCOPE........................................................................................................................................6 4. REFERENCES............................................................................................................................6 5. APPLICATION DESCRIPTION...................................................................................................7 5.1. BUSINESS OVERVIEW ..................................................................................................................................7 5.2. TECHNICAL OVERVIEW...............................................................................................................................7 6. APPLICATION SYSTEM ARCHITECTURE................................................................................8 6.1. HARDWARE...............................................................................................................................................9 6.2. SOFTWARE..............................................................................................................................................10 6.3. DATABASE..............................................................................................................................................10 6.4. NETWORKING..........................................................................................................................................10 7. ON-LINE OPERATIONS............................................................................................................11 7.1. STARTUP SEQUENCE FOR WEBLOGIC SERVER..............................................................................................11 7.1.1. Starting Server from Windows Service Console............................................................11 Starting Server from Start Menu...........................................................................12 7.2. WEBLOGIC ADMINISTRATION CONSOLE......................................................................................................13 7.3. SHUTDOWN SEQUENCE FOR WEBLOGIC SERVER...........................................................................................15 7.4. SHUTTING DOWN FROM THE CONTROL PANEL..............................................................................................15 7.5. SHUTTING DOWN FROM THE WEBLOGIC ADMINISTRATION CONSOLE...............................................................16 7.6. SETTING UP NEW USERS FOR THE XYZDIR APPLICATION............................................................................17 7.7. ADDING A WEBLOGIC USER TO THE XYZDIRUSERS GROUP..........................................................................21 8. TROUBLE SHOOTING PROCEDURES....................................................................................25 9. SANITY TEST...........................................................................................................................25 10. BATCH PROCESSING............................................................................................................26 11. CHANGE MANAGEMENT......................................................................................................27 12. PROBLEM MANAGEMENT....................................................................................................27 13. BUSINESS SYSTEM MONITORING.......................................................................................27 13.1. MONITORING XYZDIR SYSTEM VIA WEBLOGIC LOGGING COMPONENT ......................................................27 Message Severity:.............................................................................................28 Log Messages from XYZDir Application............................................................28 Configuring Logging from the WebLogic Administration Console..........................29 14. SECURITY.............................................................................................................................37 15. BACKUP AND RECOVERY....................................................................................................37
XYZ and ABC Confidential

XYZ Directory Operational Runbook

16. DISASTER RECOVERY..........................................................................................................37 17. SERVICE LEVEL AGREEMENTS...........................................................................................37

List of Tables

TABLE 1 HARDWARE.................................................................................................................9 TABLE 2 SOFTWARE...............................................................................................................10 TABLE 3 - DATABASE.................................................................................................................10 TABLE 4 - TROUBLESHOOTING.................................................................................................25 TABLE 5 - BATCH PROCESSES.................................................................................................26

XYZ and ABC Confidential

XYZ Directory Operational Runbook

1. Document Control
The document control section describes the revision history and summary of changes made in the document. It will serve as the version control for the contents of the document. Document Location The source of the document will be found on xxxx. Revision History Revision Number 1. 2. 3. 4. 5. 6. Revision Date 06/14/06 06/18/06 06/25/06 07/23/06 08/09/06 08/14/06 Draft Draft Draft Draft Final Draft Page 24 added daily to Sun-Sat batch jobs Summary of Changes Made Changed By

XYZ and ABC Confidential

XYZ Directory Operational Runbook

2. Introduction/Executive Overview
The main objective of XYZ Directory is to provide a central repository for contact information for XYZ employees and external clients. XYZ Directory resides at 123 on an Oracle 7.3.4 database that provides internal and external contact information to web based applications via Lightweight Directory Access Protocol (LDAP). XYZ Directory utilizes the application server, BEA WebLogic Server, as a layer between the user interface and XYZ Directory database. The XYZ Directory database receives data from both the MProfile database and HR PeopleSoft application. Web based applications such as XYZ.com can access this vital contact information by invoking the LDAP service. Changes can also be requested via the XYZ Directory application.

3. Scope
The purpose of this Runbook is to provide operational documentation for the XYZ Directory application. It will facilitate the management of the application in the most efficient manner possible, and thereby maximize the value of the application to XYZ, Incorporated. The primary audience of the document is the operator at 123. Operators are members of the Data Center who perform the routine system administration tasks. (i.e. site management, system monitoring, routine system maintenance). It should be used to understand the practices and procedures to be adhered to for the application. The document contains a business and technical description of the application, online operations, and troubleshooting techniques. The business and technical overview provide a high level description of the functional capabilities of the application and how it is used. The system architecture section describes the application topology. The online operations, troubleshooting and batch processing sections details specific instructions for operation and maintenance. The document also contains Appendices that address the Enterprise wide strategy/approach for standard practices such as change management, problem management, monitoring and escalation.

4. References
This table provides key contacts regarding XYZ Directory issues that may arise. Issue Networking Database Application Reference Organization XYZ I&O Group XYZ DBAs XYZ Contact Name Patrick McGee Stan Tike Manny Phillips

Please refer to Appendix 1 Contact List for additional information.

XYZ and ABC Confidential

XYZ Directory Operational Runbook

5. Application Description
The Application Description Section gives a detailed description of the application. It is organized into business and technical descriptions:

5.1. Business Overview

The main objective of XYZ Directory is to provide a central repository for contact information for XYZ employees and external clients to applications such as web based applications such as XYZ.com.

5.2. Technical Overview

The main client of the XYZ Directory database is XYZ.com. The database resides at 123 and runs on Oracle 7.3.4 and applications such as XYZ.com can access this data via LDAP. Users of XYZ.com do not have direct access to XYZ Directory. XYZ.com users can only access XYZ Directory information via applications running on their web browser. XYZ.com accesses XYZ Directory by using the BEA WebLogic Application Server. BEA WebLogic serves as the layer between the user interface and the system or database. The PeopleSoft HR application sends personnel data to XYZ Directory by using PowerMart, the data transformation engine. HR PeopleSoft is the input source and based on business rules, the information is modified before loading into the XYZ Directory database. Both the MProfile database and HR PeopleSoft application populate the XYZ Directory database. Applications such as XYZ.com, InMind, Edward, MMC Directory, Surety, and CANSYS may request data from XYZ Directory via the use of LDAP.

XYZ and ABC Confidential

XYZ Directory Operational Runbook

The following diagram displays the data flow between MProfile, XYZ Directory and LDAP.

Other Applications

External Client Contacts Added/Updated via MDIR API

MProfile

Internal Contacts External Contacts added via Perl transformation and load

XYZ Marsh
Directory

Internal Contacts External Contacts

eDirectory (LDAP)

HR PeopleSoft Web Based Apps (InMind, Edward), BBC MMC Directory, Surety, CANSYS

Figure 1. Data Flow between MProfile, XYZ Directory and LDAP

6. Application System Architecture

XYZ Directory is comprised of the following three major components: an Oracle 7.3.4 database, a BEA WebLogic application server and LDAP. The BEA WebLogic application server contains the business logic for the XYZ Directory application. It serves static and dynamic web pages as well as managing database access, security, and transaction services for XYZ Directory. LDAP (Lightweight Directory Access Protocol) is the standard protocol for maintaining and distributing directory information over the Internet. The main objective for the LDAP directory service within XYZ is to provide a standard mechanism for storing, distributing and accessing directory information. The HR PeopleSoft application and MProfile database both send personnel data such as names and addresses to XYZ Directory.

XYZ and ABC Confidential

XYZ Directory Operational Runbook

The following diagram displays the components of the XYZ Directory production environment.

XYZ Directory MarshDIRECTORY

www.xyz.com
www.marsh.com

Application Server (WebLogic API) USNYCAS67

XYZ
MPROFILE Sybase 11.0.3

Marsh DIR

Oracle 7.3.4

Staging Instance LDAP Server USNYCU09

Master Instance

Figure 2. XYZ Directory Production Environment

6.1. Hardware
Server Name USNYCAS67 USNYCU09 Nantucket Purpose XYZDirectory Application Server (BEA WebLogic) LDAP Server XYZDirectory DB (Oracle 7.3.4) Table 1 Hardware Refer to Appendix 2 List of Servers for additional information. Location 123 123 123

XYZ and ABC Confidential

XYZ Directory Operational Runbook

6.2. Software
Server USNYCAS67 Nantucket All Software Residing On This Server BEA WebLogic 6.1 sp2 Oracle Client NT 4.00.1281 Oracle 7.3.4 Table 2 Software

6.3. Database
Database Instance Name Oltp15 DB Type Oracle Version 7.3.4 Table 3 - Database Refer to Appendix 2 List of Servers for additional information. Server Name/Location NANTUCKET / Andover

6.4. Networking
Refer to Appendix 8.1 123 Data Center Network Environment for additional information.

XYZ and ABC Confidential

10

XYZ Directory Operational Runbook

7. On-line Operations
XYZ Directory is a 24 by 7 application that has no scheduled downtime.

7.1. Startup Sequence for WebLogic Server 7.1.1. Starting Server from Windows Service Console
Note: All references to myserver in the screens should be XYZdir. 1. Power up the server first.

2. The XYZDir application on WebLogic is setup as a Windows NT service. It will start automatically after a reboot. To ensure that the service is up and running, launch Windows Control Panel Services, in the row for service XYZdir, and make sure the status says Started.

<Screen: XYZdir in Started Status> 3. From the above screen, if the status is blank, then click the Startup button from the screen, the following screen will be displayed, check Allow Service to interact with Desktop, and Click OK button.

<Screen: Setting XYZdir service to be interactive>

XYZ and ABC Confidential

11

XYZ Directory Operational Runbook

Then, XYZdir screen will be displayed again. Highlight the service XYZdir, click Start button, and a console window with the following information will be displayed:

<Screen: XYZDir WebLogic Server Startup Window> Starting Server from Start Menu We can create and use the Windows shell folder and shortcuts for XYZ directory WebLogic tasks such as installing the XYZDir application as a windows service, uninstalling the service, starting WebLogic for XYZDir application and cache refreshing. To setup the Windows shell folder and shortcuts for XYZ Directory, go to C:\WINNT\Profiles\All Users\Start Menu\Programs. Then, copy the folder XYZDir WebLogic based on the supplied XYZDir WebLogic.zip file.

XYZ and ABC Confidential

12

XYZ Directory Operational Runbook

The following screen shows the shell folder for the XYZ Directory WebLogic:

<Screen: Windows Shell Folder for XYZDir WebLogic> Then a list of shortcuts for XYZDir WebLogic are setup as shown in the screen below.

<Screen: Shortcuts for XYZDir WebLogic Tasks>

7.2. WebLogic Administration Console

The WebLogic Server Administration Console runs in a Web browser such as Internet Explorer 5. It displays the components of the domain it administers, including clusters and independent WebLogic Servers, in a graphical tree in the left pane. The right pane displays details about the object selected in the left pane such as configuration pages and monitoring pages, or accessing logs. The top-level nodes in the domain tree are containers. If leaf nodes are present in those containers, you can click on the plus sign at the left to expand the tree to access the leaf nodes. Below is a sample snapshot from an Administration Console session.

XYZ and ABC Confidential

13

XYZ Directory Operational Runbook

<Screen: WebLogic Administration Console>

Launch WebLogic administration console: After starting the WebLogic Server, we can start the Administration Console by directing web browser to a URL for WebLogic console administration. To launch the WebLogic administration console using a web browser, follows the steps described below: 1. Launch a web browser such as Internet Explorer 5, and type the URL as: http://host:7001/console/, where host is the DNS name or IP address of the server which hosts the XYZDir WebLogic.

2. Then a login page will be displayed:

XYZ and ABC Confidential

14

XYZ Directory Operational Runbook

<Screen: Login page for WebLogic Administration Console > 3. Type in the user name and password, and click OK button. It is recommended to use a powerful user Id such as system or a user Id in WebLogic Administrators group to login to WebLogic administration console for administrative tasks.

7.3. Shutdown Sequence for WebLogic Server

The XYZDir WebLogic server can be shutdown by stopping the NT service XYZdir from the control panel, or by using the WebLogic administration console.

7.4. Shutting Down from the Control Panel

1. Launch Windows Control Panel Services, highlight the row for service XYZdir, then click the Stop button.

<Screen: Stop XYZdir Service>

2. The following confirmation window will be displayed:

XYZ and ABC Confidential

15

XYZ Directory Operational Runbook

<Screen: Confirm Stopping XYZdir Service> 3. Confirm the shutdown by clicking the Yes button.

7.5. Shutting Down from the WebLogic Administration Console

1. Launch and connect to the WebLogic Administration console as described in WebLogic Administration Console section. 2. On the left pane, expand the node Servers, and select XYZdir. 3. On the right pane, select the Monitoring tab as shown in the snapshot below. <Screen: Monitoring Tab for XYZdir>

XYZ and ABC Confidential

16

XYZ Directory Operational Runbook

4. Follow the link called Shutdown this Server and the following confirmation screen will be displayed:

<Screen: Confirming Shutting Down Server> 5. Click the Yes button to confirm. When WebLogic is deployed as one server, that is, as administration server and managed server, shutting down the server means shutting down both the administration server and managed server as well. No administrative work with WebLogic can be performed after shutting down the server. If we need to start the WebLogic server again, we will need to start the XYZdir service from Control Panel Services as described in the Starting Server from Windows Service Console section. If the administration server and managed servers are deployed and started separately, shutting down a managed server should not affect the administration server.

7.6. Setting Up New Users for the XYZDir Application

In order to use XYZ Directory EJBs, users, mostly client applications should be the members of a pre-defined Weblogic group called XYZdirusers. Setting up a user in order to use the XYZDir application involves creating a new WebLogic user Id, and adding the user to the XYZdirusers group. The details are described as follows: 1. Login into the WebLogic administration console with a browser, preferably Internet Explorer 5 using the following url: http://<host>:7001/console/, where host is the DNS name or IP address of the WebLogic host machine. 2. On the left pane of the WebLogic administration console, click on the tree node: /Security/Users.

XYZ and ABC Confidential

17

XYZ Directory Operational Runbook

3. The following screen will be displayed:

<Screen: Security-Users > 4. On the right side of the screen, under Create a New User, fill in the fields of name, password, confirm password, then click Create. 5. The following screen will be displayed:

XYZ and ABC Confidential

18

XYZ Directory Operational Runbook

<Screen: Saving a New User> 6. On the right side of the above screen, click the link saved to the realm implementation to make the new user information consistent. 7. The following screen will appear:

XYZ and ABC Confidential

19

XYZ Directory Operational Runbook

XYZ and ABC Confidential

20

XYZ Directory Operational Runbook

<Screen: Confirming Saving a New User> 8. Click the Yes button to confirm the change.

7.7. Adding a WebLogic User to the XYZdirusers Group

1. On the left pane of the WebLogic administration console, click on the tree node: /Security/Groups, a list of groups will be displayed on the right side as shown in the following screen print:

XYZ and ABC Confidential

21

XYZ Directory Operational Runbook

<Screen: Security-Groups> 2. Follow the link for the group name XYZdirusers. 3. The following screen will be displayed:

XYZ and ABC Confidential

22

XYZ Directory Operational Runbook

<Screen: Security - XYZdirusers> 4. Fill in the field Users with the WebLogic user to be added to the group. 5. Click the Add Members button. 6. Then, the following screen will be displayed to verify whether the change should be made.

XYZ and ABC Confidential

23

XYZ Directory Operational Runbook

<Screen: Saving Membership>

XYZ and ABC Confidential

24

XYZ Directory Operational Runbook

8. Trouble Shooting Procedures

The following table lists the possible error messages while running the test program (runtest.bat). This test program is located at D:\XYZdir. Error Description Failed initial connection !! Com.XYZmc.XYZdir.util.FatalException: failed to connect to the applications server.null at com.XYZmc.XYZdir.client.XYZdirOperations.connect( XYZdirOperations.java:411) at com.XYZmc.XYZdir.client.XYZOperations.<init>(XYZ Operations.java:140) at com.XYZmc.XYZdir.client.XYZOperations.main (XYZOperations.java:554) <Jun 21, 2001 11:19:57 AM EDT> <Info> <ConnectionManager> <Bootstrap unable to get a direct: 'Protocol: 't3''connection to: '0S:127.0.0.1: [7001,-1,-1,-1,-1,-1,-1]' on port: '7001' java.rmi.ConnectException: Destination unreachable; nested exception is: java.net.ConnectException: Connection refused: no further information Table 4 - Troubleshooting Result Exits Test Program Possible Causes Wrong User for WebLogic connection Wrong password for WebLogic connection Wrong host or protocol, in general incorrect WebLogic URL

WebLogic server is not running at the specified URL WebLogic server is not ready yet to accept calls from clients

9. Sanity Test
The best test to see whether the WebLogic server is running is trying to connect with a browser to the administration console. In the browser put the address: http://host:port/console where host and port are WebLogic host and port configured for this instance. If the console does not show up, a good way to troubleshoot is to launch the server using the desktop interaction as described in the Starting Server from Windows Service Console section. Remember the WebLogic log file can be used even if the server didnt start and we are not able to use the console.

XYZ and ABC Confidential

25

XYZ Directory Operational Runbook

10. Batch Processing

The following table displays all the batch processes associated with XYZ Directory. Batch Name Description Filename and Location (Server Name, Directory) m2mprocess.prl Nantucket, /var/opt/XYZdir/bi n b_HRPeopleSoft _Load usnycu79 Scheduling Tool Used Crontab Day & Time Batch is Run Daily; SunSat 4:30 AM Language Written Perl

Mprofile to XYZDIR

PeopleSoft to XYZDIR

GINA/MNA to XYZDIR

XYZDIR to Flat File

XYZDIR to LDAP Refresh

Synchronizes MDIR with updated information from the Mprofile system Updates XYZDIR staging tables with the latest PeopleSoft data. This is a PowerMart process which consists of multiple mappings and transformation components. All are self contained within the PowerMart servers. Updates XYZDIR staging tables with the latest Company & Client data. This also is a PowerMart process. Creates a current subset image of XYZDIR data for use by other XYZ Applications. Refresh job synchronizes the LDAP directory with changes in XYZDIR.

PowerMart Scheduler

Daily; SunSat 3:00 AM

PowerMart

b_xyzdir_gina_de lta usnycu79

PowerMart Scheduler

Tue, Wed, Thu, Fri 7:00 AM

PowerMart

b_xyzdir_extract usnycu79

PowerMart Scheduler

Daily; SunSat 7:00 AM Daily; SunSat 6:30 AM

PowerMart

run.csh usnycu09,/opt/net scape/server4/XY ZDIR/Apps/Ldap Batch

Crontab

Java

Table 5 - Batch Processes

XYZ and ABC Confidential

26

XYZ Directory Operational Runbook

11. Change Management

123 handles change management for XYZ Directory. There are no special change management procedures related to XYZ Directory. Refer to Appendix 3.3 123 Change Management Control Procedure for additional information.

12. Problem Management

123 handles problem management for XYZ Directory. There are no special problem management procedures related to XYZ Directory. Refer to Appendix 4.2 123 Problem Management for additional information.

13. Business System Monitoring 13.1. Monitoring XYZDir System via WebLogic Logging Component
XYZDir WebLogic Log Messages Log messages in WebLogic is a useful way to analyze systems and detect problems, and to track down the source of a fault. Log messages generated by native WebLogic and XYZDir are all stored in server log files under folder: <WebLogic installation directory>\config\mydomain\logs: access.log: messages for server access; wl-domain.log: filtered important messages from servers to domain; WebLogic.log: messages for server subsystems. These files are the source for the logging component of WebLogic administration console, they can be examined for possible problems. Among the three type of log files, server log file WebLogic.log is the one examined most. Each log message in a server log file consists of the values of the attributes described below: Message Attributes Timestamp: The time and date when the message originated, in a format that is specific to the locale. Severity: Indicates the degree of impact or seriousness of the event reported by the message. See the details for Message Severity described below. Subsystem: This attribute denotes the particular subsystem of WebLogic Server that was the source of the message. For example, XYZDirAPI, EJB, RMI, JMS. Server Name: For this deployment, the server name is XYZdir. Machine Name: The machine which hosts the XYZDir WebLogic. Thread ID:

XYZ and ABC Confidential

27

XYZ Directory Operational Runbook

 Transaction ID: These four attributes identify the origins of the message. Transaction ID is present only for messages logged within the context of a transaction. User ID: The user from the security context when the message was generated.

Message ID: A unique six-digit identifier. Message IDs through 499999 are reserved for WebLogic Server system messages. Message Text: Short Description of the message.

Message Severity: Each WebLogic Server log messages have an attribute called severity which reflects the importance or potential impact on users of the event or condition reported in the message. Defined severities are described below. Severities are listed in order of severity with Emergency being the highest severity. Informational: Used for reporting normal operations.

Warning: A suspicious operation or configuration has occurred but it may not have an impact on normal operation. Error: A user error has occurred. If the subsystem is XYZDirAPI, it indicates XYZDir application has a serious error, it usually needs human involvement to resolve the problem. Otherwise, the error message is from subsystems of WebLogic, The system or application is able to handle the error with no interruption, and limited degradation, of service. Notice: A warning message: A suspicious operation or configuration has occurred which may not affect the normal operation of the server. Critical: A system or service error has occurred. The system is able to recover but there might be a momentary loss, or permanent degradation, of service. Alert: A particular service is in an unusable state while other parts of the system continue to function. Automatic recovery is not possible; the immediate attention of the administrator is needed to resolve the problem. Emergency: The server is in an unusable state. This severity indicates a severe system failure or panic. The following is an example of a log message: ####<Jun 21, 2001 1:59:56 PM EDT> <Info> <Management> <tor_p10827> <XYZdir> <ExecuteThread: '1' for queue: '__weblogic_admin_html_queue'> <system> <> <141039> <MBean=(mydomain:Name=XYZdir,Type=Server) Attribute=(LogRemoteExceptionsEnabled) modified from (false) to (true).> In this example, the message logs the attributes of: Timestamp, Severity, Subsystem, Machine Name, Server Name, Thread ID, User ID, Transaction ID, Message ID, and Message Text. The log message has a severity of Info. Log Messages from XYZDir Application In addition to the messages from native WebLogic subsystems, XYZDirAPI application also logs the following messages to WebLogic server log file (located in D:\XYZDir\log). The possible messages from XYZDirAPI are described as follows:

XYZ and ABC Confidential

28

XYZ Directory Operational Runbook

 <Severity> <Subsystem> <Message Text> <Info> <XYZDirAPI> <XYZDirAPI successfully started ...> <Error> directory> <Error> > <Error> <XYZDirAPI> <XYZDirAPI> <XYZDirAPI> <XYZDIR_001: Cannot get configuration for XYZ < XYZDIR_002: Missing LDAP server connection < XYZDIR_003: Missing Database connection. > < XYZDIR_004: Cannot start topLink server < XYZDIR_005: Missing Sybase < XYZDIR_006: SQL Exception while calling < XYZDIR_007: Exception during WebLogic

<Error> <XYZDirAPI> session ><Error> <XYZDirAPI> connection.> <Error> <XYZDirAPI> Sybase stored procedure.> <Error> <XYZDirAPI> startup.>

Among those XYZDirAPI messages, there is only one message having the severity of Info and message text of XYZDirAPI successfully started .., which indicates that XYZDir WebLogic server has successfully started. This message should be checked from the server log after XYZdir service is started to make sure that the XYZDir application has started successfully. The rest of the messages have severity Error, which indicates a serious error from XYZDir application. Each message text has the format: ErrorCode followed by error description. For Example: XYZDIR_002: Missing LDAP server connection. A severity level Error log message from XYZDirAPI indicates a serious error from XYZDir application, human involvement may be needed to solve the problem. Configuring Logging from the WebLogic Administration Console The WebLogic logging component can be configured via WebLogic administration console. First launch and login to the administration console, then click on the tree node of Servers | XYZdir on the right side, and select the tab of Logging, a screen will be displayed as follows:

XYZ and ABC Confidential

29

XYZ Directory Operational Runbook

<Screen: XYZdir | Logging | General Page> From the general page of XYZdir Logging configuration, we can change the log file name and check log to stdout. From the above screen, click tab Rotation, a rotation configuration screen will be displayed:

XYZ and ABC Confidential

30

XYZ Directory Operational Runbook

<Screen: XYZdir | Logging | Rotation> We can select rotation type of By Time, and select the Time Threshold 24 hours as show above. This means that WebLogic will create a different log file when the duration exceeds the value specified in Time Threshold. The log file names will be: weblogic.log, weblogic.log00001, weblogic.log00002, Query WebLog Server Log Messages via Administration Console Instead of examining the complete list of log messages from WebLogic.log files directly, we can use the WebLogic administration console to filter out the log messages that are of interest to us. The steps are described in detail as follows:

XYZ and ABC Confidential

31

XYZ Directory Operational Runbook

XYZ and ABC Confidential

32

XYZ Directory Operational Runbook

XYZ and ABC Confidential

33

XYZ Directory Operational Runbook

1. Launch and login in to the WebLogic administration console. 2. Click on the tree node Servers | XYZdir.

<Screen: Servers | XYZdir | Logging | General>

XYZ and ABC Confidential

34

XYZ Directory Operational Runbook

3. On the right side of the screen, follow the link of View server log. This side displays the log messages based on the default selection criteria.

<Screen: Log Messages based on Default Criteria Setting> 4. We can configure the selection criteria to select the server log messages that we are interested in. Follow the link of Search and Customize and we can modify the criteria.

XYZ and ABC Confidential

35

XYZ Directory Operational Runbook

<Screen: Customizing Selection Criteria for Log Messages> The above diagram shows how to customize log message criteria: In the Show these columns box, the Chosen box specifies the attributes to appear in each log message; In the Search by subsystem box, the Chosen box specifies the components as sources for log messages; In the Search by severity box, the Chosen box specifies the severity level of the log messages. The messages with higher severity levels will also be displayed. In the Show Messages from last box, specify the duration of the log messages. After specifying all the criteria, click the View button to see the customized list of log messages.

XYZ and ABC Confidential

36

XYZ Directory Operational Runbook

14. Security
There are no specific security measures for XYZ Directory other than those provided by the basic Oracle DB configuration and that which is provided by the 123 data center general security precautions. Security is present via J2EE security model section 7.7. Refer to Appendix 9 Security for additional information.

15. Backup and Recovery

The 123 data center is responsible for backup and recovery of the XYZ Directory server. SQL Backtrack is used to back up the database on a daily basis. A full backup is done around 8pm every day and retained on disk for 1 day. Backup tapes are sent off-site to Iron Mountain every day. XYZ can request data restores by opening a Service Center Ticket. The SLA on a restore is 48 hours. At that point 123 will request that days tape from Iron Mountain. This tape will be delivered to 123 by the next business day. 123 will then restore the tape to an existing server hosted at 123. The person performing the restore will inform the requester, allowing him/her access to the data and close the open Service Center Ticket. Refer to Appendix 6 Backup and Recovery for additional information.

16. Disaster Recovery

There is no formal disaster recovery procedure in place. However, data can be reloaded from tapes, if necessary. Refer to Appendix 7 Disaster Recovery for additional information.

17. Service Level Agreements

Refer to Service Level Agreement for additional information.

XYZ and ABC Confidential

37

XYZ Directory Operational Runbook