BRKCCT-2015 TG Uccx

Introduction to New Features with
Troubleshooting on Unified Contact Center

Express
BRKCCT-2015
Ron Rodriguez
Technical Solutions Manager
Agenda
 Virtualization Best Practices
 Refresh Upgrade
 SE Linux
 Unified Sign On
 Cisco Unified Intelligence Center
 Web Chat Integration with SocialMiner
 Troubleshooting the CCX Database
 CBABU CCX ES Process
BRKCCT-2015 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Virtualization Improvements
Specs-based Virtualization Support
 Support models  Benefits
– UC on UCS Tested Reference – More flexibility in deployments
Configuration (TRC) – Leverage existing investments
– Third-party server Specs-based – Support more deployment models
UC on UCS TRC 3rd Party
Basic Approach Configuration Based Rules Based
Required Level of Medium/Low High

Virtualization/Server Experience
Cisco Tested Hardware? Yes No
Server Model, Hardware Less (tradeoff of less hardware More (tradeoff of more
Choices for more UC predictability) design/test ownership for more
hardware flexibility)
TAC Support for UC Apps? Yes (performance guaranteed) Yes (when all rules followed;
performance guidance only)
TAC Support for Server? Yes No
Single OVA & Resource Reservation
Moving from one profile to another

requires upgrade process
Single OVA
Contains all three profiles
Resource Reservation
900 MHz per vCPU
Boot from SAN
Fiber Channel
ESXi SAN
 SAN Support  Boot from SAN Support
– ESXi 4.0/4.1/5.0 – ESXi 4.1/5.0
– VMWare installed on local ESXi – VMWare installed on FC attached
disks disk array
– UC installed on FC attached disk – UC installed on FC attached disk
array array
vMotion
Fiber Channel
ESXi SAN ESXi

 Move live software to another server during production
 CCX VM must be installed on SAN
 Source and destination physical servers must be connected to same SAN
 VMWare ‘Long Distance vMotion’ not supported
 vMotion for real time load balancing not supported
New Identity
 Export freshly installed (no configuration) CCX VM as an OVA
 Import OVA into another VM host
 Use Answer File Generator for initial configuration
 Import into new CCX machine using CLI
– utils import config
ESXi ESXi
Tips
 Upgrading ESXi from 4.1 to 5.0 may cause slow client TCP problems
– Disable Large Receive Offload Change from 1 to 0:
Net.VmxnetSwLROSL
Net.Vmxnet3SwLRO
Net.Vmxnet3HwLRO
Net.Vmxnet2SwLRO
Net.Vmxnet2HwLRO
 Max configurable IVR ports determined by
the hardware configuration
– OVA template determines port count, regardless
of the number of premium licenses installed
8.x to 9.x Migration – Refresh Upgrade
Why Refresh Upgrade?
 L2 Upgrade  Refresh Upgrade
– From/To sides have same OS – From/To sides have different OS
version versions
– ‘To side’ binaries executed in Active – ‘To side’ binaries executed in
context newly installed RHEL 5 context
– RHEL 5 binaries cannot run in RHEL – Fresh install on inactive partition
4 context – Embedded data migration
Unified CCX
Inactive
Active Partition
Partition
RHEL 4 RHEL 5
8.x.y SUz 9.0.y SUz
User Experience
L2 Upgrade Refresh Upgrade
Install Time 60+ Minutes 90+ Minutes
Reboots None 2 to 4
Call Center Availability Single Node/HA: Up During Upgrade Single Node: Down During Upgrade
HA: Active During Upgrade
On Upgrade Failure Active Side Continues to Run Reboot into Previously Active Version
Displays Failure Notification Email Notification Sent
User Experience Install Log Scrolling via GUI/CLI Blue Install Screen
Dots/Pinwheel during Switch-Version Dots/Pinwheel during Switch-Version
Possible when
refresh upgrade .cop
file is installed
Refresh Upgrade Process
Apply .cop 8.x

Node 1 8.x Active -- 8.x Active
& Reboot Standby
8.x Apply .cop 8.x

Node 2 -- 8.x Active
Standby & Reboot Standby
.cop .cop
allows 8.x allows 8.x
Fully
to Detect CCX Fails to Detect CCX Fails
Functional
L2 or RU Over L2 or RU Over
System
Upgrade Upgrade
Type Type
Reboot & 9.x Inactive
Begin 9.x
Node 1 8.x Active Begin 9.x Install in
Installation Inactive Install Progress
8.x
Node 2 -- -- 8.x Active
Standby
Refresh
Upgrade Inactive CCX Fails
Fully Detected, Installation Over,
Functional Platform begins Functional
System Data from 8.x
Backed Scratch System
Up
9.x Inactive
8.x Reboot into 8.x
Node 1 Install
Standby Completes 8.x Standby
Node 2 8.x Active -- -- 8.x Active
Platform
9.x
Data
Functional Installed Functional
Restored
8.x on 8.x
on
System Inactive System
Inactive
Partition
Partition
8.x
Node 1 -- -- 8.x Active
Standby
Reboot & 9.x Inactive

Begin 9.x
Node 2 8.x Active Begin 9.x Install in
Install Inactive Install Progress
Refresh
Upgrade Inactive CCX Fails
Functional Detected, Installation Over,
8.x Platform begins Functional
System Data from 8.x
Backed Scratch System
Up
Node 1 8.x Active -- -- 8.x Active
9.x Inactive 9.x Inactive

Reboot into 8.x
Node 2 Install in Install
Progress Completes 8.x Standby
Platform
9.x
Data
Functional Installed Functional
Restored
8.x on 8.x
on
System Inactive System
Inactive
Partition
Partition
Initiate
Node 1 8.x Active Switch- 9.x Active 9.x Active
Version
Initiate
8.x 9.x
Node 2 8.x Active Switch-
Standby Version Standby
Functional
Data Data
8.x
Migrated Migrated Functional
System,
to 9.x, to 9.x, 9.x
9.x on
CCX Fails CCX Fails System
Inactive
Over Over
Partition
Tips
 Do not reboot in the middle of a switch-version operation
– Can cause unrecoverable corruption to the database
– Use CLI to perform switch-version, gives notification spinner to user
 Always verify MD5 checksum in release notes to image being used
 DNS setup and configuration is mandatory for CCX 9.0(x)
– Installation will halt without it
 Always take a regular back up before any upgrade
 When upgrading from 5.x/7.x to 8.x/9.x, use the PUT tool
– PUT tools are posted with the new version
– If not PUT tool is posted, use the latest on that train
– Stop the Engine before taking the PUT backup
SELinux
What is Security-Enhanced Linux?
 Implementation of Mandatory Access Control mechanism
 Verifies allowed operations after Discretionary Access Controls checked
 Contained within RHEL kernel
 SELinux policies augmented for UCOS platform
 CCX further augments UCOS policies to implement specific security policies
– CCX 9.0 policies installed during installation
SELinux Modes
 SELinux works in one of the following three modes of operation
– Enforced: Blocking and Logging
– Permissive: Logging only
– Disabled: No blocking or logging
 New CLI commands

– utils os secure status
– utils os secure permissive
– utils os secure enforce
– utils create report security
Requires SFTP server
SELinux Troubleshooting
 For any given issue, if SELinux is suspected as the root cause:
– Set SELinux to Permissive mode
– Reproduce the issue
 If the issue still persists, most likely it is not caused by SELinux
 If the issue does not persist
– Verify the issue is not a legitimate security concern
– Collect the SELinux debugging information
 Set SELinux to Disabled

– Login with remote support (root) credentials (requires reboot)
 Execute /usr/local/platform/bin/controlsel disable
– Customers are not recommended to disable SELinux in their production environment
– Should only be done for short periods of time to overcome blocking scenarios
SELinux Logs
 utils create report security to collect necessary data
 file get activelog syslog/security-diagnostics.tar.gz to
download the data
 What’s in the logs?
– Syslog messages
Nov 17 18:11:04 selinux-uccx user 6 setroubleshoot: SELinux is preventing UCCX_Cvd (uccxd_t)
"write" to <Unknown> (uccxd_t). For complete SELinux messages. run sealert -l 13a0aff9-a988-
4a22-ad93-76c4c6d5d8c9
– Raw audit violations

host=selinux-uccx type=AVC msg=audit(1321533664.392:11657): avc: denied { write } for
pid=25272 comm="UCCX_Cvd" laddr=0000:0000:0000:0000:0000:ffff:7f00:0001 lport=34490
faddr=0000:0000:0000:0000:0000:ffff:7f00:0001 fport=8889
scontext=system_u:system_r:uccxd_t:s0-s0:c0.c1023 tcontext=system_u:system_r:uccxd_t:s0-
s0:c0.c1023 tclass=tcp_socket
Unified Sign On
Overview
 CCX 9.0(1) allows the Application user to traverse seamlessly between CCX
Web Applications
– Possible when UCM is not accessible
CCX Web Application 8.x 9.x
Unified CCX Application No (only before initial CCX Yes

Administration Application Administration
Configuration)
Cisco Unified Serviceability Yes Yes
Cisco Unified CCX No Yes

Serviceability
Cisco Desktop Administrator No Yes
Cisco OS Administrator No No
DRS No No
Upload from Script No Yes
Single Sign On Troubleshooting
CCX UCM
CCX/UCM Connection Broke

Application User Can Still Log In
Enable User as Administrator
Cisco Unified Intelligence Center
30
Overview
 Web 2.0 based reporting platform that is packaged as part of CCX 9.0(x)
 CUIC Standard with unlimited users packaged with CCX Premium, Enhanced,
or Standard at no additional cost
 CUIC serviceability integrated with CCX serviceability page
 Integrated user management via AppAdmin
 Used for Historical Reports, option to switch between HRC and CUIC in 9.0(x)
 Supports all existing out-of-the-box HRC reports
– Same stored procedures and data source used
Enable CUIC
Reporting Configuration
Enable CUIC
Defaults to HRC
CUIC Flow Diagram
User CUIC Web UI CCX CUCM
DB
Login
Authenticate
AXL Authentication
AXL Response
Authentication Response
Load Main
Run Report
Call Stored Procedure
SQL results
Report Results
CUIC Flow Diagram
User CUIC Web UI Who Can Login?
Platform Administrator
‘Reporting’ Users
CUIC Access Log:

Login [06/Mar/2013:11:08:57 -0500] 10.77.85.103 10.77.85.103 - - 8444
GET /cuic/Login.htmx HTTP/1.1 200 11226 147
CUIC Log:
10.77.85.101: Mar 06 2013 11:18:29.631 -0500: %CCBU__CUIC_SECURITY-6- Trace Settings in
AUTHENTICATING_USER: References
%[MARKER=START][MESSAGE=attemptAuthentication][OPERATION_TYPE=START]:
Authenticating User
10.77.85.101: Mar 06 2013 11:18:29.634 -0500: %CCBU___________CUIC-7-DEBUG:

cuicSubsystem.requestUserLogin() return true to allow login for user CCX\cuicadmin
<- CuicAuthenticationProcessingFilter.java:115
[com.cisco.ccbu.cuic.util.spring.CuicAuthenticationProcessingFilter.attemptAuthenti
cation]
CUIC Flow Diagram
CUIC Web UI CCX CUCM
MADM Log:
Mar 06 11:18:29.891 EST %MADM-UCCX_WEBSERVICES-
7-UNK:The Agent ID received here is: cuicadmin

7-UNK:ID of Application/client which is invoking
the REST service: CUIC
Authenticate
AXL Authentication
MADM Log:
<executeSQLQuery><sql>
SELECT eu.pkid, eu.userid, eu.firstname, eu.lastname, eu.middlename, eu.nickname, eu.mailid,
eu.assocpc, eu.department, eu.manager, eu.title, eu.building, eu.site, ca.aakeypadmapping,
ca.aapromptname, ca.fnamealtpronun, ca.lnamealtpronun, cs.administratorlevel, cs.issupervisor FROM
enduser AS eu LEFT OUTER JOIN crsapplication AS ca ON ca.fkenduser = eu.pkid LEFT OUTER JOIN
crsuserclusterspecific AS cs ON cs.fkenduser = eu.pkid AND cs.clustername = '1330625185000'
WHERE eu.tkuserprofile = 1 AND eu.status = 1 AND lower(eu.userid) = 'cuicadmin'
</sql></executeSQLQuery>
CUIC Flow Diagram
MADM Log:
CCX CUCM <doAuthenticateUser>
<userid>CUICAdmin</userid><password>XXXXXX</password>
</doAuthenticateUser>
<axl:doAuthenticateUserResponse
xmlns:axl="http://www.cisco.com/AXL/API/7.1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
AXL Authentication <return>
<userAuthenticated>true</userAuthenticated><code>0</code>
AXL Response </return>
</axl:doAuthenticateUserResponse>
Mar 06 11:18:30.269 EST %MADM-LIB_AXL-7-

UNK:AXLUserAuthentication.getUserAuthentication() -
userAuthenticatedElementValueNode:[#text: true]
MADM Log:
Mar 06 11:18:30.269 EST %MADM-UCCX_WEBSERVICES-7-UNK:AuthenticateResource::doCapabilityCheckForClientApp
Finding the user capability checker for app CUIC
Mar 06 11:18:30.269 EST %MADM-UCCX_WEBSERVICES-7-UNK:CUICAppCapabilityChecker : Verifying the capability

of CUICAdmin
CUIC Flow Diagram
User CUIC Web UI CCX
MADM Log:
7-
UNK:AuthenticateResource::doCapabilityCheckForCl
ientApp Is Capability Present : true

Authentication Response 7-UNK:The entity set in response
Load Main AuthenticateSuccessResponseXML [event= post ,
source= /authenticateagentID= CUICAdmin data=
[[LoginXML[ agentID CUICAdmin success true]]]]
CUIC Log:
10.77.85.101: Mar 06 2013 11:18:30.496 -0500: %CCBU___________CUIC-7-DEBUG: Authentication
Success.
10.77.85.101: Mar 06 2013 11:18:30.496 -0500: %CCBU___________CUIC-7-DEBUG: POST

https://localhost:8443/uccx-webservices/authenticate/cuicadmin returned a response status of 200
OK
10.77.85.101: Mar 06 2013 11:18:30.528 -0500: %CCBU_______CUIC_JSP-7-LOAD_PAGE:

%[PAGE=/cuic/Main.htmx][REQUEST={{cmdTree : chooseProtocol}}]: Page was loaded.
CUIC Flow Diagram
CUIC Log:
User CUIC Web UI CCX 10.77.85.100: Mar 21 2013 16:18:38.654 -0400:
%CCBU_CUIC_MODEL_OBJECTS-7-
DB JSON_VALUES:%[ARGUMENT=[{"viewName":"Average+Cha
t+Duration+by+Date+Chart”… ]: Values for JSON
Object
10.77.85.100: Mar 21 2013 16:18:50.227 -0400:

%CCBU_CUIC_MODEL_OBJECTS-7-CREATE_OBJECT:
Run Report %[ARGUMENT={"filterParams":[{"relativeDate":true
Call Stored Procedure , … "value":"THISMTH","date":"03/21/2013
23:59:59” …
CUIC Log:
10.77.85.100: Mar 21 2013 16:18:50.475 -0400: %CCBU_CUIC_MODEL_OBJECTS-7-POPULATING_DATASET:
%[DATASET_ID=][FILTER_ID=8E9A9CE11000013D3FEEB4CD0A4D5564] … Populating Dataset
10.77.85.100: Mar 21 2013 16:18:50.622 -0400: %CCBU_CUIC_DATA_PROCESSING-7-

DPC_REPORT_RUN_REQUESTED: %[DATASET=8E9A9E571000013D55C477EF0A4D5564]: A request has been
received to run a report.
10.77.85.100: Mar 21 2013 16:18:50.721 -0400: %CCBU_CUIC_DATA_PROCESSING-7-DPC_WORKER: … NEW DS

ID: 8E9A9E9E1000013D5279C4FC0A4D5564 … QueryText=[{call sp_chat_traffic_analysis('2013-03-01
04:00:00','2013-03-22 03:59:59','0','0')}];
CUIC Flow Diagram
User CUIC Web UI CCX CUIC Log:
10.77.85.100: Mar 21 2013 16:18:51.013 -0400:
DB %CCBU_CUIC_DATA_PROCESSING-7-DPC_WORKER: …
DSINFO: 8E9A9E571000013D55C477EF0A4D5564 …
Report Def: Chat Traffic Analysis Report
Definition : populate the footer and formula
fields time : 138 ms
Report Results
10.77.85.100: Mar 21 2013 16:18:51.013 -0400:
SQL results %CCBU_CUIC_DATA_PROCESSING-7-DPC_WORKER: …
DSINFO: 8E9A9E571000013D55C477EF0A4D5564 …
runReport() completed successfully. Proceeding
CUIC Log: with saving data and marking execution as
10.77.85.100: Mar 21 2013 16:18:55.463 -0400: success.
%CCBU_CUIC_DATA_PROCESSING-7-
DPC_THROTTLER_ROWS_CHECKEDOUT:
%[THROTTLER_ROWS_AVAIL=249991][THROTTLER_ROWS_CHECKEDOU
T=9]
10.77.85.100: Mar 21 2013 16:18:55.943 -0400:

%CCBU_______CUIC_JSP-7-LOAD_PAGE: %[PAGE=/cuic/WEB-
INF/jsp/report/grid/GridHtml.jsp] … {datasetId :
8E9A9E9E1000013D5279C4FC0A4D5564} … Page was loaded.
Running Reports
CUIC Redirection
Users Connect to Either Node in HA
Redirect to Appropriate Node
1. User redirected to non-engine master node where

Node 1 1 Node 2 CDS/HDS enabled and CUIC is running.
M 2. User redirected to the database master node where

2 CUIC is running.
3. If Tomcat or CUIC is not running on non-engine master
3 node, no redirection is done.
10.77.85.100 10.77.85.101 4. If Tomcat or CUIC is not running on node accessed,
no redirection is done, connection fails.
Report Throttling
 Throttling mechanism prevents CUIC servers from freezing or encountering an
Out-of-Memory situation when they are pushed to their limits
 How does it work?
– Report rows are loaded into memory
– If a violation happens while CUIC is reading data from the data source, the report
execution aborts
 The report is marked as failed
– If a violation occurs while CUIC is in the process of preparing the HTTP response for a
browser (HTML of a grid, JSON string of a gauge, XML of a chart, etc.) the request to
display the data is rejected
 The user received an error message that explains the report cannot be rendered due to low
server resources
CUIC Log:
10.77.85.100: Mar 21 2013 16:18:55.463 -0400:
%CCBU_CUIC_DATA_PROCESSING-7-DPC_THROTTLER_ROWS_CHECKEDOUT:
%[THROTTLER_ROWS_AVAIL=249991][THROTTLER_ROWS_CHECKEDOUT=9]
Report Scheduler
Report Scheduler
Destination
Scheduled Execution
• Verify SMTP configuration
• Scheduled reports run from
• Verify remote settings
Publisher
• Verify user has write permissions
• Ensure CUIC Reporting Service
on remote server
on Publisher is IN SERVICE
• Collect CUIC Log • Verify network connectivity
Permalinks
 CUIC permalinks provide direct report access
– HTML: Grid, Gauge, Chart views
– Excel: Grid view
– XML: Grid view
 User accessing permalink does not need to log into CUIC
 Reference CUIC User Guide for time zone specific considerations
Permalinks
Permalinks
HTML
Excel
Refresh Report to obtain Lasted Data
XML
Dashboard
Custom Reports
 CUIC is installed with a standard CUIC license
– Cannot create custom reports
 To create a custom report
– Install standalone CUIC server with Premium license
– Target data source as CCX server
– Create report with query or stored procedure to be used in report definition
– Export the report definition
– Import the report definition into CUIC on CCX
Customize Existing Reports
Create Different Views
Chart, Gauge, Grid
Customize View
Utilization Hourly Report
 Available with all CCX license levels, but only with CUIC, not HRC
 Reports on utilization of
– Inbound ports: actual number of inbound ports being used in system
– Outbound ports: actual number of outbound ports being used across all IVR campaigns
– Agent seat count: Number of voice seats being used concurrently
 Each agent usage of CAD counted as one seat
 Each supervisor usage of CAD/CSD either alone or together counted as one seat
 New historical table called LicenseUtilization added to database
– For HA system the table is replicated using Informix Enterprise Replication
– Table purged according to configured automatic or manual purge settings
– New stored procedure called sp_license_utilization provides data to report
– Data written to table every minute from the engine
 True maximum numbers may occur at different time than when samples collected
Utilization Hourly Report
User Permissions
System Configuration Administrator Security Administrator

• Full access to the Data Sources • Full access to Security drawer
drawer
Login User • Full access to the Scheduler
• Log in to CUIC drawer Dashboard Designer
• Access to user page • Delete, import, export stock reports • Full access to Dashboard drawer
Report Designer Report Definition Designer Value List Collection Designer

• Full access to Reports drawer • Full access to Report Definition • Full access to Value List drawer
• Execute access to Data Sources drawer • Execute access to Data Source
and Value Lists drawers • Execute access to Data Sources drawer
• Access to Scheduler drawer to and Value Lists drawers
work with own reports
CUIC Serviceability
CUIC Services
Displayed if CUIC
Activated
CUIC Trace Settings
CUIC Serviceability
CUIC Process Monitor
CUIC Service Monitor
CUIC Serviceability
CUIC PerfMon Counters
CUIC Logs
CUIC Serviceability
– show cuic properties [options]
– show cuic tech [options]
– show cuic component-status [subsystem]
– show cuic trace cuicserver [subsystem]
CUIC Serviceability
– set cuic properties
– unset cuic properties host-to-ip
– set cuic trace
– set cuic syslog
– utils cuic purge
CUIC Data Mismatch
 CUIC database tables stored in Platform database
 Check CUIC replication state
– utils dbreplication runtimestate
 Reset replication (from Publisher)

– utils dbreplication stop all
– utils dbreplication reset all
 Correct data mismatch (from Publisher)
– utils dbreplication forcedatasyncsub
Tips
 Historical Report data not pulling data correctly
– Verify replication is active and running
– Verify that CDS/HDS is enabled
– Check if a RTMT alert is raised
 Run the HR File Restore from CCX Administration
 Real time reports are based on data in memory, not the database
– Reset at midnight every night
 Database not in sync alert in RTMT

– Run a repair on the datastores from CLI or Serviceability UI
Web Chat
Overview
 CCX Web Chat only available with CCX Premium license
 Requires Cisco SocialMiner
– Endpoint that hosts widgets that end users and agents use for chat interaction
– SocialMiner sold separately, no licensing associated with the SocialMiner node
 SocialMiner accepts chat request, communicates with CCX to allocate an
agent, then establishes the connection between the end user and agent
 CCX queues chat requests, allocated agents, monitors the chat session, and
provides real-time and historical reports for chat
 Web Chat compatibility defined in CCX Compatibility Guide
Chat Media
Deployment Models Signaling/Configuration
Customer Website in DMZ Customer Website in Cloud

End User End User
Web Server
Internet Internet Hosted Website
SocialMiner SocialMiner
DMZ DMZ
Agent CCX Agent CCX
Corporate Network Corporate Network
Fault Tolerance
 SocialMiner does not support High Availability (HA) SocialMiner
 Fault Tolerance is provided via CCX HA
 SocialMiner configured to communicate with both CCX nodes
– When new contact arrives, both CCX nodes notified
– Master node queues the request
– Secondary node ignores request
 If CCX nodes failover CCX CCX
– Existing chats between agents and end users stay connected
M S
– SocialMiner requeues all existing contacts to new Master node
– Agent must log out and log back into Web Chat
Configuration: SocialMiner& CCX
Configuration: SocialMiner& CCX
 Review.. 5 easy steps
– Create web chat CSQs
 Add skills to CSQs
– Configure SocialMiner for CCX
 Use SocialMiner Platform Administrator credentials
 Ensure DNS entries exist for both SocialMiner and CCX
– Configure chat system parameters
 RONA time
 Chat/Voice behavior
– Create chat widget
 Default and custom form fields
 Chat widget code for webpage
– Configure chat teams
Configuration: Agent
Cisco Desktop Administrator

Enable Integrated Browser
Create Worksite
Use One CCX URL
Auto Redirect to Active Server
Web Chat in CAD
Configuration: Supervisor
Cisco Supervisor Desktop

Enable Integrated Browser,
Cisco Web Chat
Web Chat in CSD
Web Chat Flow Diagram
Web Server SocialMiner CCX Agent
Chat Request
New Contact Notification
Contact Dialog Event
Open Reply Template

Agent Joined
Reply Template
End Session
Chat Session Complete

Web Server SocialMiner CCX SocialMiner Chat Proxy Log:
10.77.85.106: Mar 15 2013 12:59:42.572 -
0400: %CCBU_____CHATSERVER-6-
CHAT_SESSION_CREATED:
%[session.id=E6B7E9C82968F80F1FC553DB16071E6
F][session_timeout=300]: chatserver: new
chat session
Chat Request 10.77.85.106: Mar 15 2013 12:59:42.654 -

New Contact Notification 0400: %CCBU______CHATPROXY-6-
USER_CHAT_ROOM_CREATED:
Trace Settings in %[Nickname=Ron][Room=socialminer_chat.1@conf
References erence.127.0.0.1][social_contact_id=null]: A
user and chat room were created
SocialMiner Runtime Log:

10.77.85.106: Mar 15 2013 12:59:42.960 -0400: %CCBU___NOTIFICATION-6-NOTIFICATION_SUCCEEDED:…
campaignId=100021,tags=[ccx_chat_req],id=100023,name=CCX Chat Notification for HA – Backup …
A notification was sent out successfully
10.77.85.106: Mar 15 2013 12:59:43.012 -0400: %CCBU___NOTIFICATION-6-NOTIFICATION_SUCCEEDED:…

campaignId=100021,tags=[ccx_chat_req],id=100022,name=CCX Chat Notification …
A notification was sent out successfully
CCX Agent CCX Primary MADM Log:
Mar 15 12:59:42.970 EDT %MADM-UCCX_WEBSERVICES-7-UNK:Notify for
new contact
12:59:43.011 EDT %MADM-UCCX_WEBSERVICES-7-

UNK:ContactTopicNotifier.notifyChatContactTopic() - Successfully
notified ChatContactTopic.
Contact Dialog Event CCX Secondary MIVR Log:

Mar 15 12:59:42.971 EDT %MIVR-SS_CHAT-7-UNK: …
CCX Primary MIVR Log: Ignoring this event for the secondary node
Mar 15 12:59:42.853 EDT %MIVR-SS_CHAT-7-
UNK:com.cisco.wf.subsystems.chat.listener.xmpp.SMCampaignListener : Received contact event
: SMContactXML [campaignPublicId=CCX_Chat_Campaign …
Mar 15 12:59:42.967 EDT %MIVR-SS_ROUTEANDQUEUE-7-

UNK:com.cisco.wf.subsystems.routeandqueue.allocation.Allocator : offerContactToQueuer
contact … contactType=INCOMING, mediaType=CHAT … UserExtensionField [name=Support Contract
Number, value=12345] … UserExtensionField [name=h_widgetName123456, value=chat_widget2] …
chatCSQTag = Chat_Csq3
17234: Mar 15 12:59:42.968 EDT %MIVR-SS_ROUTEANDQUEUE-7-UNK:ALLOCATED Contact

[6EFE26F11000013D0000000B0A4D556A] with Agent [agent2]
Web Server SocialMiner Agent CCX Primary MADM Log:
Mar 15 13:00:00.543 EDT %MADM-
UCCX_WEBSERVICES-7-UNK:Received contact
update request for contactid
[6EFE26F11000013D0000000B0A4D556A]
Mar 15 13:00:00.543 EDT %MADM-

UCCX_WEBSERVICES-7-UNK:Received request
Open Reply Template for agent operation 0 …
Agent Joined Mar 15 13:00:00.550 EDT %MADM-

UCCX_WEBSERVICES-7-UNK: … - Successfully
notified AgentDialogTopic.
SocialMiner Chat Proxy Log:

10.77.85.106: Mar 15 2013 13:00:01.772 -0400: %CCBU______CHATPROXY-6-PRESENCE_EVENT_RECEIVED: % …
[SocialContactID=6EFE26F11000013D0000000B0A4D556A][Status=joined][Username=socialminer_chat.1@conference.
127.0.0.1/agent2 (Customer Rep)]: Received presence event
10.77.85.106: Mar 15 2013 13:00:11.839 -0400: %CCBU______CHATPROXY-6-CHAT_MESSAGE_RECEIVED:

%[[email protected]/agent2 (Customer Rep)]…
[SocialContactID=6EFE26F11000013D0000000B0A4D556A][Text=Hello]: Received chat message
Web Server SocialMiner Agent
End Session

10.77.85.106: Mar 15 2013 13:00:38.436 -0400: %CCBU______CHATPROXY-6-PRESENCE_EVENT_RECEIVED:
%[CustomerNickname=Ron][Nickname=agent2 (Customer
Rep)][SocialContactID=6EFE26F11000013D0000000B0A4D556A][Status=left][Username=socialminer_chat.1@conferen
ce.127.0.0.1/agent2 (Customer Rep)]: Received presence event
10.77.85.106: Mar 15 2013 13:00:54.775 -0400: %CCBU_____CCP-DS-API-6-CCP_DS_API_MESSAGE: ###### Saving

Chat TRanscript: ChatTranscript for SC :6EFE26F11000013D0000000B0A4D556A
SocialMiner CCX Agent CCX Primary MADM Log:
Mar 15 13:00:38.573 EDT %MADM-
UCCX_WEBSERVICES-7-UNK:Received State change
request for agent agent2 with new stateId =3
and reasonCode =32755
Mar 15 13:00:38.606 EDT %MADM-

UCCX_WEBSERVICES-7-
UNK:ContactTopicNotifier.notifyAgentTopic() -
Successfully notified AgentTopic.
CCX Primary MIVR Log:
Mar 15 13:00:38.351 EDT %MIVR-SS_CHAT-7-UNK:com.cisco.wf.subsystems.chat.listener.xmpp.SMCampaignListener
: Received contact event : SMContactXML [campaignPublicId=CCX_Chat_Campaign,
id=6EFE26F11000013D0000000B0A4D556A, publishedDate=1363366782571,… status=handled,
statusReason=chat_agent_ended …
Mar 15 13:00:38.351 EDT %MIVR-SS_ROUTEANDQUEUE-7-

UNK:com.cisco.wf.subsystems.routeandqueue.aggregator.observer.ContactObserver : Contact Notification
received ContactNotification [requestId=, contactId=6EFE26F11000013D0000000B0A4D556A,
contactState=TERMINATED, handlerQueue=3, startTime=1363366782966, endTime=1363366838351,
queueWaitDuration=17593, handlingDuration=37792,dispositionReason=chat_agent_ended]
Web Chat CSQ Real-Time Reports
Web Chat CSQ Real-Time Reports
Web Chat CSQ Historical Reports
New Historical Reports

Capture Agent and Chat Utilization Activity
Chat Agent Detail Report

Information on each chat received per agent, one row per chat
CCX Primary MADM Log:

… for contactid [6EFE26F11000013D0000000B0A4D556A] …

… [SocialContactID=6EFE26F11000013D0000000B0A4D556A] …
Chat Agent Summary Report

Summary information on chats received per agent
Contact CSQ Activity Report

Summary information on chats received per CSQ
Chat CSQ Agent Summary Report

Summary information on each chat received per agent
Chat Traffic Analysis Report

Summary information on chats received each day
SocialMiner Search
Specific Keywords, Phrases, Tags
Autocomplete Suggestions
Serviceability
System Administration
Accessible only by SocialMiner Administrator
System Version Information
Statuses for Main Services

Hover over icons to display tool tip
System Load Snapshot
Serviceability
RTMT for SocialMiner
Downloadable from System Administrator tab
SocialMiner RTMT
Provides OS Serviceability tools
Trace & Log Viewer

Provides OS level logs only
Serviceability
Logs for SocialMiner
Downloadable from System Administrator tab
Serviceability
System Snapshot
Snapshot of system parameters
API Statistics
Feeds
• Min/Max/Avg Request Time
• Requests per Second JVM Statistics
• Heap Size
JMX mbean Values
Campaigns • CPU Samples
Disk Usage • Up Time
Hardware Resource Counts
Impact to
Purge Settings System Conditions
performance while
file is downloading Application Version
Notification Rules
Serviceability
Platform Administration for SocialMiner
Viewable from System Administrator tab
Serviceability
Web Chat Services OpenFire XMPP Interface
Send state changes to
WebChat user interface
REST API Functionality

Between SocialMiner & CCX,
Agents Allocation CCX & Agent Desktop
Manage Chat Contacts

Serviceability
Chat Subsystem
Route and Queue Subsystem
Serviceability
OpenFire logs located at

/var/log/active/uccx/log/openfire
WebServices
Troubleshooting CCX Web Chat
Configuration Status
Checked when page loads
Not Configured
Configured Incorrectly
Configured
 Any time the Web Chat feed is removed from SocialMiner, the Web Chat form
must be redeployed once the CCX configuration is saved
– Web Chat feed id changes
Indicates item not configured in SocialMiner
No Chats in Campaign
Chats not being created
Web Server SocialMiner
Verify connectivity between Web Redeploy chat widget code

Server and SocialMiner
SocialMiner CCX
CCX does not receive Incoming Chats

Chats not being routed
Ensure correct ports open

http://docwiki.cisco.com/wiki/SocialMiner
Verify ping/nslookup working by
FQDN
Verify services IN SERVICE
Compare system snapshots before

and after submitting a web chat
notificationsFailed and lastFailureCause
SocialMiner Agent
Agent Desktop Reply Pane does not Load
Chat not being routed to agent
Ensure correct ports open

http://docwiki.cisco.com/wiki/SocialMiner
Verify ping/nslookup working for
CCX by FQDN
Install SocialMiner certificate
Access chat from browser outside

of agent desktop
IE8 must be installed
Database
Overview
CCX
db_cra db_hist db_cra_repository db_frascal
DB
Config Datastore Historical Datastore Repository Datastore Agent Datastore

Stores configuration Stores data necessary Stores scripts, prompt Stores agent config
data from CCX to generate Historical files, custom info, agent report data,
AppAdmin reports documents, and and metadata for
grammars recorded files
show uccx dbserver disk

Displays db space location
Size Restrictions
show uccx dbserver disk
Displays datastore size
Server Type Server Disk Config DB Historical DB Repository Agent DB

Size Size Size DB Size Size
7816 1x160 GB .5 GB 11 GB 3.0 GB .5 GB
Datastore Size
Platform specific
1x250 GB .5 GB 11 GB 3.0 GB .5 GB
7825 2x160 GB .5 GB 12 GB 3.0 GB .5 GB
2x250 GB .5 GB 12 GB 3.0 GB .5 GB
7835 2x146 GB .5 GB 13 GB 3.0 GB .5 GB
7845 4x146 GB .5 GB 20 GB 3.0 GB .5 GB
4x300 GB .5 GB 26 GB 3.0 GB .5 GB
Replication
 Several flavors of replication
– Informix Enterprise Replication
 Replication is symmetric
 Platform tables stored in separate database from CCX Application tables
– Application Replication: sync in atomic transaction using 2 phase commit
 Only enabled or disabled from CCX Serviceability page
 Used for configuration tables
– Wallboard: not replicated, only updated on current db master
Informix Enterprise Replication
Source Server Transport Target Server
Application
Database Database
Snoopy
Grouper Network Ack Queue Data Sync

Logical Log
Spool Send Queue Receive Queue
Reset Informix Enterprise Replication
 Teardown
– utils uccx dbreplication teardown
DB Master DB Standby
 Setup
– utils uccx dbreplication setup
 Repair
– utils uccx dbreplication repair
 Reset Replication commands
symmetric, can be
– utils uccx dbreplication reset executed on either node
– Performs above actions in sequence
Should be repaired in
maintenance window as
it may generate a lot of
network traffic
Application Replication
 Config Datastore does not participate in Informix Enterprise Replication
– Configuration updates are pushed to both nodes
– If one node is offline and CDS/HDS enabled, the update is not committed
 Reset application replication Source Server Destination Server
– Disable CDS/HDS
 Break CDS sync CDS CDS
 Teardown HDS replication
– Enable CDS/HDS HDS HDS
 Dump CDS source data to csv file
 Truncate destination CDS RDS RDS
 Restore destination CDS with source data
 Setup replication for HDS ADS ADS
 Initiate repair for HDS
Enable CDS/HDS over WAN may take a while, UI may timeout.
Application Replication
 When encountering problems with CDS/HDS enable/disable function
– Check dumptable.log on source server to verify all tables were dumped correctly
– Check synctable.log on destination server to verify all tables were synced correctly
synctable.log:
[Tue Mar 5 14:23:13 EST 2013] : Truncate local
dumptable.log:
table resource
[Tue Mar 12 22:38:48 EDT 2013] : Unloading table
…
configschema
[Tue Mar 5 14:23:14 EST 2013] : Truncating the
table 'resource' succeeded...
profileidmapping
[Tue Mar 12 22:38:51 EDT 2013] : generating
resource
dbload command for table 'resource'
…
[Tue Mar 12 22:38:51 EDT 2013] : Done with dbload
for table 'resource'
 If dialing list too large to dump/sync, manually purge table
Data Synchronization
Force Data Sync
Useful where a nodes

database becomes corrupted
Source Server Destination Server
Only applicable for datastores CDS CDS

participating in Informix
Enterprise Replication HDS HDS
RDS RDS
ADS ADS
Synchronize Data
Switch Version
CCX
DB Copy db spaces
Is database Create ontape
corrupted?
no backup
from active partition
to inactive partition
8.5(1)
SU3
Abort yes
Perform
Bring up database in
schema
inactive partition
migration
uccx-install.log:
Cisco Unified CCX DB appears to Dump existing table to csv file
be corrupt. Please use the CLI
command "utils uccx switch- Truncate table
version [db-check | db-recover]"
to recover the CCX DB before Alter table schema
retrying the switch version. … May take a while, do not
reboot node during this
time, the database will Reload table with csv data
9.0(1)
become corrupted
Database Logs
 online.uccx.log
– Informix activity log, good indicator of database health
– If database problems, compare good and bad copies of log
 uccx_repl_output_util.log
– Outputs CLI command execution
 dumptable.log (source server) and synctable.log (destination server)
– CDS/HDS enable/disable logs
 uccx_spl.trace
– Purge log
 DbConfigIntegrity.txt
– Generated via CLI utils uccx database dbserver integrity
– Verifies database health
Database Health
 utils uccx database dbserver integrity
– Executes oncheck –cc, oncheck –ce, and oncheck –pr to file
– Verifies database connectivity, health of tables
– Look for ISAM error: TBLspace does not exist

 Indicates corruption of control information in the root dbspace
Database Serviceability CLI Commands
 show uccx tech dbserver log diagnostic
– Shows the existence of Informix assertion failure (*.af) and shared memory dump
(*.shmem) logs, which indicate Informix has crashed
 show uccx tech dbserver status

– Captures result of ‘onstat –a’
Database Serviceability CLI Commands
 show uccx tech dbserver all
– Equivalent to running previous two commands
 tech dbserver log message

– Displays last 20 lines of the online.uccx.log
Tips
 When changing the purge setting, avoid a large differential
– Can cause a sudden spike in the purge time, possibly overlapping with business hours
 If the connection between CCX nodes is broke, or node 2 is down, Informix will
buffer the replication updates
– Resulting data propagation can cause a lot of traffic
– Recommended to restore the connection off hours is possible
 Replication commands should be during a maintenance window
– Repair replication
– Enable/Disable CDS/HDS
CBABU UCCX ES Process
Previous CBABU Process
 Interim Fix Delivery Mechanism was the ET (Engineering Special)
 Built on per customer basis, not general availability.
 Not cumulative. Situations arise when multiple, conflicting ETs are necessary.
 Risky Application. Little testing. Requires root access to install.
 No defined rollback mechanism.
 Not documented. Stored in single location or numbered consistently.
 Ad-hoc, require large overhead from DE teams, Escalation teams, and TAC
teams.
 No release schedule.
New CBABU ES Process
 One ES (Engineering Special) per Month on Latest SU (Service Update)
– Applicable for the last two current major releases (i.e. 8.5(x) and 9.0(x).
– Exceptions in release schedule may occur.
– Posted by TAC via Special File Publish.
– Given to customer on need by need basis.
 Severity 1 & 2 defects. Severity 3 with high impact.
– Only defect fixes, no new features or functionality will be introduced.
– Any ET’s produced since last ES.
 Cisco will support customers running any ES in production.
– Same process, procedures, and SLAs are applicable.
– Support of break-fix issues, defect fixes and remediation/development fixes when
necessary.
ES Release Vehicle
 UCCX ESs will be Released via .cop Files
– .cops to be installed on top of Major.Minor.SU for which they are designed.
– Rollback .cop will be provided for each Major.Minor.SU.
– Rollback .cop will revert server to Major.Minor.SU.
– .cops will be named with ES number and build number.
 ESs will reflect in both CLI and AppAdmin
– AppAdmin: 8.5.1.11003-32 (ES01-69)
– CLI: ‘show version active’ shows ES installed.
 Each ES will contain Release Notes
– Build number and filename with MD5 hash.
– Installation instructions, rollback instructions.
– Version on which ES can be installed.
– Information about compatibility with external components (if necessary).
– Resolved caveats.
Tips
 UCM qualification policy
– CCX compatibility matrix publishes qualified major versions
– All CCX SU’s are presumed qualified with qualified UCM versions
– All UCM SU’s are presumed qualified with qualified CCX versions
– All UCM ES’s are presumed qualified with qualified CCX versions
ES1 ES9
SU1 SU2 SU3 SU1 SU2 SU3
CCX 9.0(1) UCM 9.0(1)
 SU images can only be used for upgrades, never base installations

Conclusion
Topics of Discussion
 Virtualization Best Practices
– Specs based hardware; Boot from SAN; New Identity
 Refresh Upgrade
 SE Linux
– Separate enforcement of security decisions from the security policy; Replaces CSA
 Unified Sign On
 Cisco Unified Intelligence Center
– Customized Reports; Integrated User Management, Serviceability; Permalinks
 Web Chat Integration with SocialMiner
– Integrated CCX Configuration; Easy CAD/CSD Configuration & Integration
 Troubleshooting the CCX Database
 CBABU CCX ES Process
Complete Your Online Session Evaluation
 Give us your feedback and
you could win fabulous prizes.
Winners announced daily.
 Receive 20 Cisco Daily Challenge
points for each session evaluation
you complete.
 Complete your session evaluation
online now through either the mobile
app or internet kiosk stations.
Maximize your Cisco Live experience with your
free Cisco Live 365 account. Download session
PDFs, view sessions on-demand and participate in
live activities throughout the year. Click the Enter
Cisco Live 365 button in your Cisco Live portal to
log in.
Cisco Collaboration Community and User Group
Connect with Peers and Cisco
 Discuss strategy, market trends, IT architecture,

product features, user adoption, and more with peers
contact centers, social software, telepresence, unified
communications, and web conferencing
 Interact with Cisco Product Managers, Technical

Marketing Engineers and Services Consultants
 Hear announcements and see demos of new solutions
 Join the Collaboration User Group www.cisco.com/go/joinconversation
– Influence product direction Visit Collaboration Booth - Cisco Campus:

 Explore the Collaboration Community
– Participate in advisory groups & early product releases
 Join the Collaboration User Group (CUG)
– VIP perks & exclusive roadmap sessions at Cisco Live  CUG Members: get gift ticket & badge ribbon
References & Further Information
References
 SELinux
– http://magazine.redhat.com/2007/05/04/whats-new-in-selinux-for-red-hat-enterprise-
linux-5/
– http://www.centos.org/docs/4/html/rhel-selg-en-4/selg-preface-0011.html
 Virtualization Improvements
– http://docwiki.cisco.com/wiki/Virtualization_for_Unified_CCX
– http://docwiki.cisco.com/wiki/Unified_Communications_Virtualization_Supported_Applic
ations#Contact_Center_Applications
– http://docwiki.cisco.com/wiki/UC_Virtualization_Supported_Hardware
– http://www.cisco.com/web/cuc_afg/index.html
– http://docwiki.cisco.com/wiki/Disable_LRO
References
 SocialMiner
– http://docwiki.cisco.com/wiki/SocialMiner
 CUIC
– http://www.cisco.com/en/US/products/ps9755/products_user_guide_list.html
 Database
– http://publib.boulder.ibm.com/infocenter/idshelp/v115/index.jsp?topic=%2Fcom.ibm.adr
ef.doc%2Fids_adr_0387.htm
CUIC Flow Diagram Traces
 Cisco Unified CCX Engine
– LIB_AXL: Debugging
– USER_MGR: Debugging, XDebugging4
– LIB_JDBC: Debugger
– SS_DB: Debugger
 Cisco Unified CCX Administration
– LIB_AXL: Debugging
– CCX_WEBSERVICES: Debugging
 Cisco Unified Intelligence Center Services
– CUIC: Detailed
– CUIC Security: Detailed
– CUIC JSP: Detailed
– CUIC Model Objects: Detailed
Web Chat Flow Diagram Traces
 Cisco Unified CCX Engine
– SS_CHAT: Debugging, XDebugging4
– SS_ROUTEANDQUEUE: Debugging, XDebugging4
 Cisco Unified CCX Administration
– UCCX_WEBSERVICES: Debugging
 SocialMiner
– Chat Proxy Log
– Runtime Log

BRKCCT-2015 TG Uccx

Uploaded by

Copyright:

Available Formats

BRKCCT-2015 TG Uccx

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKCCT-2015 TG Uccx

Uploaded by

Copyright:

Available Formats

Introduction to New Features with

Troubleshooting on Unified Contact Center

Required Level of Medium/Low High

Moving from one profile to another

ESXi SAN ESXi

8.x.y SUz 9.0.y SUz

Apply .cop 8.x

8.x Apply .cop 8.x

Node 2 8.x Active -- -- 8.x Active

Reboot & 9.x Inactive

Node 1 8.x Active -- -- 8.x Active

9.x Inactive 9.x Inactive

 New CLI commands

 Set SELinux to Disabled

– Raw audit violations

Unified CCX Application No (only before initial CCX Yes

Cisco Unified CCX No Yes

Upload from Script No Yes

CCX/UCM Connection Broke

Enable User as Administrator

CUIC Access Log:

10.77.85.101: Mar 06 2013 11:18:29.634 -0500: %CCBU___________CUIC-7-DEBUG:

Mar 06 11:18:29.892 EST %MADM-UCCX_WEBSERVICES-

Mar 06 11:18:30.269 EST %MADM-LIB_AXL-7-

Mar 06 11:18:30.269 EST %MADM-UCCX_WEBSERVICES-7-UNK:CUICAppCapabilityChecker : Verifying the capability

Mar 06 11:18:30.494 EST %MADM-UCCX_WEBSERVICES-

10.77.85.101: Mar 06 2013 11:18:30.496 -0500: %CCBU___________CUIC-7-DEBUG: POST

10.77.85.101: Mar 06 2013 11:18:30.528 -0500: %CCBU_______CUIC_JSP-7-LOAD_PAGE:

10.77.85.100: Mar 21 2013 16:18:50.227 -0400:

10.77.85.100: Mar 21 2013 16:18:50.622 -0400: %CCBU_CUIC_DATA_PROCESSING-7-

10.77.85.100: Mar 21 2013 16:18:50.721 -0400: %CCBU_CUIC_DATA_PROCESSING-7-DPC_WORKER: … NEW DS

10.77.85.100: Mar 21 2013 16:18:55.943 -0400:

Redirect to Appropriate Node

1. User redirected to non-engine master node where

M 2. User redirected to the database master node where

System Configuration Administrator Security Administrator

Report Designer Report Definition Designer Value List Collection Designer

CUIC Trace Settings

CUIC Process Monitor

CUIC Service Monitor

CUIC PerfMon Counters

 Reset replication (from Publisher)

 Database not in sync alert in RTMT

Customer Website in DMZ Customer Website in Cloud

Internet Internet Hosted Website

Agent CCX Agent CCX

Corporate Network Corporate Network

Cisco Desktop Administrator

Cisco Supervisor Desktop

Open Reply Template

Chat Session Complete

Chat Request 10.77.85.106: Mar 15 2013 12:59:42.654 -

SocialMiner Runtime Log:

10.77.85.106: Mar 15 2013 12:59:43.012 -0400: %CCBU___NOTIFICATION-6-NOTIFICATION_SUCCEEDED:…

12:59:43.011 EDT %MADM-UCCX_WEBSERVICES-7-

Contact Dialog Event CCX Secondary MIVR Log:

Mar 15 12:59:42.967 EDT %MIVR-SS_ROUTEANDQUEUE-7-

17234: Mar 15 12:59:42.968 EDT %MIVR-SS_ROUTEANDQUEUE-7-UNK:ALLOCATED Contact

Mar 15 13:00:00.543 EDT %MADM-