Edge Support – TAC

April.2020

Overview
©2016 Harmonic Inc. All rights
©2016reserved
Harmonic
worldwide.
Inc. All rights reserved worldwide. 1
 • Escalation processes & Support Models

• Ongoing Operation

Agenda • High level Monitoring Solution

• TAC – Cookbooks

• 24/7 NOC Team

• Cases severity
©2016 Harmonic Inc. All rights reserved worldwide. 2
 How to Contact the TAC

Contact the 24x7 Technical Assistance Center (TAC)

Use one of the following options to contact our TAC team:
 Call our hotline number - +1(833)817-7030 for emergency issues
 Email our 24/7 team - DL EDGE TAC 24/7 Support at this email address:
[email protected]
Your call/email must include the following information:
 A detailed description of the problem
 Your contact information
 Instructions to enable us to connect to the system remotely

Case Flow Process

 Based on the problem description, your case will be escalated to the relevant engineers
 Once the issue is identified, you will get an update for the resolution
 In extreme cases, we will consider an onsite visit

 
©2016 Harmonic Inc. All rights reserved worldwide. 3
 The portal is becoming the TAC team’s “auto pilot”

• Calendar coordination for MW’s and review meetings with the customer
• Localizing all information in one place
• Automatic reminders
• Different Queue for each support level
• Fire cases queue
• Workflow that allows for better tracking of the support case resolution
• Reduce the needs for reports
4
©2016 Harmonic Inc. All rights reserved worldwide. 4
During outages, the portal becomes the HUB of all necessary actions and info.

• Screenshots that show the exact issue and this shortens resolutions
• Enables remote access permission
• Timeline of the events
• Transparency of all available resolutions during the event

©2016 Harmonic Inc. All rights reserved worldwide. 5

Remote Monitoring Service Model

Customer System

Health & Performance Triggered Alerts Sent to the

Checks Monitored 24x7 customer Designated
by our NoC Contacts
Customer

Tier 1,2,3 Service

Onsite Support Engineers Technical Support/R&D Delivery Manager

©2016 Harmonic Inc. All rights reserved worldwide. 6

Remote Monitoring Service

Detect the problems Restore Analyze Internal Communication External Communication

24/7 Team 24/7 Team 24/7 Team 24/7 Team 24/7 Team
- Grafana T2 Team T2 Team T2 Team T2 Team
- Kibana R&D R&D R&D SDM
- Slack Channels - Cookbooks New dashboards SFDC Confluence
- Hot line and tools Jira SFDC
- Mails Confluence
- Portal

©2016 Harmonic Inc. All rights reserved worldwide. 7

Detecting problems improvements

Detect the problems

- Developing the Grafana system

- Developing new dashboards using other tools such a Kibana

- Monitor Slack Channels

©2016 Harmonic Inc. All rights reserved worldwide. 8

Restoration improvements

Detect
Restore
the problems

- Develop and maintain Cookbooks for every new problem detected

- Clear escalation process between SJ and Kiev T2 group

- Clear escalation between the TAC and engineering

©2016 Harmonic Inc. All rights reserved worldwide. 9

Analysis improvements

Analyze
Detect
Restore
the problems

- 24/7 Can analyze problems and associate them with existing PR’s

- T2 perform deep analysis for problems reported by the 24/7

- Develop new tools and dashboards to better analyze problems

©2016 Harmonic Inc. All rights reserved worldwide. 10

Internal Communication improvements

Internal Communication
Analyze
Detect
Restore
the problems

- Clear escalation process between the 24/7 and the T2 based on SFDC

- Clear escalation process between the T2 and R&D team

- SDM communicates to engineering directly from the field

- Ongoing weekly meetings between the TAC and engineering

- Daily reports

- Weekly and boomerang reports

©2016 Harmonic Inc. All rights reserved worldwide. 11

 External Communication Improvements

External Communication
Analyze
Reports
Detect Internal
Restore
the problems

- Incident reports are sent to customer shortly after an issue detected

- SDM to work with the customer and maintain the open issues list

- Weekly meetings and reports

https://confluence360.harmonicinc.com/pages/viewpage.action?spaceKey
=EDGE&title=Buckeye+Central+Hubs

©2016 Harmonic Inc. All rights reserved worldwide. 12

High level Monitoring Solution

Public IP Internet Connection

TCP 22, 81, 443
TCP 443

VF Internal Infrastructure Harmonic

R&D Team
TAC Team
TCP 22, 2022, SNMP Monitoring Team

TCP 10050, 10051

©2016 Harmonic Inc. All rights reserved worldwide. 13

 Grafana - Monitoring System

• The NOC team has access to Grafana to monitor the system 24x7

• Dashboards include information about

• CableOS Core
• CRE Information
• RPD Information
• SG Utilization
• Environmental information (CPU, RAM, Temp, etc)

• Alerts will include notifications for:

• Processes that crashes
• Modems that are offline
• Cluster that is not protected
• And more….

©2016 Harmonic Inc. All rights reserved worldwide. 14

Processes for quick restoration of the systems
LCCE is Unprotected

NO YES
Cleared by itself in 5-10 Case closed
minutes?

Notify NOC, get on Collect logs from both

bridge Cores and RPD/RPS

• Working with clear “CookBooks”

NO YES
Any RPD stuck in
“Configure” state?

Commit minor
changes to the config
Request MW
(e.g. change RPD
description)

YES NO Try “RPD warm attach”

Issue resolved?

NO YES
Reboot LCCE Master
(service-affecting) Issue resolved?

YES NO
Issue resolved?

Escalate

Close the bridge

©2016 Harmonic Inc. All rights reserved worldwide. 15
24/7 NOC Team

• Operation software

• Cookbooks automation based on the alarms received

• Automating the escalation process – notification based

• Will open a case and escalate to engineering as necessary

• Reports creation

• Open a War room

©2016 Harmonic Inc. All rights reserved worldwide. 16

 24/7 support for LATAM region
• Proactive Maintain Services - 24/7.
• Active system monitoring
• Resolve system outages
T1 - 24/7 Argentina
• Fast escalation to T2 and engineering
• Drive product quality improvements
• Predict expected problems
• Notify customers pro-actively of outages
• System health and reports T2 and 24/7 Kiev
T2 SJ

©2016 Harmonic Inc. All rights reserved worldwide. 17

 Shift Leader Role

Responsibilities

• Shift handover
• Emails, Portal cases, Jira process control
• Provide direction and feedback to 24/7 engineers during the shift
• Timely response to customer/management requests

©2016 Harmonic Inc. All rights reserved worldwide. 18

 Harmonic TAC Performance - Q3 2019
Total Number of events 633
Escalated to RnD, #56, 9%

Processed by 24/7, #329, 52%

Processed by T2, #248, 39%

Processed by 24/7 Processed by T2 Escalated to RnD

©2016 Harmonic Inc. All rights reserved worldwide. 19

Thank You

©2016 Harmonic Inc. All rights reserved worldwide. 20