Remote Services Execution

Process and Best Practices

Remote Services Execution
Customer
Activity
Focused
• All Customer Planned Activities and Changes covering ( Configuration, Software/Patch Organization

Upgrades, Diagnostics & Maintenance Activities, Installation) Simple

Customer
Experience
• Governed by Change Management Process via HEAT CRM Tool – No Exceptions Workflow
Centralized
• Daily Process Oversight Remote
Services
• Weekly Governance and Failure Analysis For Continuous Improvement Execution
Reduced
Handovers
• Close coordination between the Regional and Central Function with clear
Accountability
• Scalable to meet Customer Demands and Emergencies
Standardized
• Metrics Driven management Success KPIs
Remote Services Execution Process
Customer Approved
Approval Directly From Customer
Production Activity Identified
RPS/GPS acquire the approval
and convey this to TAC Engineer To Perform the Activity is Identified
Opens Change Entry and Assigns To Engineer
Engineer Performing the Activity Join Customer
Bridge Execute Work Order Notify Customer
with post-checks Complete and Submit Record in
HEAT TAC
Leadership
Any degradation/issue during execution needs to be Escalation
communicated to customer and tracked with ticket
No
WO
TAC Executing Engineer is required to
Failed?
Check based on 75% of WO time Yes complete the HC/PAMS
completed. GPS to discuss with
Customer for any MW extension 1 hr Send WO Failure Notification
before MW ends
TAC
Check Change Record in HEAT for completeness

No
Is Activity
Complete?
For any question regarding creating Change Yes
entry or Planning – contact TAC Yes
Email - [email protected]
TAC
Check the “TAC Verified & Complete” Page 3
Best Practices Governance
• Ask Yourself Training – Mandatory For Every Engineer Working on Customer System

• Buddy Check

• No Mavenir Executed Activity On Production System Without Change Management Entry

• No MOP – No Work

• No Deviations From MOP Without Technical Leadership Approval

• Software Releases & Patches Always Through Official Channel – Official ST Release Notes

• Daily Calendar Reminder Reinforcing Best Practices

• Costly Mistakes Avoidance – DON’T MOVE FORWARD WITH THE ACTIVITY

• When In Doubt – Escalation Procedure

• Known Issues Recap

• Standard Communications Practices

©2020 Mavenir. Proprietary and Confidential. Page 4
Change Management In HEAT
 •
Metrics Driven Management
Weekly Global Ops Leadership Review
• Monthly Executive Review
• Leadership held accountable for >99.5% KPI excellence
for Success Rate

November 2018 Success Rate 99.6 % Vs Target Failure Analysis

>99 1. PAMS 43452 – XXX SPS-2 (EK) upgrade failed due to data_center config not being
right. Post upgrade to the new SPS release, the AM card went to the Maintenance
mode as SPS couldn't connect to the Cassandra database & mstoreMgr process didn't
report INS status. The Configuration was not caught in ST because ST wont have 2
Datacenter sites. We corrected the config in the next MW on 15th Nov.
2.
E
- TMO VMAS – Stale entry deletion caused deletion of subscriber data.

L
MP
SATop Work Generating Customers
• XX – 1228, YY – 12, ZZ – 11, YY – 8, ABC – 5
Improvement Measures
• RM RMA Analysis (Triggered with Supply Chain) in Progress
• Overall process enhancements to improve execution rate and management oversight to
prevent catastrophic failures. (Complete)
• Based on the failures testing coverage is being followed up on the specific failures and will
evaluated how we can close that gap (New Action in Progress)
• Whenever a FOA fails in prod, backtracking with RN, ST sanity suite is done to make sure, all
follow ups are done to make sure same issue never occurs again in production directly.
• Updating Customer Lab config to be as close as possible to the production config.
Ask Yourself Training – A Staple for Every Engineer
• Educates & Trains The Support Team on Remote Services Anchor Tenets
• Mandatory Certification for Every Engineer
Move To Quarterly Re-Certification For Every Engineer
Anchor Tenets for Remote Service Execution – Rules of Engagement
• NO MOP NO WORK

• NO MATTER HOW GOOD YOU ARE WITH THE PRODUCT – ALWAYS FOLLOW MOP and PROCESS.
No MOP No Work
• NOTIFY Customer When You Start Activity Per NODE

• NOTIFY Customer When You Complete Activity per NODE Communicate – Keep Customer Informed
• EVERY NEW NODE We Touch Inform Customer.

• USA Maintenance Window at 4.30 AM CST – Going Out of Maintenance Window Should Ring Alarm Bells Maintenance Window Is Sacrosanct
• If An Activity Will Exceed Allotted Time, Proactively Reach Out to Customer and Involve Your Manager

• Do Not Work An Issue in Isolation, Inform, Communicate and Escalate - BRING IN MORE PEOPLE - ESCALATE Know Your System
• Taking Node Out of Rotation May Be The Safest Way to Mitigate Impact

• NEVER do restarts/reboots/config changes without taking customer approval and track all changes through heat ticket. ALWAY KEEP YOUR SESSION LOG RUNNING.

• Official Software Release with formal Release Note to check the content and md5sum. Don’t take patches and releases over Skype/ Chat Official Software Release / RN
• Ensure Pre-Activity health checks are clean. If not, ensure that you have a clear understanding of why. If not clear, cancel the activity rather than risk piling on issues. If the pre-checks have issues and

you have concerns, contact your Manager right away.

Safe Than Sorry
• It is mandatory that all customer site accessed sessions (xshell/putty) are logged. This will help clarify (if there is an issue) where we went wrong, and what to do/not do in the next attempt.

• Ensure MoPs are peer reviewed and signed off (preferably also by the customer). Don’t do last minute changes . Cancellation is easier to manage with the customer than an outage!

• Always follow the MOP step by step – there may be custom changes you are unaware of. Prepare for the activity by reviewing the MOP before hand as far as possible. WHEN IN DOUBT ASK. We can

deviate from MOP if there is an obvious issue with it. Always correct the MOP and submit it to customer after the WO is completed

• DO NOT sign off from a Customer call or Work Order Session without checking with the Customer contact.
When In Doubt Ask
• Unplanned config changes should be avoided even if a customer asks on the fly during a WO. Please request Customer to open a TT or a WO and get it approved before moving ahead

Communicate – Inform – Escalate

Daily Timely Alerts Reminders

Daily Maintenance Window Notice

Buddy Check – Common Sense Alerts

Standard Communications Templates

Quick Reference Process Links

TMO Remote Services

Know Issues, Costly Mistakes To Avoid At Finger Tips

Living Document Updated As And When Required !

Shows Up In Daily Maintenance Window Reminder Alert !

p le
x a m
E
TMO Remote Services
Internal Escalation Matrix
1st level escalation for Tech Ops and
Support Engineer
• MOP Issue
• Doubts and Clarification
• Degradations While Activity In Progress
• Concerns regarding timely completions
• Leads or Senior Engineer In Team

2nd level escalation

• Help formulate Customer communication
• Decision stop activity or postpone
•
• Director, Voice / Messaging Portfolio •
Maintenance window delays
Rollback
• Director, Regional Ops Leadership • MOP Mistakes and Rectification

3rd level escalation and Communication

• Customer Communication and Take Action

©2020 Mavenir. Proprietary and Confidential. Page 11

Key Takeaways For Engineers

• Process is not an overhead it is a failsafe. Due attention and accuracy in documenting the

activity is as important as the activity

• No MOP No Work.

• Abort when in doubt, don’t continue with the activity, we will provide full air cover for

cancellation. Don’t end up creating an outage.

• Software Fixes should be taken from official channels only. Release notes should be used

verify the content. No Fixes should be taken via Skype or Chat.

• Escalate when in doubt.

• ACCESS TO CUSTOMER NETWORK IS A PRIVILEGE NOT A RIGHT

©2020 Mavenir. Proprietary and Confidential. Page 12

Key Takeaways On Change Management System Process

• Governance Umbrella Provided By Change Management Process

• Metrics Based Management

• Continuously Improving – Direct Feedback Loop, Known Issue Updates Via Daily Alerts

• Strong Guidelines Minimizing Ad-Hoc Decision Making

• Clear Accountability and Escalation Path

• Engineer Training and Quarterly Certification

©2020 Mavenir. Proprietary and Confidential. Page 13

THANK YOU