APM210+ +Core+APM+I +essentials+ +Student+Guide

Download as pdf or txt
Download as pdf or txt
You are on page 1of 199

Core APM I: Essentials

APM210

Student Guide
Contents

Introduction to AppDynamics (APM211) ................................................................................ 2


Introduction to AppDynamics and APM ............................................................................. 4
Enterprise Topology and AppDynamics ........................................................................... 19
The Application Dashboard.............................................................................................. 32
Business Transactions..................................................................................................... 41
Troubleshooting Basics (APM212) .......................................................................................56
Troubleshooting Overview ............................................................................................... 58
Diagnostic Sessions ........................................................................................................ 65
Troubleshooting with Transaction Snapshots .................................................................. 70
Troubleshooting Slow Transactions ................................................................................. 86
Troubleshooting Error Transactions ................................................................................. 96
Advanced Troubleshooting and Tools (APM213)................................................................111
Too Many / Slow Database Calls ................................................................................... 113
Thread State Analysis.................................................................................................... 118
Data Collectors .............................................................................................................. 122
Troubleshooting Node Level Issues ............................................................................... 126
Memory Management .................................................................................................... 130
Monitoring the Health of Your Application (APM214) ..........................................................143
Service Endpoints and Information Points ..................................................................... 145
Introduction to the Alert and Respond Model ................................................................. 153
Metrics and Baselines.................................................................................................... 155
Health Rules .................................................................................................................. 165
Events ........................................................................................................................... 171
Actions and Policies....................................................................................................... 177
Custom Dashboards ...................................................................................................... 184
Additional Monitoring ..................................................................................................... 188

© 2020 AppDynamics, LLC. i


Introduction to AppDynamics (APM211) Student Guide

Introduction to AppDynamics (APM211)

© 2020 AppDynamics, LLC. 2


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 3


Introduction to AppDynamics (APM211) Student Guide

Introduction to AppDynamics and APM

When an application experiences a performance issue, common questions arise:


Why is it slow? How long as it been slow? What is causing the slowness?

AppDynamics helps to answer these questions and provides capabilities to:


 Troubleshoot problems such as slow response times and application errors.
 Automatically discover application topology and how components in the application
environment work together to fulfil key business transactions for users.
 Measure end-to-end business transaction performance, along with the health of
individual application and infrastructure nodes.
 Receive alerts based on custom or built-in health rules, including rules against
dynamic performance baselines that alert you to issues in the context of business
transactions.
 Analyze your applications at the code execution level using snapshots.

© 2020 AppDynamics, LLC. 4


Introduction to AppDynamics (APM211) Student Guide

APM at AppDynamics stands for Application Performance Management. We have an


entire APM suite of solutions which includes end user monitoring for browser, mobile
and IoT, business performance analytics, and logs, server and network monitoring with
Infrastructure Visibility.
APM can also stand for Application Performance Monitoring, which is one element of
our solution, and the one that we will focus on in this course.

© 2020 AppDynamics, LLC. 5


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 6


Introduction to AppDynamics (APM211) Student Guide

AppDynamics currently supports these languages.

© 2020 AppDynamics, LLC. 7


Introduction to AppDynamics (APM211) Student Guide

Extensions provide the ability to use customized metrics or partner with other
technologies to extend capabilities. Before creating your own extension from scratch,
review GitHub to see what extensions may be available to download:

https://github.com/Appdynamics/

https://developer.cisco.com/codeexchange/search/?products=AppDynamics&order=pro
ducts&q=.net

© 2020 AppDynamics, LLC. 8


Introduction to AppDynamics (APM211) Student Guide

Developers and Operations/Support personnel can get enormous value from


AppDynamics. They understand performance, find problems, and can use
AppDynamics to see how their code is performing.

© 2020 AppDynamics, LLC. 9


Introduction to AppDynamics (APM211) Student Guide

The Customer Experience is the responsibility of everyone in an organization. Using


AppDynamics, everyone from the Help Desk to the C-Level executives can and do get
value from AppDynamics.
QA and testing can perform load performance tests on sandbox and pre-production
environments. They can compare these results to production to see how well production
is performing. They can also perform full function testing and if something goes wrong,
they can go in and see what went wrong and share that with the development team.
Help Desk and Business Owners can use reporting and dashboards to get a general
view of performance, which directly affects customer satisfaction and retention.

© 2020 AppDynamics, LLC. 10


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 11


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 12


Introduction to AppDynamics (APM211) Student Guide

User feedback on application issues are typically anecdotal, and do not necessarily help
to determine the root cause.

With real-time visibility and insight into IT environments, AppDynamics allows your
organization to stop the "blame game" and instead align DevOps and business owners
around information that helps to deliver flawless customer experiences at scale.

© 2020 AppDynamics, LLC. 13


Introduction to AppDynamics (APM211) Student Guide

One of the results of applications that experience slow performance or downtime is an


angry customer.

© 2020 AppDynamics, LLC. 14


Introduction to AppDynamics (APM211) Student Guide

Source: IDC Study - The cost of downtime https://www.techrepublic.com/resource-


library/whitepapers/idc-study-the-cost-of-downtime/

© 2020 AppDynamics, LLC. 15


Introduction to AppDynamics (APM211) Student Guide

You may not currently know the answer to these questions, but they are important
questions that you should be asking about your application.
AppDynamics not only can help answer them, it can help reduce the amount of time to
detect, diagnose and repair.

© 2020 AppDynamics, LLC. 16


Introduction to AppDynamics (APM211) Student Guide

AppDynamics can take this locked down, complex application that communicates with
other applications and databases, and provide meaningful insight by bringing in all of
the performance information from production. It allows users both see the big picture
and provides them with the ability to zoom in to a more detailed look at specific
problems.
AppDynamics provides a single view of the application where everyone can see exactly
what is going on via the Controller. App Agents are installed on JVMs, .NET
applications, PHP web servers and Node.js. These agents send performance data to
the controller.

© 2020 AppDynamics, LLC. 17


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 18


Introduction to AppDynamics (APM211) Student Guide

Enterprise Topology and AppDynamics

Let’s walk through how it works at a high level using the example of a user logging on to
your application.
A user accesses the login page on the web or intranet. When the login request is sent to
the webserver, Application Performance Monitoring tags and follows the request for it’s
duration. It looks at the communication stream of that request – how much time did the
request take to process? Were there any downstream calls? How many? Were there
any errors? What were they?
If a database is involved, metrics around database communication and processing can
be captured as well. Calls to external web services are also tracked.
Before we look at how AppDynamics does this, let’s look at the Application – Tier –
Node structure within AppDynamics.

© 2020 AppDynamics, LLC. 19


Introduction to AppDynamics (APM211) Student Guide

Nodes:
A node is the basic unit of processing that AppDynamics monitors. Nodes belong to
tiers and a node cannot belong to more than one tier.
In AppDynamics, a node is instrumented and monitored by an AppDynamics agent.
An AppDynamics tier represents an instrumented service (such as a web application) or
multiple services that perform the exact same functionality, and may even run the same
code. In the AppDynamics model, a tier is composed of one node or multiple redundant
nodes. For example the inventory tier may have one node whereas the ecommerce tier
may use 3 nodes. Each node in a multi-node tier typically provides identical
functionality. One example of a multi-node tier is when you have a set of clustered
application servers or services.
Tiers:
Tiers, along with applications, are solely logical (not physical) grouping mechanisms.
In order to “map” your application environment correctly to the AppDynamics model, you
need to understand what an application means in AppDynamics.
Application:
An application is a grouping of one or more related customer applications that are
closely related by business function.
You can map multiple applications that are connected as a single AppDynamics
monitored application, or keep them separate. AppDynamics can track cross-application
requests, so if a transaction makes a call to another application, you can still trace the
call and troubleshoot.

© 2020 AppDynamics, LLC. 20


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 21


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 22


Introduction to AppDynamics (APM211) Student Guide

Let’s look at how this might be set up in AppDynamics.


Let’s say you have two applications – a Customer Portal Application and a Billing
System Application. Each of these has a UI Tier and a Service Tier. The billing System
Application also connects to a database.

© 2020 AppDynamics, LLC. 23


Introduction to AppDynamics (APM211) Student Guide

A user wants to pay their bill. That request starts in the UI tier of the Customer Portal
and then makes a call to the Billing System Service Tier.
In this configuration, if the “Pay Bill” request has an error, you would need to check both
applications in AppDynamics in order to determine the root cause of the error.

© 2020 AppDynamics, LLC. 24


Introduction to AppDynamics (APM211) Student Guide

A better way might be to create one AppDynamics application and then prefix the tiers
with the system names.
This setup means that you only have to check one application to find the error in the
‘Pay Bill’ request. Generally, you would be able to more easily see the full lifecycle of
transactions.
One potential downside is that you will end up with a potentially more complicated
AppDynamics application with many monitored transactions. And there is a 200
transaction limit per Application.
Now that we know how applications are structured, let’s look at how AppDynamics
monitors your application.

© 2020 AppDynamics, LLC. 25


Introduction to AppDynamics (APM211) Student Guide

Agents are plug-ins or extensions that monitor the performance of your application
code, runtime, and behavior.
There are a number of different agents in AppDynamics - including the Analytics Agent
and the EUM Agent. We're going to focus on Application Agents, or App Agents.
Once deployed, App agents immediately monitor every line of code.
Unique tags called GUIDs or (Global Unique Identifiers) are assigned to every request
and passed along to every method call and every downstream request header.
This allows AppDynamics to trace every transaction from start to finish—even in
modern, distributed applications.

© 2020 AppDynamics, LLC. 26


Introduction to AppDynamics (APM211) Student Guide

Notice that the communication arrows only go one way - from the Agent to the
Controller. However, please note that while the communication is always initiated by the
Agent, the communication can go both ways once the communication channel has been
established. Once the communication channel is established, the Agent can pull new
property configurations from the Controller.
Agents communicate with the controller in the following manner:
 The agent configuration channel queries the Controller for any new configuration
changes every 60 seconds and downloads the changes when they are available.
 The agent metric channel posts all new periodic metrics, including JMX, Windows
performance counters, and business transaction metrics to the Controller every 60
seconds.
 If there are new business transactions that haven’t been seen before by the agent,
they are posted to the Controller for registration every 10 seconds.
 If the agent has collected any new snapshots or events, they are posted to the
Controller every 20 seconds.

Configuration changes can sometimes take 2 to 3 minutes for the changes to actually
show up.

© 2020 AppDynamics, LLC. 27


Introduction to AppDynamics (APM211) Student Guide

Agents capture performance activity and send it to the Controller, which is updated in
real-time. The Controller helps monitor, troubleshoot and analyze your entire application
landscape—from backend infrastructure to the end user—in one simple interface.
AppDynamics supports the ability to trace calls to your applications even if these calls
jump from one application to another. Sometimes calls can cross Controller boundaries,
where one Controller gets call data from the next app in the call chain. AppDynamics
supports the concept of “drill-down” if you need to track a call-chain across this
Controller boundary.

© 2020 AppDynamics, LLC. 28


Introduction to AppDynamics (APM211) Student Guide

The Controller displays your application structure in a flow map. A flow map is an
interactive, graphical display of your application, tiers, nodes, and backends It shows
the tiers and nodes traversed by the transaction, the number of calls, and the
processing time. It also uses color to denote the health of the tiers, as well as any
downstream applications - green for healthy, yellow for warning, and red for critical.
With flow maps, you can:
 Quickly determine the health of your application and tiers
 Access any additional linked applications
 View metric call data for tiers and nodes
 Click items to drill down into the next level

You can also create custom flow maps, for example, if you were only interested in a
particular set of functions, you could create a flow map to only show those tiers.
Flow maps exist throughout the Controller. All applications, a single application, tier and
node flow maps, and transaction flow maps show you varying levels of granularity.

© 2020 AppDynamics, LLC. 29


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 30


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 31


Introduction to AppDynamics (APM211) Student Guide

The Application Dashboard

The Application Dashboard is the first thing you see when you open your app, and is
sort of grand central station for health and metric data for your application.
It contains your application flow map, as well as key metric charts that can show you the
general health of your application.
Across the top, you can access other tabs that contain information specific to this
application – events, top business transactions, and transaction snapshots, for example.

© 2020 AppDynamics, LLC. 32


Introduction to AppDynamics (APM211) Student Guide

One of the important things to keep in mind about what you’re seeing on the application
dashboard is that the metrics are show for the selected time period. You can change the
time period by selecting from a list of commonly used standard time periods, or you can
create your own custom time period.

Time Zones - Rather than translate the timestamp on snapshots and metrics to another
user's time zone, you can adjust the time zone setting to collaborate and troubleshoot
issues with more accuracy. Align your time zone with other users by going to the Gear
Icon in the top right > My Preferences, and use the Display Time Zone dropdown to
change your time zone. You will then need to reload the browser.

© 2020 AppDynamics, LLC. 33


Introduction to AppDynamics (APM211) Student Guide

There are three graphs across the bottom of the Application Dashboard.
They show you the overall load, response time, and number of errors for the entire
application for the selected period of time. It’s a good place to look for spikes, or even
trends or patterns.

© 2020 AppDynamics, LLC. 34


Introduction to AppDynamics (APM211) Student Guide

You can also click and drag your mouse over areas of interest in these graphs, and
AppDynamics will ask if you want to set the time range to match your selection. This is
extremely useful in focusing on issues.

© 2020 AppDynamics, LLC. 35


Introduction to AppDynamics (APM211) Student Guide

The Transaction Scorecard is both a visual representation of how your transactions are
performing based on your thresholds. It’s also interactive, and allows you to click the
different thresholds to open a list of those transactions.

© 2020 AppDynamics, LLC. 36


Introduction to AppDynamics (APM211) Student Guide

The Transaction Score is opens a detailed view of the performance of your transactions
over time. You can click the items in the legend to add or remove that performance
level. For example, if you want to remove the normal transactions so that you’re just
looking at slow, very slow, stalled and error, you would click Normal.

© 2020 AppDynamics, LLC. 37


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 38


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 39


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 40


Introduction to AppDynamics (APM211) Student Guide

Business Transactions

Application traffic is organized into business transactions. In AppDynamics, a business


transaction represents a distinct logical user activity (login, search, checkout, etc.) , or
something that has been invoked.
For example, on this banking website, you have a menu of tasks that you can perform.
You can sign in, check your balance, or check the current savings account interest
rates.
Each of these tasks is a business transaction in AppDynamics. AppDynamics traces the
transaction context across all tiers, including JVMs, CLRs, and calls to HTTP, JMS,
SOAP, databases, third-party web services, etc. The traced activities are bundled
together to make a business transaction. So for example, all the activities that happen
as a user logs in are grouped together and represented by one business transaction
called Login.
Business Transactions allow you to view your application traffic in a way that aligns with
the primary functions of your web business. This allows you to examine the
performance of your application as your users experience it.
Most performance monitoring and troubleshooting activities occur at the Business
Transaction level, so it’s important to understand how they work.
Let’s take a closer look at what this looks like in a flow map format.

© 2020 AppDynamics, LLC. 41


Introduction to AppDynamics (APM211) Student Guide

Let’s look at the high level anatomy of this transaction in a flow map format.
The transaction starts in the mz_UI tier, which took 1 ms to process or 32.5 percent of
the total transaction time. It then made an HTTP call to the mz_service tier which took .3
ms. The mz_service tier then took .7 ms to process the request, and made a database
call. That call took 1 ms. You can add up the total time of each piece to determine the
total transaction time.

© 2020 AppDynamics, LLC. 42


Introduction to AppDynamics (APM211) Student Guide

When AppDynamics is installed, it begins to detect business transactions using default


automatic detection rules. Configuring transaction detection is covered in the advanced
course.
You can access and manage your business transactions from the left menu.
The Business Transactions page allows you to see all of your currently monitored
business transactions as well as their performance metrics. You can view summary
information for each transaction’s performance. You can sort, filter, and search for
specific transactions. And you can use the more actions to rename, delete, exclude, and
configure thresholds for each transaction.

© 2020 AppDynamics, LLC. 43


Introduction to AppDynamics (APM211) Student Guide

It’s important to identify the key transactions that are the most important to you in terms
of business value and customer impact. This is especially true with larger applications
that have a lot of transactions, but even with smaller applications, determining which
transactions are key to your business helps keep you focused.
Identify the 10-20 most critical functions of the application—the ones that prompt a
midnight call if they fail.

© 2020 AppDynamics, LLC. 44


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 45


Introduction to AppDynamics (APM211) Student Guide

You can configure thresholds for each key transaction, which means you have control
over the definition of slow, very slow, or stalled for each key transaction.
Thresholds can be set in milliseconds, percentage relative to average, and the number
of standard deviations for a specific time period.

© 2020 AppDynamics, LLC. 46


Introduction to AppDynamics (APM211) Student Guide

Until now, we’ve talked about business transactions at the top level. AppDynamics
monitors every execution of a business transaction in the instrumented environment,
and the metrics reflect all executions.
However, for troubleshooting purposes, AppDynamics takes snapshots of specific
instances of a transaction. A transaction snapshot gives you a view of the processing
flow for a single invocation of a transaction.
Snapshots can be taken periodically (every 10 minutes), or when something significant
happens such as a slow, stalled, or error transaction. Diagnostic Sessions can also be
used to capture snapshots. Diagnostic Sessions will be covered in detail later.
Snapshot collection limits prevent excessive resource consumption at the node level:
 Originating transaction snapshots are limited to a maximum of 20 originating (5
concurrent) snapshots per node per minute.
 Continuing transaction snapshots are limited to a maximum of 200 (100 concurrent)
snapshots per node per minute.

AppDynamics applies snapshot collection limits to error transactions as well. As a


result, not every error occurrence that is represented in an error count metric, for
example, will have a corresponding snapshot. For error transactions, the following limits
apply:

© 2020 AppDynamics, LLC. 47


Introduction to AppDynamics (APM211) Student Guide

 For a single transaction, there is a maximum of two snapshots per minute.


 Across transactions, the maximum is limited to five snapshots per minute. (Specified
by the node property max-error-snapshots-per-minute.)

© 2020 AppDynamics, LLC. 48


Introduction to AppDynamics (APM211) Student Guide

Transaction Snapshots give you a cross-tier view of the processing flow for a single
invocation of a transaction. They include the user experience based on thresholds that
are default in AppDynamics and which you can customize.

© 2020 AppDynamics, LLC. 49


Introduction to AppDynamics (APM211) Student Guide

Transaction snapshots can contain full or partial call graphs. All call graphs list the
methods in a call stack and provide information about each call.

A full call graph lists the methods in a call stack and provides information about each
call.

A partial call graph represents the subset of the call sequence, typically from the point at
which the transaction has been determined to be slow or have errors.

Both periodic transaction snapshots and those snapshots captured as part of a


diagnostic session will contain full call graphs.

All slow, very slow, and stalled transactions will have either a partial or full call graph in
the snapshot.

Error transactions may or may not have call graph information but will contain exception
information. That's why you sometimes won't see the icon next to them.

© 2020 AppDynamics, LLC. 50


Introduction to AppDynamics (APM211) Student Guide

A transaction snapshot depicts a set of diagnostic data, for an individual transaction


across all app servers though which the transaction has passed. It is often the vehicle
for troubleshooting the root causes of performance problems.
Call drill downs, where available, detail key information including slowest methods,
errors, and remote service calls for the transaction execution on a tier. A drill down may
include a partial or complete call graph. Call graphs reflect the code-level view of the
processing of the business transaction on a particular tier.

© 2020 AppDynamics, LLC. 51


Introduction to AppDynamics (APM211) Student Guide

This is the call graph view for this transaction and tier.

© 2020 AppDynamics, LLC. 52


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 53


Introduction to AppDynamics (APM211) Student Guide

© 2020 AppDynamics, LLC. 54


Introduction to AppDynamics (APM211) Student Guide

To differentiate yourself and your company in an increasingly competitive market,


please consider becoming AppDynamics certified.

For more information, please copy and paste this URL into a separate tab in your
browser: https://learn.appdynamics.com/certifications

For information on specific certification tracks, please copy and paste the appropriate
URLs below:

AppDynamics Certified Associate Performance Analyst:


https://learn.appdynamics.com/certifications/performance-analyst

AppDynamics Certified Associate Administrator:


https://learn.appdynamics.com/certifications/administrator

AppDynamics Certified Implementation Professional:


https://learn.appdynamics.com/certifications/implementer

© 2020 AppDynamics, LLC. 55


Troubleshooting Basics (APM212) Student Guide

Troubleshooting Basics (APM212)

© 2020 AppDynamics, LLC. 56


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 57


Troubleshooting Basics (APM212) Student Guide

Troubleshooting Overview

Everybody has been in one of those meetings where something has gone wrong with
the application, and everyone is arguing about which component is at fault. Transaction
snapshots can take the guesswork out of this, and put you on the path to fixing the
issue.

© 2020 AppDynamics, LLC. 58


Troubleshooting Basics (APM212) Student Guide

So here we’re looking at a snapshot. We can see that it’s a very slow snapshot. Let’s
see how we can find out more information.

© 2020 AppDynamics, LLC. 59


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 60


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 61


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 62


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 63


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 64


Troubleshooting Basics (APM212) Student Guide

Diagnostic Sessions

Diagnostic sessions allow you to collect extra snapshots on Business Transactions to


help you troubleshoot issues within your application. These “extra” snapshots can be
helpful because not only do they allow you to see more instances of a business
transaction, they also provide “full call graphs” within the snapshot. A full call graph
captures the entire call sequence for the business transaction invocation, and contains
information on each monitored node involved in the processing of the business
transaction.

Under normal circumstances, snapshots are only collected periodically or when there is
a performance issue; many of these snapshots contain partial call graphs, or no call
graphs at all. Snapshots with full call graphs are more likely to help diagnose
performance issues, allowing you to optimize the flow of a complex Business
Transaction.

The system can trigger diagnostic sessions manually through the user interface or you
can configure them to start automatically when thresholds for slow, stalled, or error
transactions are reached. If the diagnostic session is triggered manually, the diagnostic
session collects snapshots on each node that the selected Business Transaction
passes through. If the diagnostic session is triggered to start automatically, the
diagnostic session collects snapshots on the triggering node only.

© 2020 AppDynamics, LLC. 65


Troubleshooting Basics (APM212) Student Guide

Diagnostic sessions can be started manually, or automatically, either in response to a


performance issue, or as a response to an event.

Let’s take a closer look at each of these.

© 2020 AppDynamics, LLC. 66


Troubleshooting Basics (APM212) Student Guide

You can manually trigger a diagnostic session when you need one or configure them to
start up automatically.

To start a diagnostic session manually, go to the Business Transaction dashboard and


select the Business Transaction you want to run the session on.

Click Actions > Start Diagnostic Session, then specify the snapshot collection
duration at the bottom of the Start Diagnostic Session window. The captured
snapshots are available from the Transaction Snapshots tab > Diagnostic Sessions.

When configuring a manual or automatically triggered diagnostic session, the duration


and frequency to take snapshots must be specified. The duration can be from 1 to 10
minutes, and the frequency can be from 1 to 10 snapshots. You should not run a
diagnostic session continuously. You can set a wait period between sessions and
increase the time as needed.

© 2020 AppDynamics, LLC. 67


Troubleshooting Basics (APM212) Student Guide

Diagnostic sessions are automatically triggered when there are a high number of slow
or error transactions. For example, if more than 10% of the requests in a minute are
slow or have errors, a diagnostic session begins. And like other features in
AppDynamics, you can adjust the specific thresholds based on your own environment.
To configure the threshold value, go to Configuration > Slow Transaction Thresholds
> Diagnostic Sessions Settings.

© 2020 AppDynamics, LLC. 68


Troubleshooting Basics (APM212) Student Guide

Diagnostic sessions are also available as and action in AppDynamics. We’ll look at
policies in a later module. Policies ultimately allow you to take an event in AppDynamics
and determine a response or action to that event. Running a diagnostic session is one
of those available actions, and can be useful, because you’re telling AppDynamics to
collect full call graph snapshots when something happens, for example, a transaction
breaches a performance threshold.

© 2020 AppDynamics, LLC. 69


Troubleshooting Basics (APM212) Student Guide

Troubleshooting with Transaction Snapshots

Remember that a Business Transaction represents a distinct logical user activity (login,
search, checkout, etc.).

Business Transactions allow you to examine the performance of your application as


your users experience it, and they are at the root of most monitoring and
troubleshooting activities.

© 2020 AppDynamics, LLC. 70


Troubleshooting Basics (APM212) Student Guide

And transaction snapshots give you a cross-tier view of the processing flow for a single
invocation of a transaction.

Transaction snapshots can contain full or partial call graphs. All call graph list the
methods in a call stack and provides information about each call.

A partial call graph represents the subset of the call sequence, typically from the point at
which the transaction has been determined to be slow or have errors.

Both periodic transaction snapshots and those snapshots captured as part of a


diagnostic session will contain full call graphs.

All slow/very slow/stalled transactions will have either a partial or full call graph in the
snapshot.

Error transactions may or may not have any call graph information but will contain
exception information. That's why you sometimes won't see the icon next to them.

© 2020 AppDynamics, LLC. 71


Troubleshooting Basics (APM212) Student Guide

Drilling down on a tier using the Drill Down button on the Overview tab flow map on
the Transaction Details screen opens the code-level details for the Business
Transaction execution on that particular tier. This is how you access the call graph
information.
 The Overview tab includes a problem summary, execution times, CPU time stamp,
tier, node, process ID, thread name, etc.
 The Call Graphs tab lists call graphs showing the execution flow for the transaction
on a given tier. Full snapshots have complete call graphs. Partial snapshots can
have partial call graphs or none at all.
 The Slow Calls and Errors tab lists all the slow method calls and calls that resulted
in an error. You can use the Hot Spots slider to sort calls by execution time with the
most expensive calls in the snapshot at the top.
 The Error Details tab exposes exception stack traces and HTTP error codes.
 The DB & Remote Service Calls tab shows all SQL query exit calls to databases
and exit calls to other remote services such as web services, message queues or
caching servers.
 The Server tab displays graphs for hardware (CPU Memory, Disk IO, Network IO),
Memory (Heap, Garbage Collection, Memory Pools), JMX, and more. If you
have Server Visibility, you'll have access to full performance details for the server
hardware and operating system

© 2020 AppDynamics, LLC. 72


Troubleshooting Basics (APM212) Student Guide

 For customers using Network Visibility, the Network tab shows charts related to the
impact of the network on the transaction and other pertinent data.
 The More tab shows hows metrics for the node that deviate the most from the
established baselines as Node Problems. It also shows all the Service Endpoints
invoked during the snapshot and the Servliet URI and Process ID of the transaction.

© 2020 AppDynamics, LLC. 73


Troubleshooting Basics (APM212) Student Guide

One of the results of applications that experience slow performance or downtime is an


angry customer.

And a great first place to look is Business Transactions via transaction snapshots.

© 2020 AppDynamics, LLC. 74


Troubleshooting Basics (APM212) Student Guide

Let’s walk through how you might use Business Transactions and transaction snapshots
to find the root cause of an issue. Each of the highlighted areas will get you to
transaction snapshots although some take a couple of clicks, such as Business
Transactions and Tiers and Nodes in the left nav.

In this case, imagine that support has contacted you about users receiving login errors.

The methods for opening transaction snapshots on this page all eventually open a
general transaction snapshots page, where you could then filter and search for the
specific transaction - in this case, Login. But there’s an even faster way to get to the
transaction snapshots for this transaction.

© 2020 AppDynamics, LLC. 75


Troubleshooting Basics (APM212) Student Guide

You can click Business Transactions on the left menu to see a list of transactions and
then drill down from there.

Here’s the Business Transactions screen, and we can see that something’s going on
with the Login transaction. So let’s open that up by double-clicking it.

© 2020 AppDynamics, LLC. 76


Troubleshooting Basics (APM212) Student Guide

Now we're looking at just the snapshots for Login (Business Transactions > Login).
Here we could filter to only show those that are slow, very slow, or error transactions.

Or, we could click the Slow and Error Transactions subtab to start with those
snapshots with issues. Let’s do that.

© 2020 AppDynamics, LLC. 77


Troubleshooting Basics (APM212) Student Guide

On the Slow and Error Transactions subtab, you can use search and filter to further
narrow down to the specific Business Transaction.

We’ve got a very slow login transaction right here which has a full call graph, so let’s
look at that one.

Double-clicking the snapshot opens the snapshot details.

© 2020 AppDynamics, LLC. 78


Troubleshooting Basics (APM212) Student Guide

The transaction snapshot details page offers you additional information on


troubleshooting. Potential issues are listed on the left.

The Slow Calls and Errors tab allows you to focus only on slow and very slow calls,
and errors.

The Waterfall View tab shows the execution of an individual Business Transaction
broken into execution segments.

Segment List shows the various legs of the transaction in descending order of duration
and give access to their snapshots and allows you to drill down into their details.

And the Drill Down links over the tiers in the flow map allow you to drill down into the
details of what was going on with that transaction in the tier.

As the communication between the service tier and the database seems to be very
slow, let’s drill down there.

© 2020 AppDynamics, LLC. 79


Troubleshooting Basics (APM212) Student Guide

The call graph screen lets you see where the most time was spent in the transaction.
You can see that the last call is taking up most of the time. So let’s look at the details of
that call. You can view the details by selecting the call and clicking the details button on
the toolbar.

© 2020 AppDynamics, LLC. 80


Troubleshooting Basics (APM212) Student Guide

In this case, it looks like a sleep call was causing the slowdown on the login transaction.

© 2020 AppDynamics, LLC. 81


Troubleshooting Basics (APM212) Student Guide

To get the information out of AppDynamics so that you can send it to the appropriate
audience, you can either export the call graph information using the Export button on
the toolbar, or you can right-click the call and select Copy to Clipboard. You can then
paste the info into an email or document.

© 2020 AppDynamics, LLC. 82


Troubleshooting Basics (APM212) Student Guide

You can export the snapshot details at the overview level, or on any of the other tabs
here by clicking the Actions drop down and selecting Export.

The chain let’s you send a link to the snapshot details. If you want the link to be
available for longer than the standard two weeks, archive the snapshot.

© 2020 AppDynamics, LLC. 83


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 84


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 85


Troubleshooting Basics (APM212) Student Guide

Troubleshooting Slow Transactions

It looks like loan applications are starting to be very slow. Loan applications make up a
large portion of our bank’s business so we need to figure out what’s going on.

© 2020 AppDynamics, LLC. 86


Troubleshooting Basics (APM212) Student Guide

When troubleshooting slow and very slow transactions, start by clicking Business
Transactions in the left menu. Even though there’s a green check-mark, more calls are
starting to come in. So let’s open that transaction by double-clicking it.

© 2020 AppDynamics, LLC. 87


Troubleshooting Basics (APM212) Student Guide

Now we can look specifically at the slow and error transactions for the Apply for Loan
Business Transaction.

© 2020 AppDynamics, LLC. 88


Troubleshooting Basics (APM212) Student Guide

And here’s a very slow transaction with a full snapshot. So let’s open this one.

© 2020 AppDynamics, LLC. 89


Troubleshooting Basics (APM212) Student Guide

The transaction details opens to the Overview tab, and we can see the details of this
transaction. On the flow map, it looks like both tiers have issues, but let’s start with the
drill down link on the UI tier.

© 2020 AppDynamics, LLC. 90


Troubleshooting Basics (APM212) Student Guide

Here we can see the entire call graph, and easily see where the time was spent. To see
more details about the slow call, click to select it, and click the Details button.

© 2020 AppDynamics, LLC. 91


Troubleshooting Basics (APM212) Student Guide

And here are the details for that call.

© 2020 AppDynamics, LLC. 92


Troubleshooting Basics (APM212) Student Guide

Automated Transaction Diagnostics uses statistical analysis to identify transaction


snapshots for anomalies that could cause issues in the future. These snapshots may
not have triggered a Health Alert.

ATD delivers a prioritized list of causal factor transactions that you can use to manage
problematic business transactions before they become actual production issues or
outages. ATD's guided root-cause analysis clearly exposes the offending anomalies
along with the contributing tiers, exit calls, or inter-tier network issues.

© 2020 AppDynamics, LLC. 93


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 94


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 95


Troubleshooting Basics (APM212) Student Guide

Troubleshooting Error Transactions

An error transaction can be:


 An unhandled exception
 HTTP error codes from 400 to 505
 An exception that is handled but logged with a severity of Error or Fatal using Log4j
or java.util.logging NLog or Log4Net
 A handled exception that occurs during a call to an exit point

They can be accessed using the Transaction Scorecard, and via the Exceptions
summary section underneath the Transaction Scorecard on the Application
Dashboard. You can also open errors from the left navigation pane under
Troubleshoot > Errors.
If a transaction experiences an error, it is counted as an error transaction and not as a
slow, very slow or stalled transaction even if the transaction was also slow or stalled.
An application server exception is a code-logged message outside the context of a
Business Transaction.
There is not a one-to-one correspondence between the number of errors and the
number of exceptions. For example, a Business Transaction may experience a single
code 500 error in which several exceptions were logged as the transaction passed
through multiple tiers.

© 2020 AppDynamics, LLC. 96


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 97


Troubleshooting Basics (APM212) Student Guide

You can troubleshoot transaction errors from this screen. The error transaction
snapshots are listed in the lower part of the viewer. Select the snapshot you want to
investigate, and click the View Transaction Snapshot button to view the snapshot
details including where the error occurred and information about the error.

© 2020 AppDynamics, LLC. 98


Troubleshooting Basics (APM212) Student Guide

Exceptions that are thrown and handled within a Business Transaction are not captured
by AppDynamics, and do not appear in the Exceptions tab.

If a stack trace for the exception is available, you can access it from the Exceptions tab
in the Controller UI. A stack trace is available for a given exception if the exception was
passed in the log call. For example a logging call in the form of
logger.log(Level.ERROR, String msg, Throwable e) would include a stack trace,
whereas a call in the form of logger.log(Level.ERROR, String msg) would not.

© 2020 AppDynamics, LLC. 99


Troubleshooting Basics (APM212) Student Guide

AppDynamics error detection settings are flexible, and the configuration changes you
make will be applied to all tiers in the application. Access the error configuration screen
by navigating to Configuration > Instrumentation > Error Detection.

© 2020 AppDynamics, LLC. 100


Troubleshooting Basics (APM212) Student Guide

Certain types of exceptions, loggers or log messages as transaction error indicators


may not reflect events that should be counted as transaction errors in your environment.
They may include exceptions raised by application framework code or by user login
failures.

When you configure an exception to be ignored, the agent detects the exception,
increments the exception count, and displays the exception in Exceptions lists in the UI,
but the Business Transaction in which the error is thrown is not considered an "error
transaction" for monitoring purposes. The transaction snapshot would not show the
exception in the Summary or Error Details section and the user experience for the
transaction instance would be unaffected by the exception.

To configure exceptions for the agent to ignore, click the Add New Exception to
Ignore button in the error configuration window.

Enter the class name of the exception to be ignored and the match condition for the
exception message. For the match condition, if you do not need to filter for specific
messages in the exception, select "Is Not Empty". If you want to filter for Null, select "Is
Not Empty" and use the gear icon to select NOT, which, in effect, tests for "is empty".

© 2020 AppDynamics, LLC. 101


Troubleshooting Basics (APM212) Student Guide

Support starts to get calls about payments not going through correctly, and users are
getting error messages.

Troubleshooting errors is similar to troubleshooting slow transactions.

© 2020 AppDynamics, LLC. 102


Troubleshooting Basics (APM212) Student Guide

You can either get to errors by clicking the Errors link in the Transaction Scorecard, or
by expanding the Troubleshoot menu item on the left nav, and selecting Errors. In this
case, we know the problem is with the online Loan Application transaction, so we can
go directly to that transaction on the Business Transactions page.

© 2020 AppDynamics, LLC. 103


Troubleshooting Basics (APM212) Student Guide

With error transactions, you don’t have to worry about full or partial snapshots. Just
double-click the error to open it.

© 2020 AppDynamics, LLC. 104


Troubleshooting Basics (APM212) Student Guide

The error details screen gives you more information. The potential issues area is
populated with an error icon.

You can see on the flow map that the error occurred on the Admin_UI tier, and you can
drill down to find out more.

You can also click the Slow Calls and Errors tab to find out more. Let’s go that route
and see what’s there.

© 2020 AppDynamics, LLC. 105


Troubleshooting Basics (APM212) Student Guide

The Slow Calls and Errors tab can give you more details on specific slow calls,
methods, and errors. In this case, we need to see the Errors tab, which shows us the
specific error.

© 2020 AppDynamics, LLC. 106


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 107


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 108


Troubleshooting Basics (APM212) Student Guide

© 2020 AppDynamics, LLC. 109


Troubleshooting Basics (APM212) Student Guide

To differentiate yourself and your company in an increasingly competitive market,


please consider becoming AppDynamics certified.

For more information, please copy and paste this URL into a separate tab in your
browser: https://learn.appdynamics.com/certifications

For information on specific certification tracks, please copy and paste the appropriate
URLs below:

AppDynamics Certified Associate Performance Analyst:


https://learn.appdynamics.com/certifications/performance-analyst

AppDynamics Certified Associate Administrator:


https://learn.appdynamics.com/certifications/administrator

AppDynamics Certified Implementation Professional:


https://learn.appdynamics.com/certifications/implementer

© 2020 AppDynamics, LLC. 110


Advanced Troubleshooting and Tools (APM213) Student Guide

Advanced Troubleshooting and Tools (APM213)

© 2020 AppDynamics, LLC. 111


Advanced Troubleshooting and Tools (APM213) Student Guide

© 2020 AppDynamics, LLC. 112


Advanced Troubleshooting and Tools (APM213) Student Guide

Too Many / Slow Database Calls

One problem we see frequently is too many database queries per


request/transaction. In this context, “too many” means more than necessary.

There could be a number of database issues. In this topic, we’re just going to discuss
the following three issues.
 More data is requested than actually required in the context of the current
transaction. E.g., requesting all account information instead of only requesting the
pieces we need to display on the current screen.
 The same data is requested multiple times in the same transaction. This usually
happens as a result of misplaced code.
 Multiple queries are executed to retrieve a certain set of data. This is often a result of
not taking full advantage of complex SQL statements or stored procedures and
pagination to retrieve the data in one batch.

© 2020 AppDynamics, LLC. 113


Advanced Troubleshooting and Tools (APM213) Student Guide

For troubleshooting transactions that have too many database calls, step through this
process – looking at performance for each query as you go, and evaluate whether
you’re using “too many” database queries. Be aware that database connections as well
as database processing time tend to be precious application resources, which should be
conserved.

© 2020 AppDynamics, LLC. 114


Advanced Troubleshooting and Tools (APM213) Student Guide

The Slow Calls and Errors tab on the Transaction Snapshot details screen has three
subtabs for the slowest calls to DB and remote services, the slowest methods, and
errors.

© 2020 AppDynamics, LLC. 115


Advanced Troubleshooting and Tools (APM213) Student Guide

You can take a broader look at the slowest calls by tier by clicking Database Calls on
the left menu. This opens the database dashboard which displays communication, load,
and error health. Click the Slowest Database Calls sub-tab to see a list of calls with
long execution times. You can then drill into snapshots.

© 2020 AppDynamics, LLC. 116


Advanced Troubleshooting and Tools (APM213) Student Guide

© 2020 AppDynamics, LLC. 117


Advanced Troubleshooting and Tools (APM213) Student Guide

Thread State Analysis

There is no question that synchronization is necessary to protect shared data in an


application. Too often, application developers make the mistake of over-
synchronization, e.g., excessively-large code sequences are synchronized. Under low
load—such as on the local developers workstation—performance will not suffer
significantly. However, in a high load environment, such as a production enterprise
application, over-synchronization results in severe performance and scalability
problems.

© 2020 AppDynamics, LLC. 118


Advanced Troubleshooting and Tools (APM213) Student Guide

Since the stall and the deadlock are monitored from 2 independent sources it is possible
that a stall in a Business Transaction can appear as a deadlock along the timeline.
However not every deadlock will appear as a stall only those that are within a Business
Transaction.

You may have a number of different Business Transactions that are slowing down.
When you look at the snapshots you may see various slow, very slow, and stalled
transactions. All of these could be attributed to a single bottleneck.

© 2020 AppDynamics, LLC. 119


Advanced Troubleshooting and Tools (APM213) Student Guide

Waiting threads cannot use the resource until the other thread has released the specific
object. This results in slow or blocked performance.
Multithreading is a type of concurrent processing that uses resources more efficiently.
However, multi-threading can create conditions that result in slow or blocked
performance.
For example: two threads attempt to access the same resource (or related resource)
this could result in performance suffering.
Synchronizing methods or statements can prevent thread interference, memory
consistency and deadlock patterns.
However synchronization could introduce thread starvation and livelock.

© 2020 AppDynamics, LLC. 120


Advanced Troubleshooting and Tools (APM213) Student Guide

Business Transaction Thread Contention

To view details about thread contention for a business transaction:

Select a transaction snapshot > Transaction Flow Map > Potential Issues.

The issues labelled as Thread Contentions display the blocked time for the thread.

Double-click one of thread contention issues to display further information about the
method that is blocked. Select Drill Down into Call Graph.
The actual wait state could be as a result of any of the following:
 Thread.sleep
 Object.wait
 Thread.join
 LockSupport.parkNanos
 LockSupport.parkUntil
 LockSupport.park

© 2020 AppDynamics, LLC. 121


Advanced Troubleshooting and Tools (APM213) Student Guide

Data Collectors

Data collectors can collect any data passing through an application as long as it can be
displayed as a string. They can help determine whether data that a transaction passed
into an application is causing problems.

When you configure data collectors, AppDynamics accesses information in application


code arguments, return values, and getter chain and displays the information in the
transaction details drill down screen.

Data Collectors are also used to collect data for the analytics engine.

© 2020 AppDynamics, LLC. 122


Advanced Troubleshooting and Tools (APM213) Student Guide

Method Invocation Data Collectors


 Capture code data such as method arguments, variables, and return values
 Use to create custom fields to collect business related data

HTTP Requests
 Capture the URLs, parameter values, headers, and cookies of HTTP messages
exchanged in a business transaction
 When HTTP data collectors are configured, the following information can be
collected (see list on slide)

© 2020 AppDynamics, LLC. 123


Advanced Troubleshooting and Tools (APM213) Student Guide

© 2020 AppDynamics, LLC. 124


Advanced Troubleshooting and Tools (APM213) Student Guide

© 2020 AppDynamics, LLC. 125


Advanced Troubleshooting and Tools (APM213) Student Guide

Troubleshooting Node Level Issues

Click Tiers & Nodes in the left menu to open a list of the tiers and nodes in your
application. From the main screen, you can select a node and click Actions to view the
health rule violations that have occurred on that node.

You can also use the Show Data drop down and select from Health, Hardware,
Memory, and Network to see data for each of these.

You can also drill down into each node by either selecting the node and clicking Details,
or by double clicking a node.

© 2020 AppDynamics, LLC. 126


Advanced Troubleshooting and Tools (APM213) Student Guide

As with the Application and Business Transactions dashboards, each node has a
dashboard where you can view the node flow map, load, error and response time
information, and a transaction scorecard. You can also view exceptions and errors, and
any events that occurred during the selected time range.

© 2020 AppDynamics, LLC. 127


Advanced Troubleshooting and Tools (APM213) Student Guide

In addition to the dashboard information, there are sub-tabs across the top that give you
access to additional information for that node - from server and memory metrics, to
transaction snapshots and score. Let’s take a closer look at Transaction Score.

© 2020 AppDynamics, LLC. 128


Advanced Troubleshooting and Tools (APM213) Student Guide

You can click the Transactions tab on the Tier dashboard screen to see all
transactions.

For a more detailed view of the performance of transactions on that tier over time, click
the Transaction Score tab. The graph in the tab shows the user experience for that
tier, showing its performance relative to the performance thresholds as a bar chart.

For each time slot, the bar reflects the total calls, while the color segments—green,
yellow and so on—indicate the relative number of those calls that were normal, slow,
very slow, stall or errors.

The solid blue line in the graph indicates the average response time for the transaction
over time, while the dotted line shows the baseline, if available.

© 2020 AppDynamics, LLC. 129


Advanced Troubleshooting and Tools (APM213) Student Guide

Memory Management

If major garbage collections occur frequently, or consume a lot of time, application


latency might increase, as many garbage collection algorithms suspend other
application tasks as they compact the memory pools. A large amount of time spent
garbage collecting also affects application throughput; CPU cycles spent on garbage
collection reduce the amount of processor time available to process user requests.

Frequent major garbage collections are also an indication that the heap may be
improperly tuned. Short-lived objects should be collected during minor garbage
collections, before they get promoted to long-lived (“old generation”) memory pools. If
the memory pools reserved for short-lived (“young generation”) objects are too small,
objects may be promoted prematurely, and as a result both minor and major garbage
collections occur more frequently than needed

© 2020 AppDynamics, LLC. 130


Advanced Troubleshooting and Tools (APM213) Student Guide

In grid view, in the Tiers & Nodes List > Memory (Java) dropdown. The running Java
JVMs in your environment are displayed.

Sort the nodes by GC Time Spent in descending order. You can then identify where
garbage collection is taking too long and possibly hindering app performance.

© 2020 AppDynamics, LLC. 131


Advanced Troubleshooting and Tools (APM213) Student Guide

You can view memory health for each tier/node by selecting Memory (Java) or Memory
(.NET) on the Tiers & Nodes page. Memory health should be fairly consistent across
nodes of a given tier.

© 2020 AppDynamics, LLC. 132


Advanced Troubleshooting and Tools (APM213) Student Guide

Memory metrics vary between Java and .NET.

© 2020 AppDynamics, LLC. 133


Advanced Troubleshooting and Tools (APM213) Student Guide

 ***CMS = ‘Concurrent-Mark-Sweep’
 Object Allocation Rate - tracks the number of objects created per minute.

© 2020 AppDynamics, LLC. 134


Advanced Troubleshooting and Tools (APM213) Student Guide

 Object creation almost always happens in the Eden space.


o This metric is important for tuning Young Gen size, as a very high allocation rate
means Eden space fills up quicker, which leads to more frequent minor garbage
collection (a Stop-The-World event).
o By understanding the relationship between the allocation rate and two other
metrics: free rate and promotion rate, the young heap can be sized to fill up less
frequently.
 Object Promotion Rate - tracks the number of objects promoted from young to old
gen per minute.
o This metric is important for tuning both young and old size, as a very high
promotion rate means Eden (or Survivor) space may need to be increased, and
leads to quicker filling of the old gen, which leads to more frequent major
garbage collection (a Stop-The-World event if using default Parallel/Throughput
collector memory collection and might for the others).
 Live Data Size - tracks the memory footprint of the application.
o This metric is useful for understanding the effect of application changes or
changes in load characteristics on the amount of memory the application needs
to work efficiently.
o Any change that results in a higher memory footprint needs to be analyzed –
having this metric allows analysis of memory footprint regression.
 Object Free Rate - tracks the number of objects freed from young gen per minute.

o This metric is important for tuning the size of the young generation and can give
some clues to possible coding issues.
o It can be used to analyze the effect of young gen tuning, and can be compared to
Object Allocation Rate to anticipate and detect memory leaks.

© 2020 AppDynamics, LLC. 135


Advanced Troubleshooting and Tools (APM213) Student Guide

© 2020 AppDynamics, LLC. 136


Advanced Troubleshooting and Tools (APM213) Student Guide

OIT tracks objects that are created, but never released. Thus helps pinpoint a type of
memory leak.

Once the object types and memory allocation patterns have been identified, you can
then start an allocation tracking session to identify the Business Transactions and code
paths that are responsible for the excessive object allocations.

© 2020 AppDynamics, LLC. 137


Advanced Troubleshooting and Tools (APM213) Student Guide

A collection is also called a ‘container.’


Common collection types are any class that implements JDK 'Map' or 'Collection'
interface, for example:
 Hash maps
 Vectors
 Linked lists

To be a candidate, by default the collection must be at least 5mb in size and have 1000
or more elements.

© 2020 AppDynamics, LLC. 138


Advanced Troubleshooting and Tools (APM213) Student Guide

If you track the long-lived (old generation) memory pools over multiple major garbage
collection cycles, and observe that the minimum amount of memory used (the valleys in
the graph) is steadily increasing over time, the application may be suffering from a
memory leak.

The Java Agent offers an automatic leak detection feature to help identify the source of
the memory leak. This feature tracks collections over time to identify growing
collections. The minimum collection age and size is configurable. If a potential memory
leak is found, you can inspect the collection content or use access tracking to determine
which code paths are accessing the growing collection.

© 2020 AppDynamics, LLC. 139


Advanced Troubleshooting and Tools (APM213) Student Guide

A frequent problem is too many database queries per request/transaction and/or


queries that are too slow, but what is “too many” or “too slow” in this context? There are
3 specific instances:
 More data is requested than is actually required in the context of the current
transaction.
 The same data is requested multiple times by in a single transaction.
 Multiple queries are executed to retrieve a certain set of data. This is often a result of
not taking full advantage of complex SQL statements or stored procedures to
retrieve the data in one batch.

Too often people make the mistake of over-synchronization (e.g. excessively-large code
sequences are synchronized). In a high-load or production environment over-
synchronization results in severe performance and Memory Leaks.

It is important to release object references as soon as they are no longer needed. It is


important to understand that Java’s Automatic Garbage Collection does not prevent
memory leaks. Additionally, web application reloading can affect parts of the memory
heap and cause unsuspected memory leaks.

© 2020 AppDynamics, LLC. 140


Advanced Troubleshooting and Tools (APM213) Student Guide

© 2020 AppDynamics, LLC. 141


Advanced Troubleshooting and Tools (APM213) Student Guide

To differentiate yourself and your company in an increasingly competitive market,


please consider becoming AppDynamics certified.

For more information, please copy and paste this URL into a separate tab in your
browser: https://learn.appdynamics.com/certifications

For information on specific certification tracks, please copy and paste the appropriate
URLs below:

AppDynamics Certified Associate Performance Analyst:


https://learn.appdynamics.com/certifications/performance-analyst

AppDynamics Certified Associate Administrator:


https://learn.appdynamics.com/certifications/administrator

AppDynamics Certified Implementation Professional:


https://learn.appdynamics.com/certifications/implementer

© 2020 AppDynamics, LLC. 142


Monitoring the Health of Your Application (APM214) Student Guide

Monitoring the Health of Your Application (APM214)

© 2020 AppDynamics, LLC. 143


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 144


Monitoring the Health of Your Application (APM214) Student Guide

Service Endpoints and Information Points

© 2020 AppDynamics, LLC. 145


Monitoring the Health of Your Application (APM214) Student Guide

A business transaction offers a view of application performance that cuts across the
environment, reflecting all services that participate in fulfilling the transaction. Another
way of viewing application performance, however, focuses on the performance of a
particular service independently of the business transactions that use it. For example,
imagine that you’re in charge of the eligibility check that users have to go through to get
a loan. That check involves multiple transactions, but neither individual transaction
gives you the information you need. A Service Endpoint solves this by focusing on your
specific service.
Like business transactions, service endpoints give you key performance indicators,
metrics, and snapshots. However, they provide that information exclusively in the
context of that service, omitting business transaction context or downstream
performance data.
Service endpoints are similar to business transactions except that they report metrics
only at the entry point and do not track metrics for any downstream segments. Service
endpoints support the same entry point types as business transactions and you can
configure them in a similar way.

© 2020 AppDynamics, LLC. 146


Monitoring the Health of Your Application (APM214) Student Guide

Like Business Transactions, Service Endpoints give you key performance indicators,
metrics, and snapshots. However, they provide that information exclusively in the
context of that service, omitting BT context or downstream performance data.

© 2020 AppDynamics, LLC. 147


Monitoring the Health of Your Application (APM214) Student Guide

An Information Point is used to instrument a method. Information Points capture


numeric data that fall into two categories: system metrics and custom (or business)
metrics. Code metrics evaluate how a method is performing across the application.
Business Metrics, also called Custom metrics, evaluate data from a method’s
parameters or return values, or anything that could be called with a getter chain.
Custom metrics are aggregated – either averaged or summed over time.

For example, if you set up an Information Point to get metrics about a particular method
invocation, such as credit card authentication, any time that authentication method is
invoked the Information Point is triggered.

© 2020 AppDynamics, LLC. 148


Monitoring the Health of Your Application (APM214) Student Guide

Information points can capture how a method is performing, as well as data from a
method’s parameters or return values to report on the performance of the business.

© 2020 AppDynamics, LLC. 149


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 150


Monitoring the Health of Your Application (APM214) Student Guide

You may be wondering how the Data Collectors differ from Information Points. They are
very similar as they collect information about data passed to an application, and both
must be explicitly configured.
Data Collectors add STRING data to a Business Transaction Snapshot. Data from Data
Collectors appear in the panels of the Transaction Snapshot call drill down - HTTP
DATA, COOKIES, or USER DATA - depending on the type of the Data Collector. Data
Collectors do not appear in the Metric Browser and cannot be used in Health Rules.
You can filter Transaction Snapshots based on the value of a Data Collector in the
Transaction Snapshot list or using the AppDynamics REST API.
Information Points aggregate numeric data about invocations of a method outside the
context of any Business Transaction. Use Information Points to get metrics about
method invocations across zero, one or multiple Business Transactions. Any time the
method configured for the Information Point is invoked, no matter from where, the
Information Point is triggered.
Like Business Transactions, Service Endpoints give you key performance indicators,
metrics, and snapshots. However, like Information Points, they provide that information
independent of the context of a Business Transaction or downstream performance data.

© 2020 AppDynamics, LLC. 151


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 152


Monitoring the Health of Your Application (APM214) Student Guide

Introduction to the Alert and Respond Model

In this module we are going to take a look at the proactive alerting mechanism, and the
individual pieces that make up this mechanism.

The AppDynamics alert and respond model starts with metrics. Specifically those
metrics that are critical. Once those critical metrics have been defined, health rules can
be created to monitor those metrics.

A policy connects a trigger and an action. The trigger could be a health rule violation, an
error, or other event. The action could be a notification, or a remediation script.When
you create a policy, you’re specifying both the trigger and the action.

Three pieces work together to enable this feature, so before we discuss each piece let’s
take a quick look at the whole to understand the relationship.
In this model, you’ve set up your performance thresholds and health rules. An event
occurs. This could be a health rule violation or other AppDynamics event. These events
serve as the trigger.

© 2020 AppDynamics, LLC. 153


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 154


Monitoring the Health of Your Application (APM214) Student Guide

Metrics and Baselines

Let’s start our look at this model by focusing on Performance Metrics and Health Rules.

© 2020 AppDynamics, LLC. 155


Monitoring the Health of Your Application (APM214) Student Guide

AppDynamics reports metrics for monitored systems over time. The AppDynamics UI
provides a variety of graphs and tools to help you visualize metric data so you can use it
effectively for analyzing your application's performance.
The AppDynamics user interface organizes the infrastructure metrics for your
application by tiers. In addition to performance metrics for the tiers, for each application,
you can see metrics gathered by the machine agent (when installed). In general, agents
upload metrics to the controller once a minute at staggered intervals. The controller then
rolls up the data and makes it available in the UI.

© 2020 AppDynamics, LLC. 156


Monitoring the Health of Your Application (APM214) Student Guide

The Metric Browser is the main interface through which you inspect metrics. The Metric
Browser displays all the metric values collected over the selected time period as an
expandable tree.

In the browser, you can drag and drop metrics from the tree onto graphs to compare
metrics and analyze patterns.

To see the metrics in each category, access the Metric Browser and expand the
categories in the left pane of the browser until you get the metric you want to display.
They are organized first by category, then tier, then sub-categories.

© 2020 AppDynamics, LLC. 157


Monitoring the Health of Your Application (APM214) Student Guide

Baselines are a set of values calculated from metrics within a given time range that
show historic performance patterns and averages.

AppDynamics automatically calculates dynamic baselines for your metrics, and defines
what is 'normal' for each metric based on actual usage. The platform uses these
baselines to identify subsequent metrics whose values fall out of this normal range.

Static thresholds that are tedious to set up and, in rapidly changing application
environments can be prone to errors, are no longer needed.

© 2020 AppDynamics, LLC. 158


Monitoring the Health of Your Application (APM214) Student Guide

You can view metrics in the metric browser against their baselines by selecting the
check mark in the Base column for that metric.

In this example, notice the Call per Minute value this morning and how it relates to the
calculated baseline. You can specify the time range used to calculate the baseline, as
well as take the daily, weekly, and monthly trends into consideration.

© 2020 AppDynamics, LLC. 159


Monitoring the Health of Your Application (APM214) Student Guide

Dynamic baselines can be calculated on a daily, weekly, or monthly basis. Which one
you use will depend on what you’re trying to achieve. For example, if your application is
busier in the morning than it is at night, you would want the daily trend. With the daily
trend, AppDynamics would calculate the baseline by the hour. So at 9 a.m., your current
application performance is compared to the average value for all data from 9 to 10 a.m.
for the past 31 days. If you have more traffic on Monday rather than Sunday, you don’t
want to be comparing 9 am Monday with 9 am Sunday. Here, you would use weekly.

If you had an employee timesheet application that was busy on the 30th of each month,
you would select the monthly trend. With this option, AppDynamics will compare metrics
from the past year so that you’re looking at the same day from each previous month.

There are times when the dynamic baseline isn't helpful. Let’s say there was a
significant downtime event two months ago. In this case, you might want to create a
custom time range and exclude that week where the downtime occurred so that it
doesn't affect your baseline.

For more information on Dynamic Baselines, see:


https://docs.appdynamics.com/display/PRO45/Dynamic+Baselines

© 2020 AppDynamics, LLC. 160


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 161


Monitoring the Health of Your Application (APM214) Student Guide

You can also include baseline comparison in the flow maps throughout the controller
from the tier level all the way down to transaction flow maps. The outer circle continues
to reflect the overall health of the tier, while the inner circle represents the health of the
tier relative to the baseline.

© 2020 AppDynamics, LLC. 162


Monitoring the Health of Your Application (APM214) Student Guide

Click the Legend link in the flow map to see what flow map elements are indicating.

© 2020 AppDynamics, LLC. 163


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 164


Monitoring the Health of Your Application (APM214) Student Guide

Health Rules

A health rule defines a condition or set of conditions in terms of metrics. The condition
compares the performance metrics that AppDynamics collects with some static or
dynamic threshold that you define. If performance exceeds the threshold, a health rule
violation event is triggered.

AppDynamics comes pre-loaded with a set of default health rules. You can view the
details of the health rules used in your application at Alert and Respond > Health
Rules.

Custom health rules can be created with thresholds based on dynamic or static
parameters, and different critical or warning conditions. You can set different actions for
each condition.

Health rules can be applied to the whole application, or only apply to specific entities in
your application. Also, health rules can be set up to run only at specific times, for a set
time period, and you can specify how much recent data to use to evaluate a health rule.

© 2020 AppDynamics, LLC. 165


Monitoring the Health of Your Application (APM214) Student Guide

In AppDynamics, Health Rules can be set for any number of items, from overall
application performance down to individual Business Transactions.

Health Rules can also be used in other AppDynamics monitoring products which
monitor databases, browser performance and end user experience, and business
analytics.

© 2020 AppDynamics, LLC. 166


Monitoring the Health of Your Application (APM214) Student Guide

When the conditions set in those rules are violated, the Health Rule Violation alert
appears on the Events tab of the Application Dashboard.

© 2020 AppDynamics, LLC. 167


Monitoring the Health of Your Application (APM214) Student Guide

You can open the violation to see more details about the nature and duration of the
violation, the timeline, and any actions executed.

From this screen, you can also open the Health Rule to view its contents, or click View
Dashboard During Health Rule Violation to open the dashboard for the affected
business transaction at the time the violation occurred.

© 2020 AppDynamics, LLC. 168


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 169


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 170


Monitoring the Health of Your Application (APM214) Student Guide

Events

An event is a change in application state that can potentially be interesting to you.


AppDynamics detects 9 different types of events automatically, and lists them in the
Events tab on the application dashboard.

You can drill down into events of any type for more information. In the case of some
events, such as health rule violations, you can view the state of the dashboard at the
moment when the particular event occurred.

A summary of the event is available on the application dashboard, and you can access
the Event List page from the Summary header, or by clicking on Events on the left hand
navigation pane. You can apply filters to narrow down the list, and drill down into any of
the specific events from here also.

Let’s look at a few common events, and the kind of information captured for each.

© 2020 AppDynamics, LLC. 171


Monitoring the Health of Your Application (APM214) Student Guide

An event is a change in application state that can potentially be interesting to you.


AppDynamics detects 9 different types of events automatically, and lists them in the
Events tab on the application dashboard.

You can drill down into events of any type for more information. In the case of some
events, such as health rule violations, you can view the state of the dashboard at the
moment when the particular event occurred.

A summary of the event is available on the application dashboard, and you can access
the Event List page from the Summary header, or by clicking on Events on the left hand
navigation pane. You can apply filters to narrow down the list, and drill down into any of
the specific events from here also.

Let’s look at a few common events, and the kind of information captured for each.

© 2020 AppDynamics, LLC. 172


Monitoring the Health of Your Application (APM214) Student Guide

When AppDynamics detect problems with your application code, it logs the finding as a
code problem event. It could be a code deadlock, or resource pool limit has been
reached. Either way, you can drill down to identify the specific segment of code to be
examined.

© 2020 AppDynamics, LLC. 173


Monitoring the Health of Your Application (APM214) Student Guide

Changes in an application can provide an insight to the change in performance. The


Application Changes events include the following types.
 Application Deployment - this event is manually registered. See Monitor
Application Change Events in the AppDynamics documentation. Alternatively you
can Create a Custom Event.
 Application Configuration Change - this event is manually registered.
 App Server Restarts - this event is generated when an app server is restarted.

© 2020 AppDynamics, LLC. 174


Monitoring the Health of Your Application (APM214) Student Guide

Event Discovery Types:


 New Application Discovered
 New Tier Discovered
 New Node Discovered
 New Machine Discovered
 New Business Transaction Discovered

© 2020 AppDynamics, LLC. 175


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 176


Monitoring the Health of Your Application (APM214) Student Guide

Actions and Policies

We’re going to come back to policies. Let’s look at the actions you can set up to
respond to triggering events.

© 2020 AppDynamics, LLC. 177


Monitoring the Health of Your Application (APM214) Student Guide

An action is a predefined, reusable, automated response to an event. AppDynamics


allows for a number of different actions. A common action is to send a notification by
email or SMS to a recipient list. The text of the notification is automatically generated by
the event that triggered the action.

Another common action is to start a Diagnostic Session to collect snapshots or direct


the agent to take a thread dump for a specified number of samples (maximum of 50)
with each sample lasting for a specified number of milliseconds (maximum of 500 ms).

Beyond these typical responses, you could also use an action to run local scripts in a
node. The script executes on the machine from which it was invoked or on the machine
specified by the remediation action configuration.

© 2020 AppDynamics, LLC. 178


Monitoring the Health of Your Application (APM214) Student Guide

Finally, let’s look at policies.

© 2020 AppDynamics, LLC. 179


Monitoring the Health of Your Application (APM214) Student Guide

Ultimately, policies let you connect triggers/events to actions. Policies give you the
ability to anticipate problems and take actions to address those problems before they
cause a severe slowdown or outage.

The action is triggered automatically as soon as the event occurs.

Because the definition of health rules is separate from the definition of actions, and both
health rules and actions can be very precisely defined, you can take different actions for
breaching the same thresholds based on context, for example, which tier or node the
violation occurred in.

© 2020 AppDynamics, LLC. 180


Monitoring the Health of Your Application (APM214) Student Guide

Policies can be used to respond to a variety of situations with appropriate actions. Here
are some of the conditions you can set policies for, and typical remediation you can use.

© 2020 AppDynamics, LLC. 181


Monitoring the Health of Your Application (APM214) Student Guide

No log.

© 2020 AppDynamics, LLC. 182


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 183


Monitoring the Health of Your Application (APM214) Student Guide

Custom Dashboards

The ability to create Custom Dashboards is one of the most powerful features of
AppDynamics.

You can group monitoring metrics that are relevant to you in one consolidated
dashboard.

© 2020 AppDynamics, LLC. 184


Monitoring the Health of Your Application (APM214) Student Guide

You can share dashboard data within your organization, and even with vendors outside
of your organization.

And dashboards are read only. All could be internal and external.

© 2020 AppDynamics, LLC. 185


Monitoring the Health of Your Application (APM214) Student Guide

Depending on your role, you may have a specific set of metrics you are interested in
tracking. If you work in Operations, you are interested in metrics that deal with server
performance; if your colleague is a developer, he or she may be more interested in
comparing metrics that have to do with performance of the application code.

AppDynamics offers the ability to create custom dashboards, so everyone can have a
dashboard that presents all the data that’s relevant to their responsibilities.

© 2020 AppDynamics, LLC. 186


Monitoring the Health of Your Application (APM214) Student Guide

Custom dashboards have several features that enable you to do more with
AppDynamics. You can use your company color and logos to custom-brand the
dashboard. They are HTML5 so you can check your application’s health from anywhere,
at any time and with any device.

By default only users logged into the controller with the necessary permissions can view
a Custom Dashboard. However, if you share a dashboard, it becomes available to
anyone possessing the shared URL, regardless of whether they can log into
AppDynamics or not. You can also stop sharing a dashboard if you don’t wish to make
data available to a wider audience indefinitely.

Further, with the use of the iFrame widget, you can embed HTML content or insert data
from other monitoring applications and make this custom dashboard the single pane of
glass to view all available application performance information.

Custom dashboards can present baseline information along with actual metrics, so it’s
easy to compare and detect anomalies, and the status light feature presents health rule
status visually so you know the health status right away. And if you detect anomalies,
you can drill down to the root cause of the problem right from the dashboard.

Additionally, you can import & export dashboards, so you can use the same dashboard
in dev, QA, and production environments without creating each one separately.

© 2020 AppDynamics, LLC. 187


Monitoring the Health of Your Application (APM214) Student Guide

Additional Monitoring

Additional monitoring tools exist for browser, mobile and IoT, and for database, server,
and network monitoring.

© 2020 AppDynamics, LLC. 188


Monitoring the Health of Your Application (APM214) Student Guide

End user monitoring gives you insight into application performance from the perspective
of the end user.

© 2020 AppDynamics, LLC. 189


Monitoring the Health of Your Application (APM214) Student Guide

AppDynamics agents collect business data, enabling operational visibility.

© 2020 AppDynamics, LLC. 190


Monitoring the Health of Your Application (APM214) Student Guide

There are three components that fall under Infrastructure Visibility: Database
Monitoring, Server Visibility Monitoring, and Network Monitoring.

© 2020 AppDynamics, LLC. 191


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 192


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 193


Monitoring the Health of Your Application (APM214) Student Guide

With Network Monitoring, you can:


 Answer the key question quickly and easily: Is the network to blame for any
performance issues in my application?
 Identify the root cause of performance issues in the application, the network, or in
application/network interactions (i.e., how an application or server utilizes the
network).
 Pinpoint the traffic flows, individual nodes, and transport connections where network
or application/network issues are occurring.
 Collect detailed metrics that show how the network is performing and how well your
application uses network connections and resources.
 Collect and report targeted troubleshooting information to network, IT, DevOps, and
other teams.
 Drill down to contributing factors. For example: Retransmission Timeouts (RTOs) are
a sign of network packet loss, which results in retransmission of data when the TCP
retransmission timer expires (Timer to make sure data is ACK'ed).Typically this timer
varies from 200ms-3 sec by default (different for each operating systems and their
versions). It causes severe performance degradations as considerable amount of
time gets wasted to resend the lost data.TCP falls back to the "Slow Start" phase
impacting the performance even more.

© 2020 AppDynamics, LLC. 194


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 195


Monitoring the Health of Your Application (APM214) Student Guide

To differentiate yourself and your company in an increasingly competitive market,


please consider becoming AppDynamics certified.

For more information, please copy and paste this URL into a separate tab in your
browser: https://learn.appdynamics.com/certifications

For information on specific certification tracks, please copy and paste the appropriate
URLs below:

AppDynamics Certified Associate Performance Analyst:


https://learn.appdynamics.com/certifications/performance-analyst

AppDynamics Certified Associate Administrator:


https://learn.appdynamics.com/certifications/administrator

AppDynamics Certified Implementation Professional:


https://learn.appdynamics.com/certifications/implementer

© 2020 AppDynamics, LLC. 196


Monitoring the Health of Your Application (APM214) Student Guide

© 2020 AppDynamics, LLC. 197

You might also like