Everand Logo

Welcome to Everand!

  • Discover millions of ebooks, audiobooks, and so much more with a free trial

    Only $9.99/month after trial. Cancel anytime.

    Monitoring Elasticsearch
    Monitoring Elasticsearch
    Monitoring Elasticsearch
    Ebook322 pages1 hour

    Monitoring Elasticsearch

    By Dan Noble

    ()

    Read preview

    About this ebook

    About This Book
    • Understand common performance and reliability pitfalls in ElasticSearch
    • Use popular monitoring tools such as ElasticSearch-head, BigDesk, Marvel, Kibana, and more
    • This is a step-by-step guide with lots of case studies on solving real-world ElasticSearch cluster issues
    Who This Book Is For

    This book is for developers and system administrators who use Elasticsearch in a wide range of capacities. Prior knowledge of Elasticsearch and related technologies would be helpful, but is not necessary.

    LanguageEnglish
    PublisherPackt Publishing
    Release dateJul 27, 2016
    ISBN9781784391812
    Monitoring Elasticsearch

    Related to Monitoring Elasticsearch

    Related ebooks

    Internet & Web For You

    View More
    View More

    Related podcast episodes

    Related articles

    • Article

      Basic Concepts

      Jul 2, 2019

      A messaging system such as Kafka enables you to send messages between processes, applications and servers. Applications connect to Kafka to send or get data. Strictly speaking, a Kafka ‘topic’ is a unit of storage in Kafka: data in Kafka is stored in

      1 min read

    • Article

      Brave Browser 1.48.171

      Apr 4, 2023

      Brave Browser 1.48.171
      2 min read

    • Article

      Data Fabric

      Aug 13, 2020

      Data Fabric
      3 min read

    • Article

      How It Secures The Data?

      Jul 1, 2021

      How It Secures The Data?
      1 min read

    • Article

      In Brief

      Jun 1, 2021

      Mu is a code editor for many forms of Python. We can write standard Python 3 code, create web apps and write code for microcontrollers such as the new Raspberry Pi Pico. Mu is designed for new users and does away with complicated IDEs in favour of a

      1 min read

    • Article

      How To Develop A RESTful Client In Go

      Nov 16, 2021

      Mihalis Tsoukalos is a systems engineer and technical writer. He’s the author of Go Systems Programming and Mastering Go. You can reach him at @mactsouk. The subject of this month’s tutorial is RESTful services. In particular, you’re going to learn h

      9 min read

    • Article

      IT For A New World

      Jun 10, 2021

      IT For A New World
      6 min read

    • Article

      Join the Pod, Man!

      May 30, 2023

      Join the Pod, Man!
      8 min read

    • Article

      Tech

      Aug 5, 2024

      Tech
      3 min read

    • Article

      Linux Invades Windows

      Apr 2, 2024

      Linux Invades Windows
      1 min read

    • Article

      What Is The Future Of Game Streaming Now That Stadia Is Dead?

      Oct 31, 2022

      Once hyped as being ‘the future of gaming’, the Google Stadia game streaming service was officially, just three years after launch and before even making it to Australian shores. When game streaming first launched we did have some apprehension about

      2 min read

    • Article

      Perl at 34

      Feb 8, 2022

      Perl at 34
      7 min read

    • Article

      What is ELT?

      Apr 1, 2021

      It stands for extract, load, and transform- the processes a data pipeline uses for replicating the data from a source system into a target system such as a cloud data warehouse. 1. Extraction is the first step in which data is copied from the source

      6 min read

    • Article

      Route Traffic Between Networks Using A Pi

      Jun 2, 2020

      A deep-dive into Pi networking solutions resulted in this tutorial. The goal was to uncover a Pi configuration that would enable the routing of network traffic from a wired network to a wireless network. The aim is to build a network router using a R

      10 min read

    • Article

      Distro Watch

      May 3, 2022

      Distro Watch
      1 min read

    Related categories

    Reviews for Monitoring Elasticsearch

    0 ratings

    0 ratings0 reviews

    What did you think?

    Tap to rate

    Review must be at least 10 words

      Book preview

      Monitoring Elasticsearch - Dan Noble

      Table of Contents

      Monitoring Elasticsearch

      Credits

      About the Author

      About the Reviewers

      www.PacktPub.com

      eBooks, discount offers, and more

      Why subscribe?

      Preface

      What this book covers

      What you need for this book

      Who this book is for

      Conventions

      Reader feedback

      Customer support

      Downloading the example code

      Downloading the color images of this book

      Errata

      Piracy

      Questions

      1. Introduction to Monitoring Elasticsearch

      An overview of Elasticsearch

      Learning more about Elasticsearch

      Data distribution, redundancy, and fault tolerance

      Full-text search

      Similar technologies

      Apache Lucene

      Solr

      Ferret

      Monitoring Elasticsearch

      Resourcefulness and problem solving

      Summary

      2. Installation and the Requirements for Elasticsearch

      Installing Elasticsearch

      DEB/RPM installation

      The yum and apt-get repositories

      Ubuntu/Debian and apt-get

      CentOS/RHEL and yum

      Verification

      Configuration files

      Configuring an Elasticsearch cluster

      Cluster name

      Memory configuration

      Open file limit

      The maximum file limit

      Updating max file descriptors on Ubuntu Linux

      Enabling pluggable authentication modules

      Verifying the open file limit

      Disabling swapping

      Understanding your cluster

      Installing Elasticsearch-head

      Installing Bigdesk

      Marvel

      Cluster requirements

      Summary

      3. Elasticsearch-head and Bigdesk

      Cluster setup

      Cluster configuration

      Sample data

      Elasticsearch-head

      The Overview tab

      Cluster states

      Node and index actions

      The Indices tab

      The Browser tab

      The Structured Query tab

      The Any Request tab

      The official website

      Bigdesk

      The Elasticsearch cat API

      Background

      Count

      Health

      Indices

      Shards

      Summary

      4. Marvel Dashboard

      Setting up Marvel

      Upgrading Marvel

      Configuring Marvel

      Marvel agent configuration settings

      Marvel index configuration

      Understanding the Marvel dashboard

      Overview dashboard

      Indices dashboard

      Nodes dashboard

      Monitoring node failures

      Summary

      5. System Monitoring

      Working with Kopf

      Installing Kopf

      The cluster page

      The nodes page

      The rest page

      The more dropdown

      Working with Logstash and Kibana

      ELK

      Installation

      Installing Logstash

      Loading NGINX logs

      Installing Kibana

      Working with Nagios

      Installing Nagios

      Command line tools for system and process management

      top

      tail

      grep

      ps

      kill

      free

      du and df

      Summary

      6. Troubleshooting Performance and Reliability Issues

      System configuration

      The fielddata cache

      Analyzing queries

      Slow log

      Improving query performance

      High-cardinality fields

      Querying smaller indices

      Cold indices

      The shard query cache

      Script queries

      Testing meticulously

      System and data architecting

      Hot-Warm architecture

      Master nodes

      Hot nodes

      Warm nodes

      Reducing disk size

      Compression

      Storing the _source and analyzed fields

      Optimizing data ingestion

      Bulk indexing operations

      Drive configuration

      Case studies

      Node configuration

      Query optimization

      Web application performance

      Summary

      7. Node Failure and Post-Mortem Analysis

      Diagnosing problems

      OutOfMemoryError exceptions

      Shard failures

      Slow queries

      Resolving OutOfMemoryError exceptions

      Elasticsearch process crashes

      Disk space

      Resolving the issue

      Reviewing some case studies

      The ES process quits unexpectedly

      Query requests slow and timing out

      Summary

      8. Looking Forward

      Elasticsearch 5 overview

      Performance and reliability

      Data loss

      Upgrading to Elasticsearch 5.0

      When to upgrade

      Monitoring Elasticsearch 5

      Summary

      Index

      Monitoring Elasticsearch

      Monitoring Elasticsearch

      Copyright © 2016 Packt Publishing

      All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

      Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

      Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

      First published: July 2016

      Production reference: 1200716

      Published by Packt Publishing Ltd.

      Livery Place

      35 Livery Street

      Birmingham B3 2PB, UK.

      ISBN 978-1-78439-780-7

      www.packtpub.com

      Credits

      Author

      Dan Noble

      Reviewers

      James A. Cubeta

      Joseph E McMahon

      Parkhe Kishor B.

      Acquisition Editor

      Sonali Vernekar

      Content Development Editors

      Amey Varangaonkar

      Merint Mathew

      Technical Editor

      Hussain Kanchwala

      Copy Editor

      Priyanka Ravi

      Project Coordinator

      Judie Jose

      Proofreader

      Safis Editing

      Indexer

      Rekha Nair

      Graphics

      Kirk D'Penha

      Production Coordinator

      Melwyn Dsa

      Cover Work

      Melwyn Dsa

      About the Author

      Dan Noble is a software engineer with a passion for writing secure, clean, and articulate code. He enjoys working with a variety of programming languages and software frameworks, particularly Python, Elasticsearch, and frontend technologies. Dan currently works on geospatial web applications and data processing systems.

      Dan has been a user and advocate of Elasticsearch since 2011. He has given talks about Elasticsearch at various meetup groups, and is the author of the Python Elasticsearch client rawes. Dan was also a technical reviewer for the Elasticsearch Cookbook, Second Edition, by Alberto Paro.

      I would like to thank my beautiful wife, Julie, for putting up with me while I wrote this book. Thanks for supporting me every step of the way.

      I would also like to thank my friends and colleagues James Cubeta, Joe McMahon, and Mahmoud Lababidi, who shared their insight, time, and support. I would like to give a special thanks to Abe Usher – you have been an incredible mentor over the years.

      Finally, thanks to everyone at Packt Publishing for helping to make this book happen. A special thanks to Merint Mathew, Sonali Vernekar, Hussain Kanchwala, and Amey Varangaonkar for your valuable and careful feedback.

      About the Reviewers

      James A. Cubeta is a computer scientist with more than 20 years of experience, spanning government contracting, commercial companies, and research organizations. With a technical emphasis on data management, ETL, multi-tier application development, and relational and NoSQL databases, he has also helped author Java courses at Sun Microsystems and served as a technical reviewer for O'Reilly Media's extremely successful book Head First Java. He is currently a senior technologist at the HumanGeo Group, LLC.

      Joseph E McMahon is a computer scientist with over 25 years of experience developing distributed systems with a focus on effective and performant messaging components. He has vast experience integrating geospatial solutions and data collection architectures. Currently, he leads a division in The HumanGeo Group supporting a variety of government customers. When not coding, he enjoys cooking, woodworking, and traveling with his wife and three children.

      Parkhe Kishor B. received a BSc degree and an MTech degree in industrial mathematics and computer applications from the University of Pune. He joined High Mark Credit Information Service in 2012, where he worked as senior software engineer in the research and development department. He also has experience working in information retrieval, big data, and distributed computing. He is currently working as a design engineer at Introp Software Solutions, India. He is frequently a consultant to the industry in the area of big data, NoSQL, machine learning, artificial intelligence, and business intelligence.

      www.PacktPub.com

      eBooks, discount offers, and more

      Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

      At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

      https://www2.packtpub.com/books/subscription/packtlib

      Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

      Why subscribe?

      Fully searchable across every book published by Packt

      Copy and paste, print, and bookmark content

      On demand and accessible via a web browser

      Preface

      Welcome to Monitoring Elasticsearch!

      There are many books and online tutorials that cover the Elasticsearch API and how to configure a cluster. But, until now, there hasn't been a thorough, accessible resource for monitoring and troubleshooting purposes. We've found that Elasticsearch monitoring tools drastically improve our ability to solve cluster issues and greatly increase cluster reliability and performance as a result. We wrote this book to share those use cases and the insights that came out of them.

      This book covers how to use several popular open source and commercial Elasticsearch monitoring tools, namely, Elasticsearch-head, Bigdesk, Marvel, Kopf, and Kibana. There's also a section on the Elasticsearch cat API and how to use Nagios to perform general system monitoring. Moreover, we will discuss several case studies with real-world examples of troubleshooting Elasticsearch issues using these tools.

      We believe that the best way to learn is to do. In this book, we'll go over how to set up a sample Elasticsearch cluster and load it with data. At times, we'll deliberately introduce problems into the cluster so that we can see how the errors are tracked using our various monitoring tools. Following along with these examples in your own cluster will help you learn both how to use the monitoring tools and how to tackle new and unknown issues that may arise.

      After reading this book, we hope that you will be better equipped to run and maintain an Elasticsearch cluster. You will also

      Enjoying the preview?
      Page 1 of 1