Cassandra Monitoring with Grafana & InfluxDB (Using Dockers)

What is Cassandra ?

Apache Cassandra is a free and open-source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. Cassandra offers robust support for clusters spanning multiple data-centers, with asynchronous master less replication allowing low latency operations for all clients.

Monitoring a cassandra cluster:

Since cassandra is a distributed system, monitoring of cassandra cluster will become very difficult as cluster size increases.

It’s always important to understand the performance characteristics of your cassandra cluster for diagnosing issues and for capacity planning.

Different methods are available for monitoring cassandra.

  1. Nodetool utility
  2. JMX (Jconsole)

Nodetool utility is something which comes along with cassndra setup and using this one can read the output and statistics of a particular command at a given time.

JMX : Cassandra exposes a number of statistics and management operations via Java Management Extensions. Java Management Extensions (JMX) is a Java technology that supplies tools for managing and monitoring Java applications and services. Any statistic or operation that a Java application has exposed as an MBean can then be monitored or manipulated using JMX.

There are different third party software’s available in the market for monitoring cassandra but most of these software’s would comes with licence.

In this post i will try to show Cassandra monitoring tool setup with open-source softwares.

Before i jump on to configuration, lets looks into docker first. In this post i will be using docker images for setting up cassandra monitoring.

What is Docker?

Docker is an open-source project that automates the deployment of applications inside software containers. Quote of features from Docker web pages:

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

To setup docker on Ubuntu 14 system use below link.

https://bigdatatechcomputing.wordpress.com/2017/02/22/docker-setup

or follow docker official page (below mentioned) to setup docker engine different unix flavors.

https://docs.docker.com/engine/installation

Cassandra Monitoring:

  • Setup docker engine on monitoring box.As mentioned above follow the guide for setting up docker environment.
  • Setup and configure Cassandra cluster or single node.
    • Install required version of apache Cassandra on all nodes.
    • Update below parameters in cassandra.yaml
      • cluster_name (Should be unique across nodes in the cluster)
      • seeds (Comma separated IPAddresses)
      • listen_address (Local box IP)
      • rpc_address (Local box IP)
    • Update/Add below into cassandra-env.sh file.
      • Add below line
        JVM_OPTS=”$JVM_OPTS -Dcassandra.metricsReporterConfigFile=influxdb.yaml“influxdb.yaml file will have configuration related to influx db. Please find content of this file below.
      • Set LOCAL_JMX to noif [ “x$LOCAL_JMX” = “x” ]; then
        LOCAL_JMX=no
        fi
      • Update jmxremote.authentication to falseJVM_OPTS=”$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false
      • Add below lineJVM_OPTS=”$JVM_OPTS -Djava.rmi.server.hostname=
    • Download and place graphite-metrics jar from below link into CASSANDRA_HOME/lib location.http://central.maven.org/maven2/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar
    • Create influxdb.yaml in CASSANDRA_HOME/conf location and add below content to it.graphite:

      period: 60
      timeunit: ‘SECONDS’
      prefix: ”
      hosts:
      – host: ”
      port: 2003
      predicate:
      color: “white”
      useQualifiedName: true
      patterns:
      – “.*”
    • Above prefix will be useful in case of multi node cassandra monitoring to distinguish which metric belongs to which one.period is a gap between each metrics push from cassandra node. Change this values according to the use case.Based on this configuration file database name with “graphite” will be created in influxdb right after the cassandra startup and metrics will be saved in this database.
    • Once done with all above configuration on all Cassandra nodes lets move on to next step.
  • Pull required images into local machine.To setup monitoring system for Cassandra , it’s required to install multiple software’s and have to perform lot of configuration changes.To avoid this complexity we can use pre built docker images available from docker hub repository.Required software’s/ Docker images:
    • GrafanaGrafana is an open source metric analytics & visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics but many use it in other domains including industrial sensors, home automation, weather, and process control.Use below command on monitoring box for pulling grafana image into local system.docker pull grafana/grafana
    • InfluxDBInfluxDB is an open source time series database with no external dependencies. It’s useful for recording metrics, events, and performing analytics.Use below command on monitoring box for pulling influxdb image into local system.docker pull influxdb:latest 

With above we are done with all configurations and now it’s time to test monitoring on cassandra.

Follow below steps to start all required services.

  1. Start containers on monitoring box.
    1. Start grafana container.Create two directories to persist grafana configuration files in local (monitoring) box.mkdir -p /monitoring/lib/grafana
      mkdir -p /monitoring/etc/grafanaGive required write permissions to the user.and launch grafana container with below command.docker run -d –net host –name grafana -v /monitoring/lib/grafana:/var/lib/grafana -v /monitoring/etc/grafana:/etc/grafana grafana/grafana
    2. Start influxdb container.Create two directories to persist influxdb data & configuration files in local (monitoring) box.mkdir -p /monitoring/lib/influxdb
      mkdir -p /monitoring/etc/influxdbGive required write permissions to the user.and launch influxdb container with below command.docker run -d -e INFLUXDB_GRAPHITE_ENABLED=true –net host –name influxdb -v /monitoring/lib/influxdb:/var/lib/influxdb -v /monitoring/etc/influxdb:/etc/influxdb  influxdb:latest
    3. Once done with above check the container status using docket ps command as below.tmp
  2.  Start cassandra on all nodes/single node.Once done with cassandra startup, give some time for cassandra to settle down.

    After few minutes connect to influxdb container on monitoring box and check different metrics available from different cassandra nodes.

  •  ssh monitoring_host
  • Connect to influxdb containerdocker exec -it shInfluxdb container id can be found from “docker ps” command as mentioned above.
  • Once after the successful connection to influxdb container connect to influx database using “influx” command and check available database.tmp
  • Once done with above run below command to check different available metrics transferred from different nodes (In my case i have only one node) in influxdb.show series;orshow series limit 10;There will be hundreds of metrics available from each node. To show the sample output here am limiting the output to 10 entries.tmp

    Like wise influx will have hundreds of metrics transformed from different nodes in cassandra cluster. Here hadoop1 is my test node hostname.

    As show above each metric name in influx db will act as a table and it will hold the actual values.

    To check the data available in each metric , run similar query like below in influx db.

    select * from “hadoop1.org.apache.cassandra.metrics.BufferPool.Size”;

    and the output will look like below.

    tmp

With above we got some idea like how Cassandra is transforming metrics to influxdb and how Influxdb is storing that information in the form of time series format.

Now let’s see how to configure grafana dash boards to see the meaningful output out of each metric stored in influxdb.

Connect to grafana web console with http://:3000

You will get a login page like below. Use username and password — admin/admin to login

tmp

Once after successful login, below page will be displayed.

Click on “Add data source” to configure connection to influxdb where our metrics data is available.

tmp

Add relevant details and click on add.

tmp

In influxDB details use username and password as admin/admin. If everything works fine then you will get below message right after clicking on add button.

tmp

Click on cancel.

Now go to grafana menu  and select

menu –> dashboards –> new

tmp.png

This will create a empty dashboard.

Now select a row with graph type.

tmp.png

Click on panel head and select edit to fetch and required metric from influxdb to grafana dashboard.
tmp1
Click on below highlighted area to edit the content of query.

tmp2

click on select measurement to get available metrics from influxdb.

tmp3

As mentioned this is how grafana will display available metrics data from influxdb. Select required metric for display purpose.

tmp4

Here i have selected below query for display purpose.

tmp5

To get better visibility of the graph go to display tab and change draw modes.
tmp6

Similar to above you can add multiple queries with metrics from different nodes in the same dash board to see the performance difference.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s