How to install and configure the monitoring tool "Graphite" on Ubuntu

Introduction

An application, system, product, or process has several important performance metrics that aim to give you a glimpse of the state and strength of that system. One of these important metric roles is how a particular parameter or data point works over time. What if you want to monitor database latency in seconds, or if you want to monitor the number of hits on your API endpoint in a particular time frame, or just the time it takes to run a particular process? A single data point captured at this time does not provide much information on its own. However, tracing the same points over time reveals much more, such as the impact of changes on a particular metric.

For example, if you want to know how a new user interface design or API documentation affects the number of API hits, or how a particular software fix / upgrade affected database latency, do the current value earlier. Compare with the value. It helps to provide this insight before the changes are introduced. This is the value of time series data.

Graphite was developed on the basis of an open source tool that monitors time series data and displays information on easy-to-read dashboards. Hosted Graphite, a product of MetricFire, runs Graphite, providing reliability and ease of use that are difficult to do in-house. You can now sign up for a free trial of Metric Fire and start sending metrics today. Check out this guide for installing Graphite on Ubuntu to find out what you need to do it yourself.

Explanation of Graphite

There are several tools whose primary purpose is to collect, monitor, and provide useful insights using time series data. An example of such a tool is Graphite.

Graphite is an enterprise-scale monitoring and graphing tool that runs equally well on cloud infrastructure or inexpensive hardware. Enterprises can use Graphite to track the performance of applications, websites, network servers, and business services. This was a pioneering product in the generation of monitoring tools. With the advent of Graphite, sharing, retrieving, storing and visualizing time series data has never been easier.

Basically, Graphite does two things. Saves time series data of numbers and renders a graph of this data on demand. Graphite itself does not collect data, so in order to collect it, you need to collect it with minimal effort and code changes. Currently, there are many tools that can be used to send data to graphite. A list of such tools can be found on the graphite documentation page.

The Graphite architecture breakdown shows that it consists of three software components:

-** Carbon **- Graphite does not collect data, but uses the Carbon component, a Twisted daemon that passively listens for time series data.

-** Whisper -This is a simple database library for storing time series data (design is similar to RRD) - Graphite WebApp **-Django web app that renders graphs on demand using Cairo.

In the simplified use case of Graphite, you need to write an application that collects numerical time series data that you are interested in graphing (this data is usually collected by another tool above). The app then sends the data to Graphite's processing backend, Carbon, which stores the data in Graphite's dedicated database, Whisper. The data can then be visualized through Graphite's web interface, Graphite Web App.

Having given you an overview of Graphite, the rest of this article will focus specifically on how to install Graphite on the Ubuntu operating system.

Installing Graphite using Docker

Installing Graphite from source can be a complex process, but fortunately there is Docker in this era! If you use docker to install the Graphite image, you can launch and run Graphite in just a few minutes. This allows for a fast and very portable installation. To install Graphite using docker, just run the following command:

docker run -d\
 --name graphite\
 --restart=always\
 -p 80:80\
 -p 2003-2004:2003-2004\
 -p 2023-2024:2023-2024\
 -p 8125:8125/udp\
 -p 8126:8126\
 graphiteapp/graphite-statsd

This command launches a container named graphite, which contains the following components:

-** Graphite -Front-end dashboard - Nginx ** --Reverse proxy for Graphite dashboard -** Carbon -Backend daemon that listens for time series data - Grafana -Front-end dashboard (more sophisticated than Graphite) - Statsd **-UDP-based backend proxy

Also note that if the corresponding port listed in the above command is already occupied by the host, you are free to change the mapping of the container port to any host port. Also, it is not mandatory to map all ports. For example, using the above command means that the Grafana component will not be exposed on any port. This requires you to use the Graphite web app as your app's front-end dashboard. If you want to use Grafana as your front-end dashboard, use the following command instead.

docker run -d\
 --name graphite\
 --restart=always\
 -p 80:80\
 -p 81:81\
 -p 2003-2004:2003-2004\
 -p 2023-2024:2023-2024\
 -p 8125:8125/udp\
 -p 8126:8126\
 hopsoft/graphite-statsd

This configures Graphana port 80 and Graphite port 81. Using the previous command, Graphite is mapped to port 80. A detailed representation of the mapped port is in the following table.

Once you're logged in, update the login details for your default Django admin user account. The default is not safe.

http：// localhost / account / login

You can access the login page (or http: // localhost: 81 / account / login if Grafana is enabled) and log in with both values using the default username and password. After logging in, you can update the root user's profile at the following URL

http: // localhost / admin / auth / user / 1 / (or http: // localhost: 81 / admin / auth / user / 1 / graphana is enabled).

You can access the Graphite dashboard using the following link:

http: // localhost / Dashboard (or http: // localhost: 81 / dashboard if Grafana is enabled).

If Docker isn't right for you, you can install Graphite directly using the linux apt-get command.

Installing Graphite on Ubuntu 16.04

Graphite installation

First, run the following command to update the local package index to the latest stable version.

sudo apt-get update -y
sudo apt-get upgrade -y

As mentioned earlier, Graphite consists of several components: Graphite web application, Carbon storage backend, and database library, Whisper. Each of these components is in the Ubuntu default repository. These can be installed simply by running the following command:

sudo apt-get install graphite-web graphite-carbon -y

As part of the installation, I have the following questions:

"The / var / lib / graphite / whisper directory contains whisper database files. Keep these database files even if you have completely removed Graphite carbon in case you reinstall later. Do you want to delete the database files when purging Graphite carbon? "

The question itself is self-explanatory and the answer depends on the use case.

PostgreSQL installation and configuration

Next, configure the database for your Django mobile app. The Graphite data itself is processed by the Whisper database library and Carbon, but the Graphite web application is built as a Django Python application, so you need to store the data somewhere.

By default, it is configured to use SQLite3 database files, but these are not as robust as a real relational database management system. Instead, use the PostgreSQL database to configure your app. This is because the data entry is much more rigorous and catches exceptions that may cause problems in the future. Run the following command to install and configure PostgreSQL.

sudo apt-get install postgresql libpq-dev python-psycopg2 -y

This command installs and runs PostgreSQL. However, to use PostgreSQL, you need to create the desired user before creating the Graphite database. This is achieved as follows:

1. First, log in to the PostgreSQL shell using the following command:

sudo -u postgres psql

2. Create a Graphite user and protect it with a password as follows:

postgres=# CREATE USER graphite WITH PASSWORD 'password';

3. Create a Graphite database and assign ownership to the user.

postgres=# CREATE DATABASE graphite WITH OWNER graphite;

4. Exit the PostgreSQL shell using the following command:

\q

Graphite configuration

Now that the database and users are ready, we need to change the Graphite settings to use the configured components. Open the Graphite web app configuration file local_settings.py.

sudo nano /etc/graphite/local_settings.py

Make the following changes to the file:

1. You need to set the secret key to any value that will be used as the salt when creating the hash. Uncomment the SECRET_KEY parameter and change the value to something unique, long and easy to remember.

SECRET_KEY = 'something_unique' 

2. Specify the time zone. This is important to set as it affects the time displayed on the graph. Edit the TIME_ZONE variable in the file to set the time zone.

TIME_ZONE = 'America/New_York'

3. I would like to configure authentication to store graph data. This is done by uncommenting the next line.

USE_REMOTE_USER_AUTHENTICATION = True 

4. Find the Databse Configuration section of the file. Change the value to reflect the Postgres configuration. This includes NAME, ENGINE, USER, PASSWORD, and HOST keys.

When finished, it should look like this:

DATABASES = {
    'default': {
        'NAME': 'graphite',
        'ENGINE': 'django.db.backends.sqlite3',
        'USER': 'graphite',
        'PASSWORD': 'password',
        'HOST': '127.0.0.1',
        'PORT': ''
    }
}

When you're done, press CTRL + O, Enter, CTRL + X to save and close the file and exit the file editor.

Then run the following command to synchronize the database and create the correct structure.

sudo graphite-manage migrate auth
sudo graphite-manage syncdb 

You will then be prompted to create a database superuser account and be prompted to enter certain values. Simply follow the on-screen instructions and enter the appropriate values for each question to complete Graphite configuration.

Make up Carbon

Then edit the service configuration file, graphite-carbon, to configure the Graphite storage backend, Carbon.

sudo nano /etc/default/graphite-carbon

1. To start carbon at system boot, change the value of CARBON_CACHE_ENABLED to true.

CARBON_CACHE_ENABLED=true

When you're done, save the file and close it.

2. Next, open the Carbon configuration file carbon.conf.

sudo nano /etc/carbon/carbon.conf

3. Set the variable ENABLE_LOGROTATION to true to enable log rotation.

ENABLE_LOGROTATION = True

4. Save the file, edit the storage schema file to configure the storage schema, and tell Carbon how long to save the value and how / what the metric to keep.

sudo nano /etc/carbon/storage-schemas.conf

This file has two rulesets that are already defined. These are divided into sections. The words in parentheses are the section headers used to specify the new definition. Below each section is a pattern definition and retention policy.

A pattern definition is a regular expression used to collate the information sent to Carbon. This information includes the metric names that this pattern definition checks. In the first example, the pattern checks if the metric in question starts with the string "carbon".

The retention policy is defined by a series of colon-separated numbers. Each set consists of a metric interval (which specifies how often the metric is recorded), a colon, and the length of time to store those values. You can define multiple sets of numbers separated by commas.

To demonstrate, define your own schema that matches the test values you will use later. Add the following section to the end of the file.

[test]
pattern = ^test\.
retentions = 10s:10m,1m:1h,10m:1d

When you're done, save the file and close it.

This matches all metrics that start with the value "test". Save the collected data 3 times and change the details. The first definition (10s: 10m) creates a data point every 10 seconds and stores the value for only 10 minutes.

The second archive definition (1m: 1h) creates data points per minute. Collect all the data for the last minute (6 points if the previous archive created points every 10 seconds) and aggregate it to create a single point. By default, this is done by averaging the points, but you can adjust it later. Data created at this level of detail will be stored for an hour.

The final archive definition created (10m: 1d) creates data points every 10 minutes and aggregates the data as it did in the second archive. Save the data for the day.

Therefore, when you request information from Graphite, the data is returned from the most detailed archive that measures the time frame you are requesting. For example, requesting metrics for the last 5 minutes will return information from the first archive. If you requested a graph for the last 50 minutes, the data will be taken from the second archive.

5. Next, you need to define how to perform the aggregation. This can be done by editing a file called storage-aggregation.conf to set the aggregation specifications, but this tutorial uses the example of a storage aggregation configuration file. Copy the sample configuration file to the Carbon configuration directory.

sudo cp /usr/share/doc/graphite-carbon/examples/storage-aggregation.conf.example /etc/carbon/storage-aggregation.conf

6. Finally, start the Carbon service with the following command:

sudo systemctl start carbon-cache

Configure Apache for Graphite

Apache must be installed and configured before you can use the Graphite web interface. You can install Apache with the following command:

sudo apt-get install apache2 libapache2-mod-wsgi -y

After the installation is complete, you need to disable the default virtual host file. This is because it conflicts with the new file.

sudo a2dissite 000-default

Then copy the Graphite Apache virtual host file to the available sites directory.

sudo cp /usr/share/graphite-web/apache2-graphite.conf /etc/apache2/sites-available

Then enable the virtual host file by typing:

sudo a2ensite apache2-graphite

Reload the service to implement the changes.

sudo service apache2 reload

Access the Graphite web interface

Finally, Graphite is installed and you can access its web interface. Open your favorite web browser and enter the URL.

http://your_server_domain_name_or_IP

Congratulations. I installed and configured Graphite.

Even if you know the basics of working with Graphite, scaling Graphite to a production level can be a huge challenge. If you are considering running Graphite at production level, [MetricFire](https://www.metricfire.com/japan/?utm_source=blog&utm_medium=Qiita&utm_campaign=Japan&utm_content=How%20to%20Install%20and%20Configure" % 20Graphite% 20on% 20 Ubuntu) [Sign up for demo](https://www.metricfire.com/demo-japan/?utm_source=blog&utm_medium=Qiita&utm_campaign=Japan&utm_content=How%20to%20Install%20and%20Configure%20Graphite% 20on% 20Ubuntu) and get support.