Introducing 4 ways to monitor Python applications using Prometheus

image.png

Introduction

Although Prometheus was originally designed for single-process, multi-threaded applications rather than multi-process, it is now popular for monitoring Python applications (https://www.courseduck.com/programming/python/). Is becoming a tool for.

Prometheus was developed in the Soundcloud environment and was inspired by Google's Borgmon. In the original environment, Borgmon relies on the simple method of Service Discovery. You can easily find all the jobs running in your cluster.

Prometheus inherits these assumptions, assuming that one target is a single multithreaded process. We will assume that the Prometheus client library comes from different libraries and subsystems of multiple threads of execution running in a shared address space.

To get started, go to MetricFire [Free Trial]( Please sign up at https://www.hostedgraphite.com/accounts/signup-metricfire/?signup=japan&utm_source=blog&utm_medium=Qiita&utm_campaign=Japan&utm_content=How%20to%20monitor%20Python%20Applications%20with%20Prometheus). In this trial, you can start using Prometheus on your platform right away and put into practice what you have learned from this article.

Problems integrating Prometheus into Python WSGI applications

When I run a Python app on a WSGI application server, I start getting problems. In WSGI applications, requests are assigned to many different workers rather than a single process. Each of these workers becomes a multi-process application by being deployed using multiple processes.

When this type of application is exported to Prometheus, Prometheus gets several different workers responding to scrape requests. Each worker responds with known values, so Prometheus can scrape the countermetric and return it as 100, immediately after it is returned as 200. Each worker exports its own value, so the countermetric measures random information rather than the entire job.

There are four solutions to handle these issues:

1. Sum all worker nodes

You can label each metric uniquely so that you can query them all at once and efficiently query the entire job. For example, if you label each worker as worker_name, you can write a query like this:

sum by (instance, http_status) (sum without (worker_name) (rate(request_count[5m])))

This aggregates all worker nodes for a job at once. However, the problem with this is that the number of metrics you have will skyrocket.

2. Multi-process mode

This method is recommended by us at MetricFire. This method uses the Prometheus Python Client (https://github.com/prometheus/client_python), which handles multi-process apps on the gunicorn application server, and monitors your own application with Prometheus.

For information on how to use the Python client with MetricFire to monitor your own services, check out the article Monitoring Python Web Apps with Prometheus (https://qiita.com/MetricFire/items/57b95f6de1f608ef7c44). please look. That article describes each step in monitoring a Python web app using Prometheus.

3. Django Prometheus client

This method specifies each worker as a completely separate target. The Django Prometheus client configures each worker to listen for Prometheus scrape requests over its own port.

4. StatsD exporter

This method rejects the notion that Prometheus needs to rub the application directly. Instead, configure Prometheus to export metrics from your app to a locally running StatsD instance and get a StatsD instance on your behalf. This gives you more control over what each counter counts.

Summary

Prometheus cannot natively monitor multi-process applications, but these four solutions are good workarounds. This allows Prometheus to be the primary monitoring tool for the entire enterprise for both IT resources and APM.

For more information on how to use Prometheus to monitor Python apps, see the article on Python-based exporters (https://www.metricfire.com/blog/first-contact-with-prometheus) and Kubernetes. See the series on Developing and Deploying Python APIs Using (https://www.metricfire.com/blog/develop-and-deploy-a-python-api-with-kubernetes-and-docker).

To try Prometheus and put into practice what you've learned from this article, [MetricFire](https://try.metricfire.com/japan/?utm_source=blog&utm_medium=Qiita&utm_campaign=Japan&utm_content=How%20to%20monitor%20Python% 20Applications% 20with% 20Prometheus) [Free Trial](https://www.hostedgraphite.com/accounts/signup-metricfire/?signup=japan&utm_source=blog&utm_medium=Qiita&utm_campaign=Japan&utm_content=How%20to%20monitor%20Python%20Applications%20with Check% 20 Prometheus). You can use Prometheus directly on our platform to start monitoring metrics without setup. Also, please contact us directly by Booking a demo. .. We are always happy to hear from you about your company's monitoring needs.

See you in another article!

Recommended Posts

Introducing 4 ways to monitor Python applications using Prometheus
Introducing Python 2.7 to CentOS 6.6
Post to Twitter using Python
Start to Selenium using python
How to install python using anaconda
Introducing Python using pyenv on Ubuntu 20.04
Monitor Python web apps with Prometheus
6 ways to string objects in Python
5 Ways to Create a Python Chatbot
Migrate Django applications running on Python 2.7 to Python 3.5
From Python to using MeCab (and CaboCha)
Introduction to Discrete Event Simulation Using Python # 1
Log in to Slack using requests in Python
Try to operate Excel using Python (Xlwings)
Dump BigQuery tables to GCS using Python
Python beginners publish web applications using machine learning [Part 2] Introduction to explosive Python !!
Introduction to Discrete Event Simulation Using Python # 2
[Python] List Comprehension Various ways to create a list
3 ways to parse time strings in python [Note]
#Monte Carlo method to find pi using Python
Procedure to use TeamGant's WEB API (using python)
I want to email from Gmail using Python.
[Improved version] Script to monitor CPU with Python
Convert STL to Voxel mesh using Python VTK
Updated to Python 2.7.9
Start using Python
Scraping using Python
"Backport" to python 2
How to set up a Python environment using pyenv
(Python) Try to develop a web application using Django
How to auto-submit Microsoft Forms using python (Mac version)
Push notifications from Python to Android using Google's API
Error due to conflict between python when using gurobi
Easy way to scrape with python using Google Colab
MessagePack-Call Python (or Python to Ruby) methods from Ruby using RPC
How to make a Python package using VS Code
Write data to KINTONE using the Python requests module
How to exit when using Python in Terminal (Mac)
Copy S3 files from Python to GCS using GSUtil
How to retrieve multiple arrays using slice in python.
[Introduction to Python] How to stop the loop using break?
Try to extract high frequency words using NLTK (python)
Output to "7-segment LED" using python on Raspberry Pi 3!
How to execute a command using subprocess in Python
[Introduction to Python] How to write repetitive statements using for statements
What I was addicted to when using Python tornado
Query from python to Amazon Athena (using named profile)
I tried to access Google Spread Sheets using Python
[Technical book] Introduction to data analysis using Python -1 Chapter Introduction-