Visualize fluctuations in numbers on your website with Datadog

Introduction

Agent Check to get the numerical value posted on the website by Web Scraping and visualize it using Datadog. I made datadoghq.com/ja/guides/agent_checks/).

https://github.com/mounemoi/datadog-web-scraper

Datadog is a cloud service for server monitoring, but you can use your own Agent Check and custom metrics to monitor various user-specified numbers. Using this function, you can import the numerical values obtained by Web Scraping into Datadog and visualize them with Datadog's various graph drawing functions.

Acquisition sample

I made some samples using this Agent Check.

1. Changes in the total number of Wikipedia pages

I got the total number of pages in Wikipedia and graphed the fluctuation with Datadog. The graph shows how many pages have changed since one day. (Calculation of this starting point is performed by Datadog)

This value is obtained by Web Scraping from the following numerical part of Wikipedia top page.

Looking at the graph, you can see that the number of pages increases linearly. It's about 900 pages a day. It is amazing.

2. Changes in the number of items listed in the auction

This is the number of listings in a certain category on a certain auction site.

The number of listings is decreasing every day from 21:00 to 24:00. The day on the far right is Sunday night. After all, the auction is most exciting just before the end, so are there many sellers who set the end time according to the time that everyone often sees? (The origin of the y-axis is adjusted so that the numerical change is easy to understand. This is also done by the Datadog function.)

3. Tokyo wind speed

Here, I tried to get the wind speed of Tokyo from the site that distributes AMeDAS information. I often see graphs showing changes in temperature, so I dare to use the wind speed.

(To make it easier to understand the change from the previous day, the numerical value (black line) at the same time on the previous day is displayed at the same time. This is also a function of Datadog)

It was raining the morning before, so it seems that the wind speed was also strong. We also found that this distribution site updates information every hour.

In addition, I think that you can combine the functions of Datadog and Integration to "record price fluctuations on EC sites and notify slack when a certain amount of money is reached".

How to install and configure Agent Check

For the installation method and the method of specifying the value to be acquired by Web Scraping, see README.ja.md in the repository. It is described in md).

To use it, you need a Datadog account and a server with the Datadog Agent installed.

Note that this Agent Check web scraping simply GETs HTML, so you cannot get the value drawn by JavaScript or the value of the page that requires login.

Summary

Datadog can get various numbers with custom metrics by using Agent Check and SDK. With this function, it is possible to visualize other than server monitoring information, such as usage information of services running on the server, but I made it as a practice.

Also, since Agent Check is written in python, I thought it would be easy for python developers to get along with.

Datadog is interesting.