[JAVA] Create a dashboard for Network devices with Django!

NetOpsCoding Advent Clender This is the 7th day article. In this article, we use Django to get information about network devices. As a web application, we will share how to create a dashboard that allows you to check the status collectively. (The target for information acquisition is mainly routers, not switches) スクリーンショット 2016-12-07 5.09.55.png

** The dashboard created this time is uploaded to the following git. Qiita does not explain the details, so please see the actual item for more details ** https://github.com/Mabuchin/netboard

Introduction

Various tools have been developed from ancient times to the present as tools for acquiring the status of network devices. Cacti, MRTG, Zabbix, Munin and so on ... Of course, these tools are very useful tools for checking network status if used well. In fact, it should be the standard in various operation sites. However, when using it in the actual operation site, I want another function, I wish I could see this as well ... I think there are many such phenomena. The network operation sites are not all at once, and each operation site should have different things that you want to check at a glance. Especially when checking the status by troubleshooting, what you see first depends on what you are monitoring.

This time, while using the above ** monitoring tool regularly **, I made a dashboard that allows you to see only the information you want in case of trouble and during work ** on one page of the Web **. to watch.

Acquisition information and acquisition method

This time, ** BGP and [AS operation](https://www.nic.ad.jp/ Assuming a network operator who is doing ja / basics / terms / as.html) **, determine the information to be acquired. If you want to make it yourself, you can make it according to the requirements of each organization, but this time I will think above, which is the easiest to imagine. The items to be confirmed this time are as follows.

--Life and death of each router itself (Ping monitoring) --Interface state

This time, let's create a dashboard that shows the above.

Installation

The main external libraries used this time are as follows. Python is obtained from pip, and various front display modules such as JavaScrip are obtained from the Web.

Since the Python module has posted require.txt on Github, you can also download it all at once as follows.

cd ./netboard
pip install -r requirements.txt

If you want to use it from 1, you need to set up a Django project. Cast the spells of creating a Django project and creating an application as shown below.

% django-admin.py startproject netopboard
% cd netopboard
% python manage.py createsuperuser
(Snip)
% python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
%python manage.py startapp device_mon

When you're done, check if Django is working

% python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).
December 05, 2016 - 08:16:56
Django version 1.10.4, using settings 'netopboard.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Creating a model that manages router information

When you get information from a device, you need credentials to access the router, In most cases, you will need the target IP address, SNMP Community, login password, etc. This time, I'll define that information in Django's Model so that I can easily create a database. The definition of Django's DB model is managed by the application inside it, not by the project unit, so it is defined under the created application device_mon. This time, the behavior must be changed depending on the OS of the device, so the OS information of the device is also included.

If you define these as Django models, they look like this:

device_mon/models.py


class Device(models.Model):
    hostname = models.CharField('Hostname',max_length=255,unique = True,)
    management_ip = models.GenericIPAddressField('Management IP address',null = True,blank = True,)
    snmp_community = models.CharField('SNMP community',max_length = 255,null = True,blank = True,)
    os = models.CharField('OS',max_length = 255,validators = [AsciiValidator],)
    user = models.CharField('User Name',max_length = 255,null = True,blank = True,)
    password = models.CharField('User Password',max_length = 255,null = True,blank = True,)
    enable_password =  models.CharField('Enable Password',max_length = 255,null = True,blank = True,)
    status = models.PositiveIntegerField('Status',default = 0)

    class Meta:
        verbose_name = 'Device'
        verbose_name_plural = 'Devices'

    def __unicode__(self):
        return self.hostname

Normally, it is better to decide the input rule with Validator etc., but this time I will omit that point. After defining the model, it is necessary to actually create a DB, so let's build a DB as follows.

% python manage.py makemigrations device_mon
Migrations for 'device_mon':
  device_mon/migrations/0001_initial.py:
    - Create model Device

At this point, the foundation is complete: v:

** Not all of the explanations that follow are explained. Since only the main points are scraped, some parts are omitted. See the Github code for details **

Creating a home screen

First, let's make a home screen. If you create a page that will be the home of your dashboard, you will have a page like this! スクリーンショット 2016-12-06 5.15.37.png

Views and template

First, display the list of network devices and create a top page for checking the alive state and transitioning to the detailed page. Since I want to display the router information in list format here, I made html be drawn from Views.py as follows. html is a description using the template engine in Django.

views.py


from django.shortcuts import render
from device_mon.models import Device

def device_list(request):
    return render(request, 'device_mon/index.html', dict(devices=Device.objects.all().order_by('hostname')))

In views, all Device objects are acquired and passed to template. In template, it receives it and draws html based on that information. In template, it is possible to branch using other if to display the received information. It is a simple template that receives Device objects and draws as many table columns as there are while reflecting the contents. If the part of life and death monitoring is NG, the button is changed by ʻif` so as not to transition to the detailed information screen.

template/index.html


    <table class="table table-bordered table-condensed sortable table-hover" data-sort-key="0" id="bgp_statuses">
      <thead>
      <tr>
        <th>Hostname</th>
        <th>Address</th>
        <th>Access</th>
        <th>Detail</th>
      </tr>
      <tbody>
            {% for device in devices %} 
            <tr>
                <div class="form-group">
                    <td><strong>{{ device.hostname }}</strong></td>
                    <td class="target-data">{{ device.management_ip }}</td>
                    {% if device.status == 1 %}
                    <td><button type="submit" class="btn btn-success btn-xs btn-block " disabled="disabled">OK</button></td>
                    <form class= "form-inline" action="{% url 'device_mon:result' %}" method="post" role="form">
                        <input type="hidden" name="routerinfo_id" value="{{ device.id }}">
                        {% csrf_token %}
                        <td><button type="submit" class="btn btn-primary btn-xs btn-block">Check</button></td>
                    </form>
                    {% else %}
                    <td><button type="submit" class="btn btn-danger btn-xs btn-block" disabled="disabled">NG</button></td>
                    <td><button type="submit" class="btn btn-default btn-xs btn-block" disabled="disabled">Check</button></td>
                    {% endif %}
                </div>
            </tr>
            {% endfor %}
      </tbody>
      </table>

tips: Easy access to DB using management commands

On the top page, in addition to the list of routers, life and death monitoring of each router is performed. This is just to ping regularly behind the scenes and store the result, but at that time, in the procedure of pinging with a completely external program and storing it in the DB, the DB is directly stored. You will need to do something like editing. However, as you can see from Views.py etc., you can easily operate the object by calling models from inside the Django application.

Django's custom management commands if you need to change or reference objects from outside, not from the web. You can run the program as an additional command in the Django app's manage.py. In this case, by substituting a program like devices_alive.py under/ device_mon / management / commands /and then executing python manage.py devices_alive, the result of life and death monitoring can be manipulated by Django's Model. Can be reflected in the database by

devices_alive.py


#! /usr/bin/env python
# -*- coding: utf-8 -*-
from django.core.management.base import BaseCommand
from device_mon.models import Device
import pyping


class Command(BaseCommand):
    def handle(self, *args, **options):

        devices = Device.objects.all()

        for device in devices:
            response = pyping.ping(device.management_ip, count=1, timeout=1000)
            if response.ret_code == 0:
                device.status = 1
            else:
                device.status = 0

            device.save()

Parsing BGP / Interface state

For the information you want to get directly by logging in to the router instead of SNMP, use the expect you know. Of course, this part can be replaced with netconf or API of various network devices. However, since I want to use it for devices that netconf etc. do not support or cannot be used due to various reasons, I will carry out a sadness expect this time. I don't go to get it on a regular basis, I only have to go get it when I need to confirm it, so I'll try to make it a structure that goes to get it when I press the Check button at the top of Django. The call flow is as follows.

  1. HTML POST
  2. Detected by urls.py and passed the operation to the target Views
  3. Call the expect script in Views.py
  4. Return the result from Views to html and reflect it on the Web

Exscript, which is a variant of pexpect, is used for acquisition. The following article was very helpful Try setting from software to router with SSH (Exscript)

Execute arbitrary command through the web screen

Since there is a limit to the information that can be grasped only by the portal, it is convenient when you want to issue a confirmation command on the router to obtain more detailed information. Like this: arrow_down: cmd_response.gif

The command you want to execute is sent to the Exscript on the back using urls.py, and the response content is reflected in the modal part.

Re-depiction of only some screens by Ajax

Since the argument to execute changes depending on the command, this time I put the command I want to get in the URL.

urls.py


    url(r'^cmd_response/(?P<device_id>\d+)/(?P<cmd>.+)$', views.get_cmd_response, name='cmd_response')

When device_id is the DB id for device identification and device_id is the command part you actually want to type, and the value is returned, Only the modal part is redrawn in Ajax, and the modal is displayed after completion.

python


  var AjaxReload = function(url,target_id) {
          $.ajax({
                 url: url,
                 dataType: 'html',
                 beforeSend: function(){
                    dispLoading("Now Loading...");
                  },
                 success: function(data) {
                     $(target_id).html(data);
                 },
                 error:function() {
                     alert('Reload Error!');
                     removeLoading();
                 },
                 complete : function() { ///Wait for the command execution result to return and then display the modal
                     removeLoading();
                     if(target_id=="#cmd_exec_result"){
                        $('#commandModal').modal('show')
                     }
                 }
          });
  }

device_mon/views.py



#plain cmd exec
def get_cmd_response(request,device_id,cmd):
    if cmd == 'None':
        cmd_result = 'None'
    else:
        cmd = cmd.replace('_',' ') # replace : show_bgp_su -> show bgp su
        device = Device.objects.get( pk = device_id )
        cmd_result = get_cmd(device,cmd)
    context = {
            'cmd_response'   : cmd_result,
            }
    return render(request,'device_mon/cmd_modal.html',context)

Real-time acquisition of Traffic, CPU, Memory

Using WebSocket, it takes the form of acquiring SNMP behind the scenes and returning the result. For here, please refer to the following of NetOps Coding Advent Calender that I wrote last year.

I tried to render the traffic in real time with WebSocket http://qiita.com/Mabuchin/items/135a9edff34a3db00744

Please see. I didn't daemonize the server-side program of Websocket, but it's okay to start it like this.

cd websock_server
nohup python if_traffic_wss.py &
nohup python process_wss.py &

As a supplement, there are many cases where you have to use the MIB of each vendor for the CPU and memory. The SNMP OIDs of IOS-XR and JUNOS use the following.

python


JUNOS_CPU_USE = '.1.3.6.1.4.1.2636.3.1.13.1.8.9.1.0.0'
JUNOS_MEM_USE = '.1.3.6.1.4.1.2636.3.1.13.1.11.9.1.0.0'
IOSXR_CPU_USE = '.1.3.6.1.4.1.9.9.109.1.1.1.1.7.2'
IOSXR_MEM_USE = '.1.3.6.1.4.1.9.9.221.1.1.1.1.18'
IOSXR_MEM_FREE = '.1.3.6.1.4.1.9.9.221.1.1.1.1.20'

IOS-XR cannot get the memory usage rate in one shot. Memory in use / (free memory + memory in use) After asking for it in the form of, I push it to the Client of WebSocket.

mem_use = int((float(mem_use) / float(mem_use+mem_free) ) * 100)

Finally

It's been a long time, but I introduced how to make your own dashboard. Recently, the number of useful tools for monitoring is increasing, but it is difficult to find one that perfectly matches your own operation policy. For regular monitoring, I think you should look at them together. However, if you need a tool that allows you to see information that should be checked immediately, such as during maintenance or failure, I think it would be a good idea to create a portal that matches the operation policy like this. I will.

Reference link summary

Netboard (dashboard made this time) Introduction to Django (1) I tried to render the traffic in real time with WebSocket Try setting from software to router with SSH (Exscript)

Recommended Posts

Create a dashboard for Network devices with Django!
Create a homepage with django
Create a file uploader with Django
Create a social integration API for smartphone apps with Django
[Python] Create a screen for HTTP status code 403/404/500 with Django
Create a model for your Django schedule
Create a Django schedule
Create a LINE BOT with Minette for Python
Create a one-file hello world application with django
Create a Todo app with Django REST Framework + Angular
I tried to create a table only with Django
Create a Todo app with the Django REST framework
code-server Online environment (2) Create a virtual network with Boto3
Create a Todo app with Django ③ Create a task list page
Create a Layer for AWS Lambda Python with Docker
Create a Todo app with Django ⑤ Create a task editing function
Create a django environment with docker-compose (MariaDB + Nginx + uWSGI)
Build a Django environment for Win10 (with virtual space)
Create an API with Django
Create a Django login screen
Create a heatmap with pyqtgraph
Create a directory with python
[Memo] Build a development environment for Django + Nuxt.js with Docker
Recommendations for django, wagtail ~ Why develop a website with python ~
Create a child account for connect with Stripe in Python
[Django] Create a model suitable for phone numbers / zip codes
Create a Twitter BOT with the GoogleAppEngine SDK for Python
Create a web API that can deliver images with Django
Create a Todo app with Django ① Build an environment with Docker
(For beginners) Try creating a simple web API with Django
Steps to create a Django project
Deploy a Django application with Docker
Create a virtual environment with Python!
Write a Residual Network with TFLearn
Build a web application with Django
Make a filter with a django template
Commands for creating SNS with Django
Create a poisson stepper with numpy.random
Create a LINE Bot in Django
Create a Python execution environment for Windows with VScode + Remote WSL
Create a temporary file with django as a zip file and return it
Create a striped illusion with gamma correction for Python3 and openCV3
Web App Development Practice: Create a Shift Creation Page with Django! (Shift creation page)
Create a color picker for the color wheel with Python + Qt (PySide)
Create an environment for Django x Apache x mod_wsgi with Vagrant (Ubuntu 16.04)
Create a development environment for Go + MySQL + nginx with Docker (docker-compose)
I made a development environment for Django 3.0 with Docker, Docker-compose, Poetry
Create a REST API to operate dynamodb with the Django REST Framework
Create a USB boot Ubuntu with a Python environment for data analysis
Create and return a CP932 CSV file for Excel with Chalice
[DynamoDB] [Docker] Build a development environment for DynamoDB and Django with docker-compose
Programming for humans with a well-defined __repr__
Compose with a neural network! Run Magenta
Create a Python function decorator with Class
Create RESTful APIs with Django Rest Framework
A simple RSS reader made with Django
Build a blockchain with Python ① Create a class
Create a dummy image with Python + PIL.
Let's create a virtual environment for Python
[Python] Create a virtual environment with Anaconda
Let's create a free group with Python