[Django] Google map display of GIS data and graphing of parameters

1.First of all

I created this article because I didn't have a simple code to upload csv data, graph it, display it on a map, and display it. The completed image is shown below.

Fig.1 Completed web page

The upper half of Fig.1 was created using a line graph of amchart.js. The display range can be freely selected for both the x-axis and y-axis. The x-axis is the time axis. The lower half of Fig.2 uses GoogleMapAPI to input and display latitude / longitude data.

2. Services and frameworks used

・ Google Map API ・ Django ・ Amchart.js

3. Implementation

3-1. Construction environment

The environment and version are as follows.

macOS Catalina version 10.15.4
Python 3.7.0

3-2. Directory structure

─root─app─templates─app─base.html
     │   |             └import.html
     │   ├migrations-・ ・ ・
     │   ├forms.py
     │   ├views.py
│ ・ ・ ・ ・
     |
     |
     ├project─settings.py
     |       ├urls.py
     |・ ・ ・
     |
     ├static─js─mychart.js
     |      |  └googlemap.js
     |      └media-temp.csv (<-A file that temporarily complements the imported csv file)
     └manage.py

It follows Django's basic directory structure.

3-3. Code

app/views.py

import csv
from django.http import HttpResponse
from django.shortcuts import redirect
from django.urls import reverse_lazy
from django.views import generic
from .forms import CSVUploadForm
from .models import Post


class PostIndex(generic.ListView):
    model = Post


class PostImport(generic.FormView):
    template_name = 'app/import.html'
    success_url = reverse_lazy('app:index')
    form_class = CSVUploadForm

    def form_valid(self, form):
        form.save()
        return redirect('app:index')


def post_export(request):
    response = HttpResponse(content_type='text/csv')
    response['Content-Disposition'] = 'attachment; filename="posts.csv"'
    #The HttpResponse object is a file-like object, so csv.You can pass it to the writer as it is.
    writer = csv.writer(response)
    for post in Post.objects.all():
        writer.writerow([post.pk, post.title])
    return response

forms.py is the most devised source code.

app/forms.py

import csv
import io
from django import forms
from django.core.validators import FileExtensionValidator
from .models import Post
import codecs

class CSVUploadForm(forms.Form):
    file = forms.FileField(
        label='CSV file',
        help_text='* Please upload the file with the extension csv.',
        validators=[FileExtensionValidator(allowed_extensions=['csv'])]
    )

    def clean_file(self):
        file = self.cleaned_data['file']

        print("file:",file)

        # csv.Convert to a text mode file with TextIOWrapper to pass to reader
        csv_file = io.TextIOWrapper(file, encoding='utf-8')
        print("csv_file",csv_file)
        reader = csv.reader(csv_file)
        print("reader",reader)
        
        self.csv_file = reader
        
        for row in reader:
            #print("reader row", row)
            #print("line_num", reader.line_num)
            if reader.line_num == 1:
                with open("static/media/temp.csv", mode="w", encoding="utf-8") as f:
                    print("row in x before", row)
                    row = ",".join(row) 
                    row = row + "\n" 
                    print("row in x after", row)
                    f.write(row)
                print("option=x")

            else:
                with open("static/media/temp.csv", mode="a", encoding="utf-8") as f:
                    #print("row in a before", row)
                    #print(type(row))
                    row = ",".join(row) 
                    row = row + "\n" 
                    #print("row in a after", row)
                    #print(type(row))
                    
                    f.write(row)
                #print("option=a")


        #A list to store the unsaved model instance created from each row

        self._instances = []
        
        try:
            for row in reader:
                post = Post(pk=row[0], title=row[1])
                self._instances.append(post)

        except UnicodeDecodeError:
            raise forms.ValidationError('Check the file encoding and the correct CSV file.')

        return file

    def save(self):
        Post.objects.bulk_create(self._instances, ignore_conflicts=True)
        Post.objects.bulk_update(self._instances, fields=['title'])
        print("save content in save func", Post)

app/urls.py

from django.urls import path
from . import views

app_name = 'app'

urlpatterns = [
    path('', views.PostIndex.as_view(), name='index'),
    path('import/', views.PostImport.as_view(), name='import'),
    path('export/', views.post_export, name='export'),
]

Of the following sources, [API-KEY] inserts API-Key. Please issue Google_API-Key by yourself.

app/templates/app/base.html

<!doctype html>
{% load static %}
<html lang="ja">
  <head>
    <!-- Required meta tags -->
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

    <!-- Bootstrap CSS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">

    <style>
      #chartdiv {
        width: 100%;
        height: 500px;
      }
      #map {
        width: 100%;
        height: 400px;
      }
      </style>
      
      <!-- Resources -->
      <script src="https://www.amcharts.com/lib/4/core.js"></script>
      <script src="https://www.amcharts.com/lib/4/charts.js"></script>
      <script src="https://www.amcharts.com/lib/4/themes/animated.js"></script>
      
      <!-- Chart code -->
      <script src="{% static 'js/mychart.js' %}"></script>
    <title>Rover sensor visualization</title>
  </head>

  <body>
    <ul class="nav justify-content-center">
      <li class="nav-item">
        <a class="nav-link" href="{% url 'app:index' %}">Index</a>
      </li>
      <li class="nav-item">
        <a class="nav-link" href="{% url 'app:import' %}">CSV read</a>
      </li>
      <li class="nav-item">
        <a class="nav-link" href="{% url 'app:export' %}">CSV output</a>
      </li>
    </ul>
  <div class="container">
    {% block content %}{% endblock %}
      
  
    <div id="map"></div>
      
    <script  src="{% static 'js/googlemap.js' %}"></script>
    <script src="https://maps.googleapis.com/maps/api/js?key=[API-KEY]"
async defer></script>

    <!-- Optional JavaScript -->
    <!-- jQuery first, then Popper.js, then Bootstrap JS -->
    <script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.9/umd/popper.min.js" integrity="sha384-ApNbgh9B+Y1QKtv3Rn7W3mgPxhU9K/ScQsAP7hUibX39j7fakFPskvXusvfa0b4Q" crossorigin="anonymous"></script>
    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js" integrity="sha384-JZR6Spejh4U02d8jOt6vLEHfe/JQGiRRSQQxSfFWpi1MquVdAyjUar5+76PVCmYl" crossorigin="anonymous"></script>
  </div>

  </body>
</html>

app/templates/app/import.html

{% extends 'app/base.html' %}

{% block content %}
<form action="" method="POST" enctype="multipart/form-data">
  {{ form.as_ul }}
  {% csrf_token %}
  <button type="submit">Send</button>
</form>
{% endblock %}

app/templates/app/post_list.html

{% extends 'app/base.html' %}

{% block content %}
<div id="chartdiv"></div>
{% endblock %}

project/static/js/mychart.js

// 2)Convert from CSV to 2D array
function csv2Array(str) {
    var csvData = [];
    var lines = str.split("\n");
    for (var i = 0; i < lines.length; ++i) {
      var cells = lines[i].split(",");
      csvData.push(cells);
    }
    return csvData;
  }
  
  function drawBarChart(data) {
    // 3)chart.Prepare an array for js dataset
    var tmpLabels = [], tmpData1 = [], tmpData2 = [];
    for (var row in data) {
      tmpLabels.push(data[row][0])
      tmpData1.push(data[row][1])
      tmpData2.push(data[row][2])
    };
  
    // 4)chart.Draw with js
    var ctx = document.getElementById("myChart").getContext("2d");
    var myChart = new Chart(ctx, {
      type: 'bar',
      data: {
        labels: tmpLabels,
        datasets: [
          { label: "Tokyo", data: tmpData1, backgroundColor: "red" },
          { label: "Osaka", data: tmpData2, backgroundColor: "blue" }
        ]
      }
    });
  }
  
  function makeLineChart(data) {
    am4core.ready(function() {
            
      // Themes begin
      am4core.useTheme(am4themes_animated);
      // Themes end
      
      // Create chart instance
      var chart = am4core.create("chartdiv", am4charts.XYChart);
      
      // Add data
      console.log(data);
      console.log(typeof data);
      console.log(data[1][1]);
      chart.data = generateChartData(data); //add to
      console.log(chart.data);
      
      // Create axes
      var xAxis = chart.xAxes.push(new am4charts.ValueAxis());
  
      //xAxis.renderer.minGridDistance = 150;
      xAxis.title.text = "Time (sec)";
      
      var yAxis = chart.yAxes.push(new am4charts.ValueAxis());
      yAxis.title.text = "ax";
      
      // Create series
      var series = chart.series.push(new am4charts.LineSeries());
      series.dataFields.valueX = "timestamp";
      series.dataFields.valueY ="ax";
      series.name = "ax";
    
      series.strokeWidth = 2;
      series.minBulletDistance = 10;
      series.tooltipText = "{valueY}";
      series.tooltip.pointerOrientation = "vertical";
      series.tooltip.background.cornerRadius = 20;
      series.tooltip.background.fillOpacity = 0.5;
      series.tooltip.label.padding(12,12,12,12)
  
      var series2 = chart.series.push(new am4charts.LineSeries());
      series2.dataFields.valueX = "timestamp";
      series2.dataFields.valueY = "ay";
      series2.name = "ay";
  
  
      var series3 = chart.series.push(new am4charts.LineSeries());
      series3.dataFields.valueX = "timestamp";
      series3.dataFields.valueY = "az";
      series3.name = "az";
  
      // Create y axis range
      var range = yAxis.axisRanges.create();
      range.label.disabled = false;
      range.label.rotation = 270;
  
      // Create x axis range
      var range_x = xAxis.axisRanges.create();
      range_x.label.disabled = false;
      range_x.label.rotation = 0;
  
  
      // Add scrollbar
      chart.scrollbarX = new am4charts.XYChartScrollbar();
      chart.scrollbarX.series.push(series);
      chart.scrollbarX.series.push(series2);
      chart.scrollbarX.series.push(series3);
      chart.scrollbarY = new am4charts.XYChartScrollbar();
      chart.scrollbarY.series.push(series);
      chart.scrollbarY.series.push(series2);
      chart.scrollbarY.series.push(series3);
      
      // Add cursor
      chart.cursor = new am4charts.XYCursor();
      chart.cursor.xAxis = xAxis;
      chart.cursor.yAxis = yAxis;
      chart.cursor.snapToSeries = series;
      chart.cursor.snapToSeries = series2;
      chart.cursor.snapToSeries = series3;
      
      // Add legend
      chart.legend = new am4charts.Legend();
  
      function generateChartData(data) {
          var chartData = [];
          var index_length = Number(data.length);
          const initial_time = data[0][0]
          console.log(index_length);
          for (var i = 0; i < (index_length-1); i++) {
      
              var data_line = data[i];
  
              chartData.push({
                timestamp: (data_line[0]-initial_time)/1000,
                ax: data_line[1],
                ay: data_line[2],
                az: data_line[3],
                wx: data_line[4],
                wy: data_line[5],
                wz: data_line[6],
                mx: data_line[7],
                my: data_line[8],
                mz: data_line[9],
                lat: data_line[10],
                lng: data_line[11],
                yaw: data_line[12]              
            });
          }
  
          return chartData;
      }
      
      });
  }
  
  function main(file_name) {
    // 1)Load CSV file with ajax
    var req = new XMLHttpRequest();
    var filePath = file_name//'acc_gyro.csv';
    req.open("GET", filePath, true);
    req.onload = function() {
      // 2)Call CSV data conversion
      data = csv2Array(req.responseText);
      // 3) amChart.js data preparation, 4) amChart.js drawing call
      makeLineChart(data);
    }
    req.send(null);
  }

  main("/static/media/temp.csv");

static/js/googlemap.js

// 2)Convert from CSV to 2D array
function csv2Array(str) {
    var csvData = [];
    var lines = str.split("\n");
    for (var i = 0; i < lines.length; ++i) {
      var cells = lines[i].split(",");
      csvData.push(cells);
    }
    return csvData;
  }

//Properties and properties in a two-dimensional array
function mapData(data) {
    var mapData = [];
    const initial_time = data[0][0]
    var index_length = Number(data.length);
    for (var i = 0; i < (index_length-1); i++) {

        var data_line = data[i];

        mapData.push({
          timestamp: (data_line[0]-initial_time)/1000,
          ax: data_line[1],
          ay: data_line[2],
          az: data_line[3],
          wx: data_line[4],
          wy: data_line[5],
          wz: data_line[6],
          mx: data_line[7],
          my: data_line[8],
          mz: data_line[9],
          lat: data_line[10],
          lng: data_line[11],
          yaw: data_line[12]              
      });
    }
    return mapData;
}


function initMap(data) {
  var map;
  mapdata = mapData(data);
  var index_length = Number(data.length);
  console.log(mapdata);
  console.log(index_length);
  initial_loc = mapdata[0];

  map = new google.maps.Map(document.getElementById('map'), {
    center: {lat: Number(initial_loc.lat), lng: Number(initial_loc.lng)},
    zoom: 15
  });
  
　for(var i = 1; i < (index_length-1); i++) {
    location_line = mapdata[i];
    //console.log(i); 
    //console.log(location_line);  
    //Marker initial settings
    var markerOpts = {
    position: {lat: Number(location_line.lat), lng: Number(location_line.lng)},
    map: map,
    title: "mark"
    };
    //Create a Marker using the Marker Options created just before
    var marker = new google.maps.Marker(markerOpts);
  }
  
}

function main(file_name) {
    // 1)Load CSV file with ajax
    var req = new XMLHttpRequest();
    var filePath = file_name;//'static/js/acc_gyro.csv';
    req.open("GET", filePath, true);
    req.onload = function() {
      // 2)Call CSV data conversion
      data = csv2Array(req.responseText);
      // 3)google map data preparation, 4)google map drawing call
      initMap(data);
    }
    req.send(null);
  }

main("/static/media/temp.csv");

project/setting.py

(Omission)
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.0/howto/static-files/

STATIC_URL = '/static/'
STATICFILES_DIRS = (
    os.path.join(BASE_DIR, "static"),
)

4. Impressions

・ It will be the first web application created using Django. I am deeply moved. -Since there was no description in the tutorial on how to set the static directory, I learned a lot. ・ Next, I would like to report the results of deploying AWS to EC2 instances.

5. References

How to implement amchart: https://www.suzu6.net/posts/56-amcharts-samples/ Implementation of csv upload function: https://qiita.com/t-iguchi/items/d2862e7ef7ec7f1b07e5