I tried to display the point cloud data DB of Shizuoka prefecture with Vue + Leaflet

What I made this time

Shizuoka Point Cloud Data Map GitHub - shizokaPCDB スクリーンショット 2020-01-16 1.35.33.png

Caution

This article is not an article that displays lidar data in Leaflet. It displays information in the point cloud database (data acquisition location, construction date, etc.).

Introduction

スクリーンショット 2020-01-16 1.49.59.png Rock Shizuoka Prefectural Office has a large number of LIDAR data of construction products as open data on its own website Shizuoka Point Cloud DB (hereinafter referred to as "Shizuoka PCDB"). It is open to the public. Looking at the source with interest, I found that it seems that data can be obtained from the outside (because JavaScript was written directly in html). However, as you can see from this site and the above image, (1) the number of markers is large and the performance is affected, and (2) every time the map area changes, the data contained in that area is acquired from the server, and the performance further deteriorates. I thought there were two problems: what I was doing. I thought that there was room for improvement if the data could be obtained from the outside, so I decided to develop it.

For that reason, we will send it in two ways: access to the database, shaping, and display at the front desk.

Technology used

Database access / formatting

access

Looking at the source of Shizuoka PCDB, access to the server and display in Leaflet were implemented in about 100 lines. The part that retrieves data from the server is shown below.


var data = { request : "MarkerSet",
                Xmax : map.getBounds().getNorthEast().lng,
                Xmin : map.getBounds().getSouthWest().lng ,
                Ymax : map.getBounds().getNorthEast().lat,
                Ymin : map.getBounds().getSouthWest().lat };
    $.ajax("ankenmapsrc",{
        type: "GET",
        data: data,
        success: function(data, dataType){
            var myIcon = L.icon({
                iconUrl: 'http://cdn.leafletjs.com/leaflet-0.5/images/[email protected]',
                iconSize: [15,25],
                iconAnchor: [5, 5],
            });
//The following processing continues

You can get the data by accessing the ankenmapsrc with the query data. Here, data has the request name "MarkerSet" and Xmax, Xmin, Ymax, Ymin indicating the area to be acquired. The four variables in the area store the area displayed in Leaflet. This process is called every time the Leaflet display area is updated. In other words, the data is acquired from the server each time.

Now let's look at the return value of this request.

30XXX01010001:2018 Nirayama Reverberatory Furnace Measurement Business:138.96214537214:35.03962001009?
28XXX00030007:Shiraito no Taki Takimi Bridge area maintenance project No. 7:138.58870495572:35.312506370532?
28XXX00030008:Shiraito no Taki Takimi Bridge area maintenance project No. 8:138.58881502806:35.312596432406?
28XXX00030009:Shiraito no Taki Takimi Bridge area maintenance project No. 9:138.58892510063:35.312686494178?
29C2001011361:2017 [29th-C2001-No. 01] Outsourced fact-finding survey of outdoor advertisements on the Izu Peninsula (Kannami Town Road)_1-Line 2):138.93794860595:35.083520492945
...

Originally there is no line break, but it is added for explanation. Looking at this data, you can see that the delimiter for each construction is "?" And the delimiter for each data element is ":".

Plastic surgery

Access Shizuoka PCDB with Flask and create an API server that responds to the formatted data.

import urllib.request, urllib.parse
import json
@app.route('/markers')
def getMarkers():
    #In order to get all records, set the latitude and longitude including the entire Shizuoka prefecture as an integer value.
    xMax = 140
    xMin = 137
    yMax = 36
    yMin = 33

    params = {
        'request':'MarkerSet',
        'Xmax':xMax,
        'Xmin':xMin,
        'Ymax':yMax,
        'Ymin':yMin
    }
    p = urllib.parse.urlencode(params)
    url = "https://pointcloud.pref.shizuoka.jp/lasmap/ankenmapsrc?" + p

    #Request to SIZUOKA POINT CLOUD DB with the URL parameter generated above and get the matter list character string
    allAnkenStr = ""
    with urllib.request.urlopen(url) as res:
        allAnkenStr = res.read().decode()

    #Create json to return
    ankensObj = {
        "ankenList":[]
    }

    ankenList = allAnkenStr.split('?')
    for anken in ankenList:
        ankenInfo = anken.split(':')
        #If there is inappropriate data, skip it
        if len(ankenInfo) != 4:
            continue

        #Convert Japanese calendar to Western calendar
        yy = int(ankenInfo[0][:2])
        #Reiwa
        if yy < 24:
            yyyy = 2018 + yy
        else:
            yyyy = 1988 + yy

        ankenObj = {
            "no":ankenInfo[0],
            "name":ankenInfo[1],
            "lon":ankenInfo[2],
            "lat":ankenInfo[3],
            "year":yyyy
        }
        ankensObj['ankenList'].append(ankenObj)
    return jsonify(ankensObj)

Problems explained at the beginning (2) Every time the map area changes, the data contained in that area is acquired from the server, and the performance is further reduced. It was solved by doing. That's because the total number of cases was about 1400, and it wasn't too late to actually get all of them, so we concluded that we should get all of them first.

We decided to parse the acquired anken data and return the construction number, construction name, longitude / latitude, and construction year as json.

Display at the front

Remaining problems (1) The number of markers is large and the performance is affected, but I thought that it would be solved if it was displayed on Mapbox GL JS, which operates quickly. However, when I implemented it, the marker display became slower than Leaflet. So, I solved this problem by adopting Leaflet's Marker Cluster.

Regarding the environment construction of Leaflet and Vue, Try # 027 – I tried to build the development environment of Leaflet and Mapbox GL JS with Vue.js I will omit it because it is detailed. In addition, install vue2-leaflet-markercluster.

npm install vue2-leaflet-markercluster

Asynchronous communication with API server with Vue

Use the Fetch API. Get the data with App.vue and pass it to MapPane.vue.

App.vue


<MapPane :ankens="ankens"/>

App.vue


  data() {
    return {
      ankens:[],
    }
  },
  created() {
    let vm = this
    fetch("/markers")
    .then(response => {
        return response.json()
    })
    .then(data => {
        //After sorting, I'm passing data
        vm.ankens = data.ankenList.sort(function (a, b) {
            if (a.no < b.no) {
                return 1
            }
            if (a.no > b.no) {
                return -1
            }
            return 0 
        }).sort(function (a, b) {
            if (a.year < b.year) {
                return 1
            }
            if (a.year > b.year) {
                return -1
            }
            return 0 
        })
    })
    .catch(error => {
        console.log(error)
        alert("An error has occurred.")
    });
  }

MapPane.vue


<template>
    <div class="mapPane">
        <l-map
            :zoom="zoom"
            :center="center"
            :preferCanvas="true"
        >
            <l-control-scale
                position="bottomleft"
                :imperial="false"
                :metric="true"
            ></l-control-scale>

            <l-tile-layer
                :name="tileProvider.name"
                :visible="tileProvider.visible"
                :url="tileProvider.url"
                :attribution="tileProvider.attribution"
            ></l-tile-layer>

            <Vue2LeafletMarkerCluster :options="clusterOptions" >
                <LMarker v-for="anken in ankens" :key="anken.no" :lat-lng="makeLatLng(anken)" @click="onMarkerClick(anken.no)">
                    <LPopup :content="makeMarkerContent(anken)" ></LPopup>
                </LMarker>
            </Vue2LeafletMarkerCluster>
        </l-map>
    </div>
</template>

MapPane.vue



props: {
    ankens:Array
},

MapPane.vue


<style scoped>
    @import "~leaflet.markercluster/dist/MarkerCluster.css";
    @import "~leaflet.markercluster/dist/MarkerCluster.Default.css";

</style>

Wrap LMarker in Vue2LeafletMarkerCluster and it will work. As a result, the close markers form a cluster (bundle) and are displayed together as one feature until it expands. As a result, problem (1) has also been resolved.

in conclusion

In the future, I would like to be able to sort and search the data list display, and display the animation being loaded. I arrived at Shizuoka PCDB when I was looking up the LIDAR data hotly, but it was an unexpected big derailment. Mr. Shizuoka, who uses such a large amount of valuable data as open data, is really Rock, and recently Hyogo Prefecture seems to have joined the ranks of Rocker. I feel that this project is a little different from the mainstream of open data utilization, but at least I think it has become a self-improvement. I would like to continue to set up an antenna in the open data area.

Recommended Posts

I tried to display the point cloud data DB of Shizuoka prefecture with Vue + Leaflet
I tried to save the data with discord
I tried to visualize the running data of the racing game (Assetto Corsa) with Plotly
I tried to find the entropy of the image with python
I tried to find the average of the sequence with TensorFlow
I tried to analyze the data of the soccer FIFA World Cup Russia tournament with soccer action
I tried to automate the watering of the planter with Raspberry Pi
I tried to expand the size of the logical volume with LVM
I tried to improve the efficiency of daily work with Python
I tried to get the authentication code of Qiita API with Python.
I tried to automatically extract the movements of PES players with software
I tried to analyze the negativeness of Nono Morikubo. [Compare with Posipa]
I tried to streamline the standard role of new employees with Python
I tried to visualize the text of the novel "Weathering with You" with WordCloud
I tried to get the movie information of TMDb API with Python
I tried to display the altitude value of DTM in a graph
I tried to predict the behavior of the new coronavirus with the SEIR model.
I tried to open the latest data of the Excel file managed by date in the folder with Python
I tried to get and analyze the statistical data of the new corona with Python: Data of Johns Hopkins University
Since it is the 20th anniversary of the formation, I tried to visualize the lyrics of Perfume with Word Cloud
I tried to touch the API of ebay
I tried to get CloudWatch data with Python
I tried to correct the keystone of the image
I tried to predict the price of ETF
I tried to vectorize the lyrics of Hinatazaka46!
I tried to easily visualize the tweets of JAWS DAYS 2017 with Python + ELK
[IBM Cloud] I tried to access the Db2 on Cloud table from Cloud Funtions (python)
I tried to rescue the data of the laptop by booting it on Ubuntu
[Verification] Try to align the point cloud with the optimization function of pytorch Part 1
The story of making soracom_exporter (I tried to monitor SORACOM Air with Prometheus)
I tried to display the infection condition of coronavirus on the heat map of seaborn
I tried to create a model with the sample of Amazon SageMaker Autopilot
I tried to automatically send the literature of the new coronavirus to LINE with Python
I tried to learn the sin function with chainer
I tried to extract features with SIFT of OpenCV
I tried to summarize the basic form of GPLVM
I tried to touch the CSV file with Python
I tried to predict the J-League match (data analysis)
I tried to solve the soma cube with python
I tried to build ML Pipeline with Cloud Composer
I tried using the API of the salmon data project
I tried to visualize the spacha information of VTuber
I tried to erase the negative part of Meros
I tried to solve the problem with Python Vol.1
I tried to analyze J League data with Python
I tried to classify the voices of voice actors
I tried to summarize the string operations of Python
I tried to make something like a chatbot with the Seq2Seq model of TensorFlow
I tried to put out the frequent word ranking of LINE talk with Python
I tried to automate the article update of Livedoor blog with Python and selenium.
I tried to visualize the characteristics of new coronavirus infected person information with wordcloud
[First data science ⑥] I tried to visualize the market price of restaurants in Tokyo
I just wanted to extract the data of the desired date and time with Django
I tried to compare the processing speed with dplyr of R and pandas of Python
The 15th offline real-time I tried to solve the problem of how to write with python
I'm tired of Python, so I tried to analyze the data with nehan (I want to go live even with corona sickness-Part 2)
I'm tired of Python, so I tried to analyze the data with nehan (I want to go live even with corona sickness-Part 1)
[Horse Racing] I tried to quantify the strength of racehorses
I tried to simulate how the infection spreads with Python
I tried to analyze the whole novel "Weathering with You" ☔️
I tried to get the location information of Odakyu Bus