Convert GRIB2 format weather data that cannot be opened with pygrib to netCDF and visualize it

Convert GRIB2 format weather data that cannot be opened with pygrib to netCDF and visualize it

Introduction

The "GRIB2" format is the industry standard for the format of meteorological data with a three-dimensional spread. There is a detailed introduction by @e_toyoda about the GRIB2 format. About the WMO (World Meteorological Organization) grid data format GRIB2

There are various tools for reading data from GRIB2 format files. For python, you can use a module called pygrib.

However, since the national synthetic radar and estimated meteorological distribution are data at 1km intervals nationwide, the amount of data is large, and unlike GSM and MSM, the compression format specified by the Japan Meteorological Agency is used. .. This doesn't seem to be supported by pygrib and there is some data that can't be opened.

I want to make a Deep Learning program using this data with Keras, so I want to be able to handle it with python. Therefore, I converted it to the netCDF format that can be handled by python using a tool called wgrib2 that supports the Japan Meteorological Agency format.

National synthetic radar that cannot be opened with pygrib

The National Synthetic Radar was downloaded from Research Institute for Sustainable Humanosphere, Kyoto University. When I open this data file with pygrib, I get the following error:

>>> import pygrib
>>> grbs=pygrib.open("Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin")
ECCODES ERROR   :  Unable to find template productDefinition from grib2/template.4.50008.def 
ECCODES ERROR   :  Unable to find template productDefinition from grib2/template.4.50008.def 

This is probably because the definition used by the Japan Meteorological Agency does not exist in pygrib.

Conversion using wgrib2

wgrib2

For wgrib2, @ysomei has summarized it below. It is a tool provided by the US National Oceanic and Atmospheric Administration (NOAA). I've just analyzed the meteorological data

There are various options and you can do various things. For example, you can output the contents of the GRIB file to CSV as follows. Except for the created CSV file, var0_1_201 and surface are included in the label that specifies the data. The last three elements are longitude, latitude, and data (“data representative value”).

python


> wgrib2 -V GRIB_formatted_file -csv tmp.csv
> cat tmp.csv
Abbreviation
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.919,22.6875,7.25
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.931,22.6875,16.5
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.944,22.6875,9.25
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.956,22.6875,0
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.969,22.6875,0
Omission

Combined radar data format (data representative value and level value)

The "data representative value" of the synthetic radar is in mm / h, which is the rainfall intensity. The original GRIB2 format file does not have a direct rainfall intensity value set, but a "level value" from 0 to 251. For example, a "level value" of "37" means that the rainfall intensity is "7.0 mm / h or more and less than 7.5 mm / h". The "data representative value" is "7.75 mm / h".

In other words, there is a conversion table from "level value" to "data representative value", but wgrib2 seems to do that too (although I did not decipher wgrib2).

Convert to NetCDF format

Use the following options to convert from GRIB2 to netCDF format using wgrib2.

wgrib.sh


wgrib2 GRIB_formatted_file -netcdf GRIB_formatted_file.nc

This will create a netCDF format file. However, please note that the file size will increase dramatically, probably because the compression is released.

ls


235702   Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin
34454836 Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin.nc

Reading data using netCDF4 with python

This file can be handled as follows using the python module netCDF4.

netcdf4_open.py


from netCDF4 import Dataset
(Omitted)

nc = Dataset( fn , 'r' )
xx = nc.variables['longitude'][:] #
yy = nc.variables['latitude' ][:] #
data = nc.variables['var0_1_201_surface'][:]
lons, lats = np.meshgrid( xx , yy )
        
_save_file = savefilenamelist[ptr_sfile]

drawmap( data[0,:,:] , levels_rai , precip_colormap , _save_file ,_flag_show , "RAIN" , norm)

We have specified var0_1_201_surface to identify the data contained in the file. When I referred to the contents of the GRIB file with wgrib2, it seemed to be divided into var0_1_201 and surface, but when I converted it to a NetCDF file, it seems to be converted to the label var0_1_201_surface. When you output the information of the netcdf file as shown below, this label appears at the end of variables.

DumpNetCDF


>>> from netCDF4 import Dataset
>>> nc=Dataset("Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin.nc",'r')
>>> nc
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF3_CLASSIC data model, file format NETCDF3):
    Conventions: COARDS
    History: created by wgrib2
    GRIB2_grid_template: 0
    dimensions(sizes): latitude(3360), longitude(2560), time(1)
    variables(dimensions): float64 latitude(latitude), float64 longitude(longitude), 
    float64 time(time), float32 var0_1_201_surface(time,latitude,longitude)
    groups:

Visualization

The python source to visualize. I am using matplotlib.

Although it is a front miso, this and that of visualization is [try to draw a "weather map-like front" by machine learning based on weather data (2) Story of weather data visualization as input](https: // qiita. com / m-taque / items / 988b08185097dca5f5b5 "Drawing a" weather map-like front "based on meteorological data by machine learning (2) The story of weather data visualization as input"). Since the color map to be visualized has a white range, the map itself is slightly colored in the visualization below. The order of overlaying the colored map and contour map is specified by zorder.

draw_routine


def drawmap( values , levels , cmap , _save_filename , flag_show , name ,norm) :
        fig,ax = plt.subplots(figsize=(5,5))
        plt.subplots_adjust(left=0.02, right=0.98, top=0.98, bottom=0.02)
     
        m = Basemap(projection='stere', llcrnrlat=25, urcrnrlat=47, llcrnrlon=125, urcrnrlon=150, lat_0=60,  lon_0=140, resolution='i' )
        
        m.drawparallels(np.arange(-80.,81.,10.))
        m.drawmeridians(np.arange(-180.,181.,10.))
        m.drawcoastlines()
        m.drawmapboundary(fill_color='#eeeeee' , zorder=-1)
        m.fillcontinents(color='#ceceee' , zorder=-1)
        
        x , y = m(lons, lats)

        m.contourf( x , y , values , levels , cmap=cmap , zorder=0 , norm=norm )

        fig.savefig(_save_filename)
        if flag_show == 'yes' :
                plt.show()
        plt.close()

You can create an image like the one shown below (UTC data at 15:00 on September 8, 2019).

gsrd.v5.201909081500.png

Summary

I described how to convert a file that is GRIB2 format but cannot be handled by pygrib to netCDF format so that it can be handled from python.

Recommended Posts

Convert GRIB2 format weather data that cannot be opened with pygrib to netCDF and visualize it
Convert mesh data exported from SpriteUV2 to a format that can be imported by Spine
Examples and solutions that cannot be optimized well with scipy.optimize.least_squares
Convert the spreadsheet to CSV and upload it to Cloud Storage with Cloud Functions
Read CSV file with Python and convert it to DataFrame as it is
Address to the bug that node.surface cannot be obtained with python3 + mecab
Convert Excel data to JSON with python
Convert FX 1-minute data to 5-minute data with Python
Items that cannot be imported with sklearn
Convert Mobile Suica usage history PDF to pandas Data Frame format with tabula-py
Format DataFrame data with Pytorch into a form that can be trained with NN