The "GRIB2" format is the industry standard for the format of meteorological data with a three-dimensional spread. There is a detailed introduction by @e_toyoda about the GRIB2 format. About the WMO (World Meteorological Organization) grid data format GRIB2
There are various tools for reading data from GRIB2 format files. For python, you can use a module called pygrib.
However, since the national synthetic radar and estimated meteorological distribution are data at 1km intervals nationwide, the amount of data is large, and unlike GSM and MSM, the compression format specified by the Japan Meteorological Agency is used. .. This doesn't seem to be supported by pygrib and there is some data that can't be opened.
I want to make a Deep Learning program using this data with Keras, so I want to be able to handle it with python. Therefore, I converted it to the netCDF format that can be handled by python using a tool called wgrib2 that supports the Japan Meteorological Agency format.
The National Synthetic Radar was downloaded from Research Institute for Sustainable Humanosphere, Kyoto University. When I open this data file with pygrib, I get the following error:
>>> import pygrib
>>> grbs=pygrib.open("Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin")
ECCODES ERROR : Unable to find template productDefinition from grib2/template.4.50008.def
ECCODES ERROR : Unable to find template productDefinition from grib2/template.4.50008.def
This is probably because the definition used by the Japan Meteorological Agency does not exist in pygrib.
wgrib2
For wgrib2, @ysomei has summarized it below. It is a tool provided by the US National Oceanic and Atmospheric Administration (NOAA). I've just analyzed the meteorological data
There are various options and you can do various things. For example, you can output the contents of the GRIB file to CSV as follows.
Except for the created CSV file, var0_1_201
and surface
are included in the label that specifies the data. The last three elements are longitude, latitude, and data (“data representative value”).
python
> wgrib2 -V GRIB_formatted_file -csv tmp.csv
> cat tmp.csv
Abbreviation
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.919,22.6875,7.25
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.931,22.6875,16.5
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.944,22.6875,9.25
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.956,22.6875,0
"2020-02-07 00:00:00","2020-02-07 00:00:00","var0_1_201","surface",121.969,22.6875,0
Omission
The "data representative value" of the synthetic radar is in mm / h, which is the rainfall intensity. The original GRIB2 format file does not have a direct rainfall intensity value set, but a "level value" from 0 to 251. For example, a "level value" of "37" means that the rainfall intensity is "7.0 mm / h or more and less than 7.5 mm / h". The "data representative value" is "7.75 mm / h".
In other words, there is a conversion table from "level value" to "data representative value", but wgrib2 seems to do that too (although I did not decipher wgrib2).
Use the following options to convert from GRIB2 to netCDF format using wgrib2.
wgrib.sh
wgrib2 GRIB_formatted_file -netcdf GRIB_formatted_file.nc
This will create a netCDF format file. However, please note that the file size will increase dramatically, probably because the compression is released.
ls
235702 Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin
34454836 Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin.nc
This file can be handled as follows using the python module netCDF4.
netcdf4_open.py
from netCDF4 import Dataset
(Omitted)
nc = Dataset( fn , 'r' )
xx = nc.variables['longitude'][:] #
yy = nc.variables['latitude' ][:] #
data = nc.variables['var0_1_201_surface'][:]
lons, lats = np.meshgrid( xx , yy )
_save_file = savefilenamelist[ptr_sfile]
drawmap( data[0,:,:] , levels_rai , precip_colormap , _save_file ,_flag_show , "RAIN" , norm)
We have specified var0_1_201_surface
to identify the data contained in the file.
When I referred to the contents of the GRIB file with wgrib2, it seemed to be divided into var0_1_201
and surface
, but when I converted it to a NetCDF file, it seems to be converted to the label var0_1_201_surface
.
When you output the information of the netcdf file as shown below, this label appears at the end of variables
.
DumpNetCDF
>>> from netCDF4 import Dataset
>>> nc=Dataset("Z__C_RJTD_20200207000000_RDR_JMAGPV_Ggis1km_Prr10lv_ANAL_grib2.bin.nc",'r')
>>> nc
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF3_CLASSIC data model, file format NETCDF3):
Conventions: COARDS
History: created by wgrib2
GRIB2_grid_template: 0
dimensions(sizes): latitude(3360), longitude(2560), time(1)
variables(dimensions): float64 latitude(latitude), float64 longitude(longitude),
float64 time(time), float32 var0_1_201_surface(time,latitude,longitude)
groups:
The python source to visualize. I am using matplotlib.
Although it is a front miso, this and that of visualization is [try to draw a "weather map-like front" by machine learning based on weather data (2) Story of weather data visualization as input](https: // qiita. com / m-taque / items / 988b08185097dca5f5b5 "Drawing a" weather map-like front "based on meteorological data by machine learning (2) The story of weather data visualization as input"). Since the color map to be visualized has a white range, the map itself is slightly colored in the visualization below. The order of overlaying the colored map and contour map is specified by zorder.
draw_routine
def drawmap( values , levels , cmap , _save_filename , flag_show , name ,norm) :
fig,ax = plt.subplots(figsize=(5,5))
plt.subplots_adjust(left=0.02, right=0.98, top=0.98, bottom=0.02)
m = Basemap(projection='stere', llcrnrlat=25, urcrnrlat=47, llcrnrlon=125, urcrnrlon=150, lat_0=60, lon_0=140, resolution='i' )
m.drawparallels(np.arange(-80.,81.,10.))
m.drawmeridians(np.arange(-180.,181.,10.))
m.drawcoastlines()
m.drawmapboundary(fill_color='#eeeeee' , zorder=-1)
m.fillcontinents(color='#ceceee' , zorder=-1)
x , y = m(lons, lats)
m.contourf( x , y , values , levels , cmap=cmap , zorder=0 , norm=norm )
fig.savefig(_save_filename)
if flag_show == 'yes' :
plt.show()
plt.close()
You can create an image like the one shown below (UTC data at 15:00 on September 8, 2019).
I described how to convert a file that is GRIB2 format but cannot be handled by pygrib to netCDF format so that it can be handled from python.
Recommended Posts