Easily write JSON and Python dataclass conversions with quicktype and dacite
background
I think there are many situations where JSON is parsed when linking API with an external service in Python. Perth is boring and you don't want to spend too much time in situations like hackathons where speed is especially required. This time, I will show you how to easily convert JSON to Python dataclass.
The code introduced in this article is available on GitHub. https://github.com/gaiax/quicktype-dacite-demo
Library to use
As the title suggests, we mainly use the libraries called quicktype and dacite.
quicktype A library that infers the type of sample data such as JSON and outputs it in the corresponding language. https://github.com/quicktype/quicktype
It's published as an npm package, but you can also easily use it as a web UI. Even if it is a Web UI, the process is completed on the client without going through the server. [^ 1] It is available from https://app.quicktype.io/.
dacite A library that converts Python dicts to dataclass. https://github.com/konradhalas/dacite
Normally, this should be the case when converting from dict to dataclass.
from dataclasses import dataclass
@dataclass
class Data:
hoge: int
fuga: str
d = {"hoge": 10, "fuga": "Python"}
Data(**d)
# Data(hoge=10, fuga='Python')
This works fine, but it doesn't work well with nested dictionaries.
from dataclasses import dataclass
@dataclass
class NestedData:
foo: int
bar: str
@dataclass
class Data:
hoge: int
fuga: str
nested_data: NestedData
d = {
"hoge": 10,
"fuga": "Python",
"nested_data": {
"foo": 20,
"bar": "Ruby"
}
}
Data(**d)
# Data(hoge=10, fuga='Python', nested_data={'foo': 20, 'bar': 'Ruby'})
# nested_data is not a NestedData class, it just stores a dictionary
dacite allows for such usually unsupported nested dictionary conversions.
from dataclasses import dataclass
from dacite import from_dict
@dataclass
class NestedData:
foo: int
bar: str
@dataclass
class Data:
hoge: int
fuga: str
nested_data: NestedData
d = {
"hoge": 10,
"fuga": "Python",
"nested_data": {
"foo": 20,
"bar": "Ruby"
}
}
from_dict(data_class=Data, data=d)
# Data(hoge=10, fuga='Python', nested_data=NestedData(foo=20, bar='Ruby'))
# nested_An instance of the NestedData class is stored in data!
It makes it very easy to convert dictionaries and dataclasses. In addition, at the time of writing the article (2020/06/10), there are the following functions.
--Type check --Optional, Union compatible --List support --Hook settings for each type --Cast
- strict mode
By the way, the conversion from dataclass to dictionary is possible by passing an instance of dataclass to dataclasses.asdict in the Python standard library.
Actually convert
As some of you may have already come up with an implementation image from the explanation so far, let's actually try converting JSON and dataclass from the definition of dataclass. Here, as an example, let's convert the return value of the Event API of connpass to dataclass.
JSON data preparation
We are a group called Gaiax Technical Meetups of connpass, which holds introductory hands-on and social gatherings for engineers. ~~ Sudden promotion ~~
This time, the API will focus on that group and prepare the data.
To make the article easier to read, we will limit the number of events to two.
GET https://connpass.com/api/v1/event/?series_id=3109&count=2
connpass.json
{
"results_start": 1,
"results_returned": 2,
"results_available": 35,
"events": [
{
"event_id": 175102,
"title": "[Online dialogue] Google Apps Script utilization talk#6",
"catch": "Online dialogue with the latest information and use cases of Google Apps Script",
"description": "<h1>GAS book crowdfunding target amount achievement campaign!</h1>\n<p>Mr. Takahashi of Plan Notes, the author of "Complete Introduction to Google Apps Script" who succeeded in funding "I want to deliver the latest" Complete Introduction to Google Apps Script "to the world as soon as possible!"<br>\n<a href=\"https://camp-fire.jp/projects/view/249472\" rel=\"nofollow\">https://camp-fire.jp/projects/view/249472</a></p>\n<p>To celebrate the achievement of the target amount, we will present the first edition of "Google Apps Script Complete Introduction" to 5 people by lottery from the event participants as a campaign limited to this event! !! !!</p>\n<h1>detail</h1>\n<p>The 6th edition of "Google Apps Script Utilization Meetup", which has become a popular and standard item in Technical Meetup, will be held while changing the format!</p>\n<p>Google's G Suite (Google Apps) is beginning to be used by many companies. Introducing the latest information and use cases of Google Apps Script, a script language that extends G Suite!</p>\n<p>IT companies are migrating from office software that is installed directly on their PCs to G Suite, which is web application-type office software. Multiple people can edit at the same time, files are automatically saved in the cloud, and there are many functions to support the work style required today.</p>\n<p>Google Apps Script extends the functionality of G Suite. Since it is based on the JavaScript language that many people are familiar with, its history is short, but its use cases are increasing rapidly.</p>\n<p>※reference</p>\n<p>Please see the URL below for past events!</p>\n<p><a href=\"http://gaiax.hatenablog.com/archive\" rel=\"nofollow\">http://gaiax.hatenablog.com/archive</a></p>\n<h1>timetable</h1>\n<ul>\n<li>19:15 Delivery start</li>\n<li>19:30 talk start</li>\n<li>20:30 Q & A</li>\n<li>20:50 questionnaire</li>\n<li>21:00 close</li>\n</ul>\n<h1>Scheduled talk</h1>\n<h3>Mr. Nobunari Takahashi, Representative Director of Plan Notes Co., Ltd.</h3>\n<p>After graduating from the University of Electro-Communications Graduate School of Electronic Informatics, he worked as a saxophone player until he was 30 years old, and experienced as a producer and marketer in the mobile content industry and e-book industry. Feeling issues in the working style, productivity, IT utilization, etc. of businessmen in Japan, he started his own business in 2015. Excel,VBA,GSuite,GAS,Worked on system tool development, consulting, training, writing, etc. using the cloud.\n Online learning service "LinkedIn Learning" trainer,Presided over the community "Skill Up Study Group for Non-Programmers".\n Blog run by himself<a href=\"https://tonari-it.com/\" rel=\"nofollow\">"Always an IT job next door"</a>Boasts a monthly popularity of 960,000 PV.</p>\n<h3>GaiaX Co., Ltd. Social Media Solutions Division Shogo Matsushita</h3>\n<p>Born in 1995 Born in Kanagawa prefecture. RPG Maker and Click in junior high school&Meet Create and hit the gates of programming through game programming. After that, he entered Yokohama Medical Information College. While attending school, engaged in Web service development as an intern at various places. Joined GaiaX in 2018. Currently, in addition to development and video production, he writes technical doujinshi and doujin video production in the social media marketing division.</p>\n<p>Recently published a douujinshi about AppMaker</p>\n<ul>\n<li>Booth <a href=\"https://godan.booth.pm/items/2035726\" rel=\"nofollow\">https://godan.booth.pm/items/2035726</a></li>\n<li>bookwalker <a href=\"https://bookwalker.jp/de809d7a5b-1b32-4973-a3d6-61019cf222f2/\" rel=\"nofollow\">https://bookwalker.jp/de809d7a5b-1b32-4973-a3d6-61019cf222f2/</a></li>\n</ul>\n<h1>Notes</h1>\n<p>The purpose of this event is to share the know-how of engineers, and we do not accept participation for the purpose of eating and drinking only. Therefore, we ask you to exchange business cards at the reception to confirm your identity. Thank you for your cooperation.</p>\n<p>Regarding recruitment LT, we do not accept talks that include LT that is not related to the theme and excessive promotion that does not match the theme of this event (excluding Google's G Suite).</p>\n<p>At the conference for all participants who will be on stage or attending<a href=\"http://ja.confcodeofconduct.com/\" rel=\"nofollow\">Code of conduct(Conference Code of Conduct) </a>Please comply with. Please note that if the event management side determines that it is a serious act of harassment, we will take any action, including dismissal from the event.</p>",
"event_url": "https://gaiax.connpass.com/event/175102/",
"started_at": "2020-05-29T19:30:00+09:00",
"ended_at": "2020-05-29T22:00:00+09:00",
"limit": null,
"hash_tag": "GAS activity",
"event_type": "participation",
"accepted": 45,
"waiting": 0,
"updated_at": "2020-05-08T14:09:19+09:00",
"owner_id": 28874,
"owner_nickname": "xtetsuji",
"owner_display_name": "OGATA Tetsuji",
"place": "online",
"address": "online",
"lat": null,
"lon": null,
"series": {
"id": 3109,
"title": "Gaiax Technical Meetups",
"url": "https://gaiax.connpass.com/"
}
},
{
"event_id": 173835,
"title": "[For beginners] Introductory Flutter Online Hands-on",
"catch": "Hands-on event for those who want to touch Flutter",
"description": "<h1>About holding online</h1>\n<p>This time, it will be held online. On the day of the event, we will email you the online participation URL.</p>\n<h1>detail</h1>\n<p>iPhone/Flutter allows you to develop Andriod apps in one environment. Although the number of cases has increased little by little, I think there are many people who have not touched on it yet.</p>\n<p>Therefore, engineers who are actually developing and releasing native applications using Flutter will explain how to create simple applications for beginners hands-on, focusing on live coding!</p>\n<p>※reference</p>\n<p>Please see the URL below for past events!</p>\n<p><a href=\"http://gaiax.hatenablog.com/archive\" rel=\"nofollow\">http://gaiax.hatenablog.com/archive</a></p>\n<h1>Please join us if you like this</h1>\n<ul>\n<li>I'm interested in knowing Flutter (I've heard of it)</li>\n<li>iPhone/Interested in Android development</li>\n<li>Swift /I have development experience with kotlin and want to know Flutter</li>\n<li>Have some development experience such as WEB service</li>\n</ul>\n<h1>timetable</h1>\n<ul>\n<li>19:15 reception</li>\n<li>19:30 hands-on start</li>\n<li>21:30 Questionnaire</li>\n</ul>\n<h1>What we want participants to prepare</h1>\n<ul>\n<li>Those who want to move their hands together<ul>\n<li>Please prepare the environment by referring to the following Qiita article etc.</li>\n<li><a href=\"https://qiita.com/tomy0610/items/896dc8ec9ba95c33194f\" rel=\"nofollow\">https://qiita.com/tomy0610/items/896dc8ec9ba95c33194f</a></li>\n</ul>\n</li>\n</ul>\n<h1>Required experience / skill set</h1>\n<p>It is intended for people who have some development experience.\n There is no explanation of the construction of the development environment, programming language or the basic mechanism.</p>\n<h1>Notes</h1>\n<p>The purpose of this event is to share the know-how of engineers, and we do not accept talks that include LT that is not related to the theme or excessive promotion that does not match the theme of this event.</p>\n<p>Please note that we will use it as a material such as photos and videos of the event scenery in order to publish the event report on blogs and various media.</p>\n<p>For all participants who will be on stage or attending the conference<a href=\"https://ja.confcodeofconduct.com/\" rel=\"nofollow\">Code of conduct(Conference Code of Conduct)</a>Please follow the. Please note that if the event management side determines that it is a serious act of harassment, we will take any action, including dismissal from the event.</p>",
"event_url": "https://gaiax.connpass.com/event/173835/",
"started_at": "2020-05-07T19:30:00+09:00",
"ended_at": "2020-05-07T22:00:00+09:00",
"limit": null,
"hash_tag": "Flutter hands-on",
"event_type": "participation",
"accepted": 106,
"waiting": 0,
"updated_at": "2020-05-07T21:44:47+09:00",
"owner_id": 11134,
"owner_nickname": "norinux",
"owner_display_name": "norinux",
"place": "online",
"address": "online",
"lat": null,
"lon": null,
"series": {
"id": 3109,
"title": "Gaiax Technical Meetups",
"url": "https://gaiax.connpass.com/"
}
}
]
}
Define dataclass with quicktype
This time I want to try it easily, so I will define a dataclass in the Web UI. Go to https://app.quicktype.io/ and paste the JSON data you prepared earlier into the input field on the left.
If you select Python and check Classes only, the conversion result should be displayed automatically on the right side.
Also check Transform property names to be Pythonic for later conversion with dacite.
By the way, if you turn off Classes only, it will also define theto_dict ()method and so on. However, it is necessary to rewrite each method every time a column change occurs.
By leaving those conversion processes to dacite, you can ease the coding burden.
Also, quicktype is well done and will add ʻOptional to properties that may or may not be included in the array. Try removing the ʻevent_id column of one of the two events in the JSON on quicktype.
By default, the class name of the generated dataclass is Welcome, so change it accordingly.
In this article, we will call it Connpass.
Finally, copy the converted dataclass in its entirety and save it as a file.
Here, save it in connpass.py according to the class name of dataclass.
Convert to dataclass with dacite
Now that we have defined the dataclass, let's use dacite to convert from JSON to dataclass.
Assuming that the original JSON is saved in a file as connpass.json, the code will be as follows.
main.py
import json
from dataclasses import dataclass
from datetime import datetime
from dacite import Config, from_dict
from connpass import Connpass
def run():
with open("connpass.json", "r") as f:
data = json.load(f)
connpass = from_dict(Connpass, data, Config({datetime: datetime.fromisoformat}))
print(connpass.events)
if __name__ == "__main__":
run()
The actual conversion is done on this line.
connpass = from_dict(Connpass, data, Config({datetime: datetime.fromisoformat}))
Config is doing type conversion using dacite's cast function.
I'm passing datetime.fromisoformat through a property whose type is datetime.
If you don't write this, you will pass str to the property of datetime and dacite.exceptions.WrongTypeError will be thrown.
connpass = from_dict(Connpass, data)
# dacite.exceptions.WrongTypeError: wrong value type for field "events.started_at" - should be "datetime" instead of value "2020-05-29T19:30:00+09:00" of type "str"
Convert with __post_init__ ()
By the way, this time the cast was enough, but if the conversion process is different depending on the property even though it is the same type, you can accept it with Union and manually convert it with __post_init__ ().
post_init_example.py
# {"text": "hello world", "created_at": "Thu Jun 04 11:27:06 +0900 2020", "timestamp": 1591237626}
from typing import Union
from datetime import datetime
@dataclass
class Sample:
text: str
created_at: Union[str, datetime]
timestamp: Union[int, datetime]
def __post_init__(self):
self.__convert_created_at()
self.__convert_timestamp()
def __convert_created_at(self):
if type(self.created_at) is str:
self.created_at =datetime.strptime(self.created_at, "%a %b %d %H:%M:%S %z %Y")
def __convert_timestamp(self):
if type(self.timestamp) is int:
self.timestamp = datetime.fromtimestamp(self.timestamp)
Refactor a little
It's painful to call connpass = from_dict (Connpass, data, Config ({datetime: datetime.fromisoformat})) every time, so we move the conversion process to the Connpass class.
connpass.py
from dataclasses import asdict, dataclass
from datetime import datetime
from typing import Any, Dict, List
from dacite import Config, from_dict
#abridgement
@dataclass
class Connpass:
results_start: int
results_returned: int
results_available: int
events: List[Event]
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "Connpass":
return from_dict(cls, data, config=Config({datetime: datetime.fromisoformat}))
def to_dict(self) -> Dict[str, Any]:
return asdict(self)
Then you can call it like this. It's easier to use.
connpass = Connpass.from_dict(data)
exported_data = connpass.to_dict()
Summary
So far, I have introduced how to easily implement JSON and dataclass conversion using dacite and quicktype.
The repository published on GitHub also includes an example of setting up a simple server and checking its operation. https://github.com/gaiax/quicktype-dacite-demo
Complementation will also work, so the development experience has improved significantly compared to playing with a plain dictionary. I'm happy.
Recommended Posts