[Python3] A story stuck with time zone conversion

I was writing a little tool to receive information including time from an external API and notify Slack, but I couldn't convert the timezone consistently and stably, which meant. Make a note of what you learned during the survey and the possible solutions.

Summary

--Use pytz.timezone.localize (datetime) when using pytz for timezone conversion. --It seems better not to use the datetime.replace that comes with datetime --There are two types of datetime: Timezeon aware / Offest native --Offset native is generated when datetime is generated in Python without being aware of it. --If an external package returns a timezone-aware datetime and converts it back to JST, generate offset native once via timestamp and then convert it using pytz's localize. -The article of pytz specifications have changed | Qiita @higitune was super helpful including the comment section, so it is also in the time zone. Those who are stumbling in handling should also see this ――Unless there is a special reason --Use timestamp as consistently as possible in the logic layer --Be sure to convert as needed when putting it out to the UI layer

The complexity of timezone processing in Python x pytz

In my case, there were several types of stumbling blocks.

  1. Timezeon aware / Offest native
  2. Pytz rigor
  3. pytz specifications

Finally, I will show you what I researched about these and what kind of code I wrote to overcome them.

Timezeon aware / Offest native

There are the above two types of datetime.

Timezone aware is a datetime with an explicitly specified timezone, and Offset native is a time without a specified timezone. The two cannot be compared.

These need to be distinguished.

import os
from datetime import datetime
import pytz

# Offest native
d1 = datetime.utcnow()                                                                                                        
print(d1)
>>> 2020-04-19 10:07:47.885883

# Timezone aware
d2 = pytz.utc.localize(d1)
print(d2)
>>> 2020-04-19 10:07:47.885883+00:00

#The two cannot be compared
d1 < d2
# >>> TypeError: can't compare offset-naive and offset-aware datetimes

Strictness of pytz

The time zone seems to be determined by the location and ** time **. There seems to be a case where the definition of time difference is changed due to historical circumstances, and our Japan also corresponds to that example.

It seems that the time difference is different between Tokyo until 1888 and after that, and before 1888 it seems to be ** + 09: 19 ** off from UTC. So, pytz takes that strict consideration, and it seems that this is taken into consideration in some methods of the time zone conversion system (datetime.replace, etc.).

import pytz

#Here DISPLAY_TIMEZONE='Asia/Tokyo'Suppose
DISPLAY_TIMEZONE = os.environ.get('DISPLAY_TIMEZONE') 
tz = pytz.timezone(DISPLAY_TIMEZONE)

tz                                                                                                                                              
>>> <DstTzInfo 'Asia/Tokyo' LMT+9:19:00 STD>

It's 19 minutes off. However, this seems to be correct as a result of proper compliance.

If your program deals with ** Offset native **, you can use timezone.localize () provided by the pytz timezone object. It will be converted properly at +09: 00.

** As long as you use pytz's localize method, the timezone conversion will happen at +09: 00 as you intended, so you don't have to worry about this shift. ** **

tz.localize(d1)                                                                                                                                 
>>> datetime.datetime(2020, 4, 19, 10, 20, 3, 201190, tzinfo=<DstTzInfo 'Asia/Tokyo' JST+9:00:00 STD>)

pytz spec

pytz's localize () does not accept Timezone aware timezones. This point needs attention.

# localize offset ative
tz.localize(d1)
>>> datetime.datetime(2020, 4, 19, 10, 7, 47, 885883, tzinfo=<DstTzInfo 'Asia/Tokyo' JST+9:00:00 STD>)

# localize timezone aware (error)
tz.localize(d2)
# >>> ValueError: Not naive datetime (tzinfo is already set)

Therefore, when dealing with ** timezone aware time, localize cannot be used as it is, and it is necessary to convert it to offset native once. ** This is often encountered when the time returned, such as using a wrapper for an external API, is timezone aware.

The content written here mentions pytz specifications have changed | Qiita @ higitune, so you should take a look there. .. It will be very helpful including the comment section.

Convert a timezone aware datetime type to another timezone

When using a package developed by someone other than yourself, it is implementer-dependent whether the time returned by the package module is timezone aware or offset native. In order to combine them and implement your own development requirements, you will have to manage a mixture of these.

Therefore, it is a good idea to stamp it once (although it is on Theory Street). The code below can JST convert both offset native / timezone aware. (It is unconfirmed whether similar results can be obtained in other countries)

import pytz
from datetime import datetime

DISPLAY_TIMEZONE = 'Asia/Tokyo'
tz = pytz.timezone(DISPLAY_TIMEZONE)

def localized_datetime(date: datetime):
    return datetime.fromtimestamp(date.timestamp(), tz=tz)


if __name__ == '__main__':
    # d1: native datetime
    d1 = datetime.now()
    print(f"d1: {d1}")

    # d2: utc localize
    d2 = tz.localize(d1)
    print(f"d2: {d2}")

    print(localized_datetime(d1))
    print(localized_datetime(d2))

>>> d1: 2020-04-19 20:15:30.974272
>>> d2: 2020-04-19 20:15:30.974272+09:00
>>> 2020-04-19 20:15:30.974272+09:00
>>> 2020-04-19 20:15:30.974272+09:00

Summary

The time zone is confusing ...

The above code is posted as a reference implementation, but it seems that the actual localized time is required mainly in the presentation layer (that is, appearance), so in the logic layer and persistence layer as much as possible with datetime I think it would be better to use timestamp instead.

However, datetime is overwhelmingly more convenient when writing code, and I think there are many cases where date operations need to be performed frequently in the logic layer.

In such a case, I think it is better to unify which of timezone awre / offset native is used, at least in the code closer to the logic layer written by myself. If the time handled by the external package is inconsistent, you can create your own layer that thinly wraps the interface you want to use, and absorb the inconsistency in the datetime type and time zone. , I think the main logic will be cleaner.

I can't decide which one to send to based on my current knowledge, so I'd appreciate it if you could give us your opinion in the comments section.

There are also subtle pitfalls in the above code. For example

"The datetime type returned by the function / method is offset native, but the time is based on UTC."

In such a case. If the API of the external service is designed to return UTC consistently, and the external package implementation that wraps it is not cool and does not generate datetime considering the time zone ... I think it will be. .. .. In that case, let's deal with it individually.

Recommended Posts

[Python3] A story stuck with time zone conversion
A story stuck with handling Python binary data
A story about a python beginner stuck with No module named'http.server'
A story about making 3D space recognition with Python
A story about making Hanon-like sheet music with Python
Problems when creating a csv-json conversion tool with python
A story about trying a (Golang +) Python monorepo with Bazel
Stumble story with Python array
Make a fortune with Python
Execution time measurement with Python With
Create a directory with python
A little stuck with chainer
Time synchronization (Windows) with Python
Determine if a string is a time with a python regular expression
I made a package to filter time series with python
A story about an amateur making a breakout with python (kivy) ②
A story about an amateur making a breakout with python (kivy) ①
[Python] What is a with statement?
Solve ABC163 A ~ C with Python
Operate a receipt printer with python
A python graphing manual with Matplotlib.
Let's make a GUI with python.
Solve ABC166 A ~ D with Python
Create a virtual environment with Python!
I made a fortune with Python.
MP3 to WAV conversion with Python
Building a virtual environment with Python 3
Solve ABC168 A ~ C with Python
Make a recommender system with python
[Small story] Get timestamp with Python
[Python] Generate a password with Slackbot
Solve ABC162 A ~ C with Python
Solve ABC167 A ~ C with Python
Solve ABC158 A ~ C with Python
Let's make a graph with python! !!
[Python] Inherit a class with class variables
I made a daemon with Python
Write a batch script with Python3.5 ~
[Python, Selenium, PhantomJS] A story when scraping a website with lazy load
A story about adding a REST API to a daemon made with Python
The story of making a standard driver for db with python.
A story about developing a soft type with Firestore + Python + OpenAPI + Typescript
The story of making a module that skips mail with python
[Pyenv] Building a python environment with ubuntu 16.04
Spiral book in Python! Python with a spiral book! (Chapter 14 ~)
A note where a Python beginner got stuck
Creating a simple PowerPoint file with Python
[Python] A program that creates stairs with #
Building a Python3 environment with Amazon Linux2
The story of making a university 100 yen breakfast LINE bot with Python
Let's make a shiritori game with Python
Install Python as a Framework with pyenv
Add a Python data source with Redash
Create a dummy image with Python + PIL.
Algorithm learned with Python 7th: Year conversion
I made a character counter with Python
The story of having a hard time introducing OpenCV with M1 MAC
[Python] Drawing a swirl pattern with turtle
I drew a heatmap with seaborn [Python]
[Python] Create a virtual environment with Anaconda
Let's create a free group with Python