Let's stop worrying about logging anymore !!

1. History

Please stop printing and import logging for log output --Qiita

I was reading while thinking, "Well, I see." But at the same time, I thought like this.

** So how do you write logging? ** ** ** Even for free, I'm not good at logging in Python and it takes time, but I can't do logger !! **

So I decided to write this article. The purpose is as follows.

――I want to share my own understanding that I found as a result of desperately facing logger --Don't waste your time worrying about logging when everyone makes apps in Python

2. After all, why shouldn't I use logging! ??

Official docs and the view from reading the Qiita article above.

logging has a hierarchical structure.

└── rootLogger
    ├── scanLogger
    │    ├── scanLogger.txt
    │    ├── scanLogger.html
    │    └── scanLogger.pdf
    └── anotherLogger

It has (or is created) multiple child loggers with a logger called rootLogger as the parent.

For example ... --Module called ʻurllib uses logger called ʻurllib --ʻApscheduler.scheduler module uses ʻapscheduler.scheduler logger

The logger for each of these packages and apps is called rootLogger.

bad_sample.py


import logging

logging.info("Taking log...")

This is synonymous with playing with rootLogger directly, so let's stop. It's similar to the concept of playing with an instance instead of playing with a class.

3. Logging sample

Even if you look at the official document or the sample, you do not know how to write it ... sweat How much fixed wording and how much can I change by myself? (Range that you can name yourself)

That's why I will present the configuration whose operation has been confirmed. If you have a problem, please copy and use it.

As much as possible, I will include a comment in the comments so that you can easily customize it.

3-1. (Pattern.1) Use Config file

3-1-1. Execution environment

** I am writing the logging config file in toml. ** ** You can refer to the toml package with pip install toml.

The notation method in yaml is here.

terminal


$ tree
.
├── app
│   ├── logger
│   │   ├── __init__.py
│   │   └── logger.py
│   ├── logging.config.toml
│   └── main.py
└── log
    ├── application.log
    └── error.log

$ cd app/
$ python app.py  # app/I am writing a sample assuming that this command is executed in the directory

3-1-2. File

Also put the Github sample code.

main.py


#The previous logger is a directory. The logger at the back is a file.
from logger.logger import get_logger

logger = get_logger()

def main():
    logger.info("start = main()")
    # do something 
    logger.error("oops! something wrong!!")
    # do something 
    logger.info("end = main()")

if __name__ == "__main__":
    main()

logging.config.toml is set to output logs to the console screen and file.

toml:logging.config.toml


version = 1
disable_existing_loggers = false  #Do not disable loggers on other modules

[formatters]
    [formatters.basic]
    #Set the log format. Details of how to write will be described later
    format = "%(asctime)s  [%(name)s][%(levelname)s]  %(message)s  (%(filename)s:%(module)s:%(funcName)s:%(lineno)d)" 
    datefmt = "%Y-%m-%d %H:%M:%S"

[handlers]
    [handlers.console]
    #Set console output here
    class = "logging.StreamHandler"  #Fixed wording
    level = "DEBUG"                  #log level is your choice
    formatter = "basic"              #Select the log format listed in formatters
    stream = "ext://sys.stdout"      #Fixed wording

    [handlers.file]
    class = "logging.handlers.TimedRotatingFileHandler"  #Fixed wording. Details of TimeRotatingFileHandler will be described later. There are notes
    formatter = "basic"
    filename = "../log/application.log"                  #If specified by relative path$Write from a directory that runs python
    when = 'D'                                           #log A unit for switching files. D= day。
    interval = 1                                         #Since day is selected above, a new file is created every day.
    backupCount = 31                                     #Since day is selected above, the log file for 31 days is retained.

    [handlers.error]
    class = "logging.handlers.TimedRotatingFileHandler"
    level = "ERROR"
    formatter = "basic"
    filename = "../log/error.log"
    when = 'D'
    interval = 1
    backupCount = 31

[loggers]
    [loggers.app_name]  # app_name is the name of the logger called from python. Arbitrarily named
    level = "INFO"
    handlers = [
        "console",
        "file",
        "error",
    ]                   #Set which of the handlers set above to use
    propagate = false

[root]
level = "INFO"
handlers = [
    "console",
    "file",
    "error"
]                       #Setting of logger's parent root Logger. I also want to keep the rootLogger in console and file

logger/__init__.py


from . import logger

logger.init_logger('logging.config.toml', 'app_name')
#First argument: configfile =The name of the config file. Relative path or absolute path of the directory executed by the python command
#Second argument: loggername =The name of the logger set in the config file

logger/logger.py


import os
import toml
from logging import getLogger
from logging.config import dictConfig

CONFIGFILE = "logging.config.toml"    #Treat as a global variable. The default value is as shown on the left. Any
LOGGERNAME = "root"                   #Treat as a global variable. The default value is root. Any

def init_logger(configfile, loggername):
    global CONFIGFILE 
    global LOGGERNAME
    CONFIGFILE = configfile
    LOGGERNAME = loggername
    dictConfig(toml.load(CONFIGFILE))

def get_logger():
    return getLogger(LOGGERNAME)

3-2. (Pattern.2) Using Python dict

This is more flexible than logging in the Config file. In the Config file, the parent directory is ../, which makes it more OS-dependent. If you set it with python dict, you can specify it with ʻos.path.pardir`.

3-2-1. Execution environment

terminal


$ tree
.
├── app
│   ├── logger
│   │   ├── __init__.py
│   │   └── logger.py
│   └── main.py
└── log
    ├── application.log
    └── error.log

$ cd app/
$ python main.py

3-2-2. File

Also put the Github sample code. main.py is [main.py above](https://qiita.com/uANDi/items/9a2b980262bc43455f2e#3-1-2-%E3%83%95%E3%82%A1%E3%82 Same as% A4% E3% 83% AB). Also, the contents of CONFIG are the same as the above logging.config.toml.

logger/__init__.py


from . import logger

logger.init_logger()

logger/logger.py


import os
from logging import getLogger
from logging.config import dictConfig

APP_NAME = os.getenv('APP_NAME', default='app_name')
CONFIG = {
    'version': 1, 
    'disable_existing_loggers': False,
    'formatters': {
        'basic': {
            'format': '%(asctime)s  [%(name)s][%(levelname)s]  %(message)s  (%(module)s:%(filename)s:%(funcName)s:%(lineno)d)', 
            'datefmt': '%Y-%m-%d %H:%M:%S'
            }
        },
        'handlers': {
            'console': {
                'class': 'logging.StreamHandler', 
                'level': 'DEBUG', 
                'formatter': 'basic', 
                'stream': 'ext://sys.stdout'
            }, 
            'file': {
                'class': 'logging.handlers.TimedRotatingFileHandler', 
                'formatter': 'basic', 
                'filename': os.path.join(os.path.pardir, 'log', 'application.log'), 
                'when': 'D', 
                'interval': 1, 
                'backupCount': 31
            }, 
            'error': {
                'class': 'logging.handlers.TimedRotatingFileHandler', 
                'level': 'ERROR', 
                'formatter': 'basic', 
                'filename': os.path.join(os.path.pardir, 'log', 'error.log'), 
                'when': 'D', 
                'interval': 1, 
                'backupCount': 31
            }
        }, 
        'loggers': {
            APP_NAME: {
                'level': 'INFO', 
                'handlers': [
                    'console', 
                    'file', 
                    'error'
                ], 
                'propagate': False
            }
        }, 
        'root': {
            'level': 'INFO', 
            'handlers': [
                'console', 
                'file', 
                'error'
            ] 
        }
    }

def init_logger():
    dictConfig(CONFIG)

def get_logger():
    return getLogger(APP_NAME)

3-3. Execution result

The output format is the same for both Pattern 1. and Pattern 2. [app_name] is the logger name.

terminal


$ cd app/
$ python main.py
2019-12-21 15:43:28  [app_name][INFO]  start = main()  (main:main.py:main:8)
2019-12-21 15:43:28  [app_name][ERROR]  oops! something wrong!!  (main:main.py:main:10)
2019-12-21 15:43:28  [app_name][INFO]  end = main()  (main:main.py:main:12)

$ cat ../log/application.log
2019-12-21 15:43:28  [app_name][INFO]  start = main()  (main:main.py:main:8)
2019-12-21 15:43:28  [app_name][ERROR]  oops! something wrong!!  (main:main.py:main:10)
2019-12-21 15:43:28  [app_name][INFO]  end = main()  (main:main.py:main:12)

$ cat ../log/error.log
2019-12-21 15:43:28  [app_name][ERROR]  oops! something wrong!!  (main:main.py:main:10)

3-4. A little commentary

3-4-1. TimedRotatingFileHandler

What is set in ʻinterval and when` in Official Document TimedRotatingFileHandler Can you do it? "Is written!

Outputs log to a new file at the set time. In the above example, 'D', that is, the file is rewritten every day. For past log files, the date surfix is added to the file name, such as ʻapplication.log.2019-12-21`.

**Caution! ** ** TimedRotatingFileHandler will not work for applications where app.py does not start all day. It works for web apps like Django and Flask, but may not work properly with cron.

3-4-2. Format

The Official Documentation LogRecord Attribute (https://docs.python.org/en/3/library/logging.html#id2) has a list of each format. Let's arrange the items you need!

4. Summary (Takeaways)

I introduced two types of how to make logging! Let's look back on each! (If you think it's perfect, skip it !!)

By the way, Takeaways means "points, points". take away = Take away → Take this away with today's presentation! It is such a nuance.

4-1. Why ʻimport logging` is bad

A number of loggers have been created with rootLogger as the parent. An image like rootLogger → Class, logger → instance.

logging.info () will directly mess with the rootLogger, so let's create your own logger.

4-2. Sample file

Then how do you make it concretely?

--Read logging settings from Config file -Toml sample -Yaml sample --Write the settings in the Python dictionary -Sample

4-3. Precautions

--If you do not set disable_existing_loggers = False, the logger of gunicorn will be broken. --When creating a Config file --Note the OS dependency of the relative path --The relative path of the log folder is from the directory where you run $ python

5. Finally

I don't think this is the correct way to write logging. You can also add a handler to your python file with ʻaddHandler`. We would be grateful if you could teach us how to set up your recommended logging.

Recommended Posts

Let's stop worrying about logging anymore !!
Let's stop sudo!
Thinking about logging, then