I want to receive the configuration file and check if the JSON file generated by jinja2 is a valid JSON

The JSON file that receives the configuration file and is generated by jinja2 is used when deploying. I want to check if this generated JSON file is a valid JSON.

Command used

An image like j2cli.

Use the following two files.

--Configuration file (.json) --template file (.json.j2)

For example, when the configuration file is as follows

{
  "name": "staging"
}

Use a template like this.

{
  "app-name": "{{name}}-app",
  "batch-name": "{{name}}-batch"
}

As a result, the following JSON is output. This is used as the actual configuration file.

{
  "app-name": "staging-app",
  "batch-name": "staging-batch"
}

In the case of dev-like settings, the output may be as follows.

{
  "app-name": "dev-app",
  "batch-name": "dev-batch"
}

problem

The generated JSON may be invalid. I hope it will finally be noticed when it is deployed in production. I want to check in advance.

things to do

The policy is as follows.

  1. Emit template with jinja2
  2. Json.loads the emitted JSON again
  3. If an error occurs, it should be invalid JSON

It looks like this in python code.

import jinja2
import json


env = jinja2.Environment(loader=jinja2.FileSystemLoader(["."]), undefined=jinja2.StrictUndefined)
t = env.get_or_select_template(template_path)
json.loads(t.render(**config))  #Error if invalid

However, there are some caveats.

Note 1 If the expected value is not included in the configuration file

{"foo": "foo-{{name}}"}

If you emit with jinja2 without setting anything in such a template, the following will occur, but no error will occur.

{"foo": "foo-"}

This can be detected by changing the undefined setting of jinja2. If you add ʻundefined = jinja2.StrictUndefined, ʻUndefinedError will occur.

Note 2 If the template contains an excessive amount of "}"

It is bad if the template is flawed and contains too many "}". Although the generated JSON is valid as JSON. Not the expected output.

If you accidentally write a template like the one below.

{
  "app-name": "{{name}}}-app"
}

The following is valid JSON, though. Not the expected output.

{
  "app-name": "staging}-app"
}

For the time being, you should also check the value of dict.

A script that checks if the result of emit with jinja2 is valid JSON

Based on the above, I made a script that checks whether the result of emit with jinja2 is valid JSON. Use it like this. If the variable environ does not exist, an error will occur as shown below. An example of using this is an error when two settings named ʻenviron` do not exist in the configuration file.

$ python check.py --conf-dir conf/ --template-dir template/ || echo ng
template/back.json.j2(conf/master.json): UndefinedError 'environ' is undefined
template/back.json.j2(conf/production.json): UndefinedError 'environ' is undefined
template/back.json.j2(conf/rook.json): UndefinedError 'environ' is undefined
ng

The implementation is as follows.

import sys
import os.path
import argparse
import jinja2
import json


def validate_conf(d, path=None):
    path = path or []
    if hasattr(d, "items"):
        for k, v in d.items():
            path.append(k)
            validate_conf(v, path=path)
            path.pop()
    elif isinstance(d, (list, tuple)):
        for i, x in enumerate(d):
            path.append(i)
            validate_conf(x, path=path)
            path.pop()
    elif isinstance(d, str):
        v = d
        if "{" in v:
            raise ValueError("invalid value: '{{' is included. path={}".format(path))
        if "}" in v:
            raise ValueError("invalid value: '}}' is included. path={}".format(path))
    return d


def check(env, config, template_path):
    t = env.get_or_select_template(template_path)
    data = json.loads(t.render(**config))
    return validate_conf(data)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--conf-dir", required=True)
    parser.add_argument("--template-dir", required=True)
    args = parser.parse_args()

    env = jinja2.Environment(loader=jinja2.FileSystemLoader(["."]), undefined=jinja2.StrictUndefined)
    status = 0
    for root, _, conf_files in os.walk(args.conf_dir):
        for conf_file in conf_files:
            try:
                confpath = os.path.join(root, conf_file)
                with open(confpath) as rf:
                    conf = json.load(rf)
            except Exception as e:
                sys.stderr.write("{confpath}: {e.__class__.__name__} {e}\n".format(confpath=confpath, e=e))
                status = 1
                continue

            for root2, _, template_files in os.walk(args.template_dir):
                for template_file in template_files:
                    try:
                        filepath = os.path.join(root2, template_file)
                        check(env, conf, filepath)
                    except Exception as e:
                        sys.stderr.write("{filepath}({confpath}): {e.__class__.__name__} {e}\n".format(
                            confpath=confpath, filepath=filepath, e=e)
                        )
                        status = 1
    sys.exit(status)


if __name__ == "__main__":
    main()

Recommended Posts

I want to receive the configuration file and check if the JSON file generated by jinja2 is a valid JSON
I created a script to check if English is entered in the specified position of the JSON file in Python.
I want to hack Robomaster S1 ① Rooting and file configuration check
I want to make a music player and file music at the same time
I want to write an element to a file with numpy and check it.
Create a python script to check if the link at the specified URL is valid 2
Create a python script to check if the link at the specified URL is valid
[Django] Test to send a file by POST and check the returned context [TDD]
I want to initialize if the value is empty (python)
I want to record the execution time and keep a log.
[Blender] Script to check if the selected one is a mesh
Check if the configuration file is read in an easy-to-understand manner
What to do if you cat or tail a binary file and the terminal is garbled
I made a function to check if the webhook is received in Lambda for the time being
I want to extract the tag information (title and artist) of a music file (flac, wav).
I want to identify the alert email. --Is that x a wildcard? ---
I made a program to check the size of a file in Python
Check if there is a specific symbol in the executable file and its dependent libraries (simplified version)
I want to get the path of the directory where the running file is stored.
Anyway, I want to check JSON data easily
I want to drop a file on tkinter and get its path [Tkinter DnD2]
I want to add silence to the beginning of a wav file for 1 second
I made a tool to generate Markdown from the exported Scrapbox JSON file
I want to get the file name, line number, and function name in Python 3.4
I want to write to a file with Python
I want to replace the variables in the python template file and mass-produce it in another file.
How to check in Python if one of the elements of a list is in another list
I want to see the file name from DataLoader
Check if the string is a number in python
I want to randomly sample a file in Python
I want to win if there is the most useless visualization grand prix in the world ・ Learn visualization by evolving the OP function
I want to find the intersection of a Bezier curve and a straight line (Bezier Clipping method)
I want to create a karaoke sound source by separating instruments and vocals using Python
I want to save the photos sent by LINE to S3
How to switch the configuration file to be read by Python
I made a function to check the model of DCGAN
I want to know the features of Python and pip
I want to map the EDINET code and securities number
I want Sphinx to be convenient and used by everyone
I want to create a Dockerfile for the time being.
I want to get information from fstab at the ssh connection destination and execute a command
I want to create a histogram and overlay the normal distribution curve on it. matplotlib edition
I want to clear up the question of the "__init__" method and the "self" argument of a Python class.
I want to specify a file that is not a character string for logrotate, but is it impossible?
Check if the password hash generated by PHP matches in Python
How to make a command to read the configuration file with pyramid
LINEbot development, I want to check the operation in the local environment
I want to create a system to prevent forgetting to tighten the key 1
Scraping and tabelog ~ I want to find a good restaurant! ~ (Work)
I want to create a pipfile and reflect it in docker
I want to connect remotely to another computer, and the nautilus command
I want to check the position of my face with OpenCV!
[Golang] Check if a specific character string is included in the character string
I want to be healed by Mia Nanasawa's image. In such a case, hit the Twitter API ♪
A super introduction to Django by Python beginners! Part 3 I tried using the template file inheritance function
I want to revive the legendary Nintendo combination by making full use of AI and HR Tech!
I want to use complicated four arithmetic operations in the IF statement of the Django template! → Use a custom template
[Golang] I want to add omitempty to the json tag of the int type field of the structure so that it will be ignored if 0 is entered.
I want to change the color by clicking the scatter point in matplotlib
[Python Kivy] How to get the file path by dragging and dropping
I tried moving the image to the specified folder by right-clicking and left-clicking