The JSON file that receives the configuration file and is generated by jinja2 is used when deploying. I want to check if this generated JSON file is a valid JSON.
An image like j2cli.
Use the following two files.
--Configuration file (.json) --template file (.json.j2)
For example, when the configuration file is as follows
{
"name": "staging"
}
Use a template like this.
{
"app-name": "{{name}}-app",
"batch-name": "{{name}}-batch"
}
As a result, the following JSON is output. This is used as the actual configuration file.
{
"app-name": "staging-app",
"batch-name": "staging-batch"
}
In the case of dev-like settings, the output may be as follows.
{
"app-name": "dev-app",
"batch-name": "dev-batch"
}
The generated JSON may be invalid. I hope it will finally be noticed when it is deployed in production. I want to check in advance.
The policy is as follows.
It looks like this in python code.
import jinja2
import json
env = jinja2.Environment(loader=jinja2.FileSystemLoader(["."]), undefined=jinja2.StrictUndefined)
t = env.get_or_select_template(template_path)
json.loads(t.render(**config)) #Error if invalid
However, there are some caveats.
{"foo": "foo-{{name}}"}
If you emit with jinja2 without setting anything in such a template, the following will occur, but no error will occur.
{"foo": "foo-"}
This can be detected by changing the undefined setting of jinja2. If you add ʻundefined = jinja2.StrictUndefined, ʻUndefinedError
will occur.
It is bad if the template is flawed and contains too many "}". Although the generated JSON is valid as JSON. Not the expected output.
If you accidentally write a template like the one below.
{
"app-name": "{{name}}}-app"
}
The following is valid JSON, though. Not the expected output.
{
"app-name": "staging}-app"
}
For the time being, you should also check the value of dict.
Based on the above, I made a script that checks whether the result of emit with jinja2 is valid JSON. Use it like this. If the variable environ does not exist, an error will occur as shown below. An example of using this is an error when two settings named ʻenviron` do not exist in the configuration file.
$ python check.py --conf-dir conf/ --template-dir template/ || echo ng
template/back.json.j2(conf/master.json): UndefinedError 'environ' is undefined
template/back.json.j2(conf/production.json): UndefinedError 'environ' is undefined
template/back.json.j2(conf/rook.json): UndefinedError 'environ' is undefined
ng
The implementation is as follows.
import sys
import os.path
import argparse
import jinja2
import json
def validate_conf(d, path=None):
path = path or []
if hasattr(d, "items"):
for k, v in d.items():
path.append(k)
validate_conf(v, path=path)
path.pop()
elif isinstance(d, (list, tuple)):
for i, x in enumerate(d):
path.append(i)
validate_conf(x, path=path)
path.pop()
elif isinstance(d, str):
v = d
if "{" in v:
raise ValueError("invalid value: '{{' is included. path={}".format(path))
if "}" in v:
raise ValueError("invalid value: '}}' is included. path={}".format(path))
return d
def check(env, config, template_path):
t = env.get_or_select_template(template_path)
data = json.loads(t.render(**config))
return validate_conf(data)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--conf-dir", required=True)
parser.add_argument("--template-dir", required=True)
args = parser.parse_args()
env = jinja2.Environment(loader=jinja2.FileSystemLoader(["."]), undefined=jinja2.StrictUndefined)
status = 0
for root, _, conf_files in os.walk(args.conf_dir):
for conf_file in conf_files:
try:
confpath = os.path.join(root, conf_file)
with open(confpath) as rf:
conf = json.load(rf)
except Exception as e:
sys.stderr.write("{confpath}: {e.__class__.__name__} {e}\n".format(confpath=confpath, e=e))
status = 1
continue
for root2, _, template_files in os.walk(args.template_dir):
for template_file in template_files:
try:
filepath = os.path.join(root2, template_file)
check(env, conf, filepath)
except Exception as e:
sys.stderr.write("{filepath}({confpath}): {e.__class__.__name__} {e}\n".format(
confpath=confpath, filepath=filepath, e=e)
)
status = 1
sys.exit(status)
if __name__ == "__main__":
main()
Recommended Posts