In omegaconf, let's pass the direct parameter file to the function

Purpose

--I created a decorator that reads parameters from a file and adds them to arguments, so I want to share how to use them. --The motivation is to make it easier to read parameters for models such as DL. --Because the code that links the arguments from the main function is troublesome ... --omegaconf It's convenient, so I want people who don't know to know it. ――If you have any other useful things, please let me know!

environment

Code sample (summary)

--Preparation --Install omegaconf

%%bash
pip install omegaconf

--Preparing the decorator

import functools
from omegaconf import OmegaConf


def add_args(params_file: str, as_default: bool = False) -> callable:
    @functools.wraps(add_args)
    def _decorator(f: callable) -> callable:
        @functools.wraps(f)
        def _wrapper(*args, **kwargs) -> None:
            cfg_params = OmegaConf.load(params_file)
            if as_default:
                cfg_params.update(kwargs)
                kwargs = cfg_params
            else:
                kwargs.update(cfg_params)
            return f(*args, **kwargs)

        return _wrapper

    return _decorator

--Prepare a parameter file to read (yaml or json) --omegaconf supports yaml, json

%%bash
cat <<__YML__ > params.yml
n_encoder_layer: 3
n_decoder_layer: 5
n_heads: 4
n_embedding: 16
__YML__
:
echo "===== [ params.yml ] ====="
cat params.yml
echo "====="

--Call

@add_args("params.yml")
def use_params(a, b, n_encoder_layer, n_decoder_layer, n_heads, n_embedding):
    assert a == 0.25
    assert b == "world"
    assert n_encoder_layer == 3
    assert n_decoder_layer == 5
    assert n_heads == 4
    assert n_embedding == 16

use_params(a=0.25, b="world")
print("OK")

Here, only a and b are specified in the use_params () function.

You can also programmatically overwrite the params.yml setting as the default by specifying as_default = True as the decorator argument, as shown below. (By the way, in the case of as_default = False (default of the decorator), the direct of the configuration file is prioritized over the actual argument specified by the program.)

@add_args("params.yml", as_default=True)
def use_params(n_encoder_layer, n_decoder_layer, n_heads, n_embedding):
    assert n_encoder_layer == 128   # notice !!
    assert n_decoder_layer == 5
    assert n_heads == 4
    assert n_embedding == 16

use_params(n_encoder_layer=128)
print("OK")

--Other --You can decorate it with the class __init__, so please give it a try. --In omegaconf, you can refer to environment variables and direct variables in the configuration file as variables. --For more information on omegaconf, see here

The remaining challenges

――It's subtle to write the same code every time, so I want to be able to pip install

Recommended Posts

In omegaconf, let's pass the direct parameter file to the function
Wagtail Recommendation (4) Let's pass the context to the template
I want to get the file name, line number, and function name in Python 3.4
Do not pass self to ProcessPoolExecutor in the class
[C / C ++] Pass the value calculated in C / C ++ to a python function to execute the process, and use that value in C / C ++.
Covector to think in function
Change the standard output destination to a file in Python
It was great to edit the Python file in the Raspberry Pi with Atom's remote function
I tried to implement the mail sending function in Python
Let's create a function to hold down Button in Tkinter
Output the key list included in S3 Bucket to a file
I want to pass the G test in one month Day 1
Create a function to get the contents of the database in Go
Attempt to extend a function in the library (add copy function to pathlib)
How to use the render function defined in .mako (.html) directly in mako
Programming to fight in the world ~ 5-1
Programming to fight in the world ~ 5-5,5-6
Programming to fight in the world 5-3
How to use the zip function
sort warning in the pd.concat function
Save the binary file in Python
Programming to fight in the world-Chapter 4
Pass arguments to Task in discord.py
Cython to try in the shortest
Programming to fight in the world ~ 5-2
How to specify a .ui file in the dialog / widget GUI in PySide
[LPIC 101] How to specify the disk partition number in the GRUB configuration file
Added a function to register desired shifts in the Django shift table
The file name was bad in Python and I was addicted to import
I made a program to check the size of a file in Python
What's in the parameter? Edit String & Expression
Django ~ Let's display it in the browser ~
OR the List in Python (zip function)
Convert psd file to png in Python
Let's claim the possibility of pyenv-virtualenv in 2021
In Jupyter, add IPerl to the kernel.
Read the file line by line in Python
Read the file line by line in Python
Various comments to write in the program
[Python] Read the specified line in the file
Function to extract the maximum and minimum values ​​in a slice with Go
Various ways to read the last line of a csv file in Python
How to pass the execution result of a shell command in a list in Python
How to pass the path to the library built with pyenv and virtualenv in PyCharm
[Python] I want to know the variables in the function when an error occurs!
[Bash] While read, pass the contents of the file to variables for each column
Handle CSV that contains the element you want to parse in the file name
[AWS] Let's run a unit test of Lambda function in the local environment
If an exception occurs in the function, it will be transmitted to the caller 2
Reinventing the Wheel: libMerge to merge function definitions in Bash library into ShellScript
Don't forget to close the file just because it's in a temporary folder
If an exception occurs in the function, it will be transmitted to the caller 1
[VBA] Function to judge whether the character string of UTF-8 file system is 255 or less and function to count UTF-8 character string in bytes