About the components of Luigi

I would like to write down the elements that make up Luigi, including the meaning of organizing within myself. I will add more as needed. I would appreciate it if you could point out that there may be some misunderstandings.

task

--The unit of processing that makes up the workflow. --Tasks are dependent on each other in the workflow. --There are two types of data passed between tasks: parameters and targets. --The success of the task is to create a file to the target specified by the output () method. --A task failure is an exception within the task. --The file generated by the success of the task is also used as a checkpoint, and if the file exists, the task does nothing and is treated as successful. --The target returned by the output () method of the dependent task can be accessed as the input () method from the dependent task.

target

--An interface to provide access to various file systems. --It includes a mechanism that provides a buffer reader / writer and a client for connecting to a specific file system depending on the mode at the time of open. --The format is used when actually accessing the file.

format

--An interface to provide access to various files. --Provides processing for writing / reading depending on the mode at the time of open () of the target.

About addiction

--There are two types: normal dependence and dynamic dependence. --Normal dependencies are used for dependencies that can be statically formed during programming = when the number of tasks to be performed is known in advance. --Dynamic dependency is used when a dependency cannot be formed at the time of programming = When the number of tasks to be performed depends on the data passed by input () of that task.

Recommended Posts

About the components of Luigi
About the ease of Python
Connected components of the graph
About the features of Python
About the inefficiency of data transfer in luigi on-memory
About the return value of pthread_mutex_init ()
About the basic type of Go
About the upper limit of threads-max
About the behavior of yield_per of SqlAlchemy
About the size of matplotlib points
About the basics list of Python basics
About the behavior of enable_backprop of Chainer v2
About the virtual environment of python version 3.7
About the arguments of the setup function of PyCaret
About the Normal Equation of Linear Regression
About the queue
About the accuracy of Archimedean circle calculation method
About the behavior of copy, deepcopy and numpy.copy
About the processing speed of SVM (SVC) of scikit-learn
A note about the python version of python virtualenv
About the development contents of machine learning (Example)
[Note] About the role of underscore "_" in Python
About the behavior of Model.get_or_create () of peewee in Python
About the behavior of Queue during parallel processing
About the * (asterisk) argument of python (and itertools.starmap)
The beginning of cif2cell
About all of numpy
The meaning of self
About assignment of numpy.ndarray
About MultiIndex of pandas
the zen of Python
The story of sys.path.append ()
About the Unfold function
About the service command
About variable of chainer
About the confusion matrix
About the Visitor pattern
Revenge of the Types: Revenge of types
A memorandum about the warning of the pylint output result
Think about the next generation of Rack and WSGI
About testing in the implementation of machine learning models
About the uncluttered arrangement in the import order of flake8
A story about changing the master name of BlueZ
Personal notes about the integration of vscode and anaconda
A reminder about the implementation of recommendations in Python
Align the version of chromedriver_binary
About max_iter of LogisticRegression () of scikit-learn
10. Counting the number of lines
The story of building Zabbix 4.4
Towards the retirement of Python2
About the camera change event of Google Maps Android API
[Apache] The story of prefork
About the garbled Japanese part of pandas-profiling in Jupyter notebook
About Japanese path of pyminizip
Tank game made with python About the behavior of tanks
Compare the fonts of jupyter-themes
Get the number of digits
Explain the code of Tensorflow_in_ROS
About the enumerate function (python)
About various encodings of Python 3
About the traveling salesman problem