Build a data analysis environment with Kedro + MLflow + Github Actions

Introduction

background

** = "Issues when creating a notebook that is all in one file in a local environment for each experiment (lightgbm_02_YYYYMMDD.ipynb, etc.)" **

What i did

Introduced Kedro as a pipeline tool

icon_kedro.png ### What is Kedro? ・ Introduction method * (Reference) [Introduction to Machine Learning Pipeline with Kedro](https://qiita.com/noko_qii/items/2395d3a3dbcd9410e5e7)

Good thing

$ kedro jupyter notebook --allow-root --port=8888 --ip=0.0.0.0 &
from kedro.framework.context import load_context
proj_path = '../../../' 
context = load_context(proj_path)
# df = catalog.load("XXX")
parameters = context.params

The remaining challenges

Introduced MLflow as an experiment management tool

icon_mlflow.png ### What is MLflow? ・ Introduction method * (Reference) [Introduction to experiment management with MLflow](https://future-architect.github.io/articles/20200626/)

Good thing

The remaining challenges

Introduced Github Actions as a CI tool

icon_githubaction.png ### What is Github Actions? ・ Introduction method * (Reference) [CI / CD to try with the new function "GitHub Actions" on GitHub](https://knowledge.sakura.ad.jp/23478/)

Good thing

The remaining challenges

Other

in conclusion

Recommended Posts

Build a data analysis environment with Kedro + MLflow + Github Actions
Build a data analysis environment that links GitHub authentication and Django with JupyterHub
Build a python data analysis environment on Mac (El Capitan)
Build a comfortable psychological experiment / analysis environment with PsychoPy + Jupyter Notebook
Create a USB boot Ubuntu with a Python environment for data analysis
Easily build a development environment with Laragon
Build a Tensorflow environment with Raspberry Pi [2020]
Build a Fast API environment with docker-compose
[Linux] Build a jenkins environment with Docker
Build a python virtual environment with pyenv
Build a modern Python environment with Neovim
[Linux] Build a Docker environment with Amazon Linux 2
Build a C language development environment with a container
Build a WardPress environment on AWS with pulumi
Prepare a programming language environment for data analysis
Build a python environment with ansible on centos6
[Python] Build a Django development environment with Docker
Create a python3 build environment with Sublime Text3
Build a Django environment with Vagrant in 5 minutes
[Memo] Build a virtual environment with Pyenv + anaconda
Build a virtual environment with pyenv and venv
Build a Django development environment with Doker Toolbox
Build a Python environment with OSX El capitan
Quickly build a Python Django environment with IntelliJ
Build a Python machine learning environment with a container
Build a python execution environment with VS Code
Data analysis with python 2
Data analysis with Python
Build a python virtual environment with virtualenv and virtualenvwrapper
Build a python environment for each directory with pyenv-virtualenv
Data analysis environment construction with Python (IPython notebook + Pandas)
Build a machine learning application development environment with Python
Build a python virtual environment with virtualenv and virtualenvwrapper
Build a development environment with Poetry Django Docker Pycharm
Build a Django environment for Win10 (with virtual space)
Build a numerical calculation environment with pyenv and miniconda3
Build a basic Data Science environment (Jupyter, Python, R, Julia, standard library) with Docker.
Build a Django development environment with Docker! (Docker-compose / Django / postgreSQL / nginx)
Build Azure Pipelies with Azure DevOps in a Linux self-hosted environment
Build a machine learning scikit-learn environment with VirtualBox and Ubuntu
[Memo] Build a development environment for Django + Nuxt.js with Docker
Build a Go development environment with VS Code's Remote Containers
Build a LAMP environment [CentOS 7]
(Now) Build a GPU Deep Learning environment with GeForce GTX 960
Build python3 environment with ubuntu 16.04
Build python environment with direnv
[Django] Build a Django container (Docker) development environment quickly with PyCharm
Tested pipenv with GitHub Actions
Build a comfortable development environment with VSCode x Remote Development x Pipenv
Build a Python environment and transfer data to the server
Creating an environment that automatically builds with Github Actions (Android)
Build a machine learning environment
How to build a python2.7 series development environment with Vagrant
Build a python environment with pyenv (OS X El Capitan 10.11.3)
<Python> Build a dedicated server for Jupyter Notebook data analysis
Create a Todo app with Django ① Build an environment with Docker
Build a Python environment offline
Build a LAMP environment with Vagrant (Linux + Apache + MySQL + PHP)
Build a Python environment with WSL + Pyenv + Jupyter + VS Code
Build a 64-bit Python 2.7 environment with TDM-GCC and MinGW-w64 on Windows 7
Build a Python environment on your Mac with Anaconda and PyCharm