Learn Digdag from Digdag Official Documentation-Architecture

Target

Translated the architecture of the document on Digdag's official website + α The final goal is to create a batch in Rails using Digdag's Ruby http://docs.digdag.io/architecture.html

#table of contents

Getting started Architecture Concepts Workflow definition Scheduling workflow Operators Command reference Language API -Ruby Change the setting value for each environment with Digdag (RubyOnRails) Batch implementation in RubyOnRails environment using Digdag v

Digdag Architecture Automating workflow with Digdag Workflow automates manual actions. Define a set of tasks as a workflow. Digdag keeps running. Tasks are defined using operator plugins, allowing you to control many types of systems from a central workflow engine.

As a plugin runtime framework, Digdag handles the rest of your workload automation issues and allows you to focus on automation. Digdag sends an alert if the task fails. Digdag sends a notification if the workflow does not complete within the expected time. Tasks can be run on your local machine, distributed server, or Docker container.

Organize tasks by groups

Automating complex workflows quickly complicates the definition. Digdag allows you to organize your tasks into groups. When checking the definition, I think we will move on to the details from the overview. It is easy to see that the upper part of the figure is the outline part and the flow of data preparation, analysis, and evaluation is what you are trying to do. Then look at the details of each group for easier debugging and review during development. Administrators can quickly see what's happening in production and how to fix the problem.

Data preparation group, analysis group, evaluation group

The task is started if there are no dependent sibling tasks, or if all sibling tasks complete successfully. When a group's parent task is executed, its child tasks are executed. When all of them complete successfully, the parent task also completes successfully. When a child task fails, the failed child task and its parent task also fail. When the root task completes or fails, the entire execution ends.

Parameters Task grouping is also used to pass parameters between tasks. Since the parent task is a child task, variables can be exported. (Example: as the UNIX shell export command sets environment variables). The parent task can generate child tasks at run time, so it can perform different tasks depending on the results of the previous task.

Workflow as code The Digdag workflow is defined in code. This provides software development best practices (version control, code reviews, testing, collaboration with pull requests). Anyone can pull it and reproduce the same result by pushing the workflow to the git repository.

Running with local mode Digdag is a single file executable command. Creating and running a new workflow is as easy as a Makefile.

*.The dig extension file is used for workflow definition.

#### **`digdag run my_workflow.The dig command runs the workflow.`**
```The dig command runs the workflow.


 Once you have developed the workflow on your local machine and tested it, you need to push it to the server to run the workflow on a regular basis.

 Running on a server
 The ``` * .dig``` files and other files that exist in the same directory are called projects.
 You can push all your projects to the Digdag server and as a result run the workflow on the server.
 It's a little early, but if you push the Workflow you created last time to the server, it will be as follows.


#### **`Start Digdag server`**
```python

$ digdag init mydag
$ cd mydag
$ digdag run mydag.dig
$ digdag server --memory

Project added to Workflow

$digdag push mydag

2020-07-09 20:39:20 +0900: Digdag v0.9.41
Creating .digdag/tmp/archive-3560803829245476890.tar.gz...
  Archiving mydag.dig
Workflows:
  mydag.dig
Uploaded:
  id: 1
  name: mydag
  revision: b8b9abb8-b156-4089-a009-a01aa0337d9f
  archive type: db
  project created at: 2020-07-09T11:39:21Z
  revision updated at: 2020-07-09T11:39:21Z

Use `digdag workflows` to show all workflows.

Running tasks on Docker You can use Docker to perform tasks in the container. If the docker option is set, the task will be executed in the Docker container.

I haven't learned about detailed tasks yet, so I will explain the code below in the Workflow definition part.

_export:
  docker:
    image: ubuntu:14.04

+step1:
  py>: tasks.MyWorkflow.step1

Digdag caches and reuses the pulled image. By default Digdag consistently uses cached images even if there are updates. pull_always: You can set the true option to have Digdag check for updates and pull the latest image of the tag each time the task is started.

_export:
  docker:
    image: ubuntu:latest
    pull_always: true

+step1:
  py>: tasks.MyWorkflow.step1