About Digdag

Getting started Architecture Concepts Workflow definition Scheduling workflow Operators Command reference Language API -Ruby Change the setting value for each environment with Digdag (RubyOnRails) Batch implementation in RubyOnRails environment using Digdag

things to do

I will create a YAML-format environment setting file for Digdag in the sample application of the Ruby on Rails tutorial, get the environment variable from the Ruby task, set it in the Digdag environment variable, and check until I can get the Digdag environment variable from another Ruby task. ..

The sample application code for the Ruby on Rails tutorial is irrelevant to this discussion, but we'll townload it and add the Digdag Workflow to that project for use in the next batch implementation.

Download sample application for Ruby on Rails tutorial

https://github.com/yasslab/sample_app スクリーンショット 2020-07-15 23.06.34.png

workflow generation

$ digdag init workflow
$ cd workflow
$ digdag run workflow.dig

If the following result is displayed, there is no problem in the Workflow environment of Digdag ~

`result`


2020-07-15 22:01:56 +0900 [INFO](0017@[0:default]+workflow+setup): echo>: start 2020-07-15T00:00:00+00:00
start 2020-07-15T00:00:00+00:00
2020-07-15 22:01:57 +0900 [INFO](0017@[0:default]+workflow+disp_current_date): echo>: 2020-07-15 00:00:00 +00:00
2020-07-15 00:00:00 +00:00
2020-07-15 22:01:57 +0900 [INFO](0017@[0:default]+workflow+repeat): for_each>: {order=[first, second, third], animal=[dog, cat]}
2020-07-15 22:01:57 +0900 [INFO](0018@[0:default]+workflow+repeat^sub+for-0=order=0=first&1=animal=1=cat): echo>: first cat
first cat
2020-07-15 22:01:57 +0900 [INFO](0019@[0:default]+workflow+repeat^sub+for-0=order=1=second&1=animal=0=dog): echo>: second dog
second dog
2020-07-15 22:01:58 +0900 [INFO](0020@[0:default]+workflow+repeat^sub+for-0=order=1=second&1=animal=1=cat): echo>: second cat
second cat
2020-07-15 22:01:58 +0900 [INFO](0022@[0:default]+workflow+repeat^sub+for-0=order=2=third&1=animal=1=cat): echo>: third cat
third cat
2020-07-15 22:01:58 +0900 [INFO](0021@[0:default]+workflow+repeat^sub+for-0=order=2=third&1=animal=0=dog): echo>: third dog
third dog
2020-07-15 22:01:58 +0900 [INFO](0017@[0:default]+workflow+repeat^sub+for-0=order=0=first&1=animal=0=dog): echo>: first dog
first dog
2020-07-15 22:01:58 +0900 [INFO](0017@[0:default]+workflow+teardown): echo>: finish 2020-07-15T00:00:00+00:00
finish 2020-07-15T00:00:00+00:00

Creating an environment variable file in YAML format

$ cd workflow
$ mkdir config
$ vi environment.yml

I set url, user, password for each development, staging and production. Browse this environment variable from the Digdag task.

`environment.yml`


development:
  url: development.jp
  user: development_user
  password: development_password

staging:
  url: staging.jp
  user: staging_user
  password: staging_password
  
production:
  url: procuction.jp
  user: procuction_user
  password: procuction_password

Create a Ruby class that sets the environment variables in the YAML file to the environment variables in Digdag.

workflowUnder the foldertasksCreate a folder Create environment.rb there. This class reads environment variables from a YAML file and saves them in Digdag's environment variables.

$ cd workflow
$ mkdir tasks

`environment.rb`


require 'yaml'

class Environment
  def load
    environment = Digdag.env.params['environment']
    environment = case environment
                  when 'stg'
                    'staging'
                  when 'prd'
                    'production'
                  else
                    'development'
                  end

    #Get the environment variables of the corresponding environment
    config = YAML.load_file('config/environment.yml')[environment]
    config.each do |key, value|
      #Save to Digdag environment variable
      Digdag.env.store(key.to_sym => value)
    end
  end
end

Add task to Workflow.dig

$ digdag init workflowDelete the code automatically generated by and add the following code

`workflow.dig`


timezone: UTC

+load_environment:
  rb>: Environment.load
  require: 'tasks/environment'
+print_environment_value:
  sh>: echo url:${url} user:${user} password:${password}
+print_ruby:
  rb>: DigdagEnvironment.show
  require: 'tasks/digdag_environment'

■ load_environment task: A task to set the corresponding environment variable to the Digdag environment variable from the environment parameters when Digdag is executed. ■ print_environment_value: A task that outputs Digdag environment variables from a shell task ■ print_ruby: Task to output Digdag environment variable from Ruby class

Try running it from the development environment.

$digdag run workflow.dig -p environment=development --rerun

`Execution result in development environment`


2020-07-15 22:49:35 +0900 [INFO](0017@[0:default]+workflow+load_environment): rb>: Environment.load
2020-07-15 22:49:35 +0900 [INFO](0017@[0:default]+workflow+print_environment_value): sh>: echo url:development.jp user:development_user password:development_password
url:development.jp user:development_user password:development_password
2020-07-15 22:49:36 +0900 [INFO](0017@[0:default]+workflow+print_ruby): rb>: DigdagEnvironment.show
url: development.jp
user: development_user
password: development_password

Next is the Staging environment

$digdag run workflow.dig -p environment=stg --rerun

`Execution result in staging environment`


2020-07-15 22:55:33 +0900 [INFO](0017@[0:default]+workflow+load_environment): rb>: Environment.load
2020-07-15 22:55:33 +0900 [INFO](0017@[0:default]+workflow+print_environment_value): sh>: echo url:staging.jp user:staging_user password:staging_password
url:staging.jp user:staging_user password:staging_password
2020-07-15 22:55:34 +0900 [INFO](0017@[0:default]+workflow+print_ruby): rb>: DigdagEnvironment.show
url: staging.jp
user: staging_user
password: staging_password

Finally the Production environment

$digdag run workflow.dig -p environment=prd --rerun

`Execution result in production environment`


2020-07-15 22:56:30 +0900 [INFO](0017@[0:default]+workflow+load_environment): rb>: Environment.load
2020-07-15 22:56:30 +0900 [INFO](0017@[0:default]+workflow+print_environment_value): sh>: echo url:procuction.jp user:procuction_user password:procuction_password
url:procuction.jp user:procuction_user password:procuction_password
2020-07-15 22:56:31 +0900 [INFO](0017@[0:default]+workflow+print_ruby): rb>: DigdagEnvironment.show

Now that the environment variables have been set, I will try to create a batch from the data of the tutorial app of RubyOnRails from the next time.