Introduction

I used AWS Batch, which can confine logic in a container and run it on any instance, conveniently, but since the current project was Azure, I tried using Azure Batch. The concept is the same as AWS. Azure still doesn't have many articles in documents and Qiita, so I hope it helps.

By the way, this time it is implemented with Java SDK, but if you define a batch from the screen or use SDK of another language, you can move it by setting the same setting value, so I think that it will be helpful.

Azure Batch

Azure Batch is a service that allows you to run any program (including Shell) on an instance of a specified size. It is possible to run programs inside the container like AWS Batch. By the way, AWS (*) used CloudWatch Events to schedule, but Azure Batch has a built-in schedule function.

In the case of AWS, you need to run it like CloudWatchEvents-> Lambda-> AWS Batch.

Preparation

The program that runs in the container looks like this. Let's put out information such as the OS to check if it is running on the specified instance properly.

By the way, https://github.com/oshi/oshi is convenient for getting OS information in Java.

public class BatchMain {

    public static void main(String[] args) {

        System.out.println("execute main method.");

        System.out.println("Environment Variables");
        System.getenv().forEach((k,v) -> System.out.println(k + ":" + v));

        var si = new SystemInfo();
        System.out.println("OS: " + si.getOperatingSystem());

        var hardware = si.getHardware();
        System.out.println("Processor: " + hardware.getProcessor());

        var memory = hardware.getMemory();
        System.out.println("Memory: " + memory.getAvailable() + " / " + memory.getTotal());

        System.exit(0);
    }
}

Now we will push this to Azure Container Registry. Push using jib as usual. Don't forget to write USERNAME and PASSWORD in gradle.properties.

jib {
    to {
        image = 'xxxxx.azurecr.io/batch:0.0.1-SNAPSHOT'
        auth {
            username = "${USERNAME}"
            password = "${PASSWORD}"
        }
    }
}

gradlew jib

Create an Azure Batch account

Select batch accounts.

Create a batch account.

Make a copy of the batch account, url, and primary access key. This is used when specifying it programmatically.

Make a pool

In Azure Batch, the execution environment is defined in Pool. It is not necessary to create a pool setting for each job, and if the environment does not change, one is fine. In AWS, it's a Computing Environment.

Supported virtual machine images (Linux) are listed here [https://docs.microsoft.com/en-us/azure/batch/batch-docker-container-workloads) but only for centos or ubuntu Become. Of course, you can prepare your own.

It's easier to find the value you specify from the Azure admin console


public class Pool {

    public static final String POOL_ID = "pool_1";

    public static void main(String[] args) throws IOException {
        BatchClient client = BatchClient.open(new BatchSharedKeyCredentials(
                "https://xxxxx.japaneast.batch.azure.com",
                "xxxx",
                "xxxxxx"));

        ContainerRegistry containerRegistry = new ContainerRegistry()
                .withRegistryServer("xxxxxx.azurecr.io")
                .withUserName("xxxxxx")
                .withPassword("xxxxxxxxx");

        ContainerConfiguration containerConfiguration = new ContainerConfiguration();
        containerConfiguration.withContainerRegistries(Arrays.asList(containerRegistry));
        containerConfiguration.withContainerImageNames(Arrays.asList("xxxx.azurecr.io/batch:0.0.1-SNAPSHOT"));

        ImageReference imageReference = new ImageReference();
        imageReference.withPublisher("microsoft-azure-batch");
        imageReference.withOffer("ubuntu-server-container");
        imageReference.withSku("16-04-lts");
        imageReference.withVersion("latest");

        // VM Configuration
        VirtualMachineConfiguration virtualMachineConfiguration = new VirtualMachineConfiguration();
        virtualMachineConfiguration.withImageReference(imageReference);
        virtualMachineConfiguration.withContainerConfiguration(containerConfiguration);
        virtualMachineConfiguration.withNodeAgentSKUId("batch.node.ubuntu 16.04");

        // Create Pool
        client.poolOperations().createPool(POOL_ID, "standard_d1_v2", virtualMachineConfiguration, 1);
    }

}

Move simply

Let's move JOB and Task based on the information defined in Pool. First, create a JOB.


public class SimpleJob {

    public static final String POOL_ID = "pool_1";

    public static final String JOB_ID = "job_1";

    public static void main(String[] args) throws IOException {
        BatchClient client = BatchClient.open(new BatchSharedKeyCredentials(
                "https://xxxxx.japaneast.batch.azure.com",
                "xxxx",
                "xxxxxx"));

        PoolInformation poolInformation = new PoolInformation();
        poolInformation.withPoolId(POOL_ID);

        // CreateJob
        client.jobOperations().createJob(JOB_ID, poolInformation);
    }
}

Run the task within the JOB.

public class SimpleTask {

    public static void main(String[] args) throws IOException, InterruptedException {

        BatchClient client = BatchClient.open(new BatchSharedKeyCredentials(
                "https://xxxxx.japaneast.batch.azure.com",
                "xxxx",
                "xxxxxx"));

        TaskAddParameter parameter = new TaskAddParameter();
        parameter.withId("task1")
                .withUserIdentity(new UserIdentity().withAutoUser(new AutoUserSpecification().withElevationLevel(ElevationLevel.ADMIN).withScope(AutoUserScope.TASK)))
                .withContainerSettings(new TaskContainerSettings().withImageName("xxxx.azurecr.io/batch:0.0.1-SNAPSHOT")
                        .withContainerRunOptions("--rm"))
                .withConstraints(new TaskConstraints().withMaxTaskRetryCount(-1)).withCommandLine("");
        client.taskOperations().createTask(ScheduleJob.JOB_ID, parameter);

        long timeout = 300 * 1000;
        long startTime = System.currentTimeMillis();

        while (System.currentTimeMillis() - startTime <= timeout) {
            CloudTask task = client.taskOperations().getTask(ScheduleJob.JOB_ID, parameter.id());

            if (task.state() != TaskState.COMPLETED) {
                Thread.sleep(1000);
            }
        }
    }
}

If you do this, you can see it on the Azure admin screen.

When you click the moved JOB, the Task screen is displayed and you can see the output contents from the screen.

Execute JOB at the specified time

Specify the JOB time with withDoNotRunUntil. It also defines the tasks that move when the JOB is executed.


public class ScheduleJob {

    public static final String POOL_ID = "pool_1";

    public static final String JOB_ID = "job_5";

    public static void main(String[] args) throws IOException {
        BatchClient client = BatchClient.open(new BatchSharedKeyCredentials(
                "https://xxxxx.japaneast.batch.azure.com",
                "xxxx",
                "xxxxxx"));

        // Create Schedule SimpleJob
        DateTime scheduleDateTime = new DateTime(2019,8,22,4,30, DateTimeZone.UTC);
        Schedule schedule = new Schedule().withDoNotRunUntil(scheduleDateTime);

        // pool
        PoolInformation poolInformation = new PoolInformation();
        poolInformation.withPoolId(POOL_ID);

        // container
        JobManagerTask task = new JobManagerTask();
        task.withId("scheduletask")
                .withUserIdentity(new UserIdentity().withAutoUser(new AutoUserSpecification().withElevationLevel(ElevationLevel.ADMIN).withScope(AutoUserScope.TASK)))
                .withContainerSettings(new TaskContainerSettings().withImageName("xxxxx.azurecr.io/batch:0.0.1-SNAPSHOT")
                        .withContainerRunOptions("--rm"))
                .withCommandLine("");

        JobSpecification jobSpecification = new JobSpecification().withPoolInfo(poolInformation).withJobManagerTask(task);

        client.jobScheduleOperations().createJobSchedule(JOB_ID,schedule, jobSpecification);
    }
}

Move repeatedly from the specified time (Cron)

Specify the time to move for the first time with DoNotRunUntil, and specify the time to move repeatedly with RecurrenceInterval. If you do not specify DoNotRunUntil, the job will be run repeatedly from the defined time.

public class CronScheduleJob {

    public static final String POOL_ID = "pool_1";

    public static final String JOB_ID = "job_7";

    public static void main(String[] args) throws IOException {
        BatchClient client = BatchClient.open(new BatchSharedKeyCredentials(
                "https://xxxxx.japaneast.batch.azure.com",
                "xxxx",
                "xxxxxx"));

        // Create Schedule SimpleJob
        DateTime startDateTime = new DateTime(2019,8,22,4,45, DateTimeZone.UTC);
        Period period = new Period(0,1,0,0);
        Schedule schedule = new Schedule().withRecurrenceInterval(period).withDoNotRunUntil(startDateTime);

        // pool
        PoolInformation poolInformation = new PoolInformation();
        poolInformation.withPoolId(POOL_ID);

        // container
        JobManagerTask task = new JobManagerTask();
        task.withId("cronscheduletask")
                .withUserIdentity(new UserIdentity().withAutoUser(new AutoUserSpecification().withElevationLevel(ElevationLevel.ADMIN).withScope(AutoUserScope.TASK)))
                .withContainerSettings(new TaskContainerSettings().withImageName("xxxxx.azurecr.io/batch:0.0.1-SNAPSHOT")
                        .withContainerRunOptions("--rm"))
                .withCommandLine("");

        JobSpecification jobSpecification = new JobSpecification().withPoolInfo(poolInformation).withJobManagerTask(task);

        client.jobScheduleOperations().createJobSchedule(JOB_ID,schedule, jobSpecification);
    }
}

[JAVA] Let's run batch in container using Azure Batch