[JAVA] How to realize huge file upload with Rest Template of Spring

1.First of all

This time, I will explain how to upload a huge file using RestTemplate of Spring Framework. The point is to use RequestCallback. For the server-side implementation of huge file upload, please refer to "How to realize huge file upload with TERASOLUNA 5.x (= Spring MVC)".

(point)

2. Source code

LargeFileRequestCallback.java


package todo.app.largefile;

import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.client.ClientHttpRequest;
import org.springframework.web.client.RequestCallback;

//★ Point 1
public class LargeFileRequestCallback implements RequestCallback {

    /**
     * LOGGER
     */
    private static final Logger LOGGER = LoggerFactory
            .getLogger(LargeFileRequestCallback.class);

    /**
     *★ Point 2
     * buffer size 1MB
     */
    private static final int BUFFER_SIZE = 1 * 1024 * 1024;

    /**
     *★ Point 3
     *File information
     */
    private final FileInfo fileInfo;

    /**
     *★ Point 3
     *constructor
     * @param fileInfo File information
     */
    public LargeFileRequestCallback(FileInfo fileInfo) {
        this.fileInfo = fileInfo;
    }

    /**
     *★ Point 4
     * @see org.springframework.web.client.RequestCallback#doWithRequest(org.springframework.http.client.ClientHttpRequest)
     */
    @Override
    public void doWithRequest(ClientHttpRequest request) throws IOException {

        LOGGER.debug("doWithRequest : start");
        //★ Point 5
        // 1. set Content-type
        request.getHeaders().add("Content-type", fileInfo.getContentType());
        request.getHeaders().add("Content-Length",
                Long.toString(fileInfo.getContentLength()));
        request.getHeaders().add("X-SHA1-CheckSum", fileInfo.getCheckSum());
        request.getHeaders().add("X-FILE-NAME", fileInfo.getDownloadFileName());

        //★ Point 6
        // 2. copy contents with buffering
        try (InputStream input = new BufferedInputStream(
                new FileInputStream(fileInfo.getFilePath()));
                OutputStream out = request.getBody();) {
            byte[] buffer = new byte[BUFFER_SIZE];
            long total = 0;
            int len = 0;
            while ((len = input.read(buffer)) != -1) {
                out.write(buffer, 0, len);
                out.flush();
                total = total + len;
                // LOGGER.debug("writed : " + total);
            }
        }

        LOGGER.debug("doWithRequest : end");
    }

}

** ★ Point 1 ** ʻDefine aRequestCallback class that implements the ʻorg.springframework.web.client.RequestCallbackinterface. This class can manipulate the request header when issuing an HTTP request forRestTemplate` and write data to the request BODY. This time, it is used to write the file data to the request BODY. This time, I defined it as a separate class so that it can be used by others, but it is also possible to write it as (1) arrow operator (lambda expression) and (2) inner class (anonymous class) in the place to be used.

** ★ Point 2 ** Defines the buffer size when writing file data. This time it was 1MB.

** ★ Point 3 ** The file information to be uploaded is linked to the RequestCallback class as a constructor argument.

** ★ Point 4 ** This is the point of this article. Override the doWithRequest method to implement the process of writing the data of the upload file to the HTTP request. The ClientHttpRequest object passed as an argument is the HTTP request issued by RestTemplate. Manipulate this directly to set the request header or write data to the request BODY.

** ★ Point 5 ** When setting the request header, get the HttpHeaders object with the getHeaders method and add it with the ʻadd` method.

** ★ Point 6 ** Write to the request BODY while buffering the file data. The request BODY is accessed as an output stream by calling the getBody method of the ClientHttpRequest object.

RestTemplateCallbackExtend.java


package todo.app.largefile;

import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.web.client.RequestCallback;
import org.springframework.web.client.ResponseExtractor;
import org.springframework.web.client.RestTemplate;

//★ Point 7
public class RestTemplateCallbackExtend extends RestTemplate {

    public RestTemplateCallbackExtend() {
        super();
    }

    /**
     *★ Point 8
     * Extend the postForEntity method so that callback can be used
     * @param url the URL
     * @param responseType the type of the return value
     * @param requestCallback object that prepares the request
     * @return the converted object
     */
    public <T> ResponseEntity<T> postForEntityWithCallback(String url,
            Class<T> responseType, RequestCallback requestCallback) {
        
        // 1. create ResponseExtractor object
        ResponseExtractor<ResponseEntity<T>> responseExtractor = responseEntityExtractor(
                responseType);
        
        // 2. send request using RequestCallback and default ResponseExtractor
        return super.execute(url, HttpMethod.POST, requestCallback,
                responseExtractor);
    }
}

** ★ Point 7 ** By default, RestTemplate has only three methods that use RequestCallback, and in that case, you need to specify ResponseExtractor as well. I've implemented RequestCallback, but ResponseExtractor wants to use the default functionality, so I'll extend RestTemplate a bit.

(reference) Since it was troublesome to create ResponseExtractor <ResponseEntity <String >> of ResponseEntity <String>, this time I extended RestTemplate. If you can easily create this ResponseExtractor object, you should call the execute method of RestTemplate` directly.

List of methods that take RequestCallback as an argument in RestTemplate


public <T> T execute(String url,
                     HttpMethod method,
                     RequestCallback requestCallback,
                     ResponseExtractor<T> responseExtractor,
                     Object... urlVariables)
              throws RestClientException

public <T> T execute(String url,
                     HttpMethod method,
                     RequestCallback requestCallback,
                     ResponseExtractor<T> responseExtractor,
                     Map<String,?> urlVariables)
              throws RestClientException

public <T> T execute(URI url,
                     HttpMethod method,
                     RequestCallback requestCallback,
                     ResponseExtractor<T> responseExtractor)
              throws RestClientException

** ★ Point 8 ** This time, I wanted a postForEntity method that can specify RequestCallback as an argument in POST, so I added the postForEntityWithCallback method. Call the protected`` responseEntityExtractor method to use the default ResponseExtractor, and RequestCallback uses the one specified by the argument to execute the ʻexecute` method.

FileUploadControllerTest.java


package todo.app.largefile;

import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.*;

import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.web.client.HttpClientErrorException;

public class FileUploadControllerTest {

    /**
     * LOGGER
     */
    private static final Logger LOGGER = LoggerFactory
            .getLogger(FileUploadControllerTest.class);

    @Test
    public void test01() {
        //★ Point 9
        // 1. prepared data
        String targetUrl = "http://localhost:8090/todo-rest/upload/chunked";
        FileInfo fileInfo = getFileInfo();

        //★ Point 10
        // 2. create RestTemplate object
        RestTemplateCallbackExtend restTemplate = new RestTemplateCallbackExtend();

        //★ Point 11
        // 3. setting SimpleClientHttpRequestFactory for timeout, streaming
        SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
        requestFactory.setBufferRequestBody(false);
        requestFactory.setConnectTimeout(1000 * 120);
        requestFactory.setReadTimeout(1000 * 120);
        requestFactory.setChunkSize(1 * 1024 * 1024);
        restTemplate.setRequestFactory(requestFactory);

        try {
            //★ Point 12
            // 4. send request using RequestCallback
            ResponseEntity<String> responseEntity = restTemplate
                    .postForEntityWithCallback(targetUrl, String.class,
                            new LargeFileRequestCallback(fileInfo));

            //★ Point 13
            // 5. assert
            assertThat(responseEntity.getStatusCode(), is(HttpStatus.CREATED));
            assertThat(responseEntity.getBody(), is("success!"));

        } catch (HttpClientErrorException e) {
            LOGGER.error(e.getMessage()); //Only "403 Forbidden" is included.
            LOGGER.error(e.getResponseBodyAsString());
            throw e;
        }

    }

    /**
     *★ Point 9
     *Get file information for testing
     * @return file information
     */
    private FileInfo getFileInfo() {
        FileInfo fileInfo = new FileInfo();
        fileInfo.setFilePath("C:/tmp/spring-tool-suite-3.8.2.RELEASE-e4.6.1-win32-x86_64.zip");
        fileInfo.setContentType("application/zip");
        fileInfo.setCheckSum("ff1a62872fb62edb52c1e040fdf9c720e051f9e8");
        fileInfo.setContentLength(418945672);
        fileInfo.setDownloadFileName("STS3.8.2.zip");
        return fileInfo;
    }
}

** ★ Point 9 ** Prepare the data to send. This time, I decided to upload the STS Zip file (a little less than 400MB) that I had. The checksum (hash value) was calculated in "How to obtain the hash value (checksum) of a huge file in Python".

** ★ Point 10 ** Instantiate the RestTemplateCallbackExtend class defined at point 7 instead of RestTemplate.

** ★ Point 11 ** When uploading a huge file, it is necessary to consider the effects of timeout, buffer data size, etc. ʻConfig in the org.springframework.http.client.SimpleClientHttpRequestFactory class and enable it in the setRequestFactory method of RestTemplate.

As explained below, this time we will send a request with a large data size, so set false.

Default is true. When sending large amounts of data via POST or PUT, it is recommended to change this property to false, so as not to run out of memory.

** ★ Point 12 ** ★ Call the postForEntityWithCallback method implemented in point 8. At that time, specify the LargeFileRequestCallback object as an argument. Now the file data is written to the request BODY using the RequestCallback implemented this time.

** ★ Point 13 ** After executing the HTTP request, it will be the same as the normal RestTemplate. Access the HTTP response status code with the getStatusCode method and access the HTTP response BODY with the getBody method. If HttpClientErrorException occurs, the getMessage method gets a summary of the exception. If you need more details, check the response BODY on exception with the getResponseBodyAsString method.

3. Finally

This time, I explained how to upload a huge file using RestTemplate and RequestCallback. I tried this time and found that there is no method that can specify RequestCallback independently in RestTemplate (there is a set with ResponseExtractor), which was a little inconvenient.

Recommended Posts

How to realize huge file upload with Rest Template of Spring
How to realize huge file upload with TERASOLUNA 5.x (= Spring MVC)
How to achieve file upload with Feign
I tried to implement file upload with Spring MVC
File upload with Spring Boot
How to test file upload screen in Spring + Selenium
How to create an Excel form using a template file with Spring MVC
How to boot by environment with Spring Boot of Maven
Extract SQL to property file with jdbcTemplate of spring boot
Let's find out how to receive in Request Body with REST API of Spring Boot
How to load a Spring upload file and view its contents
How to achieve file download with Feign
How to split Spring Boot message file
How to read Body of Request multiple times with Spring Boot + Spring Security
I want to display images with REST Controller of Java and Spring!
How to access Socket directly with the TCP function of Spring Integration
How to use MyBatis2 (iBatis) with Spring Boot 1.4 (Spring 4)
How to use built-in h2db with spring boot
How to bind to property file in Spring Boot
[Spring Boot] How to refer to the property file
How to set environment variables in the properties file of Spring boot application
If you use SQLite with VSCode, use the extension (how to see the binary file of sqlite3)
[Java] How to omit spring constructor injection with Lombok
How to deal with No template for interactive request
How to use CommandLineRunner in Spring Batch of Spring Boot
File upload with Spring Boot (do not use Multipart File)
How to decompile apk file to java source code with MAC
How to use trained model of tensorflow2.0 with Kotlin / Java
[Easy] How to automatically format Ruby erb file with vsCode
How to move another class with a button action of another class.
How to change the contents of the jar file without decompressing
How to deal with different versions of rbenv and Ruby
How to change the file name with Xcode (Refactor Rename)
How to number (number) with html.erb
How to update with activerecord-import
How to deal with the event that Committee :: InvalidRequest occurs in committee during Rspec file upload test
How to make a jar file with no dependencies in Maven
I tried connecting to MySQL using JDBC Template with Spring MVC
[With sample code] Basics of Spring JDBC to learn with Blog app
How to open a script file from Ubuntu with VS code
[chown] How to change the owner of a file or directory
Overwrite upload of file with the same name with BOX SDK (java)
[Beginner] Let's write REST API of Todo application with Spring Boot
Automatic file upload with old Ruby gem What to do with Watir
How to use Lombok in Spring
How to scroll horizontally with ScrollView
Spring with Kotorin --4 REST API design
How to get started with slim
How to use Spring Data JDBC
How to enclose any character with "~"
[How to install Spring Data Jpa]
How to set Spring Boot + PostgreSQL
How to convert erb file to haml
How to use setDefaultCloseOperation () of JFrame
Implement file download with Spring MVC
How to use mssql-tools with alpine
How to use ModelMapper (Spring boot)
How to get along with Rails
How to start Camunda with Docker
How to apply thymeleaf changes to the browser immediately with #Spring Boot + maven
How to decorate the radio button of rails6 form_with (helper) with CSS