This time, I will explain how to upload a huge file using RestTemplate
of Spring Framework
. The point is to use RequestCallback
.
For the server-side implementation of huge file upload, please refer to "How to realize huge file upload with TERASOLUNA 5.x (= Spring MVC)".
(point)
RequestCallback
interface and write the upload data to the request BODY while buffering it.RestTemplate
and add a method that allows you to specify RequestCallback
using the default ResponseExtractorRequestFactory
LargeFileRequestCallback.java
package todo.app.largefile;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.client.ClientHttpRequest;
import org.springframework.web.client.RequestCallback;
//★ Point 1
public class LargeFileRequestCallback implements RequestCallback {
/**
* LOGGER
*/
private static final Logger LOGGER = LoggerFactory
.getLogger(LargeFileRequestCallback.class);
/**
*★ Point 2
* buffer size 1MB
*/
private static final int BUFFER_SIZE = 1 * 1024 * 1024;
/**
*★ Point 3
*File information
*/
private final FileInfo fileInfo;
/**
*★ Point 3
*constructor
* @param fileInfo File information
*/
public LargeFileRequestCallback(FileInfo fileInfo) {
this.fileInfo = fileInfo;
}
/**
*★ Point 4
* @see org.springframework.web.client.RequestCallback#doWithRequest(org.springframework.http.client.ClientHttpRequest)
*/
@Override
public void doWithRequest(ClientHttpRequest request) throws IOException {
LOGGER.debug("doWithRequest : start");
//★ Point 5
// 1. set Content-type
request.getHeaders().add("Content-type", fileInfo.getContentType());
request.getHeaders().add("Content-Length",
Long.toString(fileInfo.getContentLength()));
request.getHeaders().add("X-SHA1-CheckSum", fileInfo.getCheckSum());
request.getHeaders().add("X-FILE-NAME", fileInfo.getDownloadFileName());
//★ Point 6
// 2. copy contents with buffering
try (InputStream input = new BufferedInputStream(
new FileInputStream(fileInfo.getFilePath()));
OutputStream out = request.getBody();) {
byte[] buffer = new byte[BUFFER_SIZE];
long total = 0;
int len = 0;
while ((len = input.read(buffer)) != -1) {
out.write(buffer, 0, len);
out.flush();
total = total + len;
// LOGGER.debug("writed : " + total);
}
}
LOGGER.debug("doWithRequest : end");
}
}
** ★ Point 1 **
ʻDefine aRequestCallback
class that implements the ʻorg.springframework.web.client.RequestCallbackinterface. This class can manipulate the request header when issuing an HTTP request for
RestTemplate` and write data to the request BODY. This time, it is used to write the file data to the request BODY.
This time, I defined it as a separate class so that it can be used by others, but it is also possible to write it as (1) arrow operator (lambda expression) and (2) inner class (anonymous class) in the place to be used.
** ★ Point 2 ** Defines the buffer size when writing file data. This time it was 1MB.
** ★ Point 3 **
The file information to be uploaded is linked to the RequestCallback
class as a constructor argument.
** ★ Point 4 **
This is the point of this article. Override the doWithRequest
method to implement the process of writing the data of the upload file to the HTTP request.
The ClientHttpRequest
object passed as an argument is the HTTP request issued by RestTemplate
. Manipulate this directly to set the request header or write data to the request BODY.
** ★ Point 5 **
When setting the request header, get the HttpHeaders
object with the getHeaders
method and add it with the ʻadd` method.
** ★ Point 6 **
Write to the request BODY while buffering the file data.
The request BODY is accessed as an output stream by calling the getBody
method of the ClientHttpRequest
object.
RestTemplateCallbackExtend.java
package todo.app.largefile;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.web.client.RequestCallback;
import org.springframework.web.client.ResponseExtractor;
import org.springframework.web.client.RestTemplate;
//★ Point 7
public class RestTemplateCallbackExtend extends RestTemplate {
public RestTemplateCallbackExtend() {
super();
}
/**
*★ Point 8
* Extend the postForEntity method so that callback can be used
* @param url the URL
* @param responseType the type of the return value
* @param requestCallback object that prepares the request
* @return the converted object
*/
public <T> ResponseEntity<T> postForEntityWithCallback(String url,
Class<T> responseType, RequestCallback requestCallback) {
// 1. create ResponseExtractor object
ResponseExtractor<ResponseEntity<T>> responseExtractor = responseEntityExtractor(
responseType);
// 2. send request using RequestCallback and default ResponseExtractor
return super.execute(url, HttpMethod.POST, requestCallback,
responseExtractor);
}
}
** ★ Point 7 **
By default, RestTemplate
has only three methods that use RequestCallback
, and in that case, you need to specify ResponseExtractor
as well.
I've implemented RequestCallback
, but ResponseExtractor
wants to use the default functionality, so I'll extend RestTemplate
a bit.
(reference)
Since it was troublesome to create ResponseExtractor <ResponseEntity <String >>
of ResponseEntity <String>
, this time I extended RestTemplate
.
If you can easily create this ResponseExtractor
object, you should call the
execute method of
RestTemplate` directly.
List of methods that take RequestCallback as an argument in RestTemplate
public <T> T execute(String url,
HttpMethod method,
RequestCallback requestCallback,
ResponseExtractor<T> responseExtractor,
Object... urlVariables)
throws RestClientException
public <T> T execute(String url,
HttpMethod method,
RequestCallback requestCallback,
ResponseExtractor<T> responseExtractor,
Map<String,?> urlVariables)
throws RestClientException
public <T> T execute(URI url,
HttpMethod method,
RequestCallback requestCallback,
ResponseExtractor<T> responseExtractor)
throws RestClientException
** ★ Point 8 **
This time, I wanted a postForEntity
method that can specify RequestCallback
as an argument in POST
, so I added the postForEntityWithCallback
method.
Call the protected`` responseEntityExtractor
method to use the default ResponseExtractor
, and RequestCallback
uses the one specified by the argument to execute the ʻexecute` method.
FileUploadControllerTest.java
package todo.app.largefile;
import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.*;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.web.client.HttpClientErrorException;
public class FileUploadControllerTest {
/**
* LOGGER
*/
private static final Logger LOGGER = LoggerFactory
.getLogger(FileUploadControllerTest.class);
@Test
public void test01() {
//★ Point 9
// 1. prepared data
String targetUrl = "http://localhost:8090/todo-rest/upload/chunked";
FileInfo fileInfo = getFileInfo();
//★ Point 10
// 2. create RestTemplate object
RestTemplateCallbackExtend restTemplate = new RestTemplateCallbackExtend();
//★ Point 11
// 3. setting SimpleClientHttpRequestFactory for timeout, streaming
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
requestFactory.setConnectTimeout(1000 * 120);
requestFactory.setReadTimeout(1000 * 120);
requestFactory.setChunkSize(1 * 1024 * 1024);
restTemplate.setRequestFactory(requestFactory);
try {
//★ Point 12
// 4. send request using RequestCallback
ResponseEntity<String> responseEntity = restTemplate
.postForEntityWithCallback(targetUrl, String.class,
new LargeFileRequestCallback(fileInfo));
//★ Point 13
// 5. assert
assertThat(responseEntity.getStatusCode(), is(HttpStatus.CREATED));
assertThat(responseEntity.getBody(), is("success!"));
} catch (HttpClientErrorException e) {
LOGGER.error(e.getMessage()); //Only "403 Forbidden" is included.
LOGGER.error(e.getResponseBodyAsString());
throw e;
}
}
/**
*★ Point 9
*Get file information for testing
* @return file information
*/
private FileInfo getFileInfo() {
FileInfo fileInfo = new FileInfo();
fileInfo.setFilePath("C:/tmp/spring-tool-suite-3.8.2.RELEASE-e4.6.1-win32-x86_64.zip");
fileInfo.setContentType("application/zip");
fileInfo.setCheckSum("ff1a62872fb62edb52c1e040fdf9c720e051f9e8");
fileInfo.setContentLength(418945672);
fileInfo.setDownloadFileName("STS3.8.2.zip");
return fileInfo;
}
}
** ★ Point 9 ** Prepare the data to send. This time, I decided to upload the STS Zip file (a little less than 400MB) that I had. The checksum (hash value) was calculated in "How to obtain the hash value (checksum) of a huge file in Python".
** ★ Point 10 **
Instantiate the RestTemplateCallbackExtend
class defined at point 7 instead of RestTemplate
.
** ★ Point 11 **
When uploading a huge file, it is necessary to consider the effects of timeout, buffer data size, etc.
ʻConfig in the org.springframework.http.client.SimpleClientHttpRequestFactory
class and enable it in the setRequestFactory
method of RestTemplate
.
setConnectTimeout
: Set the connection timeout (unit is milliseconds) for the URL connection.setReadTimeout
: Set the read timeout (unit is milliseconds) of the URL connection.setChunkSize
: Set the chunk size (in bytes) when writing data to the HTTP request BODY.setBufferRequestBody
: Set whether the request factory buffers internally (true: default
or false
)As explained below, this time we will send a request with a large data size, so set false
.
Default is true. When sending large amounts of data via POST or PUT, it is recommended to change this property to false, so as not to run out of memory.
** ★ Point 12 **
★ Call the postForEntityWithCallback
method implemented in point 8. At that time, specify the LargeFileRequestCallback
object as an argument.
Now the file data is written to the request BODY using the RequestCallback
implemented this time.
** ★ Point 13 **
After executing the HTTP request, it will be the same as the normal RestTemplate
.
Access the HTTP response status code with the getStatusCode
method and access the HTTP response BODY with the getBody
method.
If HttpClientErrorException
occurs, the getMessage
method gets a summary of the exception. If you need more details, check the response BODY on exception with the getResponseBodyAsString
method.
This time, I explained how to upload a huge file using RestTemplate
and RequestCallback
.
I tried this time and found that there is no method that can specify RequestCallback
independently in RestTemplate
(there is a set with ResponseExtractor
), which was a little inconvenient.
Recommended Posts