This time, I will explain how to upload a file with a large data size with TERASOLUNA 5.x (= Spring MVC). If the data size is small, you don't have to worry about it, but if it is large, you need to be careful about server resources.
(point)
content-type according to the file type instead of multipart / form-datamultipart / form-data (for example, base64).ModelAttributeMethodProcessor to hide HttpServletRequest from ControllerPlease refer to "How to realize huge file download with TERASOLUNA 5.x (= Spring MVC)" for download.
(Not good code 1) When taking an HTTP request as an argument
@RequestMapping(path = "chunked", method = RequestMethod.POST)
public ResponseEntity<String> streamUpload(HttpServletRequest httpRequest) {
// You should be getted 5 parameters from HttpServletRequest and validated it
// omitted
}
This is the old-fashioned method familiar with ʻAction of ServletandStruts. It receives HttpServletRequestas an argument of the handler method and reads the uploaded file data from BODY. The disadvantage of this method is that it handles rawHttpServletRequest` and is not testable.
(Not good code 2) When parameters are defined individually
@RequestMapping(path = "chunked", method = RequestMethod.POST)
public ResponseEntity<String> streamUpload(InputStream input,
@RequestHeader(name = "Content-Type", required = false) String contentType,
@RequestHeader(name = "Content-Length", required = false) Long contentLength,
@RequestHeader(name = "X-SHA1-CheckSum", required = false) String checkSum,
@RequestHeader(name = "X-FILE-NAME", required = false) String fileName) {
// You should be validated 5 parameters
// omitted
}
Compared to the above code, it has come to use the function of spring-mvc such as using @ RequestHeader, but it seems complicated because there are many parameters.
Also, with this method, input checking using Bean Validation and BindingResult cannot be used.
**(Caution) If you specify ʻInputStream` as an argument of the handler method, you can receive the BODY data of the HTTP request as an input stream. In other words, it will be the uploaded binary file data. ** **
** I would like to use ModelAttributeMethodProcessor to receive the class that stores the required data as an argument of the handler method.
There is a HandlerMethodArgumentResolver interface as a similar function, but this usesModelAttributeMethodProcessor because input checking is not possible. ** **
StreamFile.java
public class StreamFile implements Serializable {
private static final long serialVersionUID = 1L;
@NotNull
private InputStream inputStream;
@Size(max = 200)
@NotEmpty
private String contentType;
@Min(1)
private long contentLength;
@Size(max = 40)
@NotEmpty
private String checkSum;
@Size(max = 200)
@NotEmpty
private String fileName;
// omitted setter, getter
}
Annotate the input check of Bean Validation as well as the input check of Form class(form backing bean).
StreamFileModelAttributeMethodProcessor.java
package todo.app.largefile;
import java.io.IOException;
import javax.servlet.http.HttpServletRequest;
import org.springframework.core.MethodParameter;
import org.springframework.web.bind.ServletRequestParameterPropertyValues;
import org.springframework.web.bind.WebDataBinder;
import org.springframework.web.context.request.NativeWebRequest;
import org.springframework.web.method.annotation.ModelAttributeMethodProcessor;
//★ Point 1
public class StreamFileModelAttributeMethodProcessor extends
ModelAttributeMethodProcessor {
//★ Point 2
public StreamFileModelAttributeMethodProcessor() {
super(false);
}
//★ Point 3
@Override
public boolean supportsParameter(MethodParameter parameter) {
return StreamFile.class.equals(parameter.getParameterType());
}
//★ Point 4
@Override
protected void bindRequestParameters(WebDataBinder binder,
NativeWebRequest request) {
//★ Point 5
HttpServletRequest httpRequest = request
.getNativeRequest(HttpServletRequest.class);
ServletRequestParameterPropertyValues pvs = new ServletRequestParameterPropertyValues(
httpRequest);
//★ Point 6
pvs.add("contentType", httpRequest.getContentType());
pvs.add("contentLength", httpRequest.getContentLengthLong());
pvs.add("checkSum", httpRequest.getHeader("X-SHA1-CheckSum"));
pvs.add("fileName", httpRequest.getHeader("X-FILE-NAME"));
//★ Point 7
try {
pvs.add("inputStream", httpRequest.getInputStream());
} catch (IOException e) {
pvs.add("inputStream", null);
}
//★ Point 8
binder.bind(pvs);
}
}
** ★ Point 1 **
Defines a class that extends ʻorg.springframework.web.method.annotation.ModelAttributeMethodProcessor. ModelAttributeMethodProcessor is a class that implements the ʻorg.springframework.web.method.support.HandlerMethodArgumentResolver interface.
** ★ Point 2 **
[Constructor of ModelAttributeMethodProcessor](https://docs.spring.io/autorepo/docs/spring-framework/4.3.5.RELEASE/javadoc-api/org/springframework/web/method/annotation/ModelAttributeMethodProcessor.html#ModelAttributeMethodProcessor- Call boolean-).
This time we will create an object with the default false.
** ★ Point 3 **
ʻOrg.springframework.web.method.support.HandlerMethodArgumentResolver's supportsParametermethod. Process when the data type of the argument of the handler method isStreamFile.class`.
** ★ Point 4 **
This is the point of this method. Implement the process to bind (set) the data obtained from the HTTP request to the argument object (StreamFile this time) of the handler method in the bindRequestParameters method.
That is, it gets the BODY input stream of the HTTP request and the value of the HTTP request header and binds (sets) it to WebDataBinder.
** ★ Point 5 ** org.springframework.beans.PropertyValues Implementation class [org.springframework.web.bind.ServletRequestParameterPropertyValues](https://docs.spring.io/autorepo/docs/spring-framework/4.3.5.RELEASE/javadoc-api/org/springframework/web/ bind / ServletRequestParameterPropertyValues.html) Create an object of class.
** ★ Point 6 **
Set the value obtained from the HTTP request header with the ʻadd method of ServletRequestParameterPropertyValues`.
StreamFile this time) of the handler method** ★ Point 7 **
★ Set the BODY input stream of the HTTP request in the same way as point 6. I decided to set null when ʻIOException` occurs.
** ★ Point 8 **
Bind (set) the PropertyValues generated at point 5 with the bind method of WebDataBinder.
With this, the value of ★ points 6 and 7 will be set in the argument object (StreamFile this time) of the handler method.
FileUploadController.java
package todo.app.largefile;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.validation.BindingResult;
import org.springframework.validation.annotation.Validated;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
@Controller
@RequestMapping("upload")
public class FileUploadController {
/**
* LOGGER
*/
private static final Logger LOGGER = LoggerFactory
.getLogger(FileUploadController.class);
/**
*★ Point 9
* define max upload file size
*/
private static final long MAX_FILE_SIZE = 500 * 1024 * 1024;
/**
*★ Point 9
* buffer size 1MB
*/
private static final int BUFFER_SIZE = 1 * 1024 * 1024;
//★ Point 10
@RequestMapping(path = "chunked", method = RequestMethod.POST)
public ResponseEntity<String> streamUpload(
@Validated StreamFile streamFile,
BindingResult result) {
//★ Point 11
if (result.hasErrors()) {
LOGGER.debug("validated error = {}", result.getAllErrors());
return new ResponseEntity<String>("validated error!",
HttpStatus.BAD_REQUEST);
}
//★ Point 12
if (MAX_FILE_SIZE < streamFile.getContentLength()) {
return fileSizeOverEntity();
}
//★ Point 13
try {
File uploadFile = File.createTempFile("upload", null);
InputStream input = streamFile.getInputStream();
try (OutputStream output = new BufferedOutputStream(
new FileOutputStream(uploadFile))) {
byte[] buffer = new byte[BUFFER_SIZE];
long total = 0;
int len = 0;
while ((len = input.read(buffer)) != -1) {
output.write(buffer, 0, len);
output.flush();
total = total + len;
LOGGER.debug("writed : " + total);
//★ Point 12
if (MAX_FILE_SIZE < total) {
return fileSizeOverEntity();
}
}
}
LOGGER.debug(uploadFile.getAbsolutePath());
//★ Point 14
return new ResponseEntity<String>("success!", HttpStatus.CREATED);
} catch (IOException e) {
LOGGER.error(e.getMessage());
//★ Point 15
return new ResponseEntity<String>("error!",
HttpStatus.INTERNAL_SERVER_ERROR);
}
}
/**
*★ Point 12
* @return ResponseEntity when file size excess error occurs
*/
private ResponseEntity<String> fileSizeOverEntity() {
return new ResponseEntity<String>(
"file size is too large. " + MAX_FILE_SIZE + "(byte) or less",
HttpStatus.BAD_REQUEST);
}
/**
*Display the upload form screen
* @return upload form screen
*/
@RequestMapping(path = "form", method = RequestMethod.GET)
public String form() {
return "upload/form";
}
}
** ★ Point 9 ** Define the upper limit of the data size of the upload file and the buffer size used when saving to the file.
** ★ Point 10 **
Specify @Validated and BindingResult so that input checking is enabled as an argument of the handler method.
★ By processing StreamFileModelAttributeMethodProcessor defined in point 1, it is possible to take StreamFile that has been input checked as an argument.
** ★ Point 11 **
As with normal input check, check if there is an input check error with the hasErrors method of BindingResult.
This time, in the case of an input error, we will return a response with HttpStatus.BAD_REQUEST, that is, the HTTP response status code 400.
** ★ Point 12 **
Check if the data size of the uploaded file exceeds the upper limit defined in ★ Point 9.
Check the data size in two places: (1) the value of the Content-Length header and (2) the actually read data size.
If the limit is exceeded, we will return a response with HttpStatus.BAD_REQUEST, that is, the HTTP response status code 400. ,
** ★ Point 13 ** Read the data from the input stream of BODY of the HTTP request with the buffer size defined in point 9, and save the uploaded file as a file. This time I decided to save it in a temporary directory. Please modify here according to your business requirements.
** ★ Point 14 **
Since the uploaded file could be saved as a file on the server, we will return a response with HttpStatus.CREATED, that is, the HTTP response status code 201.
(Caution) This time, the metadata of the uploaded file (file name, content type, data size, checksum (hash value), etc.) is not saved. In the actual system, when accessing the uploaded file (downloading, opening with the application, etc.), metadata is required, so the metadata is saved in the database etc.
** ★ Point 15 **
If ʻIOException occurs during processing, this time we will return a response with HttpStatus.INTERNAL_SERVER_ERROR`, that is, HTTP response status code 500.
Spring Bean definition file (spring-mvc.xml）
<!-- omitted -->
<mvc:annotation-driven>
<mvc:argument-resolvers>
<!-- omitted -->
<bean class="todo.app.largefile.StreamFileModelAttributeMethodProcessor"/>
</mvc:argument-resolvers>
</mvc:annotation-driven>
In order to enable the StreamFileModelAttributeMethodProcessor implemented this time, add the Bean of ★ point 1 to<mvc: argument-resolvers>.
For TERASOLUNA 5.x, define it in the spring-mvc.xml file.
Application server settings (eg tomcat server.xml）
<Connector connectionTimeout="200000" port="8090"
protocol="HTTP/1.1" redirectPort="8443"
maxPostSize="-1" maxParameterCount="-1"
maxSavePostSize="-1" maxSwallowSize="-1"
socket.appReadBufSize="40960"/>
When uploading a huge file, you also need to check the application server settings.
time out
The larger the data size, the longer it will take to send the data. It must be set so that it does not time out before the data transmission is completed.
For Tomcat8.0, set with connectionTimeout. The unit is milliseconds and the default is 20000 (20 seconds).
Request data size
Generally, the application server has an upper limit on the data size of the request. It must be set according to the data size that allows uploading. If you do not make this setting, the upper limit will be caught and the application server will disconnect the request.
Some servers allow you to set a different data size for HTTP requests for file uploads (multipart / form-data). However, this method is not multipart / form-data, so it is considered as a normal HTTP request.
For Tomcat 8.0, maxPostSize, maxSavePostSize, maxSwallowSize, etc. are relevant to this setting. This time, I chose -1, which means no limit.
Request data read buffer size
It turned out that data could not be obtained from the input stream of the request with the specified buffer size (★ BUFFER_SIZE at point 9).
For Tomcat8.0, set with socket.appReadBufSize. The unit is bytes and the fault is 8192. ★ It is necessary to tune with the value of BUFFER_SIZE at point 9.
For Tomcat8.0, please refer to https://tomcat.apache.org/tomcat-8.0-doc/config/http.html.
This time, I explained how to upload a file with a large data size with TERASOLUNA 5.x (= Spring MVC).
The point is the setting of arguments using ModelAttributeMethodProcessor and the necessity of tuning the application server due to the large data size.
** In the test implemented by the client "How to upload a file with ajax without using multipart", a file of less than 400MB is uploaded in 6 seconds. I was able to do. ** **
Recommended Posts