[Java] How to simulate post object form upload to OSS in Java

8 minute read

Alibaba Cloud Object Storage Service (OSS) stores objects in resources called “buckets”. Post Object uses an HTML form to upload the file to the specified bucket.

*This blog is a translation from the English version. You can check the original from here. Some machine translations are used. If you have any translation errors, we would appreciate it if you could point them out. *

Introduction

Alibaba Cloud Object Storage Service (OSS) is an easy-to-use service that can store, backup, and archive large amounts of data in the cloud. OSS acts as an encrypted central repository, providing secure access to files around the world. In addition, up to 99.9% availability is guaranteed, making it ideal for global teams and international project management.

Alibaba Cloud Object Storage Service (OSS) securely stores objects in resources called “buckets”. OSS gives you full access to your buckets and allows you to view the logs and objects within each bucket. You can read, write, delete, and save objects in your bucket indefinitely. The high performance of OSS supports multiple reads/writes simultaneously. The data transfer to the bucket is done over SSL and is encrypted.

Post Object uploads the file to the specified bucket using an HTML form instead of Put. This will allow you to upload files to your bucket via your browser. The desire to implement Post Object in Java stems from brief instructions from various support personnel. According to them, users who need this feature face a variety of difficult problems when trying to implement it according to the official documentation. This happens because there is no official code reference.

Procedure flow

Let’s look at the steps a user should take.

  • The official website first provides HTTP request syntax. The format of HTTP request headers and multipart/form-data-encoded form fields in the message body to satisfy the parameters.
  • Next, we introduce each “required” form field such as “file” or “key” in the form field table. The user will request if any one of the form fields such as “OSSAccessKeyId”, “policy”, “Signature” etc. appears. You also need a REST request header, x-oss -meta- * user meta, and other optional form fields.
  • Posting this introduces some special uses and caveats for some form fields. It also shares a stack of information about Post Policy and Signature features and how to use them.
  • The documentation is clear to some extent, but there are many concepts that are difficult to understand, making it difficult to investigate the problem. In addition, the lack of highlights about error-prone implementations presents users with two major challenges: Users may encounter two major challenges, and two more aspects are involved.

  • Not familiar with MIME type encodings like multipart/form-data.
  • You are not familiar with the OSS implementation rules for parsing Post Object requests.

Now let’s look at the two aspects mentioned above.

See RFC 2388 for a detailed introduction to multipart/form-data. Here are some things to note. Let’s discuss them one by one.

  1. At the first point, the “multipart/form-data” request contains a set of fields. Each field has a content-disposition header of type “form-data”. This header also contains a parameter “name” to describe the contents of the form field. Therefore, all fields have a format similar to the example shown in the documentation.

Content-Disposition: form-data; name=“your_key”

Note: Both “:” and “;” follow a space.

  1. A second point to note is that if you need to upload a user file to your form, you may need the filename or other file attributes in the Content-disposition header. In addition, there is also an optional Content-Type attribute to identify the content type of the file, for any MIME type value in the form field. Therefore, the documentation lists examples of “file” form fields as follows:
Content-Disposition: form-data; name="file"; filename="MyFilename.jpg"
Content-Type: image/jpeg

Note: The “;” before the “filename” still has a trailing space. Similarly, the “:” after “Content-Type” has a trailing space.

  1. The third point is to separate the data with a border line. Let’s use a complex border to distinguish it from the main content. This can be accomplished in a way similar to the content of HTTP headers, as documented.

Content-Type: multipart/form-data; boundary=9431149156168

  1. The fourth point to note is that the structure of each form field is fixed. By design, each form field starts with “–“boundary+, followed by a carriage return (/r/n). Next comes the description of the form fields (see point 1) followed by /r/n. If the content you want to transfer is a file, the file name information also includes the type of file content after the carriage return (/r/n) (see point 2). In addition, there is another carriage return (/r/n) to start the actual content, which must end with /r/n.

  2. Also notice that the last form field ends with “–“+boundary+”–”.

  3. In addition, you also need the /r/n mark to distinguish the HTTP request header from the body information (at the branch point between the header and the first form field). This is basically an extra blank line, like the documentation, and looks like this:

Content-Type: multipart/form-data; boundary=9431149156168

- -9431149156168
Content-Disposition: form-data; name="key"

So far, we have given a general description of the request syntax provided in the OSS official documentation and a related analysis by comparison with the RFC2388 standard.

Here, we will delve into some of the processing that the OSS system uses to parse POST object requests and the relevant notes.

The general procedure for OSS to parse a POST request is shown below for reference.

image.png

The request processing flow is summarized in three core steps.

  1. Parse HTTP request header boundaries to distinguish field boundaries.
  2. Parse the contents of various fields until the flow reaches the’file’ form field.
  3. Parse the file’ form field.

Therefore, the documentation emphasizes placing the’file’ form field in the “last field”. Otherwise, the form fields after “file” may have no effect. Placing the required form field “key” after “file” ensures that the result is InvalidArgument.

Next, I will briefly explain the workflow as shown in the figure.

1) Check the existence of POLICY, OSSACCESSKEYID and SIGNATURE. This check is mandatory. If one of the three fields POLICY, OSSACCESSKEYID, SIGNATURE appears, then the other two fields are required. 2) Presence or absence of authority. Confirm the validity of Post request based on the information of POLICY, OSSACCESSKEYID, and SIGNATURE. 3) Policy rule check Check whether the settings for the various form fields in the request comply with the policy settings. 4) Check length legality.

This is intended to check the length of the option field as there is a limit on the total length of the Post request body. 5) Parse the ParseContentType with ParseFile. Parse the ContentType field of the File field. This field is not mandatory.

The above ends with Java code (Maven project) that implements Post Object upload in OSS, for reference and use by those who are familiar with OSS.

import javax.activation.MimetypesFileTypeMap;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Map;

/**
 * Created by yushuting on 16/4/17.
 */
public class OssPostObject {

    private String postFileName = "your_file";
Make sure that the file exists at the path indicated in the run code.private String ossEndpoint = "your_endpoint";  
For example: http://oss-cn-shanghai.aliyuncs.com
    private String ossAccessId = "your_accessid";  This is your access AK
    private String ossAccessKey = "your_accesskey";  This is your access AK
    private String objectName = "your_object_name";  This is the object name after you upload the file
    private String bucket = "your_bucket";  Make sure that the bucket you created previously has been created. 

    private void PostObject() throws Exception {

        String filepath=postFileName;
        String urlStr = ossEndpoint.replace("http://", "http://"+bucket+".");  This is the URL for the submitted form is the bucket domain name

        LinkedHashMap<String, String> textMap = new LinkedHashMap<String, String>();
        // key
        String objectName = this.objectName;
        textMap.put("key", objectName);
        // Content-Disposition
        textMap.put("Content-Disposition", "attachment;filename="+filepath);
        // OSSAccessKeyId
        textMap.put("OSSAccessKeyId", ossAccessId);
        // policy
        String policy = "{\"expiration\": \"2120-01-01T12:00:00.000Z\",\"conditions\": [[\"content-length-range\", 0, 104857600]]}";
        String encodePolicy = java.util.Base64.getEncoder().encodeToString(policy.getBytes());
        textMap.put("policy", encodePolicy);
        // Signature
        String signaturecom = com.aliyun.oss.common.auth.ServiceSignature.create().computeSignature(ossAccessKey, encodePolicy);
        textMap.put("Signature", signaturecom);

        Map<String, String> fileMap = new HashMap<String, String>();
        fileMap.put("file", filepath);

        String ret = formUpload(urlStr, textMap, fileMap);
        System.out.println("[" + bucket + "] post_object:" + objectName);
        System.out.println("post reponse:" + ret);
    }

    private static String formUpload(String urlStr, Map<String, String> textMap, Map<String, String> fileMap) throws Exception {
        String res = "";
        HttpURLConnection conn = null;
        String BOUNDARY = "9431149156168";
        try {
            URL url = new URL(urlStr);
            conn = (HttpURLConnection) url.openConnection();
            conn.setConnectTimeout(5000);
            conn.setReadTimeout(30000);
            conn.setDoOutput(true);
            conn.setDoInput(true);
            conn.setRequestMethod("POST");
            conn.setRequestProperty("User-Agent",
                    "Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.6)");
            conn.setRequestProperty("Content-Type",
                    "multipart/form-data; boundary=" + BOUNDARY);

            OutputStream out = new DataOutputStream(conn.getOutputStream());
            // text
            if (textMap != null) {
                StringBuffer strBuf = new StringBuffer();
                Iterator iter = textMap.entrySet().iterator();
                int i = 0;
                while (iter.hasNext()) {
                    Map.Entry entry = (Map.Entry) iter.next();
                    String inputName = (String) entry.getKey();
                    String inputValue = (String) entry.getValue();
                    if (inputValue == null) {
                        continue;
                    }
                    if (i == 0) {
                        strBuf.append("--").append(BOUNDARY).append(
                                "\r\n");
                        strBuf.append("Content-Disposition: form-data; name=\""
                                + inputName + "\"\r\n\r\n");
                        strBuf.append(inputValue);
                    } else {
                        strBuf.append("\r\n").append("--").append(BOUNDARY).append(
                                "\r\n");
                        strBuf.append("Content-Disposition: form-data; name=\""
                                + inputName + "\"\r\n\r\n");

                        strBuf.append(inputValue);
                    }

                    i++;
                }
                out.write(strBuf.toString().getBytes());
            }

            // file
            if (fileMap != null) {
                Iterator iter = fileMap.entrySet().iterator();
                while (iter.hasNext()) {
                    Map.Entry entry = (Map.Entry) iter.next();
                    String inputName = (String) entry.getKey();
                    String inputValue = (String) entry.getValue();
                    if (inputValue == null) {
                        continue;
                    }
                    File file = new File(inputValue);
                    String filename = file.getName();
                    String contentType = new MimetypesFileTypeMap().getContentType(file);
                    if (contentType == null || contentType.equals("")) {
                        contentType = "application/octet-stream";
                    }

                    StringBuffer strBuf = new StringBuffer();strBuf.append("\r\n").append("--").append(BOUNDARY).append(
                            "\r\n");
                    strBuf.append("Content-Disposition: form-data; name=\""
                            + inputName + "\"; filename=\"" + filename
                            + "\"\r\n");
                    strBuf.append("Content-Type: " + contentType + "\r\n\r\n");

                    out.write(strBuf.toString().getBytes());

                    DataInputStream in = new DataInputStream(new FileInputStream(file));
                    int bytes = 0;
                    byte[] bufferOut = new byte[1024];
                    while ((bytes = in.read(bufferOut)) != -1) {
                        out.write(bufferOut, 0, bytes);
                    }
                    in.close();
                }
                StringBuffer strBuf = new StringBuffer();
                out.write(strBuf.toString().getBytes());
            }

            byte[] endData = ("\r\n--" + BOUNDARY + "--\r\n").getBytes();
            out.write(endData);
            out.flush();
            out.close();

            // Read the returned data
            StringBuffer strBuf = new StringBuffer();
            BufferedReader reader = new BufferedReader(new InputStreamReader(
                    conn.getInputStream()));
            String line = null;
            while ((line = reader.readLine()) != null) {
                strBuf.append(line).append("\n");
            }
            res = strBuf.toString();
            reader.close();
            reader = null;
        } catch (Exception e) {
            System.err.println("Error in sending a POST request:  " + urlStr);
            throw e;
        } finally {
            if (conn != null) {
                conn.disconnect();
                conn = null;
            }
        }
        return res;
    }

    public static void main(String[] args) throws Exception {
        OssPostObject ossPostObject = new OssPostObject();
        ossPostObject.PostObject();
    }

}

pom.xmlに以下を追加する必要がありますのでご注意ください。

<dependency>
    <groupId>com.aliyun.oss</groupId>
    <artifactId>aliyun-sdk-oss</artifactId>
    <version>2.2.1</version>
</dependency>

結論

Post Objectは、ブラウザに基づいてバケットにファイルをアップロードすることを可能にします。Post Objectのメッセージ本文のエンコードはmultipart/form-dataを利用しています。Post Object の操作では、プログラムはメッセージ本文のフォームフィールドとしてパラメータを転送します。Post Object は、ポリシーの署名を計算するために AccessKeySecret を使用します。POSTフォームフィールドは、パブリックリードライトバケットをアップロードするためのオプションですが、POSTリクエストを制限するためにこのフィールドを使用することをお勧めします。

アリババクラウドは日本に2つのデータセンターを有し、世界で60を超えるアベラビリティーゾーンを有するアジア太平洋地域No.1(2019ガートナー)のクラウドインフラ事業者です。 アリババクラウドの詳細は、こちらからご覧ください。 アリババクラウドジャパン公式ページ