Create variable length binary data in Java

This article

This is a memo that made a Java program that converts variable length data (with line feed code) to variable length data with length field (LL) (without line feed code).


I mainly develop Java (Spring Boot) web applications.


The source code and input files can be found on GitHub.


――I was in charge of data migration tool development for a legacy migration project that I was involved in in my work. --Migration source is IMS (hierarchical DB) segment data --The migration destination is a PostgreSQL bytea type column ――It was difficult to create test data with a text editor, so I created a program.

Test data specifications

--Variable length binary data file with length field (LL) 2 bytes. --The contents of the binary data can be anything. --Strictly speaking, the data coming from the source IMS is a binary representation of the character code EBCDIC, but the contents of the data are not text-processed within the migration tool. ――So, I am not particular about the contents of the data for testing the migration tool. I will use the UTF-8 text file, which is easy to handle in Java text input processing, as the source material.

Preparation of original material file

--Prepare a UTF-8 variable length text file as the source material.

Source code explanation

    public static void main(String[] args) {
        // (1)
        BufferedReader br = null;
        BufferedOutputStream bos = null;
        int LLSIZE = 2; 

        try {
            br = new BufferedReader(new FileReader("input.txt"));
            bos = new BufferedOutputStream(new FileOutputStream("output.txt"));
            String line = null;
            // (2)
            while ((line = br.readLine()) != null) {
                int recSize = line.length();
                // (3)
                byte ll[] = ByteBuffer.allocate(LLSIZE).putShort((short)recSize).array();
                // (4)
                byte data[] = ByteBuffer.allocate(recSize).put(line.getBytes("UTF-8")).array();
                // (5)
                ByteBuffer lineBuf = ByteBuffer.allocate(LLSIZE + recSize);
                // (6)

-(1) Input stream uses BufferedReader and output stream uses BufferedOutputStream. --Process the input file as text data and the output file as binary data. -(2) Read one record at a time with BufferedReader (readLine). -(3) The record size read by readLine is stored in the length field LL as byte data. --LL will not include its own size (2 bytes) --recSize is an int type (4 bytes) and does not fit in LL (2 bytes), so cast it to a short type (2 bytes). -(4) Holds the record data read by readLine as UTF-8 byte data. -(5) Reserve an area with ByteBuffer to combine LL and record data into one record, and set LL and record data. -(6) Output the binary array of LL and record data to a file.

Compile and run

Compiling and running is simple.




java EditBinary

Execution result

Let's take a look at the contents of output.txt with "Hex Fiend". result.png The line feed code at the end of each record has been removed and replaced with an LL at the beginning. You can see that the value of LL is as follows. The number in parentheses is the number of bytes in decimal, that is, the record length.

--LL of the first record: 0x0064 (100) ――Second record LL: 0x00C8 (200) --Third record LL: 0x012C (300) ―― 4th record LL: 0x0190 (400) ―― 5th record LL: 0x01F4 (500)

So, I was able to create variable length binary data with LL.


Newline code as a delimiter

――I don't think there are many cases where a line feed code is used as a delimiter for binary data (not used in this article), but if you do, you need to be careful because the binary representation may change depending on the environment. It is a story.

--The line feed code is CRLF on Win and LF on Unix / Linux. --The binary (hexagonal) representation of CRLF is (0x0D 0x0A) --The binary (hexary) representation of LF is (0x0A) ――It seems that it is 2 bytes or 4 bytes depending on the character code.


--Input message format and content -Line feed code

Recommended Posts

Create variable length binary data in Java
Create JSON in Java
Read binary files in Java 1
Create hyperlinks in Java PowerPoint
Importing Excel data in Java 2
How to create a data URI (base64) in Java
Create Azure Functions in Java
Import Excel data in Java
Read binary files in Java 2
Importing Excel data in Java 3
Display Firestore data in RecyclerView [Java]
TCP communication (socket communication) in Java (ASCII, Binary)
Create Java Spring Boot project in IntelliJ
Create a TODO app in Java 7 Create Header
Create barcodes and QR codes in Java PDF
java array variable
Create a CSR with extended information in Java
Partization in Java
[Kotlin] Pass Kotlin Array as Java variable length argument [Java]
Try to create a bulletin board in Java
Changes in Java 11
About var used in Java (Local Variable Type)
[Java] Use of final in local variable declaration
Rock-paper-scissors in Java
Let's create a super-simple web framework in Java
Java variable scope (scope)
java variable declaration
Java variable scope
Organized memo in the head (Java --Data type)
I tried to create Alexa skill in Java
Pi in Java
FizzBuzz in Java
How to create a Java environment in just 3 seconds
Java variable declaration, initialization, data type (cast and promotion)
Vectorize and image MNIST handwritten digit image data in Java
I tried to create a Clova skill in Java
Make the where clause variable in Spring Data JPA
[Android] Variable of Data Binding is written in camelCase
Differences in code when using the length system in Java
[java] sort in list
Read JSON in Java
Interpreter implementation in Java
Make Blackjack in Java
Rock-paper-scissors app in Java
Constraint programming in Java
Put java8 in centos7
NVL-ish guy in Java
Combine arrays in Java
"Hello World" in Java
Callable Interface in Java
[Java] Create a filter
Comments in Java source
Ruby: Create CouchDB data (Create)
[Java] Data type ①-Basic type
Azure functions in java
Format XML in Java
Simple htmlspecialchars in Java
Regarding Java variable usage
Boyer-Moore implementation in Java
Hello World in Java
Use OpenCV in Java