[JAVA] Count the frequency of occurrence of words in a sentence by stream processing (Apache Apex)

From Counting the frequency of occurrence of words in a sentence by stream processing \ (Apache Apex ) -Bad sentence pattern

Former story

Webinar: Building Your First Apache Apex Application - YouTube

Things necessary



$ cat /etc/lsb-release


$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
$ javac -version
javac 1.8.0_131


$ git version
git version 2.7.4


$ mvn --version
Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_131, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en_US, platform encoding: ISO-8859-1
OS name: "linux", version: "4.4.0-57-generic", arch: "amd64", family: "unix"


$ hadoop version
Hadoop 2.8.0
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 91f2b7a13d1e97be65db92ddabc627cc29ac0009
Compiled by jdu on 2017-03-17T04:12Z
Compiled with protoc 2.5.0
From source with checksum 60125541c2b3e266cbf3becc5bda666
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.8.0.jar

git clone

First, clone the required repository. There are three to use: ʻapex-core, ʻapex-malhar, and DataTorrent / examples. Create a ʻapex` directory in your home and work there.


$ mkdir apex
$ cd apex


$ git clone [email protected]:apache/apex-core.git
$ git clone [email protected]:apache/apex-malhar.git
$ git clone [email protected]:DataTorrent/examples.git

Apex installation

Build and install ʻapex-core and ʻapex-malhar with maven. Skip running the test with -DskipTests and it will complete (slightly) faster. It takes time, so wait for a while: coffee:


$ ls              
apex-core   apex-malhar examples
$ cd apex-core


$ mvn clean install -DskipTests


$ cd apex-malhar


$ mvn clean install -DskipTests

Convenient tool

Have the tools ready while you wait for the build to complete. There is a script called ʻaliases in DataTorrent / examples`, so load it and make it available.


$ source examples/tutorials/topnwords/scripts/aliases

The contents of ʻaliases` are as follows.


# bash aliases and functions useful for working on input and out directories

# input and output directories
in=/tmp/test/input-dir out=/tmp/test/output-dir

# list files in input directory
alias ls-input="hdfs dfs -ls $in"

# list files in output directory
alias ls-output="hdfs dfs -ls $out"

# clean input directory
alias clean-input="hdfs dfs -rm $in/*"

# clean output directory
alias clean-output="hdfs dfs -rm $out/*"

# convenient alias to run dtcli from code repository
alias dtcli3="$HOME/src/incubator-apex-core/engine/src/main/scripts/dtcli"

# copy local file (argument) to input directory
function put-file ( ) {
    hdfs dfs -put "$1" "$in"

# make local copy of output file (argument) from output directory
function get-file ( ) {
    hdfs dfs -get "$out/$1" "$1".out

Make it easy to copy and call newapp to ~ / apex.


$ cp examples/tutorials/topnwords/scripts/newapp .

It's just a mvn command, and you can rewrite myapexapp as needed.


# script to create a new project
# change project name and archetype version as needed

mvn -B archetype:generate \
  -DarchetypeGroupId=org.apache.apex \
  -DarchetypeArtifactId=apex-app-archetype \
  -DarchetypeVersion=$version  \
  -DgroupId=com.example \
  -Dpackage=com.example.$name \
  -DartifactId=$name \

Project creation

Run newapp to create a project. When Y :: appears, enter there (Turn!).


$ bash newapp
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
$ ls
apex-core   apex-malhar examples myapexapp newapp
$ cd myapexapp

After creating the project, check the source code.


$ find src -name "*.java"

I don't need a unit test this time, so I'll leave it. This way, if you forget -DskipTests at build time, the tests will not run. This time, I saved it to / tmp, but of course you can delete it.


$ mv src/test/java/com/example/myapexapp/ApplicationTest.java /tmp

Introduced sample code

Introduce the code for DataTorrent / examples. First, move to the source directory.


$ cd src/main/java/com/example/myapexapp

Bring in the source code for tutorial / topnwords. Outputs the frequency of occurrence of each word in the input sentence. Details are omitted.

$ ls   
Application.java           RandomNumberGenerator.java
$ ls ~/apex/examples/tutorials/topnwords/webinar/
ApplicationWordCount.java      WCPair.java                    WordReader.java
FileWordCount.java             WindowWordCount.java           properties-SortedWordCount.xml
LineReader.java                WordCountWriter.java
$ cp ~/apex/examples/tutorials/topnwords/webinar/*.java .
$ ls
Application.java           LineReader.java            WindowWordCount.java
ApplicationWordCount.java  RandomNumberGenerator.java WordCountWriter.java
FileWordCount.java         WCPair.java                WordReader.java

We also need a properties file, so copy that to resources as well.

$ cd ../../../../resources/META-INF 
$ cp ~/apex/examples/tutorials/topnwords/webinar/properties-SortedWordCount.xml .
$ ls
properties-SortedWordCount.xml properties.xml

Build application

Move to the top directory of the application and build. After building, myapexapp-1.0-SNAPSHOT.apa is created in the target directory.

$ cd ~/apex/myapexapp


$ mvn clean package -DskipTests
$ ls target 
antrun                     generated-resources        maven-archiver             site
archive-tmp                generated-sources          maven-status               test-classes
classes                    generated-test-sources     myapexapp-1.0-SNAPSHOT.apa
deps                       javadoc-bundle-options     myapexapp-1.0-SNAPSHOT.jar

Run application

Use the Apex CLI Tools (https://apex.apache.org/docs/apex/apex_cli/#apex-cli-commands) to run the built application. It's under ʻapex-core, so run it in the myapexapp` directory.


$ ../apex-core/engine/src/main/scripts/apex 
Apex CLI 3.7.0-SNAPSHOT 02.06.2017 @ 09:02:30 JST rev: 22feeed branch: master

Then type launch target / myapexapp-1.0-SNAPSHOT.apa and it will be executed. You will be asked if you want to run MyFirstApplication or SortedWordCount, so if you choose 2, the application will run and ʻappId` will be issued.

apex> launch target/myapexapp-1.0-SNAPSHOT.apa
  1. MyFirstApplication
  2. SortedWordCount
Choose application: 2
{"appId": "application_1496704660177_0001"}
apex (application_1496704660177_0001) > 

Operation check

First, create an I / O directory with hdfs dfs.

$ hdfs dfs -mkdir -p /tmp/test/input-dir
$ hdfs dfs -mkdir -p /tmp/test/output-dir

There is nothing yet at this point.

$ ls-input
...Nothing is displayed
$ ls-output
...Nothing is displayed

Put a text file in the input directory using put-file defined in ʻaliaces. This time, I created an English sentence at https://www.dummytextgenerator.com and saved it as dummy.txt`.

$ ls -lh ~/*.txt
$ put-file ~/dummy.txt

If you do ls-input, you will find the same text file as before.

$ ls-input
Found 1 items
-rw-r--r--   1 vagrant supergroup      15167 2017-06-06 04:34 /tmp/test/input-dir/dummy.txt

When you do ls-output, the output file is created, so check the contents.

$ ls-output
Found 1 items
-rwxrwxrwx   1 vagrant supergroup       3028 2017-06-06 04:34 /tmp/test/output-dir/dummy.txt
$ hdfs dfs -cat /tmp/test/output-dir/dummy.txt | head
t : 75
you : 43
re : 29
sixth : 22
seasons : 22
creature : 22
every : 22
moved : 20
together : 20
subdue : 20

The frequency of appearance of words was output properly!

Recommended Posts

Count the frequency of occurrence of words in a sentence by stream processing (Apache Apex)
Counting the frequency of occurrence of words in sentences by stream processing (Apache Apex) Part 2 Coding
Count the number of occurrences of a string in Ruby
A program that counts the number of words in a List
Order of processing in the program
Call a method of the parent class by explicitly specifying the name in Ruby
Fall 2017 Security Specialist I checked the frequency of words that appeared in the morning 2
Measure the size of a folder in Java
[Java] Integer information of characters in a text file acquired by the read () method
Write a test by implementing the story of Mr. Nabeats in the world with ruby
A review of the code used by rails beginners
Format of the log output by Tomcat itself in Tomcat 8
About the problem of deadlock in parallel processing in gem'sprockets' 4.0
Image processing: The basic structure of the image read by the program
Addition of variables in iterative processing in a while statement
Branch processing by the return value of RestTemplate and the status code of ResponseEntity in Spring Boot