[JAVA] I tried to make a web application that searches tweets with vue-word cloud and examines the tendency of what is written in the associated profile

It is the one that you often see.

I will show you the result first for those who say what it is.

When you search for a tweet by search word, the word of the profile of the person who tweeted the hit is displayed in the tag cloud of Word Cloud.

Characters that appear frequently will be larger.

It seems easy to make, but when I look it up, it's mostly Python.

Right now, I'm developing with Vue.js and Java (Spring Boot), so I thought I could make it with that, so I made a little web application.

This time, it's not the content of the tweet, but the content of the profile is pulled from the content of the tweet, so that may be a little tricky.

Web application creation days: 2 days Qiita article creation: 2 days

Qiita article creation is more difficult.

Why did the title end up so long?

The flow of operation of the web application is like this.

It feels like splitting a sentence into words, but I don't know much more, so I'm sure you'll be more familiar with it.

2. Environment

Vue.js：2.6.12 Spring Boot：v2.3.3 Java：AdoptOpenJDK：11.0.6 Kuromoji：0.9.0 vue-wordcloud：1.1.1 axios：0.20.0

Vue.js is the front and Spring Boot is the server-side API without any discipline.

The rough directory structure looks like this.

┠ src
┃  ┗ main
┃     ┗ java/app/myapp
┃        ┠ common
┃        ┃  ┗ HttpAccess.java
┃        ┠ controller
┃        ┃  ┗ TwitterAnalysisRestController.java
┃        ┠ entity
┃        ┃  ┗ VueWordCloudEntity.java
┃        ┠ service
┃        ┃  ┗ TwitterAnalysisService.java
┃ Various
┃
┠ web
┃ ┠ Various
┃  ┠ src
┃ ┃ ┠ Various
┃  ┃  ┠ router
┃　┃  ┃  ┗ index.js
┃  ┃  ┠ views
┃  ┃  ┃  ┗ TwitterAnalysis.vue
┃  ┃  ┃
┃ ┃ Various
┃ Various
┃
┠ build.gradle
various

3. Procedure

3-1. Premise

Spring Boot project has been created.

3-2. Twitter API usage application

First, you need to apply for Twitter API usage on the Twitter side.

It is assumed that you have a Twitter account.

There is a site that kindly explains, so please refer to this when applying for use. Detailed explanation from the example sentence of the 2020 version Twitter API usage application to the acquisition of the API key

The information changes from day to day, so it may have changed, but it should probably be easier.

All you have to do is answer a few questions from Twitter.

By all means, the application screen is forced to write in English, but in fact it may be in Japanese ...

When I answered the question, I thought I was waiting for approval ... but that wasn't the case, and I was able to immediately go to the token (Bearer Token) display screen.

All you have to do is copy this token and the key.

It seems that it took several days to wait for approval a while ago.

3-3. Java implementation

First, get profile information using Twitter's API.

It seems convenient to use a library called Twitter4J, but I haven't used it this time.

There is no particular reason.

The API URL looks like this.

The API seems to be new recently.

https://api.twitter.com/2/tweets/search/recent?expansions=author_id&user.fields=description&max_results=100&query=<Search keyword>

For details, see Official Site

So, it is like this to get information using API.

For the token, specify the Bearer Token obtained above.

TwitterAnalysisService.java

String urlString = "https://api.twitter.com/2/tweets/search/recent?expansions=author_id&user.fields=description&max_results=100";

String method = "GET";
String bearerToken = <token>;

HttpAccess httpAccess = new HttpAccess();
String response = httpAccess.requestHttp(urlString, method, bearerToken, <Search word>);

HttpAccess.java

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;

public class HttpAccess {

    public String requestHttp(String urlString, String method, String bearerToken, String query) {

        String result = null;
        String urlStringFull = null;

        if (query == null || query.isEmpty()) return result;

        try {
            urlStringFull = urlString + "&query=" + URLEncoder.encode(query, "UTF-8");
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        }

        if (urlStringFull == null) return result;
        
        HttpURLConnection  urlConn = null;
        InputStream in = null;
        BufferedReader reader = null;

        try {

            URL url = new URL(urlStringFull);

            urlConn = (HttpURLConnection) url.openConnection();
            urlConn.setRequestMethod(method);
            urlConn.setRequestProperty("Authorization","Bearer " + bearerToken);
            urlConn.connect();

            int status = urlConn.getResponseCode();
            if (status == HttpURLConnection.HTTP_OK) {

                in = urlConn.getInputStream();
                    
                reader = new BufferedReader(new InputStreamReader(in));
                
                StringBuilder output = new StringBuilder();
                String line;
                
                while ((line = reader.readLine()) != null) {
                    output.append(line);
                }

                result = output.toString();
            }
        } catch (IOException e) {
            e.printStackTrace();
		} finally {
			try {
				if (reader != null) {
					reader.close();
				}
				if (urlConn != null) {
					urlConn.disconnect();
				}
			} catch (IOException e) {
                e.printStackTrace();
			}
        }
        
        return result;
    }
}

Next is morphological analysis. If you want to do it in Java, Kuromoji is convenient.

Just add a line to your build.gradle file.

build.gradle

dependencies {
~ Abbreviation ~
    implementation 'com.atilika.kuromoji:kuromoji-ipadic:0.9.0'
~ Abbreviation ~
}

Also, as for what format to return the data, since vue-wordcloud handles an array of objects with "name" and "value" as keys in the view, "name" and "value" are used as keys. Returns an array of objects in JSON.

• The key name can be changed

Enter the word as a result of morphological analysis in "name" and the number of times the word appears in "value".

This time, assuming that only nouns are targeted, in addition to accessing the above API, morphological analysis was performed, and an object with "name" and "value" in the fields was created as follows.

Use the Tokenizer class for morphological analysis.

TwitterAnalysisService.java

import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;

import com.atilika.kuromoji.ipadic.Token;
import com.atilika.kuromoji.ipadic.Tokenizer;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import app.myapp.common.HttpAccess;
import app.myapp.entity.VueWordCloudEntity;

@Service
public class TwitterAnalysisService {

    public List<VueWordCloudEntity> analysisTwitterProfileByQuery(String query) {

        List<VueWordCloudEntity> result = new ArrayList<>();

        String urlString = "https://api.twitter.com/2/tweets/search/recent?expansions=author_id&user.fields=description&max_results=100";

        String method = "GET";
        String bearerToken = <token>;

        HttpAccess httpAccess = new HttpAccess();
        String response = httpAccess.requestHttp(urlString, method, bearerToken, query);

        if (response == null) return result;

        ObjectMapper mapper = new ObjectMapper();
        JsonNode root = null;

        try {
            root = mapper.readTree(response);
        } catch (JsonProcessingException e) {
            e.printStackTrace();
        }

        if (root == null || root.get("meta").get("result_count").asInt() == 0) {
            return result;
        }

        Tokenizer tokenizer = new Tokenizer();
        List<Token> tokens = new ArrayList<>();

        JsonNode users = root.get("includes").get("users");

        for(int i = 0; i < users.size(); i++) {
            if (users.get(i).get("description") != null) {
                tokens.addAll(tokenizer.tokenize(users.get(i).get("description").asText()));
            }
        }

        List<VueWordCloudEntity> vueWordCloudEntityList = new ArrayList<>();
        tokens.stream()
                .filter(x -> x.getPartOfSpeechLevel1().equals("noun"))
                .map(x -> x.getSurface())
                .collect(Collectors.groupingBy(x -> x, Collectors.counting()))
                .forEach((k, v) -> {
                    VueWordCloudEntity vueWordCloudEntity = new VueWordCloudEntity();
                    vueWordCloudEntity.setName(k);
                    vueWordCloudEntity.setValue(v);
                    vueWordCloudEntityList.add(vueWordCloudEntity);
                });

        result = vueWordCloudEntityList;
            
        return result;
    }
}

VueWordCloudEntity.java

import lombok.Data;

@Data
public class VueWordCloudEntity {

    private String name;
    private Long value;
}

TwitterAnalysisRestController.java

import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import lombok.RequiredArgsConstructor;
import app.myapp.entity.VueWordCloudEntity;
import app.myapp.service.TwitterAnalysisService;

@RestController
@RequiredArgsConstructor
@RequestMapping("/api")
public class TwitterAnalysisRestController {

    @Autowired
    private TwitterAnalysisService twitterAnalysisService;

    @GetMapping("/twitter/analysis/profile/by/{query}")
    public ResponseEntity<List<VueWordCloudEntity>> analysisTwitterProfileByQuery(
            @PathVariable String query) {

        List<VueWordCloudEntity> result
                = twitterAnalysisService.analysisTwitterProfileByQuery(query);

        //CORS compatible here for test app
        HttpHeaders headers = new HttpHeaders();
        headers.add("Access-Control-Allow-Credentials", "true");
        headers.add("Access-Control-Allow-Origin", "http://localhost:<Vue.js execution environment port number>");

        return new ResponseEntity<>(result, headers, HttpStatus.OK);
    }
}

I'm sure there is an easier way if you don't use Twitter4J in Java!

I'm sure there is such a sample somewhere.

Unfortunately, this time it was my sample.

• Make an excuse that the source code has priority to "work" ...

3-4. Implementation of Vue.js

This is easy because you just install vue-wordcloud and copy and paste the official source of vue-wordcloud.

Create a project with vue-cli GUI and place it in the above directory structure location.

Then install vue-word cloud with vue-cli GUI.

Oh, and axios too.

After that, copy and paste the official sample, fix it a little and get the data from the Spring Boot API.

TwitterAnalysis.vue

<template>
    <div id="twitter-analysis">
        <input v-model="query" placeholder="Please enter a keyword" style="width:400px;">
        <button @click="analyzeProfile" style="margin-left:10px;">analysis</button>
        <wordcloud
            :data="analyzedWords"
            nameKey="name"
            valueKey="value"
            color="Accent"
            :showTooltip="true"
            :wordClick="wordClickHandler">
        </wordcloud>
    </div>
</template>

<script>
import wordcloud from 'vue-wordcloud'
import axios from 'axios'
axios.defaults.withCredentials = true

export default {
    name: 'TwitterAnalysis',
    components: {
        wordcloud
    },
    data() {
        return {
            query: '',
            analyzedWords: [],
        }
    },
    methods: {
        wordClickHandler(name, value, vm) {
            console.log('wordClickHandler', name, value, vm);
        },
        analyzeProfile: async function () {
            if (this.query == null || this.query === '') return
            await axios.get('http://localhost:<Spring Boot execution environment port number>/api/twitter/analysis/profile/by/'
                    + encodeURIComponent(this.query))
                .then(res => {
                    if (res.data != null) {
                        this.analyzedWords = res.data
                    }
                })
                .catch(err => {
                    alert(err + 'This is an error.')
                })
        },
    },
}
</script>

router/index.js

import Vue from 'vue'
import VueRouter from 'vue-router'

Vue.use(VueRouter)

const routes = [
  {
    path: '/',
    name: 'TwitterAnalysis',
    component: () => import(/* webpackChunkName: "twitteranalysis" */ '../views/TwitterAnalysis.vue')
  },
]

const router = new VueRouter({
  mode: 'history',
  routes
})

export default router

with this http: // localhost: <Vue.js execution environment port number> / To access.

When you enter a search word and press the analysis button

success! Should do.

If you look at it, there are words that don't matter, such as slash and http, so I'm sure you'll exclude these.

I was wondering if this app could show some interesting trends, but I don't really understand!

In the first place, it feels like it is.

It seems that you need to study a lot ...