[JAVA] I tried to make a web application that searches tweets with vue-word cloud and examines the tendency of what is written in the associated profile

It is the one that you often see.

I will show you the result first for those who say what it is.

図1.PNG

When you search for a tweet by search word, the word of the profile of the person who tweeted the hit is displayed in the tag cloud of Word Cloud.

Characters that appear frequently will be larger.

It seems easy to make, but when I look it up, it's mostly Python.

Right now, I'm developing with Vue.js and Java (Spring Boot), so I thought I could make it with that, so I made a little web application.

This time, it's not the content of the tweet, but the content of the profile is pulled from the content of the tweet, so that may be a little tricky.

Web application creation days: 2 days Qiita article creation: 2 days

Qiita article creation is more difficult.

Why did the title end up so long?

The flow of operation of the web application is like this.


Get profile information of tweets with Twitter API ↓ Morphological analysis of profile information ↓ Visualization
What is morphological analysis?

It feels like splitting a sentence into words, but I don't know much more, so I'm sure you'll be more familiar with it.

2. Environment

Vue.js:2.6.12 Spring Boot:v2.3.3 Java:AdoptOpenJDK:11.0.6 Kuromoji:0.9.0 vue-wordcloud:1.1.1 axios:0.20.0

Vue.js is the front and Spring Boot is the server-side API without any discipline.

The rough directory structure looks like this.

┠ src
┃  ┗ main
┃     ┗ java/app/myapp
┃        ┠ common
┃        ┃  ┗ HttpAccess.java
┃        ┠ controller
┃        ┃  ┗ TwitterAnalysisRestController.java
┃        ┠ entity
┃        ┃  ┗ VueWordCloudEntity.java
┃        ┠ service
┃        ┃  ┗ TwitterAnalysisService.java
┃ Various
┃
┠ web
┃ ┠ Various
┃  ┠ src
┃ ┃ ┠ Various
┃  ┃  ┠ router
┃ ┃  ┃  ┗ index.js
┃  ┃  ┠ views
┃  ┃  ┃  ┗ TwitterAnalysis.vue
┃  ┃  ┃
┃ ┃ Various
┃ Various
┃
┠ build.gradle
various

3. Procedure

3-1. Premise

Spring Boot project has been created.

3-2. Twitter API usage application

First, you need to apply for Twitter API usage on the Twitter side.

It is assumed that you have a Twitter account.

There is a site that kindly explains, so please refer to this when applying for use. Detailed explanation from the example sentence of the 2020 version Twitter API usage application to the acquisition of the API key

The information changes from day to day, so it may have changed, but it should probably be easier.

All you have to do is answer a few questions from Twitter.

By all means, the application screen is forced to write in English, but in fact it may be in Japanese ...

When I answered the question, I thought I was waiting for approval ... but that wasn't the case, and I was able to immediately go to the token (Bearer Token) display screen.

All you have to do is copy this token and the key.

It seems that it took several days to wait for approval a while ago.

3-3. Java implementation

First, get profile information using Twitter's API.

It seems convenient to use a library called Twitter4J, but I haven't used it this time.

There is no particular reason.

The API URL looks like this.

The API seems to be new recently.

https://api.twitter.com/2/tweets/search/recent?expansions=author_id&user.fields=description&max_results=100&query=<Search keyword>
Parameters used this time Contents
expansions=author_id Specified to get information about the person who made the tweet at the same time as the text of the tweet
user.fields=description Specified to include the profile text of the person who tweeted in the information to be acquired
max_results=100 Maximum number of acquisitions
query= ツイートに対するSearch word

For details, see Official Site

So, it is like this to get information using API.

For the token, specify the Bearer Token obtained above.

TwitterAnalysisService.java


String urlString = "https://api.twitter.com/2/tweets/search/recent?expansions=author_id&user.fields=description&max_results=100";

String method = "GET";
String bearerToken = <token>;

HttpAccess httpAccess = new HttpAccess();
String response = httpAccess.requestHttp(urlString, method, bearerToken, <Search word>);

HttpAccess.java


import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;

public class HttpAccess {

    public String requestHttp(String urlString, String method, String bearerToken, String query) {

        String result = null;
        String urlStringFull = null;

        if (query == null || query.isEmpty()) return result;

        try {
            urlStringFull = urlString + "&query=" + URLEncoder.encode(query, "UTF-8");
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        }

        if (urlStringFull == null) return result;
        
        HttpURLConnection  urlConn = null;
        InputStream in = null;
        BufferedReader reader = null;

        try {

            URL url = new URL(urlStringFull);

            urlConn = (HttpURLConnection) url.openConnection();
            urlConn.setRequestMethod(method);
            urlConn.setRequestProperty("Authorization","Bearer " + bearerToken);
            urlConn.connect();

            int status = urlConn.getResponseCode();
            if (status == HttpURLConnection.HTTP_OK) {

                in = urlConn.getInputStream();
                    
                reader = new BufferedReader(new InputStreamReader(in));
                
                StringBuilder output = new StringBuilder();
                String line;
                
                while ((line = reader.readLine()) != null) {
                    output.append(line);
                }

                result = output.toString();
            }
        } catch (IOException e) {
            e.printStackTrace();
		} finally {
			try {
				if (reader != null) {
					reader.close();
				}
				if (urlConn != null) {
					urlConn.disconnect();
				}
			} catch (IOException e) {
                e.printStackTrace();
			}
        }
        
        return result;
    }
}

Next is morphological analysis. If you want to do it in Java, Kuromoji is convenient.

Just add a line to your build.gradle file.

build.gradle


dependencies {
~ Abbreviation ~
    implementation 'com.atilika.kuromoji:kuromoji-ipadic:0.9.0'
~ Abbreviation ~
}

Also, as for what format to return the data, since vue-wordcloud handles an array of objects with "name" and "value" as keys in the view, "name" and "value" are used as keys. Returns an array of objects in JSON.

Enter the word as a result of morphological analysis in "name" and the number of times the word appears in "value".

This time, assuming that only nouns are targeted, in addition to accessing the above API, morphological analysis was performed, and an object with "name" and "value" in the fields was created as follows.

Use the Tokenizer class for morphological analysis.

TwitterAnalysisService.java


import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;

import com.atilika.kuromoji.ipadic.Token;
import com.atilika.kuromoji.ipadic.Tokenizer;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import app.myapp.common.HttpAccess;
import app.myapp.entity.VueWordCloudEntity;

@Service
public class TwitterAnalysisService {

    public List<VueWordCloudEntity> analysisTwitterProfileByQuery(String query) {

        List<VueWordCloudEntity> result = new ArrayList<>();

        String urlString = "https://api.twitter.com/2/tweets/search/recent?expansions=author_id&user.fields=description&max_results=100";

        String method = "GET";
        String bearerToken = <token>;

        HttpAccess httpAccess = new HttpAccess();
        String response = httpAccess.requestHttp(urlString, method, bearerToken, query);

        if (response == null) return result;

        ObjectMapper mapper = new ObjectMapper();
        JsonNode root = null;

        try {
            root = mapper.readTree(response);
        } catch (JsonProcessingException e) {
            e.printStackTrace();
        }

        if (root == null || root.get("meta").get("result_count").asInt() == 0) {
            return result;
        }

        Tokenizer tokenizer = new Tokenizer();
        List<Token> tokens = new ArrayList<>();

        JsonNode users = root.get("includes").get("users");

        for(int i = 0; i < users.size(); i++) {
            if (users.get(i).get("description") != null) {
                tokens.addAll(tokenizer.tokenize(users.get(i).get("description").asText()));
            }
        }

        List<VueWordCloudEntity> vueWordCloudEntityList = new ArrayList<>();
        tokens.stream()
                .filter(x -> x.getPartOfSpeechLevel1().equals("noun"))
                .map(x -> x.getSurface())
                .collect(Collectors.groupingBy(x -> x, Collectors.counting()))
                .forEach((k, v) -> {
                    VueWordCloudEntity vueWordCloudEntity = new VueWordCloudEntity();
                    vueWordCloudEntity.setName(k);
                    vueWordCloudEntity.setValue(v);
                    vueWordCloudEntityList.add(vueWordCloudEntity);
                });

        result = vueWordCloudEntityList;
            
        return result;
    }
}

VueWordCloudEntity.java


import lombok.Data;

@Data
public class VueWordCloudEntity {

    private String name;
    private Long value;
}

TwitterAnalysisRestController.java


import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import lombok.RequiredArgsConstructor;
import app.myapp.entity.VueWordCloudEntity;
import app.myapp.service.TwitterAnalysisService;

@RestController
@RequiredArgsConstructor
@RequestMapping("/api")
public class TwitterAnalysisRestController {

    @Autowired
    private TwitterAnalysisService twitterAnalysisService;

    @GetMapping("/twitter/analysis/profile/by/{query}")
    public ResponseEntity<List<VueWordCloudEntity>> analysisTwitterProfileByQuery(
            @PathVariable String query) {

        List<VueWordCloudEntity> result
                = twitterAnalysisService.analysisTwitterProfileByQuery(query);

        //CORS compatible here for test app
        HttpHeaders headers = new HttpHeaders();
        headers.add("Access-Control-Allow-Credentials", "true");
        headers.add("Access-Control-Allow-Origin", "http://localhost:<Vue.js execution environment port number>");

        return new ResponseEntity<>(result, headers, HttpStatus.OK);
    }
}

I'm sure there is an easier way if you don't use Twitter4J in Java!

I'm sure there is such a sample somewhere.

Unfortunately, this time it was my sample.

3-4. Implementation of Vue.js

This is easy because you just install vue-wordcloud and copy and paste the official source of vue-wordcloud.

Create a project with vue-cli GUI and place it in the above directory structure location.

Then install vue-word cloud with vue-cli GUI.

Oh, and axios too.

After that, copy and paste the official sample, fix it a little and get the data from the Spring Boot API.

TwitterAnalysis.vue


<template>
    <div id="twitter-analysis">
        <input v-model="query" placeholder="Please enter a keyword" style="width:400px;">
        <button @click="analyzeProfile" style="margin-left:10px;">analysis</button>
        <wordcloud
            :data="analyzedWords"
            nameKey="name"
            valueKey="value"
            color="Accent"
            :showTooltip="true"
            :wordClick="wordClickHandler">
        </wordcloud>
    </div>
</template>

<script>
import wordcloud from 'vue-wordcloud'
import axios from 'axios'
axios.defaults.withCredentials = true

export default {
    name: 'TwitterAnalysis',
    components: {
        wordcloud
    },
    data() {
        return {
            query: '',
            analyzedWords: [],
        }
    },
    methods: {
        wordClickHandler(name, value, vm) {
            console.log('wordClickHandler', name, value, vm);
        },
        analyzeProfile: async function () {
            if (this.query == null || this.query === '') return
            await axios.get('http://localhost:<Spring Boot execution environment port number>/api/twitter/analysis/profile/by/'
                    + encodeURIComponent(this.query))
                .then(res => {
                    if (res.data != null) {
                        this.analyzedWords = res.data
                    }
                })
                .catch(err => {
                    alert(err + 'This is an error.')
                })
        },
    },
}
</script>

router/index.js


import Vue from 'vue'
import VueRouter from 'vue-router'

Vue.use(VueRouter)

const routes = [
  {
    path: '/',
    name: 'TwitterAnalysis',
    component: () => import(/* webpackChunkName: "twitteranalysis" */ '../views/TwitterAnalysis.vue')
  },
]

const router = new VueRouter({
  mode: 'history',
  routes
})

export default router

with this http: // localhost: <Vue.js execution environment port number> / To access.

図2.PNG

When you enter a search word and press the analysis button

図3.PNG

success! Should do.

If you look at it, there are words that don't matter, such as slash and http, so I'm sure you'll exclude these.

I was wondering if this app could show some interesting trends, but I don't really understand!

In the first place, it feels like it is.

It seems that you need to study a lot ...

Recommended Posts

I tried to make a web application that searches tweets with vue-word cloud and examines the tendency of what is written in the associated profile
I tried to make a program that searches for the target class from the process that is overloaded with Java
I tried to clone a web application full of bugs with Spring Boot
I tried to implement a buggy web application in Kotlin
I tried to make a client of RESAS-API in Java
I tried to develop a web application from a month and a half of programming learning history
I tried to develop the cache function of Application Container Cloud Service in the local environment
I tried to express the result of before and after of Date class with a number line
I tried to make the "Select File" button of the sample application created in the Rails tutorial cool
I tried to make a machine learning application with Dash (+ Docker) part2 ~ Basic way of writing Dash ~
I tried to make a machine learning application with Dash (+ Docker) part1 ~ Environment construction and operation check ~
I tried to make a sample program using the problem of database specialist in Domain Driven Design
I tried to make a parent class of a value object in Ruby
[iOS] I tried to make a processing application like Instagram with Swift
I tried to build a Firebase application development environment with Docker in 2020
I tried to make a talk application in Java using AI "A3RT"
I tried to measure and compare the speed of GraalVM with JMH
Ubuntu 18 is the OS that adds a NIC to a server instance in Sakura's cloud and assigns a local IP.
Let's create a TODO application in Java 2 I want to create a template with Spring Initializr and make a Hello world
I tried to make a machine learning application with Dash (+ Docker) part3 ~ Practice ~
What I tried when I wanted to get all the fields of a bean
I tried to express the phone number (landline / mobile phone) with a regular expression in Rails and write validation and test
I want to get the information of the class that inherits the UserDetails class of the user who is logged in with Spring Boot.
I tried to make a product price comparison tool of Amazon around the world with Java, Amazon Product Advertising API, Currency API (2017/01/29)
[Ruby] I want to make a program that displays today's day of the week!
I tried to make a login function in Java
I tried to make FizzBuzz that is uselessly flexible
I tried to explain what you can do in a popular language for web development from a beginner's point of view.
What is the difference between the responsibilities of the domain layer and the application layer in the onion architecture [DDD]
[Java] I tried to make a rock-paper-scissors game that beginners can run on the console.
[Rails] How to temporarily save the request URL of a user who is not logged in and return to that URL after logging in
A program that searches for a character string, and when the search character string is found, displays the character string from the beginning of the line to just before the search character string.
I tried to make an application in 3 months from inexperienced
I tried to modernize a Java EE application with OpenShift.
I tried to summarize the basics of kotlin and java
Determine that the value is a multiple of 〇 in Ruby
I tried to verify this and that of Spring @ Transactional
I want to make a list with kotlin and java!
I want to make a function with kotlin and java!
I tried JAX-RS and made a note of the procedure
[Rails] Implementation of multi-layer category function using ancestry "I tried to make a window with Bootstrap 3"
Since the Rspec command is troublesome, I tried to make it possible to execute Rspec with one Rake command
After learning Progate, I tried to make an SNS application using Rails in the local environment
I tried to take a look at the flow of Android development environment construction with Android Studio
A note of what I stumbled upon and noticed in catching up with Laravel from Rails
Graph the sensor information of Raspberry Pi in Java and check it with a web browser
I tried to solve the problem of "multi-stage selection" with Ruby
I tried to summarize the words that I often see in docker-compose.yml
I tried to illuminate the Christmas tree in a life game
I tried running a letter of credit transaction application with Corda 1
I tried to build the environment of PlantUML Server with Docker
I tried using the cache function of Application Container Cloud Service
I tried to make an Android application with MVC now (Java)
I tried to check the operation of gRPC server with grpcurl
A story that struggled with the introduction of Web Apple Pay
I tried to summarize the methods of Java String and StringBuilder
How to identify the path that is easy to make a mistake
I tried to make Numeron which is not good in Ruby
I tried to make a group function (bulletin board) with Rails
Creating an ArrayList that allows you to throw in and retrieve the coordinates of a two-dimensional plane
I tried to solve the past 10 questions that should be solved after registering with AtCoder in Java