[Elasticsearch × Java] Since the execution result of the query acquired in Java was different from the assumption, I investigated → Summary of the corresponding contents

When I searched using Elasticsearch in Java, the result was not as expected, so I investigated → responded. I will summarize the contents.

• Some names have been rewritten as alternatives because they also serve as a reminder of the content that occurred during work. We would appreciate your understanding.

environment

• OS: Windows10
• Java: 11（Amazon Corretto 11）
• IDE: Eclipse 2020-03
• Elasticsearch: 7.4.2
• Kibana: 7.4.2

Investigation

Source

--Index definition --Java source code --Execution query in Kibana (equivalent to Java source code) Are as follows.

In the index definition, in addition to the mappings definition, analyzer is also set as follows. The field name positioning is as follows.

Index definition

{
  "settings": {
    "analysis": {
      "filter": {
        "my_ngram": {
          "type": "ngram",
          "min_gram": 1,
          "max_gram": 2
        }
      },
      "analyzer": {
        "my_kuromoji_analyzer": {
          "type": "custom",
          "tokenizer": "kuromoji_tokenizer",
          "char_filter": [
            "icu_normalizer",
            "kuromoji_iteration_mark"
          ],
          "filter": [
            "kuromoji_stemmer",
            "my_ngram"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "itemId": {
        "type": "integer"
      },
      "itemName": {
        "type": "text",
        "analyzer": "my_kuromoji_analyzer"
      },
      "itemNameKana": {
        "type": "text",
        "analyzer": "my_kuromoji_analyzer"
      },
      "itemNameHira": {
        "type": "text",
        "analyzer": "my_kuromoji_analyzer"
      }
    }
  }
}

As a process on the Java side, the entered search word is searched for matching any of the product name, product name (katakana), and product name (hiragana) fields, and sorted in descending order of score. It is the content.

Java source code

	/**
	 *Product Search
	 *
	 * @param keyword search word
	 * @param index index name
	 * @param limit number
	 * @param client Elasticsearch connection client
	 * @return search results
	 * @throws IOException
	 */
	public SearchResponse search(String keyword, String index, int limit, RestHighLevelClient client) throws IOException{

		//Initialize search conditions
		SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

		BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();

		boolQueryBuilder.should(QueryBuilders.matchQuery("itemName", keyword))
						.should(QueryBuilders.matchQuery("itemNameKana", keyword))
						.should(QueryBuilders.matchQuery("itemNameHira", keyword));

		searchSourceBuilder.query(boolQueryBuilder);
		//Sort order setting (sort by score)
		searchSourceBuilder.sort(new FieldSortBuilder("_score").order(SortOrder.DESC));
		//Set the number of returns
		searchBuilder.size(limit);

		SearchRequest request = new SearchRequest(index).source(searchSourceBuilder);

		return client.search(request, RequestOptions.DEFAULT);


	}

Finally, execute query in Kibana. Enter the word you want to search in the place where "Search word" is entered and search. Also, specify 5 for size (limit in Java source code).

Execution query in Kibana

POST item_list/_search
{
  "from": 0,
  "size": 5,
  "sort": {
    "_score": {
      "order": "desc"
    }
  },
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "itemName": "Enter a search word"
           }
        },
        {
          "match": {
            "itemNameKana": "Enter a search word"
          }
        },
        {
          "match": {
            "itemNameHira": "Enter a search word"
          }
        }
      ]
    }
  }
}

Candidate factors

The following are considered as candidate factors.

―― 1. The content of the query issued in Java is incorrect. ―― 2. The analyzer settings are incorrect. --3. It is necessary to correct the implicitly set contents.

I decided to investigate these in order.

Survey implementation

1. The content of the query issued in Java is incorrect

First, from the perspective of whether the content of the query issued in Java is incorrect. As a confirmation method, the Java source code SearchRequest request = new SearchRequest(index).source(searchBuilder); A breakpoint is set in the sentence to be done in 1 and the contents of searchSourceBuilder are viewed.

The contents were as follows.

Contents of searchBuilder

{"size"5,"query":{"bool":{"should":[{"match":{"itemName":{"query":"Enter a search word","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"itemNameKana":{"query":"Enter a search word","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"itemNameHira":{"query":"Enter a search word","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[{"_score":{"order":"desc"}}]}

At first glance, there seems to be no problem. Running this content in Kibana gave exactly the same result as hitting "Run Query in Kibana". The executed query is as follows.

Query executed with the contents of searchSourceBuilder

POST item_list/_search
{"size"5,"query":{"bool":{"should":[{"match":{"itemName":{"query":"Enter a search word","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"itemNameKana":{"query":"Enter a search word","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"itemNameHira":{"query":"Enter a search word","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[{"_score":{"order":"desc"}}]}

From this result, it can be said that "the content of the query issued in Java is incorrect" is not the cause. Also, when I checked the score (_score) at this stage for the acquisition result in Java (result returned to Elasticsearch → Java: return value of client.search (request, RequestOptions.DEFAULT)), Kibana It turned out that it is different from the execution result in. </ b> (This is an important point that was revealed in the survey, so I will leave it in bold)

SearchRequest{searchType=QUERY_THEN_FETCH, indices=[item_list], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":10,"query":{"bool":{"should":[{"match":{"itemName":{"query":"Enter a search word","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"itemNameKana":{"query":"Enter a search word","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"itemNameHira":{"query":"Enter a search word","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[{"_score":{"order":"desc"}}]}}

	/**
	 *Product Search
	 *
	 * @param keyword keyword
	 * @param index index name
	 * @param limit number
	 * @param client Elasticsearch connection client
	 * @return search results
	 * @throws IOException
	 */
	public SearchResponse search(String keyword, String index, int limit, RestHighLevelClient client) throws IOException{

		//Initialize search conditions
		SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

		BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();

		boolQueryBuilder.should(QueryBuilders.matchQuery("itemName", keyword))
						.should(QueryBuilders.matchQuery("itemNameKana", keyword))
						.should(QueryBuilders.matchQuery("itemNameHira", keyword));

		searchSourceBuilder.query(boolQueryBuilder);
		//Sort order setting (sort by score)
		searchSourceBuilder.sort(new FieldSortBuilder("_score").order(SortOrder.DESC));
		//Set the number of returns
		searchSourceBuilder.size(limit);

		SearchRequest request = new SearchRequest(index).source(searchSourceBuilder);
		//Search type DFS,Set to Query Then Fetch//Postscript
		request.searchType(SearchType.DFS_QUERY_THEN_FETCH); //Postscript
		return client.search(request, RequestOptions.DEFAULT);


	}