Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 2 Step 07 Memo "Evaluation"

Contents

This is a memo for myself as I read Introduction to Natural Language Processing Applications in 15 Steps. This time, in Chapter 2, Step 07, I will write down my own points.

Preparation

--Personal MacPC: MacOS Mojave version 10.14.6 --docker version: Version 19.03.2 for both Client and Server

Chapter overview

--Quantitatively evaluate the prediction accuracy of machine learning systems using various indicators --When you make improvements to an existing system, you can guarantee that the performance has not deteriorated by evaluation, and you can update the system with confidence. --Understand overfitting and prevent it from occurring

07.1 Training and test data, and overfitting and generalization

The overfitting of the classifier to the training data due to learning is called ** overfitting **.

If 100% of the feature vectors included in the training data can be identified, even noise that can be ignored becomes a fine identification surface so that it can be correctly identified. Being able to make stable predictions for data other than training data is called ** generalization **.

If the same data is used for training and evaluation, the overfitted system will be highly evaluated, so ** the test data for evaluation must be evaluated using something different from the training data **. (No matter how good the training data is, it doesn't make much sense, and you have to make sure that you are not overfitting.)

07.2 Evaluation index

item Contents
Accuracy (correct answer rate) Ratio of correct test data to all test data
Precision Percentage of correct test data out of the test data predicted for the target class
Recall Percentage of correct predictions of test data for the target class
F value An index showing the balance between Precision and Recall

Implementation

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

accuracy_score(y_true, y_pred)
precision_score(y_true, y_pred)
recall_score(y_true, y_pred)
f1_score(y_true, y_pred, average='macro')

--By specifying ʻaverage, you can calculate macro average ='macro'and micro average='micro'`. --A trade-off between precision and recall

07.3 Precautions for evaluation

item Contents
Lower limit of accuracy The lower limit of accuracy is the case of predicting without guessing.
Number of classification classes Since the difficulty level of multi-class classification is naturally higher than that of 2-class classification, the evaluation index has different values depending on the application.
Test data type Relative evaluation of different systems should be done with the same test data
Bias in the number of data It is desirable that the test data include the data of each class as evenly as possible.

Recommended Posts

Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 2 Step 07 Memo "Evaluation"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 2 Step 06 Memo "Identifier"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 2 Step 02 Memo "Pre-processing"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 4 Step 14 Memo "Hyperparameter Search"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 4 Step 15 Memo "Data Collection"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 3 Step 08 Memo "Introduction to Neural Networks"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 2 Step 05 Memo "Features Conversion"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 3 Step 11 Memo "Word Embeddings"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 3 Step 13 Memo "Recurrent Neural Networks"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 3 Step 09 Memo "Identifier by Neural Network"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 2 Step 01 Memo "Let's Make a Dialogue Agent"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 2 Step 03 Memo "Morphological Analysis and Word Separation"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 3 Step 10 Memo "Details and Improvements of Neural Networks"
Try the book "Introduction to Natural Language Processing Application Development in 15 Steps" --Chapter 1 Memo "Preliminary Knowledge Before Beginning Exercises"
[WIP] Pre-processing memo in natural language processing
Summary from the beginning to Chapter 1 of the introduction to design patterns learned in the Java language
[Chapter 6] Introduction to scikit-learn with 100 knocks of language processing
[Chapter 3] Introduction to Python with 100 knocks of language processing
[Chapter 2] Introduction to Python with 100 knocks of language processing
[Chapter 4] Introduction to Python with 100 knocks of language processing
[Job change meeting] Try to classify companies by processing word-of-mouth in natural language with word2vec
[Natural language processing] I tried to visualize the hot topics this week in the Slack community
Sentiment analysis with natural language processing! I tried to predict the evaluation from the review text
[Natural language processing] I tried to visualize the remarks of each member in the Slack community
[Python] Try to classify ramen shops by natural language processing
Summary of Chapter 2 of Introduction to Design Patterns Learned in Java Language
Chapter 4 Summary of Introduction to Design Patterns Learned in Java Language
Summary of Chapter 3 of Introduction to Design Patterns Learned in Java Language
[Introduction to RasPi4] Environment construction; natural language processing system mecab, etc. .. .. ♪
I tried to display the analysis result of the natural language processing library GiNZA in an easy-to-understand manner
100 natural language processing knocks Chapter 4 Commentary
100 Language Processing Knock Chapter 1 in Python
Web application development memo in python
Cython to try in the shortest
Preparing to start natural language processing
From the introduction of GoogleCloudPlatform Natural Language API to how to use it
I tried to solve the 2020 version of 100 language processing [Chapter 3: Regular expressions 25-29]
100 Knocking Natural Language Processing Chapter 1 (Preparatory Movement)
Introduction to Effectiveness Verification Chapter 1 in Python
Convenient goods memo around natural language processing
Let the COTOHA API do the difficult things-Introduction to "learn using" natural language processing-
I tried to solve the 2020 version of 100 language processing knocks [Chapter 3: Regular expressions 20 to 24]
I tried to solve the 2020 version of 100 language processing knocks [Chapter 1: Preparatory movement 00-04]
I tried to solve the 2020 version of 100 language processing knocks [Chapter 1: Preparatory movement 05-09]
The first artificial intelligence. I wanted to try natural language processing, so I will try morphological analysis using MeCab with python3.