In addition, there are many things to check in df.head, such as how many lines of data should be taken from the head and details of missing value processing (dropna and fillna).

Matplotlib If you feel like it, I will summarize something, but for example, the style of the graph, the setting of the title, etc. are also written in the text, so you need to check it. In particular, graph types (ax.bar, ax.scatter, ax.hist, ax.boxplot, ax.pie, etc.) are required items as the test title of data analysis.

Machine learning

There are various methods for machine learning. Decision tree, principal component analysis, NN, etc. Of course, a data analyst should be able to use everything. However, I think that it is necessary to review the algorithm, explanation of the method, scikit-learn, etc. for the method that is not used often. In the reference book, I felt that supervised learning and unsupervised learning were confused (Isn't that the case?), So you may want to study together with or without supervised learning. Hmm.

Summary of machine learning

・ ** Data preprocessing (missing value processing, feature normalization, etc.) ** For missing value processing, the above-mentioned missing value processing by Pandas is introduced. Please check the reference book for details. Feature normalization requires knowledge such as probability statistics (distributed normalization, maximum normalization, etc.). Also, in pre-processing, it is important to select the feature amount of the data and exclude outliers (outside the test range?), But for the time being, you should check the contents in the reference book.

・ ** Supervised learning ** The reference book explains the support vector machine (SVC of sklearn.svm) and the linear regression model (LinearRegression of sklearn.linear_model), so please check there for details.

・ ** Unsupervised learning ** The reference book explains the decision tree (DecisionTreeClassifier of sklearn.tree), random forest (RandomForestClassifier of sklearn.ensemble), dimension reduction (PCA of sklearn.decomposition), and clustering (KMeans of sklearn.cluster, AgglomerativeClustering of sklearn.cluster). Please check there for details.

・ ** Reinforcement learning ** Only a brief explanation appears in the reference book.

・ ** Variable encoding ** ** [Category variable encoding] ** Humans give names to objects so that they can easily recognize the world in which they live. For example, the two sensory organs attached to the front of the human head that receive light are called "eyes", and the organs for olfaction and breathing attached to the front of the human head are called "nose". Even if the object is interpreted by the character type in this way, if you want to classify it by calculation, the input and output must be a sequence (numerical value). The work of associating the numerical value with the actual target is categorical variable encoding.

Conceptual diagram of categorical variable encoding

Specifically, I will explain where the categorical variable encoding is in the program I made earlier.

Category variable encoding, One-Conceptual diagram of hot encoding

The following page is an article when classifying hook wrenches and spanner wrenches using PyTorch and Keras. https://qiita.com/moriitkys/items/316fcca8d83dfa706597 In this case, the categorical variable encoding is the assignment of the dictionary executed in cell 1, line 56 of the MyOwnNN_pytorch.ipynb program on GitHub (GitHub-moriitkys / MyOwnNN).

#How many classes are in "dataset" folder
categories = [i for i in os.listdir(os.getcwd().replace("/mylib", "") + "/dataset")]
categories_idx = {}#ex) HookWrench:0, SpannerWrench:1
for i, name in enumerate(categories):
    categories_idx[name] = i
nb_classes = len(categories)#ex) nb_classes=2

In this case, the hook wrench is converted to 0 and the spanner wrench is converted to 1. For categorical variable encoding, the method of sklean.preprocessing is introduced in the reference book.

** [One-hot encoding] ** This is also the work required to obtain the desired result by calculation. Since it is often more convenient to treat inputs and outputs as arrays in machine learning, you also need to convert the scalar value dataset as an array. For example, if there are two categories, spanner wrench: 0 and hook wrench: 1, convert the spanner wrench as [1, 0] and the hook wrench as [0, 1].

Specifically, I will explain where the categorical variable encoding is in the program I made earlier. One-hot encoding is around the 13th to 16th lines of cell 3.

y_train1=np_utils.to_categorical(y_train,nb_classes)
y_val1=np_utils.to_categorical(y_val,nb_classes)

Two methods of One-hot encoding, sklean and pandas, are introduced in the reference book.

・ ** Model evaluation ** ** [Category evaluation index] ** I can understand it, but I don't know if I memorize it, so I summarized it. This explanation is also included in the reference book, so it's a good idea to check it.

TP (True Positive): Predicted as a positive example and actually a positive example FP (False Positive): Predicted as a positive example, but actually a negative example FN (False Negative): Predicted as a negative example, but actually a positive example TN (True Negative): Predicted as a negative example and actually a negative example

** Compliance rate **: Percentage of actual positive cases out of the predicted positive cases. ・ Compliance rate formula P = TP / (TP + FP) ・ The precision rate is considered to be an index of how small the mistakes in the prediction class can be. ** Recall rate **: Percentage of actual positive cases predicted to be positive. ・ Recall rate formula R = TP / (TP + FN) ・ There is a trade-off between recall and precision. ** F value **: Harmonic mean of precision and recall. ・ F = 2PR / (P + R) ・ Indicator that improves the balance between P and R ** Correct answer rate **: Percentage of data whose forecast and actual results match ・ Correct answer rate formula A = (TP + TN) / (TP + FP)

** [Accuracy of prediction probability] ** If you know it, there is nothing wrong with it, but the reference book mentions the ROC curve (roc_curve in sklearn.metrics).

Summary

You should study properly. I don't think it's possible to do something like rote memorization. Memorizing reference books may be useful, but it should be understood.

Reference book

New textbook for data analysis using Python (Manabu Terata, Shingo Tsuji, Takanori Suzuki, Shintaro Fukushima)

Author of this article

moriitkys Takayoshi Morii Make a robot. I'm interested in AI / Robotics / 3D Graphics. Recently, I've been thinking about how to make money, and I'm planning to make hardware with that money. Qualification / Certification: G test, Python engineer certification data analysis test, AI implementation test A grade, TOEIC: 810 (2019/01/13)