EV3 x Pyrhon Machine Learning Part 3 Classification

The content of this article is in beta and is subject to change. In this article, we will use the educational version LEGO® MINDSTORMS EV3 (hereafter EV3) and the Python environment to identify colors using classification. I will go. For environment construction, please refer to the article on environment construction. Also, in the previous article, the contents of line tracing and linear regression are combined.

Machine learning with EV3 Part 1 Environment construction: here Machine learning with EV3 Part 2 Linear regression: here Machine learning with EV3 Part 3 Classification: This article

reference

The content of this article is based on the following books. The basic control of EV3 using Python is covered below.

Introduction to AI starting with robots

Environment in this article

• PC Windows10 Python 3.7.3 Development environment Visual Studio Code

• EV3 ev3dev

table of contents

1. Classification method
2. What to do
3. EV3 model and reading color
4. Creating a program
5. Program execution
6. Execution result
7. Summary

Classification method

SVM (Support Vector Machine)

A technique for drawing boundaries to separate data when it is scattered.

Draw a boundary line = predict classification, but use past data for prediction. At that time, if extra data such as outliers are used, the accuracy may decrease. Therefore, SVM uses only some data that is really necessary for prediction. The data required for prediction is called a support vector, and a machine learning method using the support vector is a support vector machine.

things to do

This time, the color sensor collects RGB (degree of red, green, and blue) and label numbers of several colors as a set. The EV3 color sensor can acquire RGB values from 0 to 255. Based on the collected data group, the boundary of what color the data is is defined and inference is performed based on the data.

The following is an example of the data collected this time

EV3 model and reading color

Read the color at the end of the course you used last time. By changing the label definition in the program, it is possible to use some colors as learning data, so it is possible to classify favorite colors. 　　　　　　

Also this time, we will use the same EV3 model "Base Robo" as last time. Since the motor is not moved this time, there is no problem if there is an intelligent block and a color sensor, but since the color sensor needs to have a gap of about 0.5 cm to 1 cm from the reading surface, the color sensor can be fixed stably. Use the model. 　　　　　

Creating a program

This time, create the following two programs.

• EV3 side program data_get_color.py
• PC side program Classification.py

As in the previous time, the data processing and inference itself are performed by the PC side program, and the EV3 side acquires and transmits the color RGB values. The relationship between each program is shown in the figure below.

EV3 side program

The EV3 side program data_get_color.py is created in the workspace on VS Code. Please refer to the following articles for how to create a workspace and transfer it to EV3. EV3 x Pyrhon Machine Learning Part 1 Environment Construction

import time
import socket
import sys
from ev3dev2.button import Button
from ev3dev2.sensor import INPUT_3
from ev3dev2.sensor.lego import ColorSensor


def dataget(color_dic, color_num):
    _rgb_data = color.raw
    _rgb_data = list(_rgb_data)
    _rgb_data.append(color_num)
    _rgb_data_str = ','.join(map(str, _rgb_data))
    s.send(_rgb_data_str.encode())
    print('\n'+'rgb_color = {}'.format(_rgb_data_str))


def predict():
    _rgb_data = color.raw
    _rgb_data = list(_rgb_data)
    _rgb_data_str = ','.join(map(str, _rgb_data))
    s.send(_rgb_data_str.encode())
    pre_color = s.recv(1024).decode()
    print('\n'+'predict_color = {}'.format(color_dic[int(pre_color[0])]))


 sensors&motors definition
button = Button()
color = ColorSensor(INPUT_3)

 gyro initialize
color.mode = 'RGB-RAW'

 variable initialize
color_dic = {
            1: 'RED',
            2: 'GREEN',
            3: 'BLUE'
            }

color_num = 1
color_max = len(color_dic)

 get gyrodate and into array
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.connect(('169.254.85.105', 50010))    # your PC's Bluetooth IP & PORTpy
    for cnt in range(color_max):
        s.send((color_dic[cnt+1]).encode())
        time.sleep(0.1)
    s.send(('END').encode())
    print('Start program...')
    while not(button.backspace):
        if button.up:
            color_num += 1
            time.sleep(0.1)
            if color_num > color_max:
                color_num = color_max
        elif button.down:
            color_num -= 1
            time.sleep(0.1)
            if color_num < 1:
                color_num = 1
        elif button.right:
            msg = 'save'
            s.send(msg.encode())
            dataget(color_dic, color_num)
        elif button.left:
            msg = 'predict'
            s.send(msg.encode())
            predict()
        print('\r'+'save_color = {}   '.format(color_dic[color_num]), end='')
        time.sleep(0.1)

print('\n'+'End program')
sys.exit()

• Please note that if you comment out the EV3 environment in Japanese, an error will occur due to the nature of the character code.

The s.connect (('169.254.207.161', 50010)) described in the latter half is rewritten according to the environment as before.

The label name when recording as data can be changed by changing the following RED, GREEN, and BLUE parts. This time we will read red, green and blue, so leave them as follows.

 variable initialize
color_dic = {
            1: 'RED',
            2: 'GREEN',
            3: 'BLUE'
            }

PC side program

In the PC side program, the color value and label sent from EV3 are recorded in a CSV file as a set, and when an inference message is sent from EV3 side, what color is currently being viewed based on the data group? Or infer and display the result.

Create Classification.py as a text document in the program folder as before, and describe the following contents.

import socket
import sys
import csv
import numpy as np
import pandas as pd
import os.path
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.multiclass import OneVsRestClassifier
from sklearn.ensemble import RandomForestClassifier as RFC
from sklearn.metrics import accuracy_score


 setting svm
C = 1.
kernel = 'rbf'
gamma = 0.01
estimator = SVC(C=C, kernel=kernel, gamma=gamma)
clf = OneVsRestClassifier(estimator)

x_data = np.zeros(0)
y_data = np.zeros(0)

color_elements = None

color_dic = {}
color_cnt = 1

# Creating a data file
if os.path.exists('color_data.csv') == False:
    writedata = ['red', 'green', 'blue', 'color']
 f = open ('color_data.csv','w', newline ='') # Open the file
    writer = csv.writer(f)
 writer.writerow (writedata) # Write data
    f.close()
 data = pd.read_csv ("color_data.csv", sep = ",") # read csv file

# Divide the read data into input data and labels
x_data = data.loc[:, ["red", "green", "blue"]]
y_data = data.loc[:, "color"]

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.bind(('169.254.85.105', 50010))  # your PC's Bluetooth IP & PORT
    s.listen(1)
    print('Start program...')
    while True:
        conn, addr = s.accept()
        with conn:
 #Create the same color dictionary created by the program on the EV3 side
            while True:
                color_elements = conn.recv(1024).decode()
                if color_elements == 'END':
                    break
                color_dic[color_cnt] = color_elements
                color_cnt += 1
            print('color_dic = {}'.format(color_dic))

 #Change behavior depending on message
            while True:
                rgb_data = conn.recv(1024).decode()
                if not rgb_data:
                    break
 # Send the value to EV3 after inference using the accumulated data
                elif rgb_data == 'predict':
                    train_x, test_x, train_y, test_y = train_test_split(x_data,
                                                                        y_data)
                    clf.fit(train_x, train_y)
                    y_pred = clf.predict(test_x)
                    rgb_data = conn.recv(1024).decode()
                    rgb_data = rgb_data.split(',')
                    pre_color = clf.predict([rgb_data])
                    print('predict_color = {}'.format(color_dic[pre_color[0]]))
                    conn.send(str(pre_color[0]).encode())
 Save the data sent from # EV3
                elif rgb_data == 'save':
                    rgb_data = conn.recv(1024).decode()
                    rgb_data = rgb_data.split(',')
                    print('rgb_data = {}'.format(rgb_data))
                    np.append(y_data, rgb_data[0:2])
                    np.append(y_data, int(rgb_data[3]))
                    writedata = rgb_data
                    f = open('color_data.csv', 'a', newline='')
                    writer = csv.writer(f)
                    writer.writerow(writedata)
                    f.close()
            print('End program')
            sys.exit()

The s.bind (('169.254.207.161', 50010)) described in the latter half is changed according to the environment like the EV3 side program. How to check and change the environment [Previous article](https://qiita.com/Hiroki-Fujimoto/items/6dae8c407e56a38625cf#%E3%83%97%E3%83%AD%E3%82%B0%E3% 83% A9% E3% 83% A0% E3% 81% AE% E4% BD% 9C% E6% 88% 90) Or check the following. [IP settings for socket communication](https://qiita.com/Hiroki-Fujimoto/items/6dae8c407e56a38625cf#%E3%82%BD%E3%82%B1%E3%83%83%E3%83%88%E9 % 80% 9A% E4% BF% A1% E3% 81% AEip% E8% A8% AD% E5% AE% 9A)

Program execution

Once you have created two programs, run each one.

1. Execute cd Desktop \ program from the command prompt (\ is synonymous with \ mark)

2. Then run python Classification.py at the command prompt

• After execution, Start program ... will be displayed and the system will be in a standby state.

3. Open the SSH terminal of EV3 connected on VS Code and execute cd ev3 workspace /

4. Execute python3 data_get_gyro.py in the SSH terminal

5. Install the EV3 on the color attached to the course, acquire the RGB value with the color sensor and send it to the PC. Since the operation is set for each button of EV3, press the button according to it and collect the data. 　　

--Top: Switch the color (label) to save --Bottom: Switch the color (label) to save --Right: When the button is pressed, the color sensor value and label at that time are sent to the PC (data collection). --Left: When the button is pressed, the value of the color sensor at that time is sent to the PC (inference)

Since the operation is set as described above, align the label with the color you want to collect with the up and down buttons, and press the right button to collect the data. Press the up and down buttons to switch the name of the label to be collected on VS Code.

When you collect color data with the right button, RGB values and labels are output on VS Code and on the command prompt as shown below, and saved in a CSV file.

Similarly, collect some data for each label.

Execution result

After collecting data to some extent, press the upper left button of EV3 to end the program once.

Confirm that CSV has been created.

Execute the programs on the PC side and EV3 side again using the same procedure, Press the left button on each collected color and check the inference result on the PC side. 　　　　

It can be confirmed that each color can be distinguished.

Summary

We were able to classify based on the data actually received from the robot. This time, we used the SVM method, but it is also possible to implement machine learning with other classification methods such as random forest by changing the description of the model specification of Scikit-learn. In recent machine learning, there are many things that are simplified in the PC, but it feels familiar in terms of being able to check the execution results on the edge (robot side) and using the data collected by yourself like this time. I wonder if it can be done.