Let's make a tic-tac-toe AI with Pylearn 2

Introduction

I want to do Deep Learning with pylearn2. But I'm not sure if I'm just doing the image discrimination in the tutorial. So, I'm going to make an AI for tic-tac-toe (so-called XX game). I really wanted to make an AI for Othello, but since this is my first time using pylearn2, I chose a simpler game. By the way, I am a beginner of machine learning and pylearn2. Please point out any mistakes. Skip the steps to install the required packages such as pylearn2 and numpy.

Neural network design

Since it is a tic-tac-toe AI, first number the squares on the 3x3 board to indicate the state on the board. 3x3.png

The input is a certain state of the 3x3 board, and the position of the next hand is the output. Therefore, consider the following network. n-net.png

The numbers on the shoulders of the input and output circles are the numbers of the squares on the board. The input is 9 inputs, that is, the state of 9 squares. The hidden layer is one layer, and the activation function is the sigmoid function. Since the output is divided into 9 classes, we will use the softmax function. Here, "divide into 9 classes" means to classify a certain input (state of the board) according to which square the next move is hit. In other words, if it is classified into 0, the next move will be taken on the 0 square.

Creation of teaching data

The AI created this time does not pursue strength. If you can proceed with the game without deviating from the teaching data, we will divide it into a letter. Therefore, instead of collecting only the battle data that is doing the best, prepare a large amount of data that is simply competing according to the rules. In order to generate a large amount of battle data, I created a Ruby script that just puts it in a place where both the first move and the second move can be placed randomly. You might be told why you don't write in python, but just because you're tired of writing just python lately doesn't mean much. The first move is 〇 and the second move is ×. You can create a game record by specifying the number of loops in the argument and executing it. This is used as teaching data. The number is the number of the square you put, and the win and lose at the end indicate the victory or defeat of the first attack.

tic_tac_toe.rb


#!/usr/bin/env ruby
#
# tic tac toe
#
# 0 : O (first)
# 1 : X (second)
# 2 : none
#

def show_board(array)
  p array[0][0].to_s + "," + array[0][1].to_s + "," + array[0][2].to_s
  p array[1][0].to_s + "," + array[1][1].to_s + "," + array[1][2].to_s
  p array[2][0].to_s + "," + array[2][1].to_s + "," + array[2][2].to_s
  p ""
end

def judge(array)
  ret = 2
  for stone in [0, 1] do
    for i in [0 ,1, 2] do
      if (array[i][0]==stone && array[i][1]==stone && array[i][2]==stone) ||
         (array[0][i]==stone && array[1][i]==stone && array[2][i]==stone)   then
        ret = stone
      end
    end
   
    if (array[0][0]==stone && array[1][1]==stone && array[2][2]==stone) ||
       (array[0][2]==stone && array[1][1]==stone && array[2][0]==stone)   then
      ret = stone
    end
  end
  return ret
end

loop_max = ARGV[0].to_i
#p "loop max=" + loop_max.to_s
cell_array = [] 
stone_array = [] 

loop_cnt = 0
until loop_cnt >= loop_max do
  cell_array = Array.new(9)
  stone_array = Array.new(3).map { Array.new(3, 2) }
  9.times do |num|
    cell_array[num] = num
  end

  i = 10
  history = []
  9.times do |num|
    #p i
    rnd = rand(i) - 1
    if num % 2 == 0
      stone_array[cell_array[rnd].divmod(3)[0]][cell_array[rnd].divmod(3)[1]] = 0
    else
      stone_array[cell_array[rnd].divmod(3)[0]][cell_array[rnd].divmod(3)[1]] = 1
    end
    history.push(cell_array[rnd])
    #show_board(stone_array)
    ret = judge(stone_array)
    if ret == 0 then
      history.push("win")    # "O" is winner.
      break
    elsif ret == 1 then
      history.push("lose")   # "O" is loser.
      break
    end
    
    cell_array.delete_at(rnd)

    i -= 1
  end
  p history.join(",")
  loop_cnt += 1
end
$ ruby tic_tac_toe.rb 500 | tee tic-tac-toe_records.log
"6,8,3,4,2,0,lose"
"7,1,4,5,0,2,8,win"
"6,2,3,0,5,8,1,7,4,win"
"3,8,4,2,6,5,lose"
"1,8,2,7,3,0,6,4,lose"
"8,0,3,6,7,4,1,2,lose"
"6,8,3,5,2,7,4,win"
"2,1,7,4,3,5,8,6,0"
"4,8,6,7,1,3,2,win"
"6,1,3,0,8,7,5,4,lose"
"8,2,7,1,4,3,0,win"
"8,6,1,2,7,0,3,4,lose"
"4,3,8,1,2,6,7,0,lose"
"8,6,3,4,1,5,7,2,lose"
"1,2,0,4,7,8,5,6,lose"
"0,5,2,3,6,7,8,4,lose"
"7,1,2,6,4,5,0,3,8,win"
"2,1,0,8,3,5,7,4,6,win"
"2,0,8,5,6,7,4,win"
...

Process this game record data into csv so that it can be used easily as an input. This time, in order to make an AI that decides the second attack, only the ones that the second attack won are extracted.

$ awk '{gsub("\"","");print $0;}' tic-tac-toe_records.log | grep lose | tee tic-tac-toe_records_lose.csv

It will be like this after processing.

tic-tac-toe_records_lose.csv


6,8,3,4,2,0,lose
3,8,4,2,6,5,lose
1,8,2,7,3,0,6,4,lose
8,0,3,6,7,4,1,2,lose
...

Now you are ready to go.

MLP model using pylearn2

First, the source code is shown.

tic_tac_toe.py


#!/usr/bin/env python
# -*- cording: utf-8 -*-

import theano
from pylearn2.models import mlp
from pylearn2.training_algorithms import sgd
from pylearn2.termination_criteria import EpochCounter
from pylearn2.datasets.dense_design_matrix import DenseDesignMatrix
import numpy as np
import csv

class TicTacToe(DenseDesignMatrix):
    def __init__(self):
        X = []
        y = []
        X_temp = [0,0,0,0,0,0,0,0,0]  # 3x3 board
        y_temp = [0,0,0,0,0,0,0,0,0]  # 3x3 board

        # (1)
        self.class_names = ['0', '3']
        f = open("tic-tac-toe_records_lose.csv", "r")
        reader = csv.reader(f)

        # (2)
        for row in reader:
            for i, cell_index in enumerate(row):
                if cell_index == "win" or cell_index == "lose":
                    X_temp = [0,0,0,0,0,0,0,0,0]
                elif i % 2 == 0:
                    temp = []
                    X_temp[int(cell_index)] = 1
                    for x in X_temp:
                        temp.append(x)

                    #print "  temp = " + str(temp)
                    X.append(temp)
                else:
                    X_temp[int(cell_index)] = 2
                    y_temp[int(cell_index)] = 3
                    #print "y_temp = " + str(y_temp)
                    y.append(y_temp)
                    y_temp = [0,0,0,0,0,0,0,0,0]

        X = np.array(X)
        y = np.array(y)
        super(TicTacToe, self).__init__(X=X, y=y)

# (3)
data_set = TicTacToe()
h0 = mlp.Sigmoid(layer_name='h0', dim=9, irange=.1, init_bias=1.)
out = mlp.Softmax(layer_name='out', n_classes=9, irange=0.)
trainer = sgd.SGD(learning_rate=.05, batch_size=200, termination_criterion=EpochCounter(5000))
layers = [h0, out]

ann = mlp.MLP(layers, nvis=9)
trainer.setup(ann, data_set)

# (4)
while True:
    trainer.train(dataset=data_set)
    ann.monitor.report_epoch()
    ann.monitor()
    if trainer.continue_learning(ann) == False:
        break

# (5)-1
next_move = [0,0,0,0,0,0,0,0,0]
inputs = np.array([[0,0,1,0,0,0,0,0,0]])
output = ann.fprop(theano.shared(inputs, name='inputs')).eval()
print output[0]
for i in range(0,9):
    if max(output[0]) == output[0][i]:
        next_move[i] = 3

print next_move

# (5)-2
next_move = [0,0,0,0,0,0,0,0,0]
inputs = np.array([[1,0,2,1,0,0,0,1,2]])
output = ann.fprop(theano.shared(inputs, name='inputs')).eval()
print output[0]
for i in range(0,9):
    if max(output[0]) == output[0][i]:
        next_move[i] = 3

print next_move

I will explain a little. Please read it in correspondence with (1), (2) ... in the code. As a premise, 〇 is "1", × is "2", and a square with nothing is "0". (It is different from the ruby code used to create the teaching data. Sorry for the incomprehension.)

(1) Setting the output value The value to take as the output value. I'm using 0,1,2 for the input, so I decided to use 3 for the output. (2) Conversion of game records Corresponds the read game record to the input and output of the neural network. For example, if the input of the game record is "6,8,3,4,2,0, lose", the expansion will be as follows.

Game record: 6,8,3,4,2,0,lose
X[n]   = [0,0,0,0,0,0,1,0,0]  : "6"In the position of"〇"That is"1"Enter
y[n]   = [0,0,0,0,0,0,0,0,3]  :The next move is"8"In the position of"×"So"8"In the position of"3"Outputs
X[n+1] = [0,0,0,1,0,0,1,0,2]  : "8"In the position of"×"That is"2",next"3"In the position of"〇"That is"1"Enter
y[n+1] = [0,0,0,0,3,0,0,0,0]  :The next move is"4"In the position of"×"So"4"In the position of"3"Outputs
X[n+2] = [0,0,1,1,2,0,1,0,2]  : "4"In the position of"×"That is"2",next"2"In the position of"〇"That is"1"Enter
y[n+2] = [3,0,0,0,0,0,0,0,0]  :The next move is"0"In the position of"×"So"0"In the position of"3"Outputs

(3) MLP structure description The hidden layer h0 is generated with dim (dimension) = 9 (number of squares) using the sigmoid function as the activation function. irange and init_bias are suitable. The output layer out is generated with n_classes (number of classifications) = 9 (number of cells) using the softmax function as the activation function. irange is suitable. [Stochastic Gradient Descent](https://en.wikipedia.org/wiki/%E7%A2%BA%E7%8E%87%E7%9A%84%E5%8B%BE%E9%85%8D Train with% E9% 99% 8D% E4% B8% 8B% E6% B3% 95). learning_rate and batch_size are appropriate. termination_criterion specifies when to end training.

ann = mlp.MLP(layers, nvis=9)

The above specifies the structure of MLP. nvis has input dimension = 9 (number of squares). (4) Training It's a training loop. A monitor of the progress is also available. (5) Test Test what the output looks like for some inputs. The place with the highest probability (output [0]) of 9 classes is the next move (next_move).

The output when the program is executed is as follows.

$ python tic_tac_toe.py
Parameter and initial learning rate summary:
	h0_W: 0.05
	h0_b: 0.05
	softmax_b: 0.05
	softmax_W: 0.05
Compiling sgd_update...
Compiling sgd_update done. Time elapsed: 0.203131 seconds
compiling begin_record_entry...
compiling begin_record_entry done. Time elapsed: 0.003911 seconds
Monitored channels: 
Compiling accum...
Compiling accum done. Time elapsed: 0.000039 seconds
Monitoring step:
	Epochs seen: 1
	Batches seen: 3
	Examples seen: 542
Monitoring step:
	Epochs seen: 2
	Batches seen: 6
	Examples seen: 1084

...

Monitoring step:
	Epochs seen: 5000
	Batches seen: 15000
	Examples seen: 2710000
Monitoring step:
	Epochs seen: 5001
	Batches seen: 15003
	Examples seen: 2710542
[ 0.07985083  0.10700001  0.00255253  0.15781951  0.08504663  0.16470689
  0.11459433  0.12293593  0.16549335]
[0, 0, 0, 0, 0, 0, 0, 0, 3]
[  2.56981722e-03   1.25923571e-01   2.05250923e-04   6.14268028e-04
   1.85819252e-02   2.43921569e-02   8.27328217e-01   3.84348076e-04
   4.45556802e-07]
[0, 0, 0, 0, 0, 0, 3, 0, 0]

From the above, If the input is [0,0,1,0,0,0,0,0,0], the next move is [0,0,0,0,0,0,0,0,3]. In other words, AI hits as shown in the figure below. example01.png

If the input is [1,0,2,1,0,0,0,1,2], the next move is [0,0,0,0,0,0,3,0,0]. In other words, AI hits as shown in the figure below. example02.png

Well, if you hit "X" at the "5" position, you win and you're done, but you're going to stop the reach of "O". Well, for the fact that it is based on a random game record, is it okay?

It's a bit annoying because I have to mess with the source code every time, but I got the next move for arbitrary input.

In the future, based on the results of one training, I will try to implement it in a tic-tac-toe game so that I can easily obtain the next move for any input. First of all, I have to make a game program of tic-tac-toe. **I made. Please see the following article. ** ** Create a tic-tac-toe AI with Pylearn2-Save and load models-

reference

http://www.arngarden.com/2013/07/29/neural-network-example-using-pylearn2/ [http://sinhrks.hatenablog.com/entry/2014/11/30/085119] (http://sinhrks.hatenablog.com/entry/2014/11/30/085119) https://www.safaribooksonline.com/blog/2014/02/10/pylearn2-regression-3rd-party-data/

Recommended Posts

Let's make a tic-tac-toe AI with Pylearn 2
Let's create a tic-tac-toe AI with Pylearn 2-Save and load models-
Let's make a GUI with python.
Let's make a breakout with wxPython
Let's make Othello AI with Chainer-Part 1-
Let's make a graph with python! !!
Let's make a supercomputer with xCAT
Let's make Othello AI with Chainer-Part 2-
Let's make a voice slowly with Python
Let's make a web framework with Python! (1)
Let's make a Twitter Bot with Python!
Let's make a web framework with Python! (2)
Let's replace UWSC with Python (5) Let's make a Robot
Let's make a Discord Bot.
[Let's play with Python] Make a household account book
Let's make Othello with wxPython
Let's make a simple game with Python 3 and iPhone
Make Puyo Puyo AI with Python
Let's make Splatoon AI! part.1
Let's make dice with tkinter
Make a fortune with Python
Let's make a spherical grid with Rhinoceros / Grasshopper / GHPython
Let's make a rock-paper-scissors game
[Super easy] Let's make a LINE BOT with Python.
Make a fire with kdeplot
Let's make a websocket client with Python. (Access token authentication)
Let's make a diagram that can be clicked with IPython
Let's make a remote rumba [Hardware]
Let's make a remote rumba [Software]
Make a sound with Jupyter notebook
Let's make a spot sale service 2
Let's make a spot sale service 1
Make a recommender system with python
Make a filter with a django template
Make a model iterator with PySide
Make a nice graph with plotly
Let's make a spot sale service 3
Let's make a WEB application for phone book with flask Part 1
Let's make a cycle computer with Raspberry Pi Zero (W, WH)
Let's make a WEB application for phone book with flask Part 2
Let's make a WEB application for phone book with flask Part 3
Let's make a WEB application for phone book with flask Part 4
Let's make a web chat using WebSocket with AWS serverless (Python)!
Make a video player with PySimpleGUI + OpenCV
Let's create a free group with Python
Make a rare gacha simulator with Flask
Make a Notebook Pipeline with Kedro + Papermill
[Easy] AI automatic recognition with a webcam!
Make a partially zoomed figure with matplotlib
Let's scrape a dynamic site with Docker
Make a drawing quiz with kivy + PyTorch
Make a cascade classifier with google colaboratory
[Python] Let's make matplotlib compatible with Japanese
Make a logic circuit with a perceptron (multilayer perceptron)
Make a Yes No Popup with Kivy
Let's make a multilingual site using flask-babel
Make a wash-drying timer with a Raspberry Pi
Make a GIF animation with folder monitoring
Let's make a combination calculation in Python
Make a desktop app with Python with Electron
Let's make a Backend plugin for Errbot