Markov Chain Chatbot with Python + Janome (1) Introduction to Janome

Preface

I wanted to implement a Markov chain chatbot in Python. Well, when I investigated what to throw in morphological analysis, Janome has no dependencies on other libraries It seems that you can install it quickly with one pip. Try it!

environment

Python 3.8.5 Janome 0.4.1

How to use

from janome.tokenizer import Tokenizer

t = Tokenizer()

s = "The red powder of the grass-colored pencils scatters and I can't sleep"

for token in t.tokenize(s):
    print(token)

Use the Tokenizer class.

t = Tokenizer()

Create a Tokenizer instance,

for token in t.tokenize(s):
    print(token)

Pass the sentence you want to parse to the tokenize method. If you display the contents one by one as above, it looks like this.

python analysis.py
Grass noun,General,*,*,*,*,grass,Kusa,Kusa
Wakaba noun,Proper noun,Organization,*,*,*,Wakaba,Wakaba,Wakaba
Colored pencil noun,General,*,*,*,*,Colored pencils,Iroempitsu,Iroempitsu
Particles,Attributive,*,*,*,*,of,No,No
Red adjective,Independence,*,*,Adjective, Auoudan,Word connection,red,Akaki,Akaki
Powder noun,General,*,*,*,*,powder,Kona,Kona
Particles,Case particles,General,*,*,*,of,No,No
Scatter verb,Independence,*,*,Five steps, La line,Uninflected word,Scatter,Chill,Chill
Is a particle,Connection particle,*,*,*,*,But,Moth,Moth
Beloved adjective,Independence,*,*,Adjective, Idan,Continuous connection,Beloved,Itoshiku,Itoshiku
Sleeping verb,Independence,*,*,One step,Continuous form,sleep,Ne,Ne
Particles,Connection particle,*,*,*,*,hand,Te,Te
Kezuru verb,Independence,*,*,Five steps, La line,Uninflected word,Kezuru,Kezuru,Kezuru
Nari particle,Connection particle,*,*,*,*,Nari,Nari,Nari

You can also take out the elements one by one. I tried to output the surface layer form, the basic form, and the part of speech.

from janome.tokenizer import Tokenizer

t = Tokenizer()

s = "I can't sleep"

for token in t.tokenize(s):
    print("==========")
    print(token.surface + " (Surface type)")
    print(token.base_form + " (Uninflected word)")
    print(token.part_of_speech + " (Part of speech)")

Execution result

python analysis.py
==========
Sleep(Surface type)
sleep(Uninflected word)
verb,Independence,*,* (Part of speech)
==========
hand(Surface type)
hand(Uninflected word)
Particle,接続Particle,*,* (Part of speech)
==========
Kezuru(Surface type)
Kezuru(Uninflected word)
verb,Independence,*,* (Part of speech)
==========
Nari(Surface type)
Nari(Uninflected word)
Particle,接続Particle,*,* (Part of speech)

We will implement sentence generation in the next chapter. Markov Chain Chatbot with Python + Janome (2) Introduction to Markov Chain

Recommended Posts

Markov Chain Chatbot with Python + Janome (1) Introduction to Janome
Markov Chain Chatbot with Python + Janome (2) Introduction to Markov Chain
Introduction to Python Image Inflating Image inflating with ImageDataGenerator
[Introduction to Python] Let's use foreach with Python
[Python] Introduction to CNN with Pytorch MNIST
[Python] Easy introduction to machine learning with python (SVM)
Introduction to Artificial Intelligence with Python 1 "Genetic Algorithm-Theory-"
Introduction to Artificial Intelligence with Python 2 "Genetic Algorithm-Practice-"
Introduction to Tornado (1): Python web framework started with Tornado
Introduction to Python language
Introduction to OpenCV (python)-(2)
Introduction to formation flight with Tello edu (Python)
Introduction to Python with Atom (on the way)
Introduction to Generalized Linear Models (GLM) with Python
[Introduction to Udemy Python3 + Application] 9. First, print with print
[Introduction to Python] How to iterate with the range function?
[Chapter 5] Introduction to Python with 100 knocks of language processing
An introduction to Python distributed parallel processing with Ray
Introduction to Mathematics Starting with Python Study Memo Vol.1
Reading Note: An Introduction to Data Analysis with Python
[Chapter 3] Introduction to Python with 100 knocks of language processing
[Chapter 2] Introduction to Python with 100 knocks of language processing
[Chapter 4] Introduction to Python with 100 knocks of language processing
Connect to BigQuery with Python
Introduction to Python Django (2) Win
Connect to Wikipedia with Python
Post to slack with Python 3
Introduction to RDB with sqlalchemy Ⅰ
Introduction to serial communication [Python]
Switch python to 2.7 with alternatives
Write to csv with Python
[Introduction to Python] <list> [edit: 2020/02/22]
Introduction to Python (Python version APG4b)
An introduction to Python Programming
Introduction to Python For, While
I tried to automatically create a report with Markov chain
Introduction to her made with Python ~ Tinder automation project ~ Episode 6
[Markov chain] I tried to read negative emotions into Python.
[Markov chain] I tried to read a quote into Python.
20200329_Introduction to Data Analysis with Python Second Edition Personal Summary
Introduction to her made with Python ~ Tinder automation project ~ Episode 5
Introduction to Python for VBA users-Calling Python from Excel with xlwings-
[Raspi4; Introduction to Sound] Stable recording of sound input with python ♪
[Introduction to Python] How to get data with the listdir function
[Introduction to Udemy Python3 + Application] 51. Be careful with default arguments
[Introduction to Udemy Python 3 + Application] 58. Lambda
[Introduction to Udemy Python 3 + Application] 31. Comments
Python: How to use async with
Link to get started with python
Introduction to Python Numerical Library NumPy
Practice! !! Introduction to Python (Type Hints)
[Introduction to Python3 Day 1] Programming and Python
[Python] Write to csv file with Python
Create folders from '01' to '12' with python
Nice to meet you with python
[Introduction to Python] <numpy ndarray> [edit: 2020/02/22]
[Introduction to Udemy Python 3 + Application] 57. Decorator
Try to operate Facebook with Python
Introduction to Python Hands On Part 1
[Introduction to Python3 Day 13] Chapter 7 Strings (7.1-7.1.1.1)
Output to csv file with Python