You will be an engineer in 100 days --Day 35 --Python --What you can do with Python

What you can do with Python today.

Click here for the last time [You will become an engineer in 100 days --Day 34 --Python --Python Exercise 3] (https://qiita.com/otupy/items/400f89fd2755bc7e47f0)

After that, how can it be used for work or research?

[] (https://youtu.be/c-yVo9l_Qvw)

Now How was the basics of programming?

If you can do all the grammar and do some exercises I think you can understand it as it is

It ’s still at this stage What you can do with programming

What is it useful for ... I think that there are many people who do not have an image.

So, while looking at the code actually used at work What programming is like Experience it again and what to do after that I would like to dig deeper into that.

Read file

with open(File Path)as variable name:
processing

#Display the contents of files placed in the same hierarchy.
with open('sample.py') as _r:
    print(_r.read())

def hello(aa): print(aa)

With variable name.read () when reading Reads all the contents of the file. In the above example, all the code in the file is read and printed.

Read CSV file

The reading part of the file is the same. CSV is a file in a format separated by , You can read by separating with ,.

Convert to list type separated by comma with string.split (',')

#Prepare a variable to store the result
res = []
#Read file
with open('sample.csv') as _r:
    for row in _r:
        #Remove line breaks and separate them with commas to make an array
        rows = row.replace('\n','').split(',')
        #Add to variable for result
        res.append(rows)

print(res)

[['aaa', 'bbb', 'ccc'], ['ddd', 'eee', 'fff'], ['hhh', 'iii', 'jjj'], ['kkk', 'lll', 'mmm']]

for row in res:
    #Display tab-separated
    print('\t'.join(row))

aaa bbb ccc ddd eee fff hhh iii jjj kkk lll mmm

Scraping

Scraping is a technology for accessing websites and acquiring information. Because the Python language has a library for scraping Information can be obtained from the website relatively easily.

requests.get (site URL) Access the website and get information.

import requests

#Access the website and get the data
html = requests.get('http://yahoo.co.jp')

#Display the first 800 characters of the acquired data
print(html.content.decode('utf-8')[0:800])

<! DOCTYPE HTML PUBLIC "-// W3C // DTD HTML 4.01 Transitional // EN" "http://www.w3.org/TR/html4/loose.dtd"> ・ ・ ・

Use of machine learning libraries

** Data visualization **

Dataframe.plot () Visualize the data with (default is a line graph)

import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
#Creating a data frame
df = pd.DataFrame([[3,4],[4,5],[6,9],[2,8]],columns=['a','b'])
#Draw data frame
df.plot()

plt.scatter (data frame, data frame) Display a scatter plot using two columns of data frames.

df = pd.DataFrame([[3,4],[4,5],[6,9],[2,8]],columns=['1','2'])
#Draw a scatter plot
plt.scatter(df['1'],df['2'])

scikit learn

scikit learn is a library for machine learning with a group of programs for creating various models. Sample data for learning is available.

datasets.load_iris() Reading training data (iris: Ayame sample data)

import pandas as pd
#Import required libraries
from sklearn import datasets, model_selection, svm, metrics

#Reading the data of the famous iris
iris = datasets.load_iris()
#Convert iris data into a data frame.
iris_data = pd.DataFrame(data=iris.data, columns=iris.feature_names)
#Show only 5 lines
iris_data.head()

#Read label data
iris_label = pd.Series(data=iris.target)
#Show only 5 lines
iris_label.head()

0 0 1 0 2 0 3 0 4 0 dtype: int64

train_test_split (training data, correct label) Divide the training data into training and testing.

#Divide the iris data into training data, test data, training label, and test label.
train_data, test_data, train_label, test_label = model_selection.train_test_split(iris_data, iris_label)
#Training data
train_data.head()

#Training label
train_label.head()

70 1 125 2 77 1 25 0 51 1 dtype: int64

#Number of training data and test data
print(len(train_data), '\t',  len(test_data))

112 38

Variable name = Learning model variable name. Class () Training model variable name.fit (training data, training label) Learn with training data.

#Definition of SVM learner
clf = svm.SVC()

#Learning with training data
clf.fit(train_data, train_label)

#Predicted by test data
pre = clf.predict(test_data)

print(type(pre))
print(pre)

<class 'numpy.ndarray'> [0 0 1 1 0 2 1 0 2 1 2 0 2 2 0 1 0 0 2 1 0 0 0 2 0 2 2 2 1 0 2 0 1 2 2 1 0 1]

ʻAccuracy_score (test label, predicted value) ` Calculate the correct answer rate of the predicted value

#Correct answer rate
ac_score = metrics.accuracy_score(test_label, pre)
print(ac_score)

0.947368421053

Since there are about 10 types of sample data alone You can try out various machine learning models.

Other

I won't put the code together to see what else you can do, but here's an example.

** Automation of routine work ** Sending and receiving emails, creating a list Collection of SNS posts and posts Creation of Excel, Word documents, etc. (report creation) PDF text extraction GUI operation

** Database operation ** Data processing, registration, addition, deletion Mass data processing

Image processing Image collection Image processing using opencv etc. Face extraction Object detection

** Statistical analysis ** Calculation of basic statistics Distribution calculation and visualization regression analysis Interval estimation Hypothesis test

** Web application development ** Website construction using frameworks such as Flask and Django

** Game development ** Game development using frameworks such as PyGame and Kivy

** Natural language processing ** Text mining Morphological analysis Dependency n-gram woed2vec

** AI development ** Machine learning DeepLearning Reinforcement learning GAN

Summary

Now that you have learned all about programming. I've come to the grammar, so I should be able to write it.

From here, let's create the program you want to create.

I've put together a list of frequently used codes.

I will post a link here, so please refer to it. https://note.com/otupy/n/n1bedb9f36e54

65 days until you become an engineer

Author information

Otsu py's HP: http://www.otupy.net/

Youtube： https://www.youtube.com/channel/UCaT7xpeq8n1G_HcJKKSOXMw

Twitter： https://twitter.com/otupython