Automatic posting of web design gary site with python + selenium (1) Environment construction

Overview

Motivation

I decided to use wordpress to create a web design gallery site specializing in treatment clinics in connection with my side business. However, it is very troublesome to save the capture for the gallery site image and update to wordpress every time. So I created an automatic posting environment after studying python. Since it will be long, I will post it in several times.

\ * Auto-posting here refers to running a python script on the terminal, taking screen captures, creating post text, logging in to worpdress, and posting.

Premise

Contents

  1. Environment construction
  2. Get screen capture-selenium
  3. Image cropping and compression-PIL, tinypng
  4. Create CSV for posting --bs4
  5. Auto Post -selenium

The actual gallery site will be here.

Folder structure

image_cap
├── cap.py #Capture save, csv write script
├── images #Where to save the captured image
├── min-images #Where to save the compressed captured image
├── list.txt #Gallery-Get the URL of the site you want to add to
├── up-list.csv #CSV used for bulk posting
├── upload.py #Script for wordpress posting
├── setting.config #Settings such as login ID, password and image directory path

For the time being, the source code is managed on github. Please click here.

1. Environment construction

Local settings

Install selenium

Software for programmatically running a web browser used for capturing captures. It seems to be used mainly for testing websites.

$ pip install selenium

Download chromewebdriver

I want to use Chrome this time, so download it from homebrew.

$ brew install chromedriver
$ which chromedriver 
  usr/local/bin/chromedriver #Set this as a selenium pass

installation of beautifulsoup4

It is used to get the title of the posted site by scraping.

$ pip install beautifulsoup4

install tinify

To use the tinypng API for image compression. The post of here was very helpful. Please also set the API KEY.

$ pip install tinify

lxml installation

Installed to use lxml as an html parser. To be honest, I don't know what the parser will do. .. Sweat

pip install lxml

I referred to here.

Read configuration file

Variables such as login ID and file path are summarized using python's ConfigParser to make it easier for future fixes. We will use this in the code below.

setting.config


[CAPTURE] #Captured image path
img_pass = /cap_git/images
min_img_pass = /cap_git/min_image

[RESIZE] #tinypng API key
tiny_key = hogehoge

[WRITE] #csv write settings
init_row = post_title,post_category,capture,url,post_type,post_status
post_type = post
post_status = publish
up_address = http://test/wp-content/themes/test/dist/images/

[UPLOAD] #Wordpress login information etc.
login_url = http://test/wp-login.php
login_id = hoge
login_pass = hogehoge
import_url = http://test/wp-admin/admin.php?import=csv
img_up_pass = http://test/wp-content/themes/sage-8.5/dist/images

So in cap.py load the variables below

cap.py


#Read configuration file
conf = configparser.ConfigParser()
conf.read('setting.config')

#API key settings
tinify.key = conf['RESIZE']['tiny_key']
#Dir with unreduced image
FROM_DIR = conf['CAPTURE']['img_pass']
#Place the reduced image dir
MIN_DIR = conf['CAPTURE']['min_img_pass']
#Setting items
INIT_ROW = list(conf['WRITE']['init_row'].split(','))
#Image URL
IM_ADRRESS = conf['WRITE']['up_address']
#Post type to set
POST_TYPE = conf['WRITE']['post_type']
POST_STATUS = conf['WRITE']['post_status']

Module configuration with configparser.ConfigParser () Specify the path of the confg file in conf.reade and read it. Since the value can be accessed in a dictionary-like format, I insert it into a variable.

Wordpress side settings

Add Really Simple CSV Importer Plugin

For bulk posting in csv. I referred to here. In addition, Advanced Custom Fields also sets custom fields for posting.

Continue to save part 2 screen capture

Recommended Posts

Automatic posting of web design gary site with python + selenium (1) Environment construction
Automatic scraping of reCAPTCHA site every day (1/7: python environment construction)
Automatic operation of Chrome with Python + Selenium + pandas
Poetry-virtualenv environment construction with python of centos-sclo-rh ~ Notes
Environment construction of python2 & 3 (OSX)
Environment construction of python and opencv
Get started with Python! ~ ① Environment construction ~
Python3 environment construction with pyenv-virtualenv (CentOS 7.3)
pytorch @ python3.8 environment construction with pipenv
Web application created with Python + Flask (using VScode) # 1-Virtual environment construction-
[Ubuntu 18.04] Python environment construction with pyenv + pipenv
Horse Racing Site Web Scraping with Python
Practice web scraping with Python and Selenium
Let's get along with Python # 0 (Environment construction)
Environment construction of Flask / MySql / Apache / mod_wsgi / virtualenv with Redhat7 (Python2.7) November 2020
Easy-to-understand explanation of Python Web application (Django) even for beginners (1) [Environment construction]
Collecting information from Twitter with Python (Environment construction)
Prepare the execution environment of Python3 with Docker
Python environment construction
[Note] Export the html of the site with python.
Environment construction (python)
Use multiple versions of python environment with pyenv
Installation of Python3 and Flask [Environment construction summary]
[Python] Web application from 0! Hands-on (1) -Design, DB construction-
python environment construction
Python --Environment construction
Python environment construction
First python ① Environment construction with pythonbrew & Hello World !!
From Python environment construction to virtual environment construction with anaconda
python environment construction
Automatic login to ServiceNow with Selenium Web Driver
Virtual environment construction with Docker + Flask (Python) + Jupyter notebook
Recommendation of building a portable Python environment with conda
[Python] OpenCV environment construction with Docker (cv2.imshow () also works)
Automatic follow on Twitter with python and selenium! (RPA)
Data analysis environment construction with Python (IPython notebook + Pandas)
Creating BINGO "Web Tools" with Python (Table of Contents)
QGIS3 Python plugin development environment construction with VSCode (macOS)
python windows environment construction
homebrew python environment construction
Python development environment construction
ScreenShot with Selenium (Python)
Python environment with docker-compose
python2.7 development environment construction
Scraping with Selenium [Python]
Python web scraping selenium
Mac environment construction Python
Python environment construction @ Win7
OpenJTalk on Windows10 (Speak Japanese with Python from environment construction)
Unify the environment of the Python development team starting with Poetry
[LDAP environment construction: 7] Add / search / change / delete users with Python
Construction of Python local development environment Part 2 (pyenv-virtualenv, pip usage)
Memo of python + numpy/scipy/pandas/matplotlib/jupyterlab environment construction on M1 macOS (as of 2020/12/24)
Until Python [Django] de Web service is released [Environment construction]
[SAP CP] Web API created with python in CF environment
The highlight of REST + Normal Site construction with Flask, Eve
Automatic operation of Chrome with Python + Selenium + pandas
Python + Anaconda + Pycharm environment construction
Web scraping with python + JupyterLab
Scraping with selenium in Python
Scraping with Selenium + Python Part 1