I dare to fill out the form without using selenium

Introduction

It's not so much as "dare", but when I'm investigating form editing in Python, selenium often appears, so I implemented it with Requests + Beautiful Soup.

Posted before is just right as a development, so I edited Qiita's profile text.

Editing function

Many editable items were found to be of type ['text','url','checkbox'] from the source, so I was able to get them with little effort. If you add'user [desctiption]]', which is an acquisition omission, and edit it, the post data is completed. For some reason, the check box for publishing email was checked by default, so I am fixing it to 0. Once you have created the data, post it using the login session and you're done.

def edit_profile(session):
    profile_page_data = get_page_data(session, profile_url)
    bs_profile_page = BeautifulSoup(profile_page_data.text, 'html.parser')
    authenticity_token = str(bs_profile_page.find(attrs={'name':'authenticity_token'}).get('value'))
    post_data = {
        'utf-8':'✓',
        '_method': 'put',
        'authenticity_token':authenticity_token
    }
    response = bs_profile_page.find_all(attrs={'type':['text', 'url', 'checkbox']})
    for i in response:
        value = i.get('value')
        post_data[i.get('name')] = value
    post_data['user[public_email]'] = 0 #1 by default for some reason
    post_data['user[description]'] = 'I edited it! !! !! !!'
    print(post_data)
    response = session.post(profile_url, post_data)
    print(response)

Execute

Add the edit_profile () described above to the program created last time to complete the program.

import requests
import os
from bs4 import BeautifulSoup
import json


user_name = 'user_name'
user_password = 'user_password'
login_url = 'https://qiita.com/login'
profile_url = 'https://qiita.com/settings/profile'
    

def get_authenticity_token(session, login_url):
    page_data = get_page_data(session, login_url)
    bs_page_data = BeautifulSoup(page_data.text, 'html.parser')
    authenticity_token = str(bs_login_page.find(attrs={'name':'authenticity_token'}).get('value'))
    return bs_page_data, authenticity_token


def get_page_data(session, url):
    response = session.get(url)
    response.encoding = response.apparent_encoding
    return response


def login(session):
    login_form = {
        'utf-8':'✓',
        'authenticity_token':'token',
        'identity':user_name,
        'password':user_password
    }
    bs_login_page, authenticity_token = get_authenticity_token(session, login_url)
    login_form['authenticity_token'] = authenticity_token
    session.post(login_url, login_form)
    

def edit_profile(session):
    bs_profile_page, authenticity_token = get_authenticity_token(session, profile_url)
    post_data = {
        'utf-8':'✓',
        '_method': 'put',
        'authenticity_token':authenticity_token
    }
    response = bs_profile_page.find_all(attrs={'type':['text', 'url', 'checkbox', 'textarea']})
    for i in response:
        value = i.get('value')
        post_data[i.get('name')] = value
    post_data['user[public_email]'] = 0 #1 by default for some reason
    post_data['user[description]'] = 'I edited it! !! !! !!'
    print(post_data)
    response = session.post(profile_url, post_data)
    print(response)
    
if __name__ == '__main__':
    session = requests.Session()
    login(session)
    edit_profile(session)

If it is executed and Response [200] is returned, it is successful.

Recommended Posts

I dare to fill out the form without using selenium
How to find out the number of CPUs without using the sar command
[Selenium] Go to the next page without pressing Next
I tried to summarize the basic form of GPLVM
I tried to approximate the sin function using chainer
I want to automate ssh using the expect command!
I tried to identify the language using CNN + Melspectogram
I tried to complement the knowledge graph using OpenKE
I tried to compress the image using machine learning
[Uncorrelated test] I tried to put out the boundary line with or without rejection
I tried to find out the outline about Big Gorilla
I want to use the latest gcc without sudo privileges! !!
I want to automate ssh using the expect command! part2
I want to move selenium for the time being [for mac]
I tried to simulate ad optimization using the bandit algorithm.
Start to Selenium using python
[TF] I tried to visualize the learning result using Tensorboard
pyenv, virtualenv Use docker to manage the environment without using
I tried to approximate the sin function using chainer (re-challenge)
I tried to output the access log to the server using Node.js
I tried to get the index of the list using the enumerate function
I wanted to challenge the classification of CIFAR-10 using Chainer's trainer
I tried to digitize the stamp stamped on paper using OpenCV
I tried to cut out a still image from the video
I calculated the stochastic integral (I to integral)
I tried to move the ball
I tried using the checkio API
I tried to estimate the interval.
I tried to understand the learning function in the neural network carefully without using the machine learning library (second half).
I tried to transform the face image using sparse_image_warp of TensorFlow Addons
I tried to execute SQL from the local environment using Looker SDK
I tried to get the batting results of Hachinai using image processing
Python -I tried to restore the dictionary comprehensive notation to its original form-
I tried to estimate the similarity of the question intent using gensim's Doc2Vec
[Python] Warning came out when I raised from Selenium 3 to 4 [Web Driver]
I learned scraping using selenium to make a horse racing prediction model.
I tried to control multiple servo motors MG996R using the servo driver PCA9685.
(Python Selenium) I want to check the settings of the download destination of WebDriver
I tried to summarize various sentences using the automatic summarization API "summpy"
I tried to extract and illustrate the stage of the story using COTOHA
I tried the common story of using Deep Learning to predict the Nikkei 225
Using COTOHA, I tried to follow the emotional course of Run, Melos!
I tried to analyze the New Year's card by myself using python
I tried to understand the learning function of neural networks carefully without using a machine learning library (first half).
I tried to get the information of the .aspx site that is paging using Selenium IDE as non-programming as possible.
I want to pin Spyder to the taskbar
I want to output to the console coolly
I tried using Azure Speech to Text.
I tried to summarize the umask command
I want to handle the rhyme part1
I tried to recognize the wake word
I tried using Selenium with Headless chrome
I want to handle the rhyme part3
I tried to classify text using TensorFlow
I tried to summarize the graphical modeling.
Reboot the router using Python, Selenium, PhantomJS
I can't get the element in Selenium!
I tried to estimate the pi stochastically
I tried to touch the COTOHA API
I want to display the progress bar
I tried using Headless Chrome from Selenium