Understand the arguments required for the HTTP request method of the Requests module

Hi, I'm a late demon moon walker. This article is an article of JSL (Japan System Giken) Advent Calendar 2020 14th.

The portable HDD I bought for backup was replaced due to an initial failure, and the mac I replaced with a new one was also initially defective and part of the screen disappeared, so I wrote it in a desperate situation: innocent:

How did you come to write this article?

A while ago, when implementing HTTP/IF in Requests, even though parse failed at the request destination because I did not understand the parameters when executing the HTTP method, "API does not connect! I once made a noise. I decided to keep it as a memorandum so that I and other engineers would not follow the same rut.

First, figure out the HTTP methods that Requests support

From requests/api.py

Of these, OPTIONS and HEAD used in addition to CRUD are excluded this time.

These commonly pass arguments such as HTTP method name to the request method in api.py.

def request(method, url, **kwargs):
    with sessions.Session() as session:
        return session.request(method=method, url=url, **kwargs)

Delve into session.request

    ....
    def request(self, method, url,
            params=None, data=None, headers=None, cookies=None, files=None,
            auth=None, timeout=None, allow_redirects=True, proxies=None,
            hooks=None, stream=None, verify=None, cert=None, json=None):
    ....

Many keyword arguments came out, but the parameters specified in the HTTP request function are

There are only three. (Headers if you use it often?)

Check the value entered in each parameter with httpbin

params

Only get uses params. The code is very simple

params_test.py


import pprint
import requests


if __name__ == '__main__':
    response = requests.get(
        'http://httpbin.org/get',
        params={'name': 'moonwalker'},
    )
    pprint.pprint(response.json())

result...

$ python params_test.py
{'args': {'name': 'moonwalker'},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.25.1',
             'X-Amzn-Trace-Id': 'Root=1-5fe9fabf-007c29751b44b5817125a68e'},
 'origin': '126.194.217.32',
 'url': 'http://httpbin.org/get?name=moonwalker'}

If you specify params, it will be specified as a query string in the url.

data

data is used in POST/PUT/PATCH. Run POST as an example.

data_test.py


import pprint
import requests


if __name__ == '__main__':
    response = requests.post(
        'http://httpbin.org/post',
        data={'name': 'moonwalker'},
    )
    pprint.pprint(response.json())

result...

$ python data_test.py
{'args': {},
 'data': '',
 'files': {},
 'form': {'name': 'moonwalker'},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Content-Length': '15',
             'Content-Type': 'application/x-www-form-urlencoded',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.25.1',
             'X-Amzn-Trace-Id': 'Root=1-5fe9fc82-43df8cad5846bad3276bb4d9'},
 'json': None,
 'origin': '126.194.217.32',
 'url': 'http://httpbin.org/post'}

Notice that inserting a dictionary into the data argument defaults the Content-Type to application/x-www-form-urlencoded. This is the same movement as below.

...
    data = urllib.parse.urlencode({'name': 'moonwalker'})
    response = requests.patch(
        'http://httpbin.org/patch',
        data=data,
        headers={'Content-Type': 'application/x-www-form-urlencoded'},
    )
...

json

Finally, json. I was at the mercy of this guy. .. .. json is specified only in the post function. I think this is mainly used for API requests accepted by json.

json_test.py


import pprint
import requests

if __name__ == '__main__':
    response = requests.post(
        'http://httpbin.org/post',
        json={'name': 'moonwalker'},
    )
    pprint.pprint(response.json())

result...

$ python json_test.py
{'args': {},
 'data': '{"name": "moonwalker"}',
 'files': {},
 'form': {},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Content-Length': '22',
             'Content-Type': 'application/json',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.25.1',
             'X-Amzn-Trace-Id': 'Root=1-5fea0046-7ff25f850de04a71401e3046'},
 'json': {'name': 'moonwalker'},
 'origin': '126.194.217.32',
 'url': 'http://httpbin.org/post'}

Notice that inserting a dictionary into the json argument defaults the Content-Type to application/json. This is the same movement as below.

import json
...
    response = requests.post(
        'http://httpbin.org/post',
        data=json.dumps({'name': 'moonwalker'}),
        headers={'Content-Type': 'application/json'},
    )
...

My failure

I used data and json at the time of post with a fluffy understanding, so I wrote it like this.

miss_test.py


...
    response = requests.post(
        'http://httpbin.org/post',
        json=json.dumps({'name': 'moonwalker'}),
    )
...

Request data is strange

$ python miss_test.py
{'args': {},
 'data': '"{\\"name\\": \\"moonwalker\\"}"',
 'files': {},
 'form': {},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Content-Length': '28',
             'Content-Type': 'application/json',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.25.1',
             'X-Amzn-Trace-Id': 'Root=1-5fea0160-4290dd525d9fb77b419f693d'},
 'json': '{"name": "moonwalker"}',
 'origin': '126.194.217.32',
 'url': 'http://httpbin.org/post'}

As you can see from the data and json in ↑, json.dumps was duplicated. That's why you can't parse at the request destination ... Even if you pass it to the data argument, you can expect the same behavior by json.dumps, so if you use it in an atmosphere, it will lead to careless mistakes.

Summary

Reference article

About the Python Requests module

Recommended Posts

Understand the arguments required for the HTTP request method of the Requests module
Mock HTTP requests via the requests module in Responses
Make the display of Python module exceptions easier to understand
What is the default TLS version of the python requests module?
Python: Find the required plate thickness for the limit buckling pressure of the exposed tube by the Brent method
How to use machine learning for work? 01_ Understand the purpose of machine learning
[Required subject DI] Implement and understand the mechanism of DI with Go
Test the version of the argparse module
Understand the contents of sklearn's pipeline
Implementation of the treatise of "Parameter estimation method for human flow simulation" (unofficial)