MultiDict: When doing form POST with a crawler, etc.

MultiDict

There can be multiple choices, so try using MultiDict from werkzeug:

$ pip install werkzeug

From the list dict:

>>> from werkzeug.datastructures import MultiDict
>>> MultiDict([('a',1), ('a', 2), ('a', 3)])
MultiDict([('a', 1), ('a', 2), ('a', 3)])

add to:

>>> d.update({'a': '4'})
>>> d
MultiDict([('a', 1), ('a', 2), ('a', 3), ('a', '4')])

update

>>> d['a']=5
>>> d
MultiDict([('a', 5)])

Get form data with BeautifulSoup

Get form from HTML with BeautifulSoup select ():

    def form_data(self, form_node):
        ''' form_node :Select from HTML()Form node
        '''

        data = MultiDict(list(
           (t['name'], t.get('value', '')) 
           for i in ['text', 'hidden', 'password', ]
           for t in form_node.select('input[type=%s]' % i)
        ) + list(
           (t['name'], t.text) 
           for t in form_node.select('textarea')
        ) + list(
           (t['name'], t.get('value', '')) 
           for t in form_node.select('input[checked]' % i)
        ) + list(
            (s.parent['name'], s.get('value', s.text))
            for s in form_node.select('option[selected]')
        ))  

        # TODO: "file", "image", 
        # TODO: HTML5 -- input(list) and datalist

        return data

When POSTing with requests, only the first parameter is passed

requests says that it supports MultiDict, but it doesn't work in my environment. .. ..

It can't be helped, so I converted it to dict for the time being:

>>> dict(d.lists())
{'a': [1, 2, 3, '4']}
    def post(self, uri, data):

        if isinstance(data, MultiDict):
            data = dict(data.lists())

        self.res = self.req.post(
            uri, data=data,
            verify=False, allow_redirects=False)
        self.last_uri = uri

Recommended Posts

MultiDict: When doing form POST with a crawler, etc.
A workaround when installing pyAudio with pip.
Error when installing a module with Python pip
Personal tips when doing various things with Python 3
I get a UnicodeDecodeError when running with mod_wsgi
A memo when creating a python environment with miniconda