Pooling mechanize with Django

Premise

Create a WEB API with Django called business improvement In-house, partly using mechanize to convert intra login and intra operations to API I was having a good time.

problem

Since it is a WEB API, access is basically stateless, and an instance of mechanize is created each time for intra-operation. Therefore, from the perspective of the intranet, it was ** a new login ** for each request to the API. In the near future, I will be angry with the information system and want to improve.

Coping

Instead of creating an instance of mechanize every time, I decided to pool it.

Original caller


worker = IntraWorker(USER,PASS)
worker.do_something()

IntraWoker is a class that uses mechanize. I log in every time with __init__. I made this below.

New caller


# get from pool
worker = IntraWorkerPool().get(USER,PASS)
worker.do_something()
# back to pool
worker.close()

Add a class called IntraWorkerPool and let it manage the pool.

IntraWorkerPool


from threading import Lock

class IntraWorkerPool(object):
    """
Mechanize pooling management

    """
    #Worker being pooled
    # user,Use the pass tuple as a key
    pooled_workers = {}

    def __init__(self):
        self.lock = Lock()
    def get(self, user, password):
        """
Remove a Worker instance from the pool
        """
        ret_aw = None
        pool = IntraWorkerPool.pooled_workers
        key = (hashed(user), password)
        with self.lock:
            if key in pool:
                #Make if not available
                if len(pool[key]) == 0:
                    pool[key].append(IntraWorker(*key, pool_control=self))
            else:
                pool[key] = []
                pool[key].append(IntraWorker(*key, pool_control=self)
            
            #Check if the worker is still able to log in
            while (ret_aw is None) or (not ret_aw.is_login()):
                ret_aw = pool[key].pop()

        return ret_aw

    def back_to_pool(self, aw_inst):
        """
Return the instance to pool
You don't actually call this directly. Call with close on the worker side

        """
        key = (hashed(aw_inst.user_no), aw_inst.user_password)
        pool = IntraWorkerPool.pooled_workers
        with self.lock:
            pool.setdefault(key, [])
            pool[key].append(aw_inst)

What we are doing is managing a tuple of usernames and passwords and a list of Worker instances with a dictionary called pooled_workers as a class variable. Also, list operations should be thread-safe, so I'm serializing them using threading.Lock (). To return to the pool, a method called back_to_pool is prepared, which is also protected by Lock, but in reality, back_to_pool is executed when the Close method on the Worker instance side is executed.

To be honest, I don't like the fact that the Pool management class and the managed class are interdependent, but I can't think of a smart implementation and the tea is muddy in this way.

I would appreciate it if anyone could tell me if there is something smarter.

Recommended Posts

Pooling mechanize with Django
Internationalization with django
Authenticate Google with Django
Django 1.11 started with Python3.6
Upload files with Django
Development digest with Django
Output PDF with Django
Markdown output with Django
Use Gentelella with django
Twitter OAuth with Django
Send email with Django
File upload with django
Use LESS with Django
Use MySQL with Django
Start today with Django
Getting Started with Django 2
Get started with Django! ~ Tutorial ⑤ ~
Create an API with Django
Do Django with CodeStar (Python3.8, Django2.1.15)
Deploy Django serverless with Lambda
Python3 + Django ~ Mac ~ with Apache
Getting Started with Python Django (1)
Create a homepage with django
Get started with Django! ~ Tutorial ④ ~
Getting Started with Python Django (4)
Web application creation with Django
Getting Started with Python Django (3)
Save tweet data with Django
Do AES encryption with DJango
Getting Started with Python Django (6)
Combine two images with Django
Getting Started with Django with PyCharm
Real-time web with Django Channels
Double submit suppression with Django
Django REST framework with Vue.js
Use prefetch_related conveniently with Django
Getting Started with Python Django (5)
Login with django rest framework
Qiita API Oauth with Django
Connection pooling with Python + MySQL
Test Driven Development with Django Part 3
reload in django shell with ipython
Steps to develop Django with VSCode
Test Driven Development with Django Part 4
Load Django modules with an interpreter
Set up social login with Django
Test Driven Development with Django Part 6
Manage Django config files with Python-decouple
Deploy a Django application with Docker
Common html to rent with Django
Django Model with left outer join
Test Driven Development with Django Part 2
Django
Django Tips-Create a ranking site with Django-
Twitter posting application made with Django
Automatically generate model relationships with Django
Run python3 Django1.9 with mod_wsgi (deploy)
Build a web application with Django
Make a filter with a django template
[Django] Use MessagePack with Django REST framework
Excel, csv import, export with Django