Handle requests in a separate process

Introduction

Another personal memo.

Since it looks like ↓, I wrote it because I wanted to separate the process according to the request source and the type of message when waiting for reception.

Source code

main.py


#!/usr/bin/env python
# -*- coding: utf-8 -*-

import procs


workers = {}
source = procs.Source()

try:
    while True:

        #Poll from source
        data = source.poll()
        key = data['key']

        #Generated if there is no Worker corresponding to key
        if key not in workers:
            workers[key] = procs.Worker()

        #Delegate processing to worker
        workers[key].delegate(data)

finally:
    #Terminate Worker when finished with Keyboard Interrupt etc.
    source.terminate()
    [ w.terminate() for _,w in workers ]

procs.py


#!/usr/bin/env python
# -*- coding: utf-8 -*-

import abc
import multiprocessing


class Proc(metaclass=abc.ABCMeta):
    
    def __init__(self):
        self._stop    = multiprocessing.Event()
        self._queue   = multiprocessing.SimpleQueue()
        self._process = multiprocessing.Process(target=self._run)
        self._process.start()

    def terminate(self):
        self._stop.set()
        self._process.join()
        self._process.terminate()

    def _run(self):
        while not self._stop.is_set():
            self._do()

    @abc.abstractmethod
    def _do(self, **kwargs):
        pass


class Source(Proc):
    
    def _do(self, **kwargs):
        #Receiving messages
        data = { 'key' : 'some-data' }
        self._queue.put(data)

    def poll(self):
        return self._queue.get()

        
class Worker(Proc):

    def _do(self, **kwargs):
        data = self._queue.get()
        #Process the received data

    def delegate(self, data):
        return self._queue.put(data)

Recommended Posts

Handle requests in a separate process
Daemonize a Python process
dict in dict Makes a dict a dict
Try loading the image in a separate thread (OpenCV-Python)
Use ujson in requests
Handle markdown in python
Handle Parquet in Python
Ingenuity to handle data with Pandas in a memory-saving manner
Process the files in the folder in order with a shell script
Take a screenshot in Python
Handle constants in Django templates
Create a function in Python
Collaborate in a remote environment
Handle environment variables in Python
Make a bookmarklet in Python
Draw a heart in Python
Implement Gaussian process in Pyro
Handle complex numbers in Python
Process the contents of the file in order with a shell script
IQ Bot Custom Logic (Python): Efficient replacement process in a loop
About psd-tools, a library that can process psd files in Python
IQ Bot Custom Logic (Python): Efficient replacement process in a loop
<Pandas> How to handle time series data in a pivot table