parallelization of class method

Class method parallelization and process monitoring

In python, parallelization can be easily realized by using multiprocessing.Pool of multiprocessing which is a parallelization library, but it cannot be used for parallelization such as class method. In this case, there is no choice but to parallelize using multiprocessing.Process, but if you run processes for each number of CPUs, when you run more processes than the number of CPUs, the CPU of the process that ended earlier will be freed and it will be wasted. It ends up. I wondered if I could run a new process from a CPU that ended up like SGE, so I forcibly created a script for it. Maybe there is a better way.

An example of a parallel process management script when parallelizing the class Method.

import multiprocessing as mp
ps = []
for i in range(1000):
    ps.append(mp.Process(target=hoge,args=(i,))) #Generate 1000 target functions or methods

finishedList       = []
presentindex       = 0
for p in ps[0:mp.cpu_count()]:
    p.start()
    presentindex += 1

while 1:
    time.sleep(1)#Monitor every second.
    for i, p in enumerate(ps[:presentindex]):
        if p.is_alive() or i in finishedList:#Judgment of end of process p
            pass
        else:
            finishedList.append(i)#Save index of terminated process
            if presentindex < len(ps):
                ps[presentindex].start()
                presentindex += 1
                print(i)

    if len(finishedList) == len(ps):#Exit if the number of terminated processes and the number of created processes are the same
        break

Recommended Posts

parallelization of class method
Clustering of clustering method
Class method static method
Summary of test method
Globalization of class instance variables
Behavior of pandas rolling () method
Profile within a class method
Parallelization
Use instance method and class method properly
[Python of Hikari-] Chapter 09-03 Class (inheritance)
class
Einsum implementation of value iterative method
Introduction of data-driven controller design method
Calculation of homebrew class and existing class
[python] -1 meaning of numpy's reshape method
class
Hit a method of a class instance with the Python Bottle Web API
Explain the mechanism of PEP557 data class
Summary of gamma distribution parameter specification method
[Learning memo] Basics of class by python
Count / verify the number of method calls.
Example of using class variables and class methods
Test of uniqueness in paired comparison method
[Python] Difference between class method and static method
Comparison of class inheritance and constructor description
[Pandas_flavor] Add a method of Pandas DataFrame
Various format specifications of str.format () method of Python3
Summary of SQLAlchemy connection method by DB
Construction of development environment for Choreonoid class
You can call the method of the class directly in Python3 without adding @staticmethod