Python --Simple multi-thread sample

Ref: [1] - Stackoverflow - Multiple Threads in Python

This pattern is usually used in Http requesting. Since computing tasks might cause a lot of context switches, which might decrease the speed.

import threading

urlTaskList = [
# your tasks here
]

parallelism = 4

# here is the task you want to run
def subTask(urlList):
  import urllib2

  for url in urlList:
    data = urllib2.urlopen(url).read()
  
    # process the data here
    # ...


# this method is used to seperate task chunk
def splitChunk(tasks):
  """ Yield successive n-sized chunks from l.
  """
  for i in xrange(0, len(l), n):
    yield l[i:i+n]


threadList = list()
chunkList = list(splitChunk(urlTaskList))


for subChunk in chunkList:
  worker = threading.Thread(target=subTask, args=(subChunk))
  worker.start()
  threadList.append(worker)
  

# wait for every thread finish its work
for worker in threadList:
  worker.join()

The method splitChunk(taskList) above is from this post.

Recommended Posts

Python --Simple multi-thread sample
Simple gRPC in Python
Ajax + Python + PostgreSQL sample
Python --Simple security software
Python Socket communication sample / simple data throwing tool
Sample data created with python
A simple sample of pivot_table.
GitHub Actions Python cache sample
# 1 [python3] Simple calculation using variables
Simple FPS measurement of python
Python higher-order function (decorator) sample
Simple regression analysis in Python
Sample usage of Python pickle
Simple HTTP Server for python
Simple IRC client in python
First simple regression analysis in Python
Python
AtCoder: Python: Daddy the sample test.
[Python] Test sample using unittest2, mock
My favorite boto3 (Python) API sample
Predictive Statistics (Practice Simple Regression) Python
Python parallel / parallel processing sample code summary
Implementing a simple algorithm in Python 2
Automate simple tasks with Python Part0
Run a simple algorithm in Python
NAOqi Python sample (make NAO walk)
Simple gacha logic written in Python
Google Cloud Vision API sample for python
Creating a simple PowerPoint file with Python
Socket communication and multi-thread processing by Python
A simple HTTP client implemented in Python
Simple Slack API client made with Python
Try drawing a simple animation in Python
Automate simple tasks with Python Part1 Scraping
Machine learning with python (2) Simple regression analysis
PHP / Python / Ruby sample hitting Path API
Create a simple GUI app in Python
Use Amazon Simple Notification Service with Python
Sample script to trap signals in Python
Write a simple greedy algorithm in Python
Sample for handling eml files in Python
Ported Python parallel computing sample to F #
Write a simple Vim Plugin in Python 3
Which is faster, Python shuffle or sample?
I made a simple blackjack with Python
Sample to convert image to Wavelet with Python