python + SCOOP distributed computing cannot be done on a Windows PC

Introduction

This is a memo when I tried distributed computing with a Python library called SCOOP on Windows. SCOOP is a Python library for task distribution, which is different from the Windows package management tool Scoop. It's complicated. Since parallelism and dispersion are amateurs, I believed that a great person could do it, but I did a lot of research, but I couldn't get the result, so I'll leave it so that there aren't any that take the same wasteful steps as myself.

Why you can't

In distributed execution on multiple PCs by SCOOP, in the process os.getpgrp() Call the function. This is only available on Unix, so I get an error. If you use [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install-on-server'Windows Subsystem for Linux'), you can use getpgrp () on it. However, since it will be executed on Linux, it will not be distributed execution on Windows. I was almost fooled because it was possible to execute in parallel on my PC, but in other words --host <host> If you specify, an error will occur.

What i did

I will leave the procedure I tried for reference below.

Preparation

Instance preparation

  1. Prepare two EC2 instances on AWS --Connection source and destination --This time, both are "Windows Server 2019" --OpenSSH is manually prepared on Windows Server 2016 --Both issue fixed IP --Add "SSH" to the inbound rule of the security group of the instance to connect to with all addresses OK (because it is troublesome) --Check "File name extension" and "Hidden file" in the folder display settings

Preparing SSH connection (public key authentication)

  1. Addition of OpenSSH server function --Settings → Apps → Manage optional features → Add features → Install "OpenSSH Server" --"Start" OpenSSH from the service and change the startup type to "Automatic" --Added "New rule ..." with port number "22" in "Inbound rule" of firewall --Confirm that SSH communication is possible between instances just in case --Check if you can log in with ssh Administrator @ <IP of the other instance>
    , if it does not time out and you are prompted for a password
  2. Public key communication settings --Set up to allow communication between instances using SSH public key --Create a private / public key pair with ssh-keygen on the connection source instance --Place the public key (***. pub) directly under the .ssh folder of the connection destination instance and rename it to "authorized_keys" --The .ssh folder should be in "C: \ Users \ Administrator \ .ssh" unless you do something special ――This time it was the first one, so I renamed it, but after the second one, add it in authorized_keys. --Comment out the last two lines of the "C: \ ProgramData \ ssh \ sshd_config" file (the line that Administrators manages to write) with "#" and restart OpenSSH.
  3. Create a config file --Create ".ssh \ config" on the connection source instance and define the connection name and connection destination as follows. --The indentation other than Host is for readability when describing multiple connection destinations, so if it is troublesome, you do not need to lower it. --In this example, if the connection is successful with
    ssh test
    , it's OK.
Host <Arbitrary name>
  HostName <IP address of the connection destination>
  User <Any user>
  Port <The port you want to use>
  IdentityFile <The path of the private key you have>
---The following is this example---
Host test
  HostName **.***.***.***
  User Administrator
  Port 22
  IdentityFile C:/Users/Administrator/.ssh/id_rsa

Install build tools for visual studio 2019

  1. Open "Visual Studio 2019 Tools" at the bottom of the Official Page screen.
  2. Download and run Build Tools for Visual Studio 2019
  3. Check C ++ Build Tools --At this time, check the two MSVC140 and 141 under the checklist that appears on the right side of the screen and click "Install".

Python ready

  1. Install python --I used the latest version 3.9.0 --Install directly without using anaconda etc. --Installing the library - pip install pyzmq -Download for SCOOP Official Manual I wrote it, but the URL file is quite old ... - pip install scoop
  2. Prepare the python program you want to distribute -Copy and paste map_doc.py from the Official Manual --Specify the function you want to execute in parallel with furures.map (function you want to distribute, argument you want to pass to the ← function)

map_doc.py


from __future__ import print_function
from scoop import futures

def helloWorld(value):
    return "Hello World from Future #{0}".format(value)

if __name__ == "__main__":
    returnValues = list(futures.map(helloWorld, range(16)))
    print("\n".join(returnValues))

Trial

If you can do so far, follow the official manual python -m scoop -n 8 map_doc.py If this is the case, it will be executed in parallel within the own PC and will succeed normally. python -m scoop -n 8 --host test map_doc.py If you do this, you will get the following error and it will fail. The Attribute Error: module'os' has no attribute'getpgrp' at the end of this error is the cause of the failure. There is nothing that isn't there.

b'Traceback (most recent call last):\r\n  File "C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python39\\lib\\runpy.py", line 197, in _run_module_as_main\r\n    return _run_code(code, main_globals, None,\r\n  File "C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python39\\lib\\runpy.py", line 87, in _run_code\r\n    exec(code, run_globals)\r\n  File "C:\\Users\\Administrator\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\scoop\\broker\\__main__.py", line 65, in <module>\r\n    sys.stdout.write(str(os.getpgrp()) + "\\n")\r\nAttributeError: module \'os\' has no attribute \'getpgrp\'\r\n'

Finally

When I was about to give up, I consulted with other people and found that there are many patterns such as "If you want to do distributed computing, try using it in a supercomputer in the future, so I write programs on the premise of Linux in the first place, I do not do much on Windows. I received a comment like "Yo". That seems to be the case. I wanted to do it on Windows this time because I wanted to execute programs written on the assumption that it would run on a Windows PC in parallel, but it can't be helped because there is no OS that has no functions.

Reference link

https://scoop.readthedocs.io/en/0.7/index.html https://docs.python.org/ja/3/library/os.html http://fx-kirin.com/python/python-scoop/ https://www.server-world.info/query?os=Windows_Server_2019&p=ssh&f=3

Recommended Posts

python + SCOOP distributed computing cannot be done on a Windows PC
What to do if PyAudio cannot be installed on Python 3.7, 3.8, 3.9 on Windows
Creating a python virtual environment on Windows
Python on Windows
Dual_boot windows10 and linux by using clover on a uefi PC that cannot be booted from nvme
Project cannot be created with Python3.5 (Windows) + django1.7.1
Simply build a Python 3 execution environment on Windows
Python environment settings (virtualenv + pip cannot be done)
Investigation when import cannot be done with python
Create a Python virtual development environment on Windows
Build a GVim-based Python development environment on Windows 10 (3) GVim8.0 & Python3.6
Build a Python extension for E-Cell 4 on Windows 7 (64bit)
Until drawing a 3D graph in Python on windows10
Create a comfortable Python 3 (Anaconda) development environment on windows
Build a GVim-based Python development environment on Windows 10 (1) Installation
python basic on windows ②
Install python on windows
Procedure for building a CDK environment on Windows (Python)
Create a decent shell and python environment on Windows
Build a GVim-based Python development environment on Windows 10 (2) Basic settings
Set-enable Python virtualenv on Windows
Install watchdog on Windows + Python 3.3
Python + Kivy development on Windows
Sphinx-autobuild (0.5.2) on Windows7, Python 3.5.1, Sphinx 1.3.5
Fastest Python installation on Windows
Build Python environment on Windows
Build python environment on windows
I ran python on windows
[Python] [Chainer] [Windows] Install Chainer on Windows
Use Python on Windows (PyCharm)
Create a Python environment for professionals in VS Code on Windows
Build a 64-bit Python 2.7 environment with TDM-GCC and MinGW-w64 on Windows 7
A record that GAMEBOY could not be done in Python. (PYBOY)
A note on using tab completion when running Python interactively on Windows
Everything from building a Python environment to running it on Windows
Building a Python environment on Mac
Python environment construction memo on Windows 10
Python 3.6 on Windows ... and to Xamarin.
Installing Kivy on Windows10 64bit Python3.5
Hit a command in Python (Windows)
Anaconda python environment construction on Windows 10
Building a Python environment on Ubuntu
Install python2.7 on windows 32bit environment
Create a Python environment on Mac (2017/4)
Install Python on Windows + pip + virtualenv
Install Pytorch on Blender 2.90 python on Windows
Integrate Modelica and Python on Windows
Installing Kivy-Designer on Windows10 64bit Python3.5
Create a Linux environment on Windows 10
Mecab / Cabocha / KNP on Python + Windows
Create a python environment on centos
Install Python development environment on Windows 10
Python CGI file created on Windows
Getting started with Python 3.8 on Windows
Build a python3 environment on CentOS7
Reproduce One-Touch Search on Python 3.7.3. (Windows 10)
Introducing a library that was not included in pip on Python / Windows
Create a Python development environment on Windows (Visual Studio Code remote WSL).
Steps to create a Python virtual environment with VS Code on Windows
Modules cannot be imported in Python on EC2 run from AWS Lambda
A python script for Mac that zips without garbled characters on Windows