Steps to attach and debug from VS Code to Jupyter Lab on a remote server

1. What to explain in this post

Follow the steps below to build an environment for debugging using Jupyter Lab and VS Code.

  1. Launch Jupyter Lab on a remote server
  2. Call a Python class from JupyterLab.
  3. From VS Code on the local PC, remotely attach to the above Jupyter Lab notebook and debug.
  4. The figure is as follows.
スクリーンショット 2020-04-08 00.09.27.png

If you watch the video below, you can get an idea of what you can do. Untitled.gif

  1. Motivation
  2. JupyterLab is convenient for writing a little code and sharing the results neatly together, but when creating a program in earnest, you probably use an editor such as VS Code.
  3. On the other hand, when verifying using the program developed in this way, it is often convenient to call it from JupyterLab.
  4. In this usage, we will explain how to set a breakpoint on the program called from JupyterLab and build an environment for debugging from VS Code.

3. Points of the environment to be built this time

--In VS Code, by using a mechanism called remote development, you can edit folders and files on the remote server almost the same as on your local PC by connecting with ssh. See here for details. --VS Code allows you to attach and debug Python processes on the remote server by installing a package called ptvsd on the remote server. See here for details. --In JupyterLab, it is possible to use any process (not necessarily Python) as an interpreter by using a mechanism called kernel. ――In this paper, by combining the above, the above explanation is realized.

4. Environment construction

4.1. Remote server environment construction

4.1.1. Prerequisites

--In this article, we built the environment based on the Docker image "tensorflow / tensorflow: 1.14.0-gpu-py3-jupyter". ――However, if you have already installed Jupyter Lab with anaconda etc., you can use it as it is. (I'll explain it below, but the point is that it works if ptvsd is installed on the remote server and set as the kernel of Jupyter Lab.)

4.1.2. Installation of required apps

# apt-get update
# apt-get install -y wget
# apt-get install -y language-pack-ja
# apt-get install -y ssh
# apt-get install -y git
# update-locale LANG=ja_JP.UTF-8

4.1.3. Installation of nodejs

Required to install the JupyterLab Extension. This step is not necessary if you just want to set the kernel, but it is better to support it for convenient use of Jupyter Lab.

# curl -sL https://deb.nodesource.com/setup_12.x | bash -
# apt-get install -y nodejs

4.1.4. Installing Python & JupyterLab

--In particular, you need jupyterlab and ptvsd. --This time, I will install numpy, matplotlib, scikit_learn. --As of April 2020, version 2 of jupyterlab is out, but since there are many extensions that are not yet supported, we will install 1.0.2 this time.

# pip install -r requirements.txt

requirements.txt


numpy==1.16.3
matplotlib==3.0.3
scikit_learn==0.20.3
jupyterlab==1.0.2
ptvsd==4.3.2

4.1.5. Start ssh to access from VS Code on the local PC side

Required to connect to a remote server from VS Code for remote development. This time, open the ssh port on port 19205.

# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.org
# sed -e s/#Port/Port/g /etc/ssh/sshd_config.org  | sed -e s/22/19205/g > /etc/ssh/sshd_config

Next, start ssh.

root@25fa703a90c4:/tf# service ssh start
 * Starting OpenBSD Secure Shell server sshd                             [ OK ]

4.2. Jupyter Lab settings

Next, set up Jupyter Lab. This time, we will start the jupyterLab server as a jupyter user.

4.2.1. Change ownership of / usr / local / share / jupyter / lab directory

First, in preparation, change ownership of the / usr / local / share / jupyter / lab directory to jupyter. If you do not do this, you will not be able to install Extensions from Jupyter Lab under startup as a jupyter user, which is inconvenient. I think there is a better way (for example, keep these files in your home directory). Here, I will simply change the ownership below.

# chown -R jupyter:jupyter /usr/local/share/jupyter/lab

4.2.2. Set ptvsd as kernel of Jupyter Lab

First, save the following file with the file name /usr/local/share/jupyter/kernels/ptvsd/kernel.json so that you can call the debug interpreter from JupyterLab. (The method of calling the interpreter set here from JupyterLab will be explained later.)

kernel.json


{
 "argv": [
  "/usr/bin/python3",
  "-m",
  "ptvsd",
  "--host",
  "0.0.0.0",
  "--port",
  "19204",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python3 Debug",
 "language": "python"
}

It is like this.

root@25fa703a90c4:/usr/local/share/jupyter/kernels# pwd
/usr/local/share/jupyter/kernels
root@25fa703a90c4:/usr/local/share/jupyter/kernels# mkdir ptvsd
root@25fa703a90c4:/usr/local/share/jupyter/kernels# cd ptvsd
root@25fa703a90c4:/usr/local/share/jupyter/kernels/ptvsd# cat > kernel.json <<EOL
{
 "argv": [
  "/usr/bin/python3",
  "-m",
  "ptvsd",
  "--host",
  "0.0.0.0",
  "--port",
  "19204",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python3 Debug",
 "language": "python"
}
EOL
root@25fa703a90c4:/usr/local/share/jupyter/kernels/ptvsd# ls kernel.json
kernel.json

This completes the Jupyter Lab settings.

4.3. VS Code settings

Next, make settings for remote development with VS Code.

4.3.1. Preparing to connect with ssh

First of all, if you haven't done so already, you need to prepare for key exchange so that you can connect to the remote server from your local PC with ssh. I will omit the detailed procedure, but I think that here etc. will be helpful.

4.3.2. Installing the Remote Development Extension Pack

Follow these steps (https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack) to install the Remote Development extension pack in Visual Studio.

4.3.3. Settings to connect to a remote server from VS Code

Follow the steps below to connect to a remote server from VS Code 1 and edit folders and files. (From the second time onward, you can also connect using the saved settings.)

  1. Click the button at the bottom left of the screen
skitch.2.png 2. When the command palette is displayed, select "Remote-SSH: Connect to Host". skitch.1.png 3. Select "Add New SSH Host ..." skitch.png 4. Enter the command to access with ssh from the command line. For example, enter: Convenient!
ssh user@host name-p port name
skitch.4.png 5. Specify the save location of ssh settings (usually, specify the one under the personal folder) skitch.3.png 6. Click the "Connect" button in the dialog that appears at the bottom right skitch.5.png

4.3.4. Installing Python extension for Visual Studio Code

Next, install the Python extension for Visual Studio Code from here to debug Python on VS Code.

This completes the connection from VS Code to the remote server. This procedure allows you to edit files and folders on the remote server with VS Code as if they were on your local PC.

5. Run the sample program

5.1. Description of sample program

It's very easy, but I will use the sample program below. First, on the remote server, save the following files in the same directory where you save the notebook.

sample.py


import numpy as np

def sample(x):
    y = np.square(x)
    return y

5.2. Access to Jupyter Lab

Next, start JupyterLab on the remote server and access JupyterLab from the browser on the local PC. (It is the same as accessing Jupyter Lab normally.)

5.2.1. Start Jupyter Lab on remote PC

jupyter lab --ip=0.0.0.0

5.2.2. Access Jupyter Lab

You can find the following message in the log when jupyterLab is started, so you can open JupyterLab by accessing the URL from your local PC. * At the time of access, the IP address will be changed to that of the remote PC.

    To access the notebook, open this file in a browser:
        file:///home/jupyter/.local/share/jupyter/runtime/nbserver-13-open.html
    Or copy and paste one of these URLs:
        http://(da9742c1f7ef or 127.0.0.1):8888/?token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

You can access JupyterLab by accessing the URL output when 5.2.1 JupyterLab starts from a browser on your local PC.

5.3. Specify Kernel and create notebook

When you access JupyterLab, you should see a kernel called "Python3 debug" in the Notebook column as shown below. (This is the kernel using ptvsd set in step 4.2.2.) Click this button to create a new notebook. JupyterLab.png

The upper right of the screen is "Python3 Debug", and you can see that the Kernel for Debug created earlier is used.

JupyterLab2.png

For the time being, create a notebook like the one below and run it.

Jupyter_and_VSCode_sample.png

5.4. Setting up remote debugging in VS Code

5.4.1. Connect to a remote server and open the project

Follow the steps in 4.3 to connect to the remote server with VS Code. Once connected, open the sample program. It is like this.

sample_py.png

5.4.2. Debug settings (launch.json settings)

Follow the steps below to configure debug settings.

VSCode2.png

The settings are as follows.

--port: Specify the port set in the kernel settings of ptvsd. --host: Specify the IP address of the remote server. --pathMappings: localRoot should be $ {workspaceFolder} as shown in the example below. remoteRoot specifies the folder on the remote server where the files are stored.

{
    //You can use IntelliSense to learn the available attributes.
    //Hover and display the description of existing attributes.
    //Check the following for more information: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
    
        {
            "name": "Python:attach",
            "type": "python",
            "request": "attach",
            "port": 19204,
            "host": "192.168.2.200",
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "/home/jupyter/proj/docker"
                }
            ]
        }
    ]
}

This completes the debug settings. Next, let's actually debug.

5.5. Attach to JupyterLab process and debug

The debugging procedure is as follows. See also the demo video at the beginning for details.

  1. After setting breakpoints on VS Code as appropriate, execute the "Python attach" set earlier.

  2. Run notebook from JupyterLab

  3. Debugging is performed because the processing is interrupted at the breakpoint.

  4. The result of the command executed in "Debug Console" on VS Code is also displayed on JupyterLab. Especially, it is very convenient because you can display internal variables with matplotlib etc. and display them as a graph!

  5. Summary So far, we have explained how to debug a Python process executed from JupyterLab by remotely attaching it from VS Code. It's very convenient, so please try it. It's a little scribbled, so there may be some deficiencies. If you have any questions, please comment and I would like to add or correct as appropriate.

Recommended Posts

Steps to attach and debug from VS Code to Jupyter Lab on a remote server
Run Jupyter notebook on a remote server
Steps to create a Python virtual environment with VS Code on Windows
[Jupyter Notebook / Lab] 3 ways to debug on Jupyter [Pdb]
Good and bad code to compare on a minimap
Building a Python environment on a Mac and using Jupyter lab
A simple way to launch Jupyter Notebook / Lab and set a password
Building a Python development environment on Windows -From installing Anaconda to linking Atom and Jupyter Notebook-
Simple code to call a python program from Javascript on EC2
How to develop containers on remote servers with VS Code Remote-Containers
Migrate from VS Code to PyCharm
How to build Python and Jupyter execution environment with VS Code
Set up a yum repository server on CentOS7 system and refer to it locally and from other servers.
I want to pass an argument to a python function and execute it from PHP on a web server
I made a tool to convert Jupyter py to ipynb with VS Code
I want to format and check Python code to my liking on VS Code
Build Linux on a Windows environment. Steps to install Laradock and migrate
Let Code Day6 Starting from Zero "1342. Number of Steps to Reduce a Number to Zero"
Execute a script from Jupyter to process
Start jupyter notebook on GPU server (remote server)
Build jupyter notebook on remote server (CentOS)
Steps to run TensorFlow 2.1 from Jupyter on supercomputer ITO front end (with GPU)
Cross-compiling Raspberry Pi and building a remote debugging development environment with VS Code
When I connect to a remote Jupyter Server with VScode, it's remote but local
Introduce VS Code and Remote Development to an offline environment to make linux development comfortable
Build a PYNQ environment on Ultra96 V2 and log in to Jupyter Notebook
Send a message from Slack to a Python server
Enable Jupyter Notebook with conda on remote server
How to Git from GCP's Jupyter Lab to GSR
From installing Flask on CentOS to making it a service with Nginx and uWSGI
Steps to quickly create a deep learning environment on Mac with TensorFlow and OpenCV
[Django] Use VS Code + Remote Containers to quickly build a Django container (Docker) development environment.
How to build a Python virtual execution environment using Visual Studio Code and pipenv on a Windows machine (also Jupyter notebook)
Steps to measure coverage and get badges on codecov.io
Steps from installing Python 3 to creating a Django app
How to make a Python package using VS Code
How to run Django on IIS on a Windows server
Remote debug Django environment created with docker-compose with VS Code
How to create a kubernetes pod from python code
I want to get rid of import warnings from Pyright and pylint in VS Code
I made a server with Python socket and ssl and tried to access it from a browser