How to use PyCharm with Glue development endpoints running inside a VPC

The basics follow the article below. Tutorial: Set up PyCharm Professional on Development Endpoint

However, since this article assumes a development endpoint that runs outside the VPC, the procedure is significantly different for endpoints that run inside the VPC.

Things to watch out for when dealing with endpoints in a VPC

――Because you are in a private VPC, do not touch it directly --You need to create a bastion server to make a connection with SFTP --In addition, you need to create a local ssh tunnel to connect with PyCharm.

Reasons to create an endpoint in a VPC in the first place

If Glue deals only with S3 as a data source, it can be outside the VPC, but when handling the JDBC system as a data source, it needs to be placed inside the VPC.

Diagram

空白の UML - 空白の UML.png

Environment / premise

Implementation

1. Creating a bastion server

--Creating a bastion server in a common way --Get not forgetting only the ssh key --Created in the same VPC as the development endpoint --The VPC to which the development endpoint belongs can be confirmed by clicking the development endpoint and checking the details of the endpoint that appears.

2. Setting up a private subnet SSH security group

--Allow ssh only from the public subnet where the platform is located in the private subnet security group

3. Create an SSH tunnel

--Create an SSH tunnel as shown below

ssh -i <Stepping stone server pem> ec2-user@<Stepping stone server public dns> -L <Local port you want to set>:<Private address of the development endpoint>:22 -N

--Give the v option when it doesn't work and you want to debug. --The private address of the development endpoint can be confirmed in detail

4. PyCharm settings

--Set up the code and libraries as described in the tutorial --The deployment part is very different from the tutorial --Host specifies localhost --Port specifies the local port set in the tunnel --ʻUser name is glue --ʻAuthentication is Key pair OpenSSH --Private key path is the private key path of the public key set in the development endpoint (not the EC2 private key) --OK if you succeed with Test Connection スクリーンショット 2020-03-15 20.48.09.png

The rest is OK if you do the same as the tutorial.

Other

--Multiple public keys can be set for the development endpoint, so there is no need to share keys. --Ssh tunnel needs to be alive when developing with PyCharm

reference

Tutorial: Set up PyCharm Professional on Development Endpoint Securely Connect to Linux Instances Running in a Private Amazon VPC | AWS Security Blog Spark development with AWS Glue-Ota's storeroom Debugging with Remote PyCharm-Help | PyCharm

Recommended Posts

How to use PyCharm with Glue development endpoints running inside a VPC
How to add a package with PyCharm
How to use Cmder with PyCharm (Windows)
Understanding how to use Jinja2 makes development with Flask smarter
How to build a python2.7 series development environment with Vagrant
Python: How to use async with
How to use virtualenv with PowerShell
How to use FTP with Python
How to use an external editor for Python development with Grasshopper
How to use OpenVPN with Ubuntu 18.04.3 LTS
How to use Ass / Alembic with HtoA
How to use Japanese with NLTK plot
How to use jupyter notebook with ABCI
How to use Anaconda interpreter in PyCharm
How to use CUT command (with sample)
How to use SQLAlchemy / Connect with aiomysql
How to use JDBC driver with Redash
[Python] Explains how to use the range function with a concrete example
How to read a CSV file with Python 2/3
How to use GCP trace with open Telemetry
A simple example of how to use ArgumentParser
How to send a message to LINE with curl
How to draw a 2-axis graph with pyplot
How to develop a cart app with Django
Flutter in Docker-How to build and use a Flutter development environment inside a Docker container
How to make a dictionary with a hierarchical structure.
How to use tkinter with python in pyenv
[Note] How to create a Ruby development environment
How to set up a local development server
How to create a multi-platform app with kivy
[Note] How to create a Mac development environment
A story about how Windows 10 users created an environment to use OpenCV3 with Python 3.5
How to use fixture in Django to populate sample data associated with a user model
How to use Docker to containerize your application and how to use Docker Compose to run your application in a development environment
How to use python multiprocessing (continued 3) apply_async in class with Pool as a member
How to convert / restore a string with [] in python
How to use xgboost: Multi-class classification with iris data
A memo of how to use AIST supercomputer ABCI
[Python] How to draw a line graph with Matplotlib
How to create a submenu with the [Blender] plugin
How to get a logged-in user with Django's forms.py
How to convert a class object to a dictionary with SQLAlchemy
How to use App Engine's local development server dev_appserver.py
[Mac OS] Use Kivy with PyCharm! [Python application development]
How to make a shooting game with toio (Part 1)
A memorandum on how to use keras.preprocessing.image in Keras
How to use python interactive mode with git bash
From buying a computer to running a program with python
Autoencoder with Chainer (Notes on how to use + trainer)
How to build a development environment for TensorFlow (1.0.0) (Mac)
Scraping with Python-Selenium is old! ?? ・ ・ ・ How to use Pyppeteer
[Python] How to create a 2D histogram with Matplotlib
Build a development environment with Poetry Django Docker Pycharm
[Python] How to draw a scatter plot with Matplotlib
How to use xml.etree.ElementTree
How to use Python-shell
How to use tf.data
How to use virtualenv
How to use Seaboan
How to use image-match
How to use shogun