How to use PyCharm with Glue development endpoints running inside a VPC
The basics follow the article below. Tutorial: Set up PyCharm Professional on Development Endpoint
However, since this article assumes a development endpoint that runs outside the VPC, the procedure is significantly different for endpoints that run inside the VPC.
Things to watch out for when dealing with endpoints in a VPC
――Because you are in a private VPC, do not touch it directly --You need to create a bastion server to make a connection with SFTP --In addition, you need to create a local ssh tunnel to connect with PyCharm.
Reasons to create an endpoint in a VPC in the first place
If Glue deals only with S3 as a data source, it can be outside the VPC, but when handling the JDBC system as a data source, it needs to be placed inside the VPC.
Diagram
Environment / premise
- Mac OS
- PyCharm Professional -Tutorial Prerequisites is over to the development endpoint in VPC for JDBC What you are doing --Set the public key for the endpoint created in the premise of the tutorial
Implementation
1. Creating a bastion server
--Creating a bastion server in a common way --Get not forgetting only the ssh key --Created in the same VPC as the development endpoint --The VPC to which the development endpoint belongs can be confirmed by clicking the development endpoint and checking the details of the endpoint that appears.
2. Setting up a private subnet SSH security group
--Allow ssh only from the public subnet where the platform is located in the private subnet security group
3. Create an SSH tunnel
--Create an SSH tunnel as shown below
ssh -i <Stepping stone server pem> ec2-user@<Stepping stone server public dns> -L <Local port you want to set>:<Private address of the development endpoint>:22 -N
--Give the v option when it doesn't work and you want to debug. --The private address of the development endpoint can be confirmed in detail
4. PyCharm settings
--Set up the code and libraries as described in the tutorial
--The deployment part is very different from the tutorial
--Host specifies localhost
--Port specifies the local port set in the tunnel
--ʻUser name is glue --ʻAuthentication is Key pair OpenSSH
--Private key path is the private key path of the public key set in the development endpoint (not the EC2 private key)
--OK if you succeed with Test Connection
The rest is OK if you do the same as the tutorial.
Other
--Multiple public keys can be set for the development endpoint, so there is no need to share keys. --Ssh tunnel needs to be alive when developing with PyCharm
reference
Tutorial: Set up PyCharm Professional on Development Endpoint Securely Connect to Linux Instances Running in a Private Amazon VPC | AWS Security Blog Spark development with AWS Glue-Ota's storeroom Debugging with Remote PyCharm-Help | PyCharm
Recommended Posts