Use AWS SDK for Python (boto) under Proxy environment

When I was making a Python script using the boto library to play with AWS, I made a note of it because it fit under the Proxy environment. Please note that it is close to bad know-how, which is not a best practice.

environment

・ Linux (RHEL6.5) ・ Python 3.3.6 -AWS SDK for Python (boto2.34.0) ・ Under HTTP Proxy environment

Event

If you use Proxy, you will get the following error when calling API.

# python3
>>> import boto.ec2
>>> conn = boto.ec2.connect_to_region(‘ap-northeast-1’)
>>> conn.get_only_instances(instance_ids=[‘i-0de8dc14’])
Traceback (most recent call last):
  File “<stdin>”, line 1, in <module>
  File “/opt/python3.3.6/lib/python3.3/site-packages/boto/ec2/connection.py”, line 623, in get_only_instances
    next_token=next_token)
  File “/opt/python3.3.6/lib/python3.3/site-packages/boto/ec2/connection.py”, line 682, in get_all_reservations
    [(‘item’, Reservation)], verb=‘POST’)
  File “/opt/python3.3.6/lib/python3.3/site-packages/boto/connection.py”, line 1166, in get_list
    response = self.make_request(action, params, path, verb)
  File “/opt/python3.3.6/lib/python3.3/site-packages/boto/connection.py”, line 1112, in make_request
    return self._mexe(http_request)
  File “/opt/python3.3.6/lib/python3.3/site-packages/boto/connection.py”, line 913, in _mexe
    self.is_secure)
  File “/opt/python3.3.6/lib/python3.3/site-packages/boto/connection.py”, line 705, in get_http_connection
    return self.new_http_connection(host, port, is_secure)
  File “/opt/python3.3.6/lib/python3.3/site-packages/boto/connection.py”, line 747, in new_http_connection
    connection = self.proxy_ssl(host, is_secure and 443 or 80)
  File “/opt/python3.3.6/lib/python3.3/site-packages/boto/connection.py”, line 796, in proxy_ssl
    sock.sendall(“CONNECT %s HTTP/1.0\r\n” % host)
TypeError: ‘str’ does not support the buffer interface

Provisional measures

It looks like SSL communication wasn't going through well, so When making a connection, I was able to connect by adding ʻis_secure = False`. However, as the setting value name suggests, it is likely to be non-SSL communication, so you should be careful about the usage scene.

>>> conn = boto.ec2.connect_to_region(‘ap-northeast-1’,is_secure=False)
>>> conn.get_only_instances(instance_ids=[‘i-0de8dc14’])
>>> [Instance:i-0de8dc14]

boto config

It seems that the boto library can be loaded with some settings at runtime.

  • /etc/boto.cfg - for site-wide settings that all users on this machine > will use
  • ~/.boto - for user-specific settings
  • ~/.aws/credentials - for credentials shared between SDKs

Boto Config — boto v2.35.1

Regarding Proxy, it seems good to describe as follows.

~/.boto


[Boto]
proxy = myproxy.com
proxy_port = 8080

However, in my environment, this did not seem to be able to be read well for some reason, so I came up with hard coding as a temporary measure. If you have any know-how in this area, please comment.

Recommended Posts

Use AWS SDK for Python (boto) under Proxy environment
boto3 (AWS SDK for Python) Note
Use Resource API rather than Client API in AWS SDK for Python (Boto3)
Authentication information used by Boto3 (AWS SDK for Python)
AWS SDK for Python (Boto3) development in Visual Studio 2017
Use smbus with python3 under pyenv environment
[Cloudian # 1] Try to access object storage with AWS SDK for Python (boto3)
How to use pip3 under proxy environment Note
Call Polly from the AWS SDK for Python
Docker under proxy environment
Python environment for projects
Fleet provisioning with AWS IoT SDK for Python v2
Use Python installed with pyenv for PL / Python execution environment
Python environment construction For Mac
Python3 environment construction (for beginners)
Python environment construction under Windows7 environment
[For organizing] Python development environment
Run AWS IoT Device SDK for Python on Raspberry Pi
Pip install (Windows) under Proxy environment
python memo pip under proxy, easy_install
Next, use Python (Flask) for Heroku!
Install Networkx in Python 3.7 environment for use in malware data science books
Python development environment options for May 2020
Emacs settings for Python development environment
AWS Layer Creation Script for python
Python3 TensorFlow for Mac environment construction
Beginners use Python for web scraping (1)
Quick reference table under Proxy environment
Beginners use Python for web scraping (4) ―― 1
I easily built an operating environment for Python3 + Tornado on AWS EC2.
(For myself) Flask_AWS_1 (Install PHP, MySQL, phpMyAdmin, Python in AWS virtual environment)
[AWS] Problem avoidance memo when installing awscli and boto3 in Python 2.7 environment
Use the pip command under an HTTP proxy environment that requires authentication
Build an environment for Blender built-in Python
Python development environment for macOS using venv 2016
Use DeepL with python (for dissertation translation)
Python project environment construction procedure (for windows)
AWS ~ For those who will use it ~
Python environment tool comparison chart for Rubyist
Let's create a virtual environment for Python
Creating amateur python environment settings (for MAC)
[Mac] Building a virtual environment for Python
[Python] Organizing how to use for statements
Install Python3 using pythonz under macports environment
How to use tensorflow under docker environment
How to use "deque" for Python data
Use pathlib in Maya (Python 2.7) for upcoming Python 3.7
Building a Python development environment for AI development
Python environment preparation (venv first time use)
Prepare the development environment for Python on AWS Cloud9 (pip install & time change)
Selenium + WebDriver (Chrome) + Python | Building environment for scraping
Use Azure ML Python SDK 2: Use dataset as input-Part 2
Building an environment for executing Python scripts (for mac)
Upgrade the Azure Machine Learning SDK for Python
Use logger with Python for the time being
Use multiple versions of python environment with pyenv
Wrap C with Cython for use from Python
Get note information using Evernote SDK for Python 3
Image Processing with Python Environment Setup for Windows
Use Python in your environment from Win Automation
Manage AWS nicely with the Python library Boto