Import your own functions on AWS Glue

Stumble by importing your own function

Simple import is not possible when writing common processing or complicated functions in a separate file on AWS Glue. That said, I just added the file path when creating the job, but I didn't know the relevant part of the document, so make a note of it.

Import method

Basically Using Python libraries with AWS Glue (https://docs.aws.amazon.com/ja_jp/glue/latest/dg/aws-glue-programming-python-libraries.html) It can be handled by referring to the document in.

The basic flow is to zip the python file, put it in S3, and set the script path.

This time we will import the following two simple functions. It is one function and one file.

hello_world.py


def hello_world(name):
    return 'Hello World, ' + str(name)

calcu.py


def sum(x, y):
    return x + y

I put these two files together in a zip and created a zip file called lib.zip this time. I will give this zip file to s3. Suppose you put it in the path s3: //example_backet/lib.zip. Put this path in the choice Python Library Path when creating or editing the job. If there are multiple zip files, enter them separated by commas.

image.png

Now you can import as usual after creating the job.

sample.py


from hello_world import hello_world
from calcu import sum

hoge = hello_world('hoge') 
sum = sum(1, 2)

Recommended Posts

Import your own functions on AWS Glue
Try implementing k-NN on your own
Easily use your own functions in Python
Note on building your own Miniconda environment
[Python] Register your own library on PyPI
Glue Studio [AWS]
Jupyter on AWS
Import your own modules in Grasshopper's Python development
Take your own peak memory usage on Linux & Python
Create your own exception
Install Docker on AWS
Notes on SciPy.linalg functions