Modules cannot be imported in Python on EC2 run from AWS Lambda

Postscript: 2017/06/19

solved. http://qiita.com/tf_qiita/items/60d6d21ffb23656949d2

Original

When the video file was placed on S3, I kicked Python on EC2 with Lambda and I was making a program that starts ffmpeg via ffmpy and processes videos. Since ffmpy cannot be imported normally, I will write it down in the information sharing when it is resolved with the current situation.

environment

The execution code is located under the home directory of ec2-user as shown below.

/home/ec2-user/hoge/
├── MAIN1_KICKED_BY_LAMBDA.py #Main method
├── fuga
│   └── hogefuga.py           #Video editing module
...

I made the following settings for the root user.

ffmpegcpl.sh


#!/bin/sh

sudo yum -y install autoconf automake cmake freetype-devel gcc gcc-c++ git libtool make mercurial nasm pkgconfig zlib-devel

mkdir ~/ffmpeg_sources

#Yasm
cd ~/ffmpeg_sources
git clone --depth 1 git://github.com/yasm/yasm.git
cd yasm
autoreconf -fiv
./configure --prefix="$HOME/ffmpeg_build" --bindir="$HOME/bin"
make
make install
make distclean

#libx264
cd ~/ffmpeg_sources
git clone --depth 1 git://git.videolan.org/x264
cd x264
PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure --prefix="$HOME/ffmpeg_build" --bindir="$HOME/bin" --enable-static
make
make install
make distclean

#libx265
cd ~/ffmpeg_sources
hg clone https://bitbucket.org/multicoreware/x265
cd ~/ffmpeg_sources/x265/build/linux
cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="$HOME/ffmpeg_build" -DENABLE_SHARED:bool=off ../../source
make
make install

#libfdk_aac
cd ~/ffmpeg_sources
git clone --depth 1 git://git.code.sf.net/p/opencore-amr/fdk-aac
cd fdk-aac
autoreconf -fiv
./configure --prefix="$HOME/ffmpeg_build" --disable-shared
make
make install
make distclean

#libmp3lame
cd ~/ffmpeg_sources
curl -L -O http://downloads.sourceforge.net/project/lame/lame/3.99/lame-3.99.5.tar.gz
tar xzvf lame-3.99.5.tar.gz
cd lame-3.99.5
./configure --prefix="$HOME/ffmpeg_build" --bindir="$HOME/bin" --disable-shared --enable-nasm
make
make install
make distclean

#libopus
cd ~/ffmpeg_sources
git clone http://git.opus-codec.org/opus.git
cd opus
autoreconf -fiv
./configure --prefix="$HOME/ffmpeg_build" --disable-shared
make
make install
make distclean

#libogg
cd ~/ffmpeg_sources
curl -O http://downloads.xiph.org/releases/ogg/libogg-1.3.2.tar.gz
tar xzvf libogg-1.3.2.tar.gz
cd libogg-1.3.2
./configure --prefix="$HOME/ffmpeg_build" --disable-shared
make
make install
make distclean

#libvorbis
cd ~/ffmpeg_sources
curl -O http://downloads.xiph.org/releases/vorbis/libvorbis-1.3.4.tar.gz
tar xzvf libvorbis-1.3.4.tar.gz
cd libvorbis-1.3.4
LDFLAGS="-L$HOME/ffmeg_build/lib" CPPFLAGS="-I$HOME/ffmpeg_build/include" ./configure --prefix="$HOME/ffmpeg_build" --with-ogg="$HOME/ffmpeg_build" --disable-shared
make
make install
make distclean

#libvpx
cd ~/ffmpeg_sources
git clone --depth 1 https://chromium.googlesource.com/webm/libvpx.git
cd libvpx
./configure --prefix="$HOME/ffmpeg_build" --disable-examples
make
make install
make clean

#FFmpeg
cd ~/ffmpeg_sources
git clone http://source.ffmpeg.org/git/ffmpeg.git
cd ffmpeg
PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure --prefix="$HOME/ffmpeg_build" --extra-cflags="-I$HOME/ffmpeg_build/include" --extra-ldflags="-L$HOME/ffmpeg_build/lib" --bindir="$HOME/bin" --pkg-config-flags="--static" --enable-gpl --enable-nonfree --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265
make
make install
make distclean
hash -r

Current status / trial

Please refrain from publishing the code to be executed. .. .. The log when executed from Lambda is as follows, and it seems that it is dropped by looking at pyenv (non-existent) of ec2-user.

Traceback (most recent call last):
  File "/home/ec2-user/hoge/MAIN1_KICKED_BY_LAMBDA.py", line 88, in <module>
    hogefuga.exeExtractWav(TMP_DIRECTORY2 + '/', nameonly)
  File "/home/ec2-user/hoge/fuga/hogefuga.py", line 175, in exeExtractWav
    extractWav(filename)
  File "/home/ec2-user/hoge/fuga/hogefuga.py", line 162, in extractWav
    ff.run()
  File "/home/ec2-user/.pyenv/versions/anaconda3-4.3.1/lib/python3.6/site-packages/ffmpy.py", line 99, in run
    raise FFExecutableNotFoundError("Executable '{0}' not found".format(self.executable))
ffmpy.FFExecutableNotFoundError: Executable 'ffmpeg' not found

When I connect with SSH and execute it directly as root, I go to see pyenv on the root side and it is executed without problems. Is there the following two solutions?

--Ec2-Build pyenv and ffmpeg environment in user environment as well --Move the executable files to root's home

In the first place, it will not be refreshing unless you clarify the cause of the path change between direct execution and Lambda execution.

(Added on 2017.05.23 17:12)

--Ec2-Build pyenv and ffmpeg environment in user environment as well --Move the executable files to root's home

I tried both, but ended up with the following error.

----------ERROR-------
failed to run commands: exit status 1
Traceback (most recent call last):
  File "MAIN1_KICKED_BY_LAMBDA.py", line 17, in <module>
    import formatchg
  File "/root/fuga/hoge/fugahoge.py", line 10, in <module>
    from ffmpy import FFmpeg
ImportError: No module named ffmpy

Recommended Posts

Modules cannot be imported in Python on EC2 run from AWS Lambda
Run Python on Schedule on AWS Lambda
[Python] Allow pip3 packages to be imported on AWS Lambda
[Python] Run Headless Chrome on AWS Lambda
Things to note when running Python on EC2 from AWS Lambda
Periodically run a python program on AWS Lambda
Operators ++,-cannot be used in python (difference from php)
[Python] Scraping in AWS Lambda
Write AWS Lambda function in Python
xgboost (python) on EC2 Spot instance environment prepared by AWS Lambda
Best practice for logging in JSON format on AWS Lambda / Python
Run AzureKinect in Python on Christmas Eve.
Python modules with "-(hyphen)" cannot be removed
Tkinter could not be imported in Python
Run Python in C ++ on Visual Studio 2017
Notes on accessing dashDB from python
Tweet from AWS Lambda
Things to note when running Python on EC2 from AWS Lambda
Notes on using MeCab from Python
Run Python on Schedule on AWS Lambda
Python calling Google Cloud Vision API from LINE BOT via AWS Lambda
Modules cannot be imported in Python on EC2 run from AWS Lambda
[Python] Run Headless Chrome on AWS Lambda
Run Python YOLOv3 in C ++ on Visual Studio 2017
Python development on Ubuntu on AWS EC2 (using JupyterLab)
Create a setting in terraform to send a message from AWS Lambda Python3.8 to Slack
TensorFlow: Run data learned in Python on Android
Matplotlib etc. cannot be imported in VirtualEnv environment
# 2 Build a Python environment on AWS EC2 instance (ubuntu18.04)
[Memorandum] Japanese keys cannot be used in python string.Template.substitute
Check types_map when using mimetypes on AWS Lambda (Python)
Deploy Python3 function with Serverless Framework on AWS Lambda
list comprehension because operator.methodcaller cannot be used in python 2.5
Execute python3 system with PHP exec () on AWS EC2
Support for Python 2.7 runtime on AWS Lambda (as of 2020.1)
Run CGI written in python on Sakura's rental server
I want to AWS Lambda with Python on Mac!
Run GPU version tensorflow on AWS EC2 Spot Instances
[Python] Retry process (Exponential Backoff) memo in AWS Lambda
[Redash] Standard library cannot be used in python function
Run python from excel
Run mysqlclient on Lambda
Tweet from AWS Lambda
Run BigQuery from Lambda
Tensorflow cannot be imported
Posted as an attachment to Slack on AWS Lambda (Python)
Run AWS IoT Device SDK for Python on Raspberry Pi
Video cannot be loaded with Spyder in Python development environment
Post images of Papillon regularly on Python + AWS Lambda + Slack
Building an environment to execute python programs on AWS EC2
I want to be able to run Python in VS Code
[AWS Lambda] Use any container Image in Python very simply
If you are having trouble with timeouts when implementing Slack's SlashCommand in Python on AWS Lambda
Run Openpose on Python (Windows)
Run automatic jobs in python
Run shell commands in python
Run Python unittests in parallel
OCR from PDF in Python
Run illustrator script from python
Query Athena from Lambda Python
Run Tensorflow 2.x on Python 3.7
Run Python CGI on CORESERVER
Run unix command on python
Python --Install MySQLDB on EC2
Run YOLO v3 on AWS v2
Run YOLO v3 on AWS
How to install OpenCV on Cloud9 and run it in Python
If you get stuck in Cannot load mkl_intel_thread.dll in Python on Windows