[EC2] What to do when selenium is stuck and processing does not proceed

[EC2] What to do when selenium is stuck and processing does not proceed

--When using selenium on EC2, it takes a long time to get the data. ――It went smoothly at first, but as the execution is repeated, the processing becomes heavier. --If you wait for a long time, you will finally get an error that you cannot reach chrome.

error


$python3 filename.py

selenium.common.exceptions.WebDriverException: 
Message: chrome not reachable

Cause

Multiple processes and drivers for the chrome browser have been launched. (Because it is headless, it cannot be visually confirmed ...)

approach

-Stop the running chrome browser & driver. -Write object.quit () at the end of the file.

Remedy procedure

1. 1. Check if multiple chrome drivers are launched.

python


#Show all running processes including "chrome"
$ ps aux | grep chrome


#If you see a lot of descriptions like the one below, multiple webdrivers are running.
Username pts/0    Sl   04:09   0:00 
chromedriver --port=49671

#If you see a lot of statements like the one below, multiple browser processes are running.
Username pts/0    S    04:32   0:00
/opt/google/chrome/chrome --type=broker

ps aux

--ps command: Display the command being executed --aux is an option --a: All user processes --u: Display user name and start time --x: Shows processes without control terminals


`| grep text`

-"| (Pipe)": Added command --grep text: Show only processes that include text


### 2. Quit chrome browser & webdriver

python


killall chrome
killall webdriver

Check the running file again.

python


$ ps aux | grep chrome


#Only the following was left and the others were finished.
Username pts/0    S+   04:35   0:00 
grep --color=auto chrome

### 3. 3. Add quit () to processing Execute quit () at the end of the .py file to close all browsers.

python


#Assuming the following description
browser = webdriver.Chrome(options=options)

browser.quit()

If the variable for starting webdriver is described in another description such as driver, match it. Example: driver.quit ()


Above, -The execution of the py file becomes much faster. -There will be no running processes after processing.

Supplement

In another article, there was something like stopping the sandbox and adding --no-sandbox as an option, but in my case, the sandbox was not started and was unnecessary.

If sandbox is displayed in the running process, it may be effective to try the following as well.

killall chrome-sandbox
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-setuid-sandbox")

The original article is here

By the way, the sandbox is an isolated environment that does not affect outside programs. (Used to run programs that are likely to contain viruses, etc.)

Recommended Posts

[EC2] What to do when selenium is stuck and processing does not proceed
What to do when Japanese is not displayed on matplotlib
[Beginner] What to do when "[Errno 2] File b'test.csv' does not exist: b'test.csv" is displayed when reading pandas csv
NameError: global name'dot_parser' is not defined and what to do when it comes up in python
What to do if the user name is changed and the pyenv library path does not pass
What to do if pyenv install does not proceed with an error
Notes on what to do when matplotlib scatter () / scatter3d () does not work
Example of what to do when the sample script does not work (OpenCV-Python)
What to do when the graph does not appear in jupyter (ipython) notebook
curl: (60) What to do when Issuer certificate is invalid.
What to do if sys / cdefs.h does not exist
What to do if pyenv is not enabled (zsh)
[Python] Type Error:'WebElement' object is not iterable What to do when an error occurs
What to do when python3 type venv does not work well on Raspberry Pi
What to do when is not in the sudoers file.This incident will be reported.
What to do when only the window is displayed and nothing is displayed in pygame Note
What to do when Python starts up in Anaconda does not come out unexpectedly
What to do if you get the error "Error: opencv3: Does not support building both Python 2 and 3 wrappers" when installing openCV 3
What to do when PyCharm font is strange or garbled
Why django-import-export import is so slow and what to do
What to do when a Missing artifact occurs in a jar that is not defined in pom.xml
What to do when Ubuntu crashes
What to do when the jupyterlab extension settings are not reflected
What to do if the inode is exhausted on EC2 Linux
What to do when the value type is ambiguous in Python?
[VSCode] unable to import'google.cloud' What to do when pylint (import-error) does not recognize the Python import statement
What to do when the result downloaded via scrapy is in English
[Mac OS] What to do when Python is not installed as a framework. Is displayed when import matplotlib is performed.
[Python] What to do when an error related to SSL authentication is returned
What to do in my case when pyenv install is not possible after upgrading to macOS Big Sur
What to do when the warning "The environment is in consistent ..." appears in the Anaconda environment
What to do when you get angry that libxml / xmlversion.h does not exist when you put lxml with pip
What to do when a warning message is displayed in pip list
What to do if you get the error RuntimeError: Python is not installed as a framework when trying to use matplitlib and pylab in Python 3.3
What to do if CERTIFICATE_VERIFY_FAILED occurs when nltk.download () is done on macOS pyhon
What to do when "TypeError: must be string, not int…" appears when using strptime
What to do when "TypeError: data type not understood" appears in python's numpy.zeros
What to do if abort is displayed when inputting camera video in OpenCV
What to do if Japanese language support is not completely installed on Ubuntu 16.04
What to do when PermissionError of tempfile.mkstemp occurs
[Python] What to do when No module named'pyproj.datadir' appears when Exe is done with PyInstaller
What to do if the progress bar is not displayed in tqdm of python
What to do if Python IntelliSense is not displayed in VS Code on Windows
What to do when "Type Error: must be _socket.socket, not socket" appears on GAE
What to do if Python does not switch from the System version in pyenv
[Small story] What to do if "Error with child process: Building resource'awsToolkitSamLocalResource'" does not work when Lambda is locally executed by SAM from VS Code
[Go 1.13] What to do when unexpected directory layout: appears
[openpyxl] What to do when IllegalCharacterError appears in pandas.DataFrame.to_excel
What to do when gdal_merge creates a huge file
What to do when raise ValueError, "unsupported hash type"
What to do when "cannot import name xxx" [Python]
What is pip and how do you use it?
pipenv shell is no longer available ... what to do?
What to do when you can't bind CaboCha to Python
What to do when there is no response due to Proxy setting in Python web scraping
What to do if the image is not displayed using matplotlib etc. in the Docker container
[Python beginner] Variables and scope inside the function (when the processing inside the function is reflected outside the function and when it is not reflected)
[AWS] What to do when you want to pip with Lambda
What to do if ʻarguments [0] .scrollIntoView ();` fails in python selenium
How to not load images when using PhantomJS with Selenium
What to do when a Remove Error occurs when updating conda