How to set browser location in Headless Chrome

When scraping, some sites may ask for your browser's (geo) location. This article is a personal note about the solution when using Selenium + Headless Chrome + Python.

environment

EC2 (Amazon Linux 2 AMI)

reference

Setting sensors (location) in headless Chrome

tool

・ Selenium ・ Chrome driver (85.0.4183.87) ・ Python3 (3.6.2)

Why you need location information

An error occurs when I try to scrape a certain site using Headless Chrome. When I checked the screenshot, the following pop-up was displayed and I could not access the specified page.

__ * "Processing is not possible because the location information could not be obtained. Please allow the location information." * __

As a result of various investigations, it seems that the cause was that the location information of the browser could not be confirmed.

How to set location information in Headless Chrome

So I searched for a way to set location information in Headless Chrome, but the information is surprisingly small. .. ..

In the first place, I was wondering if there was a way to set location information in the Chrome browser, so when I looked it up, it seems that it can be set from the developer tools.

So, when I thought, "Isn't it possible to solve the problem if the developer tools can be operated even with headless chrome?", Some people asked the same question on Stack overflow, and a solution was presented. Apparently it uses a method called execute_cdp_cmd ().

execute_cdp_cmd() It seems that you can set Google Developper tools with the method of selenium.webdriver.

The code implementation example is below

location_setting.py


from selenium import webdriver

def start_chrome_driver():
	options = webdriver.ChromeOptions()
	options.add_argument("--headless")
	options.add_argument("--disable-gpu")
	# executable_Please change the path according to each environment
	driver = webdriver.Chrome(executable_path="/usr/local/bin/chromedriver", chrome_options=options)
	return driver

#Launch Headless Chrome
driver = start_chrome_driver()

#Permission to set location information (location information cannot be set without permission)
driver.execute_cdp_cmd(
    "Browser.grantPermissions",
    {
        "origin": "https://hromssp.obc.jp/",
        "permissions": ["geolocation"]
    },
)

#Latitude, longitude, latitude / longitude error(Unit: m)To set
driver.execute_cdp_cmd(
    "Emulation.setGeolocationOverride",
    {
        "latitude": 35.689487,
        "longitude": 139.691706,
        "accuracy": 100,
    },
)

result

Now you can set location information (impersonation?) In Headless Chrome, and you can access the page you want to scrape!

Recommended Posts

How to set browser location in Headless Chrome
How to scrape pages that are “Access Denied” in Selenium + Headless Chrome
How to download files from Selenium in Python in Chrome
How to develop in Python
How to import Python library set up in EFS to Lambda
How to set the html class attribute in Django's forms.py
[Python] How to do PCA in Python
How to handle session in SQLAlchemy
How to use classes in Theano
How to write soberly in pandas
How to collect images in Python
Set opset to embed in ONNX
How to update Spyder in Anaconda
How to use SQLite in Python
How to convert 0.5 to 1056964608 in one shot
How to reflect CSS in Django
How to kill processes in bulk
How to use Mysql in python
How to wrap C in Python
How to use ChemSpider in Python
How to use PubChem in Python
How to run TensorFlow 1.0 code in 2.0
How to handle Japanese in Python
How to log in to Docker + NGINX
How to call PyTorch in Julia
How to set the output resolution for each keyframe in Blender
How to use calculated columns in CASTable
[Blender] How to set shape_key with script
[Introduction to Python] How to use class in Python?
How to suppress display error in matplotlib
How to access environment variables in Python
How to check local GAE from iPhone browser in the same LAN
How to dynamically define variables in Python
How to set optuna (how to write search space)
How to do R chartr () in Python
How to convert csv to tsv in CLI
How to set up SVM using Optuna
How to set the server time to Japanese time
How to delete expired sessions in Django
[Itertools.permutations] How to put permutations in Python
How to use Google Test in C
To set default encoding to utf-8 in python
How to implement nested serializer in drf-flex-fields
How to work with BigQuery in Python
How to set xg boost using Optuna
How to execute commands in jupyter notebook
How to do'git fetch --tags' in GitPython
How to get a stacktrace in python
How to display multiplication table in python
How to extract polygon area in Python
How to reassign index in pandas dataframe
How to check opencv version in python
How to enable SSL (TLS) in Apache
How to use Anaconda interpreter in PyCharm
How to specify non-check target in Flake8
[Python] How to set the (client) window size inside the browser with Selenium
How to handle consecutive values in MySQL
How to switch python versions in cloud9
How to adjust image contrast in Python
How to use __slots__ in Python class
How to dynamically zero pad in Python