What to do when the result downloaded via scrapy is in English

When I look at the page in a browser, it is in Japanese, but when I download it via scrapy, the English page may be downloaded. This is because Accept-Lauguage when scrapy makes a request to the web server is ʻen` by default, so you can request a Japanese page by writing the following contents in settings.py.

DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'ja,en-US;q=0.8,en;q=0.6',
}

reference: https://doc.scrapy.org/en/latest/topics/settings.html#std:setting-DEFAULT_REQUEST_HEADERS

Recommended Posts

What to do when the result downloaded via scrapy is in English
What to do when the value type is ambiguous in Python?
What to do when the warning "The environment is in consistent ..." appears in the Anaconda environment
What to do when is not in the sudoers file.This incident will be reported.
What to do when only the window is displayed and nothing is displayed in pygame Note
What to do when a warning message is displayed in pip list
[Python] What to do when PEP8 is violated in the process of importing from the directory added to sys.path
What to do if abort is displayed when inputting camera video in OpenCV
What to do when the graph does not appear in jupyter (ipython) notebook
[openpyxl] What to do when IllegalCharacterError appears in pandas.DataFrame.to_excel
curl: (60) What to do when Issuer certificate is invalid.
What to do if the progress bar is not displayed in tqdm of python
What to do when Japanese is not displayed on matplotlib
What to do when PyCharm font is strange or garbled
What to do when "SSL: CERTIFICATE_VERIFY_FAILED _ssl.c: 1056" appears in Python
What to do when "Invalid HTTP_HOST header" appears in Django
What to do when there is no response due to Proxy setting in Python web scraping
What to do if the image is not displayed using matplotlib etc. in the Docker container
What to do when a Missing artifact occurs in a jar that is not defined in pom.xml
[AWS] What to do when the ping command causes a "timeout"
How to give and what the constraints option in scipy.optimize.minimize is
What to do when the jupyterlab extension settings are not reflected
What to do if "Unnamed: 0" is added in to_csv-> read_csv in pandas
What to do if the inode is exhausted on EC2 Linux
What to do if you can't use the trash in Lubuntu 18.04.
What to do when you get "I can't see the site !!!!"
What to do when UnicodeDecodeError occurs during read_csv in pandas (pd.read_table ())
Animate what happens in frequency space when the Nyquist frequency is exceeded
What to do when ModuleNotFoundError: No module named'XXX' occurs in Python
What to do when a part of the background image becomes transparent when the transparent image is combined with Pillow
NameError: global name'dot_parser' is not defined and what to do when it comes up in python
What is "mahjong" in the Python library? ??
When the target is Ubuntu 16.04 in Ansible
What to do if there is a decimal in python json .dumps
What to do if the Pipenv environment is corrupted by updating Homebrew
[Docker] What to do when error Couldn't find the binary git appears
[Python] What to do when an error related to SSL authentication is returned
What is the fastest way to create a reverse dictionary in python?
[OSX] [pyenv] What to do when an SSL error occurs in pip
What to do when "Something is already running at port 8000" is displayed when the develop command of Gatsby is executed.
How to write what to do when an application is first displayed in Qt for Python with Designer
What to do in my case when pyenv install is not possible after upgrading to macOS Big Sur
What to do when "OS Error: [WinError 126] The specified module cannot be found" appears in import torch
Example of what to do when the sample script does not work (OpenCV-Python)
What to do if CERTIFICATE_VERIFY_FAILED occurs when nltk.download () is done on macOS pyhon
What to do if pipreqs results in UnicodeDecodeError
[python] What to do when an error occurs in send_keys of headless chrome
What is wheezy in the Docker Python image?
What to do when SSL error occurs in pip in Windows10, miniconda, VScode environment
What to do when a warning appears around Python integration in Neovim's CheckHealth
Why Docker is so popular. What is Docker in the first place? How to use
What to do if you get the error RuntimeError: Python is not installed as a framework when trying to use matplitlib and pylab in Python 3.3
[Ubuntu 18.04 LTS] What to do when the screen resolution cannot be selected [NVIDIA]
What to do when "TypeError: data type not understood" appears in python's numpy.zeros
What to do when PermissionError of tempfile.mkstemp occurs
What to do when [Errno 2] No such file or directory appears in Python
What to do to get google spreadsheet in python
What to do if a version error occurs in the selenium Chrome driver
[EC2] What to do when selenium is stuck and processing does not proceed
What to do if (base) is displayed at the beginning of the Mac terminal
What happens to the access log when the website is automatically accessed by selenium-webdriver