Using proxy service with Java crawling

Use proxy service for Java crawling process

I decided to use a proxy service with Java crawling PJ and implemented it with reference to other articles. It can be easily applied by adding the following 2 files. -Src / main / java / ProxySetConfig.java ・ Src / main / resources / proxy.properties

Create ProxySetConfig.java

import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.context.annotation.PropertySource;
import javax.annotation.PostConstruct;

@Configuration
@Profile("pro")
@PropertySource("classpath:proxy.properties")
public class ProxySetConfig {
    @Value("${proxy.host}")
    private String host;

    @Value("${proxy.port}")
    private String port;
    
    @PostConstruct
    private void setProxy() {
        // Proxy for System
        System.setProperty("http.proxyHost", host);
        System.setProperty("http.proxyPort", port);
        System.setProperty("https.proxyHost", host);
        System.setProperty("https.proxyPort", port);
    }
}

The above @Profile ("pro") is @Profile ("local") for local If it is a test, you can switch it by rewriting it to @Profile ("test").

Create proxy.properties file

proxy.host=xxx.xx.xx.xxx
proxy.port=xxxxx

In the above proxy.properties file It is very convenient because you can respond to proxy changes simply by rewriting proxy.host and proxy.port.

Reference site

https://kcf-developers.hatenablog.jp/entry/2018/09/04/112933

Recommended Posts

Using proxy service with Java crawling
Using Mapper with Java (Spring)
Using Java 8 with Bluemix (on Liberty Runtime & DevOps Service)
Try using Redis with Java (jar)
Using Java with AWS Lambda-Eclipse Preparation
Using Java with AWS Lambda-Implementation-Check CloudWatch Arguments
Using Java with AWS Lambda-Implementation-Stop / Launch EC2
Using JupyterLab + Java with WSL on Windows 10
Proxy server with squid using docker image
Game development with two people using java 2
I tried using OpenCV with Java + Tomcat
Game development with two people using java 1
Try using the Wii remote with Java
Java Dynamic Proxy
Specify ClassPath when using jupyter + Java with WSL
Install java with Homebrew
Getting started with Java programs using Visual Studio Code
Install Java with Ansible
Using Pair with OpenJDK
Comfortable download with JAVA
Get Azure App Service for Java Configuration with System.getEnv ()
Switch java with direnv
Download Java with Ansible
Let's scrape with Java! !!
Build Java with Wercker
[Java] Development with multiple files using package and import
Endian conversion with JAVA
Using multiple versions of Java with Brew on Mac + jEnv
Socket communication with a web browser using Java and JavaScript ②
Socket communication with a web browser using Java and JavaScript ①
I tried using the CameraX library with Android Java Fragment
Easy BDD with (Java) Spectrum?
Use Lambda Layers with Java
Getting Started with Java Collection
Try using RocksDB in Java
Java Config with Spring MVC
Basic Authentication with Java 11 HttpClient
Let's experiment with Java inlining
Run batch with docker-compose with Java batch
[Template] MySQL connection with Java
Rewrite Java try-catch with Optional
Install Java 7 with Homebrew (cask)
[Java] JSON communication with jackson
Java VB.net service reference halfway
Try using GloVe with Deeplearning4j
Java to play with Function
Try using view_component with rails
Try DB connection with Java
Try scraping using java [Notes]
Enable Java EE with NetBeans 9
[Java] JavaConfig with Static InnerClass
Try gRPC with Java, Maven
Let's operate Excel with Java! !!
Japaneseize using i18n with Rails
Version control Java with SDKMAN
RSA encryption / decryption with java 8
Paging PDF with Java + PDFBox.jar
Sort strings functionally with java
Object-oriented (java) with Strike Gundam
Spring with Kotorin ―― 7. Service layer
[Java] Content acquisition with HttpCliient