Using proxy service with Java crawling

Use proxy service for Java crawling process

I decided to use a proxy service with Java crawling PJ and implemented it with reference to other articles. It can be easily applied by adding the following 2 files. -Src / main / java / ProxySetConfig.java ・ Src / main / resources / proxy.properties

Create ProxySetConfig.java

import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.context.annotation.PropertySource;
import javax.annotation.PostConstruct;

@Configuration
@Profile("pro")
@PropertySource("classpath:proxy.properties")
public class ProxySetConfig {
    @Value("${proxy.host}")
    private String host;

    @Value("${proxy.port}")
    private String port;
    
    @PostConstruct
    private void setProxy() {
        // Proxy for System
        System.setProperty("http.proxyHost", host);
        System.setProperty("http.proxyPort", port);
        System.setProperty("https.proxyHost", host);
        System.setProperty("https.proxyPort", port);
    }
}

The above @Profile ("pro") is @Profile ("local") for local If it is a test, you can switch it by rewriting it to @Profile ("test").

Create proxy.properties file

proxy.host=xxx.xx.xx.xxx
proxy.port=xxxxx

In the above proxy.properties file It is very convenient because you can respond to proxy changes simply by rewriting proxy.host and proxy.port.

Reference site

https://kcf-developers.hatenablog.jp/entry/2018/09/04/112933

Recommended Posts

Using proxy service with Java crawling
Using Mapper with Java (Spring)
Using Java 8 with Bluemix (on Liberty Runtime & DevOps Service)
Try using Redis with Java (jar)
Using Java with AWS Lambda-Eclipse Preparation
Html5 development with Java using TeaVM
Using Java with AWS Lambda-Implementation-Check CloudWatch Arguments
Using Java with AWS Lambda-Implementation-Stop / Launch EC2
Using JupyterLab + Java with WSL on Windows 10
Game development with two people using java 2
I tried using OpenCV with Java + Tomcat
Game development with two people using java 1
Java Dynamic Proxy
Interact with LINE Message API using Lambda (Java)
Specify ClassPath when using jupyter + Java with WSL
Using Gradle with VS Code, build Java → run
Install java with Homebrew
Sorting using java comparator
Getting started with Java programs using Visual Studio Code
Change seats with java
Install Java with Ansible
Using Pair with OpenJDK
Comfortable download with JAVA
Scraping practice using Java ②
Get Azure App Service for Java Configuration with System.getEnv ()
Switch java with direnv
Scraping practice using Java ①
Download Java with Ansible
Let's scrape with Java! !!
Build Java with Wercker
[Java] Development with multiple files using package and import
Endian conversion with JAVA
Using multiple versions of Java with Brew on Mac + jEnv
Problems with Dijkstra's algorithm using PriorityQueue and adjacency list (java)
Socket communication with a web browser using Java and JavaScript ②
Socket communication with a web browser using Java and JavaScript ①
I tried using the CameraX library with Android Java Fragment
Easy BDD with (Java) Spectrum?
Use Lambda Layers with Java
Java multi-project creation with Gradle
Getting Started with Java Collection
Try using RocksDB in Java
Java Config with Spring MVC
Basic Authentication with Java 11 HttpClient
Let's experiment with Java inlining
Run batch with docker-compose with Java batch
[Template] MySQL connection with Java
Rewrite Java try-catch with Optional
Install Java 7 with Homebrew (cask)
[Java] JSON communication with jackson
Java VB.net service reference halfway
Try using GloVe with Deeplearning4j
Java to play with Function
Try DB connection with Java
Try scraping using java [Notes]
Enable Java EE with NetBeans 9
[Java] JavaConfig with Static InnerClass
Try gRPC with Java, Maven
Let's operate Excel with Java! !!
Japaneseize using i18n with Rails
Version control Java with SDKMAN