Overview

I had to access it via a proxy using jsoup, but it was unexpectedly difficult.

The points are the following two points.

It is set by default (from Java 8) to prevent HTTPS access via a proxy that requires authentication.
ʻAuthenticator.setDefault () `must specify the proxy server user ID and password.

environment

jsoup

`pom.xml`


<!-- https://mvnrepository.com/artifact/org.jsoup/jsoup -->
<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.12.1</version>
</dependency>

Java version: 8

OK example

//Cancel the setting "Prohibit HTTPS access via proxy that requires authentication".
//By default this property"Basic"Is specified and Basic authentication is not available, so"Basic"Overwrite with an empty string to cancel the setting of.
System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");

//Proxy server ID/Make sure to pass the password via Authenticator.
Authenticator.setDefault(new Authenticator() {
    @Override
    protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication("{Proxy server user ID}", "{Proxy server password}".toCharArray());
    }
});

Jsoup.connect("{URL you want to access}")
  .proxy("{Proxy server ID or host name}", {Proxy server port number})
  .get();

NG example

//Version 1 that uses the Authorization header
Jsoup.connect("{URL you want to access}")
  .header("Authorization", "Basic " + Base64.getEncoder().encodeToString("{Proxy server user ID}:{Proxy server password}".getBytes()))
  .proxy("{Proxy server IP address or host name}", {Proxy server port number})
  .get();

//Version that uses the Authorization header Part 2 (in the header value"Basic "Do not specify the character string of)
Jsoup.connect("{URL you want to access}")
  .header("Authorization", Base64.getEncoder().encodeToString("{Proxy server user ID}:{Proxy server password}".getBytes()))
  .proxy("{Proxy server IP address or host name}", {Proxy server port number})
  .get();

// Proxy-Version 1 that uses the Authorization header
Jsoup.connect("{URL you want to access}")
  .header("Proxy-Authorization", "Basic " + Base64.getEncoder().encodeToString("{Proxy server user ID}:{Proxy server password}".getBytes()))
  .proxy("{Proxy server IP address or host name}", {Proxy server port number})
  .get();

// Proxy-Version that uses the Authorization header Part 2 (in the header value"Basic "Do not specify the character string of)
Jsoup.connect("{URL you want to access}")
  .header("Proxy-Authorization", Base64.getEncoder().encodeToString("{Proxy server user ID}:{Proxy server password}".getBytes()))
  .proxy("{Proxy server IP address or host name}", {Proxy server port number})
  .get();

// 「jdk.http.auth.tunneling."disabled Schemes" is not specified

Authenticator.setDefault(new Authenticator() {
    @Override
    protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication("{Proxy server user ID}", "{Proxy server password}".toCharArray());
    }
});

Jsoup.connect("{URL you want to access}")
  .proxy("{Proxy server ID or host name}", {Proxy server port number})
  .get();

In either case, the following error occurred.

java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"

Reference (Acknowledgment)

It was very helpful! Thank you! https://qiita.com/kaakaa_hoe/items/d4fb11a3af035a287972

[Java] Use jsoup to access HTTPS via a proxy that requires authentication