Use native libraries from Scala via Java CPP + Java

Introduction

Unexplored Target We are using Qulacs, which is a quantum computer simulator written mainly in C ++ for the development of the 2018 gated quantum computer. Qulacs already has Python bindings, but I've always wanted to develop in Scala, which is also a more familiar programming language. So I used JavaCPP to do a lot of research to call native libraries like Qulacs from Scala via Java, and finally the sbt plugin. I was able to make.

• https://github.com/y-yu/sbt-javacpp4s

Actually, JavaCPP's GitHub organization also has an sbt plugin, but I found out that this is to make OpenCV etc. already available in JavaCPP easier to use from Scala, so that a new native library can be used from Scala. I decided to create a new one this time because it was not something to do. This article describes how to use this sbt plugin and some internal implementations. If you have any questions or improvements after reading this article, feel free to let us know in the comments.

Plugin overview

This sbt plugin does three things:

1. Create a native library like .so (or .dylib)
2. Pass the library and header file of (1) to JavaCPP to create a file for JNI (Java Native Interface).
3. Compile (2) with a processing system such as g ++ to create a native library that can be used from Java.

First of all, (1) seems to be created differently depending on the library (using make, etc.), so the OS command is passed as the settingKey of sbt. And (2) also starts a new Java process from sbt. This would have to change the options passed to the JVM as appropriate, and could have been achieved with the proper changes to sbt's run command, javaOptions, etc., but it was a big deal for the final effort. I did this because there was no change. Then, Java CPP does (3) as it is, and it becomes a dynamic link library that can be called from Java.

Settings and usage

From here, I will explain based on Sample project. Now, the C ++ source code and header files exist as follows.

cpp_src/HelloWorld.hpp

class HelloWorld {
public:
    int printN(int n);
};

cpp_src/HelloWorld.cpp

#include <iostream>
#include <string>
#include "HelloWorld.hpp"

using namespace std;

int HelloWorld::printN(int n) {
    for (int i = 0; i < n; i++) {
        cout << "Hello World!\n";
    }

    return n;
}

First, write the settings for compiling this in build.sbt.

build.sbt

includePath := (baseDirectory in Compile).value / "cpp_src"

libraryName := "libHelloWorld"

makeLibraryCommands := Seq(
  gppCompilerPath.value,
  "-I", includePath.value.toString,
  currentLibraryMeta.value.option,
  "-o",
  (libraryDestinationPath.value / s"${libraryName.value}.${currentLibraryMeta.value.extension}").toString,
  ((baseDirectory in Compile).value / "cpp_src" / "HelloWorld.cpp").toString
)

Here makeLibraryCommands is the OS command used to create libHelloWorld. I will explain in order. First, gppCompilerPath usesclang ++by default. Next, currentLibraryMeta stores processing options such asclang ++by the OS and the data structureDynamicLibraryMeta to absorb the extension of the generated native library.

DynamicLibraryMeta.scala

sealed abstract class DynamicLibraryMeta(
  val option: String,
  val extension: String
)

object DynamicLibraryMeta {
  case object Mac extends DynamicLibraryMeta("-dynamiclib", "dylib")
  case object Linux extends DynamicLibraryMeta("-shared", "so")
}

And currentLibraryMeta is automatically selected from the Java propertySystem.getProperty ("os.name")by default. By default, libraryDestinationPath also uses a dedicated directory under the target directory.

Next, create Java code that uses the native library.

src/main/java/javacpp/sbt/HelloWorld.java

package javacpp.sbt;

import org.bytedeco.javacpp.*;
import org.bytedeco.javacpp.annotation.*;

@Platform(include = {"HelloWorld.hpp"}, link = "HelloWorld")
public class HelloWorld extends Pointer {
    static {
        Loader.load();
    }

    public HelloWorld() {
        allocate();
    }

    public native void allocate();

    public native int printN(int n);
}

Go back to build.sbt and give the native library and the corresponding Java classpath.

build.sbt

nativeJavaClassPath := "javacpp.sbt.*"

enablePlugins(SbtJavaCPP4S)

You can also use wildcards like this. Also, with this plugin enabled, the work in build.sbt is finished.

Finally, create the Java part call code from Scala.

src/main/scala/javacpp/HelloWorld.scala

object HelloWorld {
  def main(args: Array[String]): Unit = {
    val instance = new HelloWorld()
    instance.printN(5)
  }
}

After that, if you sbt run this, it will be as follows.

$ sbt run
[info] Loading settings for project example from build.sbt ...
[info] Set current project to example (in build file:/Users/yyu/Desktop/javacpp-sbt/example/)
[info] Compiling 1 Scala source to /Users/yyu/Desktop/javacpp-sbt/example/target/scala-2.12/classes ...
[info] Success!
[info] running (fork) org.bytedeco.javacpp.tools.Builder -cp /Users/yyu/Desktop/javacpp-sbt/example/target/scala-2.12/classes:/Users/yyu/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/bytedeco/javacpp/1.5.1/javacpp-1.5.1.jar:/Users/yyu/.sbt/boot/scala-2.12.10/lib/scala-library.jar -Dplatform.compiler=clang++ -Dplatform.includepath=/Users/yyu/Desktop/javacpp-sbt/example/cpp_src -Dplatform.linkpath=/Users/yyu/Desktop/javacpp-sbt/example/target/libjni -d /Users/yyu/Desktop/javacpp-sbt/example/target/libjni javacpp.sbt.*mple / Compile / generateJNILibrary 0s
[info] Generating /Users/yyu/Desktop/javacpp-sbt/example/target/libjni/jnijavacpp.cpp
[info] Generating /Users/yyu/Desktop/javacpp-sbt/example/target/libjni/jniHelloWorld.cpp
[info] Compiling /Users/yyu/Desktop/javacpp-sbt/example/target/libjni/libjniHelloWorld.dylib
[info] clang++ -I/Users/yyu/Desktop/javacpp-sbt/example/cpp_src -I/Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk/Contents/Home/include/darwin /Users/yyu/Desktop/javacpp-sbt/example/target/libjni/jniHelloWorld.cpp /Users/yyu/Desktop/javacpp-sbt/example/target/libjni/jnijavacpp.cpp -march=x86-64 -m64 -O3 -Wl,-rpath,@loader_path/. -Wall -fPIC -dynamiclib -undefined dynamic_lookup -o libjniHelloWorld.dylib -L/Users/yyu/Desktop/javacpp-sbt/example/target/libjni -Wl,-rpath,/Users/yyu/Desktop/javacpp-sbt/example/target/libjni -lHelloWorld -framework JavaVM
[info] Deleting /Users/yyu/Desktop/javacpp-sbt/example/target/libjni/jniHelloWorld.cpp
[info] Deleting /Users/yyu/Desktop/javacpp-sbt/example/target/libjni/jnijavacpp.cpp
[info] running (fork) javacpp.HelloWorld
[info] Hello World!
[info] Hello World!
[info] Hello World!
[info] Hello World!
[info] Hello World!
[success] Total time: 6 s, completed 2019/11/09 4:57:50

It's working fine like this.

in conclusion

The small example worked, so now I want to be able to work with large C ++ libraries like Qulacs. This development takes a lot of effort for the amount of code. After all, I had a lot of troubles, such as not knowing the native error well and not knowing about dynamic link. As an aside, I think that the complexity of dynamic linking is one of the reasons why Go language etc. are single binary.