[JAVA] How to pass image pixel information natively from Unity

There was a case where I wanted to pass the pixel information of a texture image such as Unity's RenderTexture natively, so I had a lot of trouble (especially on the Android side), so make a note. By the way, Android is shit. I want you to disappear quickly. To be more specific, it is a case where you want to pass the image data to the Java library using WebCamera on the Unity side.

Click here for sample project https://bitbucket.org/HoshiyamaTakaaki/pixelreadstest

First of all, implementation on the Unity side

CameraOnPost.cs

RenderTexture.active = m_TargetTexture;
m_TargetTexture2D.ReadPixels(new Rect(0, 0, m_TargetTexture.width, m_TargetTexture.height), 0, 0);
m_TargetTexture2D.Apply();

var col = m_TargetTexture2D.GetPixels32();

var handle = default(GCHandle);
try
{
	handle = GCHandle.Alloc(col, GCHandleType.Pinned);
	var ptr = handle.AddrOfPinnedObject();
	libglutil_SetPixels(ptr.ToInt32(), m_TargetTexture.width, m_TargetTexture.height);
}
finally
{
	if (handle != default(GCHandle))
		handle.Free();
}

On the iPhone side, Unity's GetPixels32 is reasonably fast, so I think it's okay to get the color information normally on the Unity side and pass the pointer to the Native side. In the case of RenderTexture, it is necessary to move it to Texture2D once, and it may be necessary to be careful because that area becomes overhead.

CameraOnPost.cs

m_NativePlugin.CallStatic("sendRgbaFrame", m_Context, m_TargetTexture.GetNativeTexturePtr().ToInt32(), m_TargetTexture.width, m_TargetTexture.height);
GL.InvalidateState();

Unfortunately, Android does not perform well with Unity's ReadPixels. And the delivery of byte [] to Java is very heavy. It's too heavy to use at this stage. So on Android, the texture is passed to Java as it is, and ReadPixels is performed on the Java side. By the way, you can deliver it to NDK if you make it like you did on your iPhone. It seems that it is faster to transfer a large Byte array to Java via C # → NDK → JNI.

On the Java side, use the texture you received, draw the texture in the frame buffer with OpenGL ES, and read it with glReadPixels. But Android's glReadPixels are very heavy. It's heavy after all. Using PBO (Pixel Buffer Object) makes it a little faster (and sometimes slower), but it's still heavy. What's more, PBO requires OpenGL ES 3.0, which isn't complete in Java, so you can step on it or kick it. (You need to raise the minimum SDK API to 24 to complete Java. Can you raise it or stupid)

In order to save the acquired RGBA information in MP4 etc., it is necessary to further convert it to YUV, and when it comes to this stage, it is inevitable that it will be in a jerky state.

It may not be very useful for the hard work, but adjusting the size of the RenderTexture will improve performance to some extent.

Besides, Android had a lot of trouble. I also had a hard time finding a way to include other created so in aar. When the 64-bit version of the so file was included, it took a long time to resolve the problem that a pop-up appeared and ended the moment the application created with Unity was started.

In creating this sample, I borrowed a lot of information from our predecessors. And I saw a lot of Questions (corpses) without Answers. Everyone is in trouble with Android.

Thank you very much.