A bird's-eye view of parallel and asynchronous processing from a full stack perspective (JavaScript (Node.js), Kotlin, Go, Java)
Introduction
This post is CyberAgent Developers Advent Calendar 2019 14th day article-
I'm a full-stack engineer, but what does programming look like when you look at it from a full-stack perspective? I hope I can tell you that.
For those who are learning several languages, when learning a different language, I could do this with that language, but this language is convenient. I think you have felt that. I don't think there are many articles that explain techniques, methods, writing methods, etc. in different languages side by side, so I will dare to compare examples of the same method in different languages as one article.
This time, I would like to give you an example of when you want to execute asynchronous processing at the same time, but in this article, there are small difficulties such as "difference between parallel and parallel processing" and "difference between multithread and event loop". I will omit it. It's a long sentence because it describes multiple languages, but I hope you'll read it.
This time, I'm writing examples in the classics from old times and in the following languages that I often use these days.
--JavaScript (Example in Node.js) --Kotlin (example on Android)
- Go
- Java
Throw processing asynchronously at the same time
I think that microservices have become popular in recent years, but when it comes to microservices, the closer you get to the front side, the more APIs you hit, saying, "That data is from this API" and "This data is from this API ..." Too much, put together the results and process it for the screen. .. .. I think that the number of cases to do is increasing considerably. There are cases where it is done with the BFF (Backend for frontend) API, but I think there are quite a few cases where it is done on the client side (JS or smartphone application side). As in any case, the more APIs you hit, the faster the response speed and screen display speed will be affected, so instead of executing the APIs one by one in order, you want to process the APIs at the same time and then process the results. I think that the number of cases is increasing.
For example, it is necessary to execute an API that takes 3 seconds, an API that takes 2 seconds, an API that takes 1 second, and 3 APIs, and there is a process of processing it for the screen for 1 second, and when the results are attached, it can be displayed on the screen. In the case of.
As a premise, overhead such as class call is excluded
If you execute the API in order as shown in the figure below, it will take 7 seconds to display the screen, but
If you execute the API at the same time, it can be shortened to 4 seconds before the screen is displayed.
I think that the number of cases where the method of throwing at the same time is required is increasing, but I think that the writing method is quite different depending on the language. Basically, the flow as shown in the above figure does not change, so
--Thinking --Imagine a flow --Basic program behavior (processes, threads, etc.)
Once you have learned, even if the language changes, you can almost imagine it with a little research. Of course, the characteristics differ depending on the language, so if you do not dig deep when using it, it may lead to unexpected problems, but I think that you will not understand it unless you actually use it, so I will not write it deeply here.
Now, let's compare the programs that actually throw processing at the same time in different languages.
Asynchronous processing of JavaScript (Node.js)
In JavaScript (Node.js), use Promise. If you look at the source below, is Promise only Promise.all? I think, but if you add async to the method, all returns will be Promise. It is Promise.all that does something close to that, execute promises at the same time.
If you await in Promise.all, it will wait in the line of Promise.all until all the results of the function specified in the array of all are returned. Since test1 and test2 are executed almost at the same time, it is possible to return a response faster than executing them in sequence.
Although it is JavaScript, it is premised on Node.js. With JS on the browser, you can use Promise with chrome or recent browsers, but with IE11 or older browsers (although the number is increasing where support is discontinued), Promise can not be used, so put polyfill with webpack etc. And you need to convert it once so that it works on older browsers.
JavaScript (Node.js) source
const main = async () => {
console.log('[JavaScript] main start')
//Throw multiple processes asynchronously, Promise.Wait until all results are returned with all
const result = await Promise.all([
test1(),
test2(),
])
console.log(result[0])
console.log(result[1])
console.log('[JavaScript] main end')
}
const test1 = async () => {
console.log("test1 method")
await sleep(2000) //Assuming you are calling an API or something
return 123
}
const test2 = async () => {
console.log("test2 method")
await sleep(1000) //Assuming you are calling an API or something
return 456
}
//JavaScript is convenient Thread.Since there is no sleep-like thing, I will reproduce something close
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
//Execute main function
main()
Execution result
$ npm run main
[JavaScript] main start
test1 method
test2 method
123
456
[JavaScript] main end
Supplemental relationship between Promise and async / await
Promise is for executing asynchronous processing, but Node.js V8 or earlier (excluding beta time) does not have async / await, and when performing asynchronous processing, use new Promise () and the result of asynchronous processing It was a callback hell to make a callback chain. Callback hell makes the source less readable, so you can do async / await, it's more readable, and you don't have to write a lot of promises. If you look at the return value using TypeScript, you can see that the return value of the function with async always has a promise.
// TypeScript
const testFunc = (): Promise<number> => {
return 123
}
Promise.all remains because it is used in parallel execution, but basically it is okay not to use Promise so much, but some older libraries still return the result with callback. In such a case, once you wrap it in a Promise, you can make it async / await, so I will describe it as a supplement.
Sample source
/**
*Async the library that is the callback function/Example when you want to await
*/
const main = async () => {
console.log('[JavaScript] main start')
const result = await testFunc()
console.log(result)
console.log('[JavaScript] main end')
}
//async callback with promise/I'm wrapping the API that comes back with a callback so that it can be executed with await
const testFunc = () => {
return new Promise(resolve => {
console.log("testFunc method")
callbackMethod(ret => {
resolve(ret)
})
})
}
//Change of library that is callback
const callbackMethod = callback => {
callback(456)
}
main()
Kotlin asynchronous processing
Kotlin uses Coroutine. Coroutine works with newer versions of kotlin, but may not work with older versions. In that case, you will be using Thread from the Java era, so I will explain assuming a new Kotlin that can use Coroutine.
Coroutine is simply an easy-to-use version of Java Thread. If you look at the Thread number in debug when you run Coroutine, you can see that it is running in a different Thread.
Basically, the idea of async / await is the same as JS, should the method executing Coroutine add suspend and await to the spare side? Would you like to? I entrust the judgment or use runBlocking etc. to make it feel like returning to the synchronous processing when it is called. In this example, I wanted to execute test1 and test2 at the same time, so in doAll, I call two functions with async, wait for the two results to finish with await, and then return to the caller. There is no Promise.all like JS, so if you put the result in an array, you can reproduce the Promise.all-like method.
Kotlin source
package sample.kotlin
import kotlinx.coroutines.*
import kotlin.coroutines.resume
import kotlin.coroutines.suspendCoroutine
fun main() {
println("[kotlin] main start")
//Since doAll is suspend, do runBlocking and wait until all the processing inside is completed.
runBlocking {
val result = doAll()
println(result[0])
println(result[1])
}
println("[kotlin] main end")
}
/**
* JavaScriptPromise.implementation similar to all processing results
*Executor Service Callable in Java
*/
suspend fun doAll(): Array<Int> = coroutineScope {
val ret1 = async { test1() }
val ret2 = async { test2() }
// Promise.all wind
//When I put the method execution result in an array and return it, a promise.Become like all
arrayOf(ret1.await(), ret2.await())
}
suspend fun test1(): Int {
println("test1 method")
delay(2000) //Assuming you are calling an API or something
return 123
}
suspend fun test2(): Int {
println("test2 method")
delay(1000) //Assuming you are calling an API or something
return 456
}
Execution result
Right-click Main.kt in Android Studio and run Run.
(I thought I could put kotlinc or something and make it possible from the command line, but I used coroutine and I didn't have time to set it, so I did it from Android Studio to save time m (_ _) m)
[kotlin] main start
test1 method
test2 method
123
456
[kotlin] main end
Supplement Coroutine also tries to convert callback API to async / await
Although Coroutine is easier than Thread in Java, it has many features and which one should I use? I think it will be. This time, as an example, if you use suspendCoroutine, you can stop the callback in the same way as async / await using JS's new Promise ().
cont.resume(it)
This is js
resolve(ret)
If you execute resume, it will be returned.
This will allow the callback to be async, as in this line.
val func = async { testFunc() }
val result = func.await()
You can do what you want to do in different languages, even if you write it differently.
Sample source
package sample.kotlin
import kotlinx.coroutines.*
import kotlin.coroutines.resume
import kotlin.coroutines.suspendCoroutine
fun main() {
println("[kotlin] main start")
//In JavaScript, it feels like await. It will wait until the processing in runBlocking is completed on the street
runBlocking {
val func = async { testFunc() }
val result = func.await()
println(result)
}
println("[kotlin] main end")
}
/**
*New Promise in JavaScript()The same effect as.
*async callback with suspendCoroutine/I'm wrapping the API that comes back with a callback so that it can be executed with await
*/
suspend fun testFunc(): Int {
println("testFunc method")
return suspendCoroutine { cont ->
callbackMethod {
cont.resume(it)
}
}
}
//Change of library that is callback
fun callbackMethod(ret: (Int) -> Unit) {
ret(456)
}
Asynchronous processing of Go
go uses goroutine and chan. goroutine becomes asynchronous when you execute a function with the keyword go. Since you can't get the result with just go, use chan (channel) to receive the asynchronous result. chan has the same effect as await and will await with <-. Other than that, it works in the same flow as before. However, the movement of the contents is adopted by go as parallel processing. I won't go into that much detail about channels here, but in the case of go there is also a waitGroup. If you want to know more, please search by parallel processing and parallel processing.
I chose chan this time because I wanted to surely store the first, second and results like JS's Promise.all introduced so far. The point here is that we make two channels and reserve only one buffer for the second argument. Set the buffer to 2 in one channel as shown below.
channel := make(chan int, 2)
go Test1(chanel)
go Test2(chanel)
resultArray = append(resultArray, <- chanel)
resultArray = append(resultArray, <- chanel)
You can write it like this, but there is no guarantee that the result of Test1 will be returned in the first array. When actually executed, the result of Test2 may be included in array 0 depending on the timing of processing. By separating the channels, the result of Test1 can be surely put in channel1, so I separated it in this sample. With that as a premise, I think you should look at the following sources.
Go source
package main
import (
"fmt"
"time"
)
func main() {
fmt.Println("[go] main start")
result := doAll()
fmt.Println(result[0])
fmt.Println(result[1])
fmt.Println("[go] main end")
}
func doAll() []int {
resultArray := make([]int, 0, 2)
chanel1 := make(chan int, 1)
chanel2 := make(chan int, 1)
//Asynchronous processing with goroutine
go Test1(chanel1)
go Test2(chanel2)
// Promise.All-style execution results are packed in an array.
// <-Is like the await keyword in js and kotlin. Java is future.get()
//It feels like I'm waiting until all the execution results are bought.
resultArray = append(resultArray, <- chanel1)
resultArray = append(resultArray, <- chanel2)
close(chanel1)
close(chanel2)
return resultArray
}
func Test1(c chan int) {
fmt.Println("test1 method")
time.Sleep(time.Second * 2) //Assuming you are calling an API or something
c <- 123
}
func Test2(c chan int) {
fmt.Println("test2 method")
time.Sleep(time.Second * 1) //Assuming you are calling an API or something
c <- 456
}
Execution result
$ go run src/main.go
[go] main start
test2 method
test1 method
123
456
[go] main end
Since goroutine does not guarantee the execution order, test2 is executed first, but since it awaits and waits, the caller does not have to worry about it.
Java asynchronous processing
Since Java is an old language, there are several asynchronous processing methods, but this time we will use Executor, which was added from around Java 7. There are many ways to do it, but basically the same thing is done with Java threading. This time I wanted to receive the asynchronous execution result, so I implemented it using the Callable class. Asynchronous processing is executed when submitting,
The part where future.get () is await is await for the execution result. By putting the result in List, the execution result like Promise.all is reproduced. In the case of Java, there are more procedures for asynchronous than others, such as creating a Class or using Future. I used Executor because Kotlin runs on the JVM, so the result is Java, but at the time of writing this article, when using the Android CameraX library, I had to pass an instance of Executor, so I wanted to make a sample using Executor. It's around here.
Google CodelabsGetting Started with CameraX
Java is an old language and it takes a lot of work, but even when it comes to Kotlin, Java libraries are often called, and what is Thread in the first place? I think that it is still useful to suppress such things.
Java source
package sample;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;
class JavaApplication {
public static void main(String[] args) {
System.out.println("[Java] main start");
List<Integer> result = doAll();
System.out.println(result.get(0));
System.out.println(result.get(1));
System.out.println("[Java] main end");
}
/**
* JavaScriptPromise.implementation similar to all processing results
*Implementation similar to the result of async with Kotlin's coroutineScope
*/
private static List<Integer> doAll() {
List<Future<Integer>> futures = new ArrayList<>();
List<Integer> result = new ArrayList<>();
ExecutorService executor = Executors.newFixedThreadPool(2);
try {
//Please think that these two lines are almost the same as the async keyword or promise in js and kotlin.
futures.add(executor.submit(new Text1Class()));
futures.add(executor.submit(new Text2Class()));
// Promise.all wind
//If you put the execution result of the class that implements Callable in ArrayList and return it, promise.Become like all
for (Future<Integer> future : futures) {
result.add(future.get()); // future.get()Is like the await keyword in js and kotlin
}
} catch (Exception e) {
e.printStackTrace();
} finally {
executor.shutdownNow(); //Java doesn't let the process go down unless you explicitly stop the thread.
}
//It feels like awaiting until all the execution results are bought in this loop.
return result;
}
}
class Text1Class implements Callable<Integer> {
@Override
public Integer call() throws InterruptedException {
System.out.println("test1 method");
Thread.sleep(2000); //Assuming you are calling an API or something
return 123;
}
}
class Text2Class implements Callable<Integer> {
@Override
public Integer call() throws InterruptedException {
System.out.println("test2 method"); //Assuming you are calling an API or something
Thread.sleep(1000);
return 456;
}
}
Execution result
$ cd src
$ javac sample/Main.java
$ java sample.JavaApplication
[Java] main start
test1 method
test2 method
123
456
[Java] main end
Summary
This time, I gave an example of executing asynchronous processing at the same time using four languages.
I also combined the code of other languages in a form similar to JavaScript execution, but the basic idea is asynchronous execution in the thread and the idea of receiving a signal waiting for the thread execution completion with wait. JavaScript is a single-threaded event loop, and other languages are threads instead of threads, but the idea of ʻasync / awaitandprocessing flow can be applied even if the language is different, so the idea ` I think that it will lead to the improvement of technical ability to acquire.
This asynchronous is just an example. Rather than learning a language, it is important to use a language to understand the basics of architecture, methods, lifecycles, flows, and so on.
--Processes, threads and thread-safe programs for asynchronous processing ――If it is a life cycle, the characteristics and movements of the life cycle of each framework ――What you are good at and what you are not good at in that language --Understanding network protocols --How data flows to the screen, etc ...
The above is just an example, but it is important to understand these basic things through the program.
Finally
How can I get a full stack? Is sometimes asked, but seriously, basic is important.
However, if you try to memorize one point broadly and deeply (applied), of course, there are small differences in each language and architecture, and it takes time as it is. A long time ago, engineers were not divided into small areas, but in recent years, the division between front, native, backend, infrastructure engineers, etc. requires more advanced knowledge and experience in each field, so it is not exclusive. I think it's because catching up has become difficult.
Even so, I will go wide and deep with a full stack. Of course, I go deeper, so I work and study twice or three times as many people. I will not betray the effort, and the things that can be proposed as a whole architecture or a library that can be proposed only because I know deeply in all fields will expand dramatically. I think it's a lot of fun to be able to do that kind of development.
If you are just starting a program, it will be easier for you to understand it if you understand the basics of the program mentioned above in one language and learn the next language after you can use it freely. I will. In addition, computer science such as computers and operating systems will come up later, so if you don't know how to study, you can take the basic information processing engineer exam or applied processing engineer. You may also look at reference books such as exams. I have a qualification, but I don't remember that the qualification itself was very useful ^^; However, the basic mechanism I learned in the process is still useful.
Since programming languages and frameworks are popular, you may think that you will not be able to use what you have learned so far. However, even if the language or framework changes, the basics do not change, so even when new things appear, the ideas that we have learned will definitely be useful somewhere.
In my example, very recently, I made Channel (Blocking Queue in Java), Reentrant Lock, TCP / UDP communication, etc. with Kotlin on Android and made a thread-safe original library from a low layer, but 10 I was involved in the development of frameworks using thread-safe programs and communication-related programs in a project I participated in about a year ago. It was around this time that I thought that the ideas and design skills I learned at that time were useful now.
Technology is fashionable and obsolete in every era,
I will not betray the way of thinking that I have learned with effort
Finally, I would like to conclude with this one word.
Here is the sample project I made this time. https://github.com/tanaka-yui/parallel-async-sample
Thank you for staying with us.