In a micro-service fashion, everything is streamlined to small domains, small actions and therefore fast responses.
This is theorically true, BUT. Sometimes we may want to access to expensive in time information. Either a partner api which is slow, or a restricted access to a confidential network which takes seconds to route correctly.
You will run the risk of having the client api renewing the same request multiple times because the data has no fallback information (you cannot find a fallback for a credit card for example)
In this case, the Nagle’s algorithm’m will not work. The retries will happen each N seconds, and then the packets will get acknowledged before the grouping can happen.
It is time if not too late to group your api calls together. All the inbound retries will subscribe to the same backend call. And one successful technology to do that is the Coroutines in Kotlin.
We are going to create two roles in this process : a router to map each request to one task, and several workers whose role is to handle a task. The workers, which are not threads but coroutines, will always be changing contexts to avoid wasting cpu time.
The behavior is : The client api will send its input to the router, which will reply either with a new task or a currently running task. It will wait until completion of the task. The only intelligence there is to compare the input with the other saved inputs to see if two of them are the same. It will help the router to decide to merge two requests in one.
The repository in github is this one : https://github.com/libetl/coroutines-pooling-client
The test runs two scenarios : one having 100 requests of different calls with different inputs, and one having 100 requests exactly identical. Each backend call will be logged.
In the first scenario, each inbound call leads to a call to the backend (100 different inputs)
In the second scenario, there are 100 inbound calls but only 10 calls to the backend (you can even decrease that number if the latency of the backend increases). Why ? As you have guessed, some calls are grouped together.
Coroutines can help you implement a wide range of different strategies for your use cases, but what I foresee the most is the need for seamless implementations when you need your code to stay stupid and simple. And the coroutines framework easily help to do that by letting the business code dislocated from the boilerplate use of coroutines.
So now perf developers and craftsmen can become friends again.