Is there a *Kotlin Coroutines* based DataLoader ?

[hantsy]

I noticed the issue #413 is closed, and I have tried to create a sample project to experience the Kotlin Coroutines support.
But I still can not find a way to declare a Kotlin Coroutines version of DataLoaders.

Is there a Kotlin Coroutines based DataLoader ?

I hope there is a Kotlin Coroutines variant instead of the original BatchLoader/MappedBatchLoader which returns a CompeletionStage.

#413 (comment)

[hantsy]

For example, how to convert the following DataLoaders to Kotlin Coroutines?

@DgsDataLoader(name = "authorsLoader")
class AuthorsDataLoader(val authorService: AuthorService) : BatchLoader<String, Author> {
// there is no a Kotlin Co variant for `BatchLoader`???
    override fun load(keys: List<String>): CompletionStage<List<Author>> = supplyAsync {
        authorService.getAuthorByIdIn(keys)  // assume it returns a Flow instead.
    }
}

@DgsDataLoader(name = "comments")
class CommentsDataLoader(val postService: PostService) : MappedBatchLoader<String, List<Comment>> {

    override fun load(keys: Set<String>): CompletionStage<Map<String, List<Comment>>> {
        val comments = postService.getCommentsByPostIdIn(keys) // assume it returns a Flow instead.
        val mappedComments: MutableMap<String, List<Comment>> = mutableMapOf()
        keys.forEach {
            mappedComments[it] = comments.filter { c -> c.postId == it } 
        }
        log.info("mapped comments: {}", mappedComments)
        return supplyAsync { mappedComments }
    }

    companion object {
        val log: Logger = LoggerFactory.getLogger(CommentsDataLoader::class.java)
    }
}

[hantsy]

If possible add a new contract for Kotlin Coroutines like this.

class AuthorsDataLoader(val authorService: AuthorService) : CoBatchLoader<String, Author> {
    override fun load(keys: List<String>): Flow<Author> =  authorService.getAuthorByIdIn(keys)
}

class CommentsDataLoader(val postService: PostService) : CoMappedBatchLoader<String, List<Comment>> {
    suspend override fun load(keys: Set<String>): Map<String, List<Comment>> {}
}

[hantsy]

Add kotlinx-coroutines-jdk8 dep(which provides some extensions to CompletableFuture, Stream, etc.) to overcome this issue temporily.

@DgsDataLoader(name = "authorsLoader")
class AuthorsDataLoader(val authorService: AuthorService) : BatchLoader<String, Author> {
    override fun load(keys: List<String>): CompletionStage<List<Author>> = GlobalScope.future {
        authorService.getAuthorByIdIn(keys).toList()
    }
}

@DgsDataLoader(name = "comments")
class CommentsDataLoader(val postService: PostService) : MappedBatchLoader<String, List<Comment>> {

    override fun load(keys: Set<String>): CompletionStage<Map<String, List<Comment>>> = GlobalScope.future {
        val comments = postService.getCommentsByPostIdIn(keys).toList()
        val mappedComments: MutableMap<String, List<Comment>> = mutableMapOf()
        keys.forEach {

            mappedComments[it] = comments.filter { c -> c.postId == it }
        }
        log.info("mapped comments: {}", mappedComments)
        mappedComments
    }

    companion object {
        val log: Logger = LoggerFactory.getLogger(CommentsDataLoader::class.java)
    }
}

[paulbakker]

The difficulty with this is that the data loader API comes from this library: https://github.com/graphql-java/java-dataloader, which graphql-java depends on. That means we don't control the API in the DGS framework.

So although I like the suggestion in theory, I don't see a practical way to implement it.

[BCantos17]

Bump

Has there been thoughts or updates on how to implement this? Or perhaps a work around? Currently working on my own Kotlin coroutines implementation and this is a pretty signification problem as we are using data loaders quite frequently.

@hantsy
I came across your example on dgs kotlin coroutines (which has been fantastic btw!) but the use of the @OptIn(DelicateCoroutinesApi::class) found here https://github.com/hantsy/spring-graphql-sample/blob/3cc1ca1fef10b7938a8d5338f1195aab6a10eb29/dgs-kotlin-co/src/main/kotlin/com/example/demo/DataLoaders.kt#L24

seems to go against the Kotlin documentation found here https://kotlin.github.io/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-global-scope/

I noticed you are using Flow when calling the service from the dataloader, does this help keep the application non-blocking? The Kotlin documentation seems to suggest that the use of GlobalScope keeps the thread active consuming resources as it waits for a network call to be completed

[srinivasankavitha]

We do not plan on addressing this in the near future due to other priorities, since kotlin is not predominantly used within Netflix. We do welcome contributions from folks who can help with this.

[paulbakker]

The problem is still that it's not something we can fix in DGS - it sits a level deeper in graphql-java. Not much we can do about it in the framework. @hantsy @BCantos17 definitely do correct me if I'm wrong about this.

[BCantos17]

This means that the whole graphql-java framework would have to be based on coroutines right? That would be a significant undertaking...

[paulbakker]

It probably could be an optional code path in graphql-java, similar to how we have it as an option (for data fetchers) in the DGS framework.

[hantsy]

@paulbakker Spring GraphQL provides Reactor DataLoader variants, if Dgs add similar features, it is easy to convert Reactor to Kotlin Coroutines by reactor Kotlin extensions.

[fluff-shark]

While working at Carbon Health I built a small library to support this our custom context. @muratakbal was nice enough to give me permission to extract and publish it. I have done that here.

We used these in prod without seeing any issues, but our scale was much smaller than Netflix would have to deal with. Feedback or suggestions for improvement welcome!