Best approach for efficiency fetching from multiple databases

[DenyWatanabe]

I need to build an app which fetches data from multiple fields from multiple databases (DynamoDB, Neptune and Redshift). The app will likely receive tens of thousands of requests per second, so it needs to be as efficient as possible.

So my design idea is as follows:

1. Check which fields were requested (easy to do with INFO)
2. If fields requested from DynamoDB, query DynamoDB with the requested fields (example: id, title) - also easy with INFO
3. If fields requested from Neptune, query Neptune with the requested fields (example: likes, shares) - also easy with INFO
4. If fields requested from Redshift, query only the requested fields (example: score) - also easy with INFO
5. The API response will only contain the requested fields

I'm currently stuck in the last step, how to request data from multiple databases and then resolve them to a single object, assigning only its attributes which were requested and returned from the database calls.

creating a single resolver which returns Object(attr1=value1, attr2=value2, ...) doesn't seem the best option as not all attributes will be returned every time.

I have considered an extremely ugly solution but there must be a better way:

@strawberry.federation.type
class myObject:
  id: strawberry.ID
  attr1: str
  attr2: str
  attr3: str
  attr4: str

@strawberry.federation.type
class Query:
  @strawberry.field
  def myResolver(self, id: strawberry.ID, info:Info) -> myObject
    requestedFields = [field.selections[0].name for field in info.selected_fields]
    dbResultSet = db.getItem("myTable", {"id": id}, ",".join(requestedFields))

    retVal = myObject

    if "attr1" in dbResultSet:
      retVal.attr1 = dbResultSet["attr1"]

    if "attr2" in dbResultSet:
      retVal.attr2 = dbResultSet["attr2"]

    if "attr3" in dbResultSet:
      retVal.attr3 = dbResultSet["attr3"]

    if "attr4" in dbResultSet:
      retVal.attr4 = dbResultSet["attr4"]

    return retVal

Important requirements:

1. If no fields are requested, for example, from Neptune, the corresponding database call will not be executed
2. If a database provides fields "a", "b", "c", "d" and the API only receives a request for "a", then only "a" will be requested and fetched from the database

What would be the best design/implementation approach in my case?

[jkimbo]

@DenyWatanabe you probably want to define resolvers on your myObject type instead of trying to do it in myResolver. For example:

def dynamo_resolver(myObject: myObject, info: Info) -> str:
  return db.getItem("myTable", {"id": myObject.id}, info.field_name)

@strawberry.federation.type
class myObject:
  id: strawberry.ID
  attr1: str = strawberry.field(resolver=dynamo_resolver)
  attr2: str = strawberry.field(resolver=dynamo_resolver)
  attr3: str = strawberry.field(resolver=neptune_resolver)
  attr4: str = strawberry.field(resolver=redshift_resolver)

@strawberry.federation.type
class Query:
  @strawberry.field
  def myResolver(self, id: strawberry.ID, info:Info) -> myObject
    return myObject(id=id)

That way you'll only resolve the attributes that were requested by the client.

To prevent multiple DB queries you can use a dataloader for each type of resolver:

# Cache data loaders
DYNAMO_LOADERS = {}

def get_dynamo_dataloader(id: str):
	if id in DYNAMO_LOADERS:
		return DYNAMO_LOADERS[id]

	async def load_from_dynamo(keys: List[str]) -> List[str]:
    	result_map = db.getItem("myTable", {"id": id}, ",".join(keys))
		return [result_map[key] for key in keys]

	return DataLoader(load_fn=load_from_dynamo)

async def dynamo_resolver(myObject: myObject, info: Info) -> str:
  loader = get_dynamo_dataloader(myObject.id)
  return loader.load(info.field_name)

@strawberry.federation.type
class myObject:
  id: strawberry.ID
  attr1: str = strawberry.field(resolver=dynamo_resolver)
  attr2: str = strawberry.field(resolver=dynamo_resolver)
  attr3: str = strawberry.field(resolver=neptune_resolver)
  attr4: str = strawberry.field(resolver=redshift_resolver)

@strawberry.federation.type
class Query:
  @strawberry.field
  def myResolver(self, id: strawberry.ID, info:Info) -> myObject
    return myObject(id=id)

[DenyWatanabe]

Hi Jonathan, thank you for the quick reply, I have two questions, sorry if they're very basic but I'm just getting started with graphql and on top of that it's for a huge high-profile project, that's why I'm so concerned with code efficiency.

First, in your reply the myObject class is referenced by the dynamo_resolver function before its definition, I've seen there's a lazy/Annotation feature in strawberry but not sure if that's what should be used in this case, and if it is, how to implement it.

Second, I've noticed the resolvers have strong typed return values, what happens if the definition of myObject looks like the example below? (asterisks used for emphasizing references to other objects)

class Location:
  id: strawberry.ID
  name: str

class Author:
  id: strawberry.ID
  name: str
  location: Location*

class Category:
  id: strawberry.ID
  name: str

class myObject:
  id: strawberry.ID
  title: str
  author: Author*
  category: Category*
  interactions: int
  relevanceScore: int

[DenyWatanabe]

Quick update:

I've taken a somewhat simpler approach, still thanks to your suggestion:

@strawberry.federation.type
class Location:
  id: strawberry.ID
  name: str

@strawberry.federation.type
class Author:
  id: strawberry.ID
  name: str
  location: Location

@strawberry.federation.type
class Category:
  id: strawberry.ID
  name: str

GLOBAL_BUFFER = {}

def resolveAuthor() -> Author:
  return Author(id=GLOBAL_BUFFER["author"]["id"], 
         name=GLOBAL_BUFFER["author"]["name"], 
         location=Location(id=GLOBAL_BUFFER["author"]["location"]["id"],
            name=GLOBAL_BUFFER["author"]["location"]["name"]))

def resolveTitle() -> str:
  return GLOBAL_BUFFER["title"]

def resolveCategory() -> Category:
  return Category(id=GLOBAL_BUFFER["category"]["id"], name=GLOBAL_BUFFER["category"]["name"])

def resolveInteractions() -> int:
  return GLOBAL_BUFFER["interactions"]

def resolveRelevanceScore() -> int:
  return GLOBAL_BUFFER["relevanceScore"]

@strawberry.federation.type
class myObject:
  id: strawberry.ID
  title: typing.Optional[str] = strawberry.field(resolver=resolveTitle)
  author: typing.Optional[Author] = strawberry.field(resolver=resolveAuthor)
  category: typing.Optional[Category] = strawberry.field(resolver=resolveCategory)
  interactions: typing.Optional[int] = strawberry.field(resolver=resolveInteractions)
  relevanceScore: typing.Optional[int] = strawberry.field(resolver=resolveRelevanceScore)

def loadSingleObject(id: str, info: Info):
    global GLOBAL_BUFFER
    requestedFields = [selection.name for field in info.selected_fields for selection in field.selections]
    GLOBAL_BUFFER= db.getItem("myTable", {"id": id}, ",".join(requestedFields))["Item"]

Now, this works perfectly:

@strawberry.federation.type
class Query:

  @strawberry.field
  def objectByID(self, id: strawberry.ID, info: Info) -> myObject:
    loadSingleObject(id=id, info=info)
    return myObject(id=id)

However when I try to fetch multiple objects, only the last one is returned by uvicorn (same object is repeated multiple times), even though the data being fetched from the database is correct (objects returned are effectively different from each other):

@strawberry.field
def listObjects(self, qty: int, info: Info) -> typing.List[myObject]:
  ids = [el["id"] for el in db.scan("myTable", "id")["Items"]]
  maxItems = min(qty, len(ids))
  retList = []
  for i in range(maxItems):
    loadSingleObject(id=ids[i], info=info)
    retList.append(myObject(id=ids[i]))

  return retList

Is this a bug or am I doing something wrong?

[DenyWatanabe]

On my previous question, I've figured out that resolvers are defined in the schema and then executed when responding to a request, now trying to come up with a way for resolvers to have access to parameters or even parent object attributes (id in this case) so they can resolve to the correct data row.