Skip to content

Scan large DynamoDB tables faster with parallelism

License

Notifications You must be signed in to change notification settings

whodeen/dynamodb-parallel-scan

 
 

Repository files navigation

dynamodb-parallel-scan CircleCI npm (scoped)

Scan DynamoDB table concurrently (up to 1,000,000 segments), recursively read all items from every segment

Install

$ yarn add @shelf/dynamodb-parallel-scan

Usage

const {parallelScan} = require('@shelf/dynamodb-parallel-scan');

(async () => {
  const items = await parallelScan(
    {
      TableName: 'files',
      FilterExpression: 'attribute_exists(#fileSize)',
      ExpressionAttributeNames: {
        '#fileSize': 'fileSize'
      },
      ProjectionExpression: 'fileSize'
    },
    {concurrency: 1000}
  );

  console.log(items);
})();

Read

Publish

$ git checkout master
$ yarn version
$ yarn publish
$ git push origin master

License

MIT © Shelf

About

Scan large DynamoDB tables faster with parallelism

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • TypeScript 77.6%
  • JavaScript 22.4%