Skip to content

Latest commit

 

History

History
118 lines (84 loc) · 5.2 KB

readme.md

File metadata and controls

118 lines (84 loc) · 5.2 KB
Raw

PDF to ALTO

Service that listens for incoming messages from SQS. On receipt of a message this service will download reference PDF, extract ALTO file per-page and upload to specified S3 bucket.

If COMPLETED_TOPIC_ARN env var specified a notification will be raised.

Messages Format

The incoming message is in the shape:

{
  "pdfLocation": "https://www.hq.nasa.gov/alsj/a17/A17_FlightPlan.pdf",
  "pdfIdentifier": "a17_flightplan",
  "outputLocation": "s3://pdf-to-alto/a17_flightplan_alto"
}

Where

  • pdfLocation - the URL where PDF can be downloaded from.
  • pdfIdentifier - unique identifier for PDF. where i is 0-based page index. If omitted a random uuid will be used.
  • outputLocation - s3 location where final ALTO files will be output. With or without preceding s3:// and no trailing /.

(See sample.json)

The completed notification message echos back the original message with "numberOfFiles" property added.

The generated alto file will be placed in outputLocation. The format of each file will depend on value of PREPEND_ID envvar. This format will be (where i is the page number):

  • If true: f"{pdfIdentifier}-{i:04d}.xml"
  • else f"{i:04d}.xml",

Technology

This is a Python script that utilises the following libraries:

  • pdfalto - C lib used to generate ALTO files.
  • PyMuPDF - Python lib used to query PDF object for page dimensions.
  • lxml - Used to parse ALTO files and update scaled values.
  • requests - Used to download PDF files.

Configuration

The following environment variables can be used to configure the app:

Env Var Description Default
DOWNLOAD_CHUNK_SIZE Chunk size for downloading PDF 2048
WORKING_FOLDER Local working folder for storing generated files ./work
REMOVE_WORK_DIR Whether to clean up working dir on completion True
RESCALE_ALTO Whether to rescale generated ALTO to page True
MONITOR_SLEEP_SECS How long to sleep long polling operations if no messages received 30
AWS_REGION AWS region being used eu-west-1
INCOMING_QUEUE The name of the SQS queue to monitor for incoming messages. Required
COMPLETED_TOPIC_ARN The ARN of a topic to post completion notifications to
LOCALSTACK If using LocalStack False
LOCALSTACK_ADDRESS Address for LocalStack instance http://localhost:4566

(See .env.dist for sample .env file)

Running Locally

There is a multi-stage Dockerfile that builds the pdfalto binary and copies it to a new stage.

docker-compose.yml will build and start the main Python app alongside a LocalStack instance.

# build and start image using LocalStack
docker-compose up

# build image
docker build --tag pdf-to-alto:local .

# run docker image and listen to queue
docker run --env-file .env -it --rm --name pdf-to-alto pdf-to-alto:local

# run docker image to process 1 single api
docker run -it --rm --name pdf-to-alto \
  pdf-to-alto:local \
  opt/app/app/pdf_processor.py https://text.example/test.pdf my-pdf-identifier s3://pdf-bucket/alto

Note: Building pdfalto from source takes a few minutes

LocalStack

The docker-compose.local.yml file will spin up a LocalStack instance and configure a few resource for local testing:

  • An S3 bucket titled "pdf-to-alto"
  • An SNS topic "incoming-topic" with an SQS subscription to "incoming"
  • An SNS topic "completed-topic" with an SQS subscription to "completed"
docker-compose -f docker-compose.local.yml up

To use LocalStack set the LOCALSTACK and LOCALSTACK_ADDRESS env vars (see above).

When using the aws-cli with LocalStack the --endpoint-url needs to be specified. Below are some handy commands to use when testing:

# raise sample notification using sample.json
aws --endpoint-url=http://localhost:4566 sns publish --topic-arn arn:aws:sns:eu-west-1:000000000000:incoming-topic --message file://sample.json --region eu-west-1

# clear incoming queue
aws --endpoint-url=http://localhost:4566 sqs purge-queue --queue-url "http://localstack:4566/000000000000/incoming" --region eu-west-1

# check number of 'completed' notifications raised
aws --endpoint-url=http://localhost:4566 sqs get-queue-attributes --queue-url "http://localstack:4566/000000000000/completed" --attribute-names All --region eu-west-1

# check contents of s3
aws --endpoint-url=http://localhost:4566 s3 ls pdf-to-alto --recursive --region eu-west-1