Data Ingestion API

A NestJS application that fetches JSON data from URLs and stores it in MongoDB. The application includes scheduled tasks for regular data ingestion and comprehensive test coverage.

Features

• JSON data ingestion from any URL
• In-memory MongoDB for development
• Scheduled hourly data ingestion
• Comprehensive test coverage
• RESTful API for querying ingested data

Prerequisites

• Node.js (v20.10.0 or higher)
• npm or yarn
• MongoDB (for production)

Installation

# Install dependencies
npm install

Development

The application uses an in-memory MongoDB instance in development mode, which is automatically populated on startup.

# Start the development server
npm run start:dev

The development server will:

1. Start an in-memory MongoDB instance
2. Initialize the database with sample data from the test's API S3 File
3. Start the NestJS application
4. Begin scheduled hourly data ingestion

Production

For production, you'll need to set up a MongoDB instance and configure the connection:

# Set the MongoDB connection string
export MONGO_URI=mongodb://your-mongodb-uri

# Start the production server
npm run start:prod

API Endpoints

Query Data

GET /

Query Parameters:

• field=value - Exact match for numeric fields
• field=min,max - Range query for numeric fields
• field=text - Case-insensitive partial text match

Examples:

1. Exact price match:

GET http://localhost:3000/?priceForNight=252

2. Price range query:

GET http://localhost:3000/?priceForNight=100,120

3. Combined price range and name search:

GET http://localhost:3000/?priceForNight=50,120&name=Rustic%20Cabin

4. Multiple field search:

GET http://localhost:3000/?priceForNight=100,200&name=Cabin&location=Mountain

5. Text search with spaces (URL encoded):

GET http://localhost:3000/?name=Cozy%20Mountain%20Retreat

Note: All text searches are case-insensitive and support partial matches.

Scheduled Tasks

The application includes a scheduled task that runs every hour to:

1. Fetch JSON data from configured URLs
2. Validate the data format
3. Store the data in MongoDB

Testing

The application includes comprehensive test coverage for all modules and services:

# Run unit tests
npm run test

Test coverage includes:

• JsonFetchService - Testing URL fetching and error handling
• IngestionService - Testing data ingestion and validation
• TasksService - Testing scheduled tasks
• AppStatusController - Testing API endpoints

Development Mode

In development mode (NODE_ENV=development):

• Uses in-memory MongoDB
• Automatically populated on startup
• No need for external MongoDB instance
• Scheduled tasks run every hour