ETag comparison

[GrumpyGiuseppe]

Prerequisites

• I have checked the Wiki and Discussions and found no answer

• I have searched other issues and found no duplicates

• I want to request a feature or enhancement and not ask a question

The problem

High bandwidth usage if big filter lists get updated frequently.

Proposed solution

Compare the ETag of the local filter file with the one on the server before downloading.

Alternatives considered and additional information

I update filters with curl now because it also uses way less RAM and IO.
Example (thx to ChatGPT):

#!/bin/bash

# URL of the file to download
URL="https://example.txt"

# Local file to save the downloaded content
LOCAL_FILE="/mnt/example.txt"

# File to store the ETag
ETAG_FILE="/mnt/etagexample.txt"

# Function to download the file
download_file() {
    echo "Downloading file..."
    curl -o "$LOCAL_FILE" -D - "$URL" | grep -i etag > "$ETAG_FILE"
}

# Check if the ETag file exists
if [ -f "$ETAG_FILE" ]; then
    # Get the saved ETag
    SAVED_ETAG=$(cat "$ETAG_FILE" | awk -F'"' '/etag/ {print $2}')

    # Make a HEAD request to get the current ETag from the server
    CURRENT_ETAG=$(curl -I "$URL" 2>/dev/null | grep -i etag | awk -F'"' '/etag/ {print $2}')

    # Compare the ETags
    if [ "$SAVED_ETAG" != "$CURRENT_ETAG" ]; then
        echo "ETag has changed. Downloading new file..."
        download_file
    else
        echo "ETag has not changed. No download needed."
    fi
else
    echo "No previous ETag found. Downloading file..."
    download_file
fi