Skip to content

Extend TTL for your 3rd party (non ioc2rpz) DNS Firewall (RPZ) feeds

Jump to bottom
Vadim edited this page Jul 20, 2020 · 2 revisions

Overview

When you subscribe on DNS Firewall feeds the feed provider fully maintains them. Indicators are added or removed when they are discovered or expired if not resubmitted. In case if you security policies mandate you to block indicators for a longer period of time (for example newly registered/observed domains) usually you should generate and maintain DNS Firewall feeds yourself and ioc2rpz can perfectly help you with that.
Because you want the same content just for the longer period of time (accepting a possibility of higher rate of false positives) you may think about if there is a short cut?
Yes, there is and ioc2rpz can help you to implement it (don't forget to check your license agreement if it is allowed).

Pulling RPZ feeds

The idea is that you should periodically pull a feed you are interested in and store the content in a file. You need to keep the history as long as you wish to extend TTL (time to life) of the feed. For the feed generation you need to merge indicators from the all files into a single list.
The script below implements this logic for host/domain based feeds. For IP based feeds you need to implement some extra logic.
You need to add it to your ioc2rpz installation (may run on a host server), update the following fields: Server, Feed, TKey_Name, TKey and execute by cron.

#!/bin/sh
#  Extended TTL for an RPZ feed
#  (c) Vadim Pavlov

KeepDays="3"
Feed="rpz-feed" # update
TKey_Name="TSIG Key Name" # update 
TKey="TSIG Key" # update
Server="127.0.0.1" # update
Prefix="/opt/ioc2rpz/include"
Prefix_out="/opt/ioc2rpz/cfg/include"

NewSerial=`/usr/bin/dig @$Server -y$TKey_Name:$TKey $Feed SOA | grep "IN SOA ns" | awk '{print $7}'`
CurrSerial=`cat ${Prefix}/${Feed}_serial.txt`

[ -z "$CurrSerial" ] && CurrSerial=0

if [ "$NewSerial" != "$CurrSerial" ]; then
    echo -n $NewSerial > ${Prefix}/${Feed}_serial.txt
    #keep
    TODAY=`date "+%F_%H_%M"`
    /usr/bin/dig @$Server -y$TKey_Name:$TKey $Feed ixfr=$CurrSerial | grep -v "^*." | tail -n +10 | sed -e "s/\.$Feed.*//" -e '/^$/d'  > ${Prefix}/${Feed}_${TODAY}.txt
    #remove
    OLDREMOVE=`date +%F --date="-${KeepDays} days"`
    rm -f ${Prefix}/${Feed}_${OLDREMOVE}_*.txt
    cat ${Prefix}/${Feed}_*.txt | sort | uniq >${Prefix_out}/${Feed}.txt
fi

To optimize data transfer incremental zone transfer (IXFR) is used and a zone serial is tracked.

Create Sources and modified RPZ feeds

The script above creates a local file /opt/ioc2rpz/include/${Feed}.txt (replace ${Feed} by a feed name). When you configure a source you just need to read the file with the same interval the feed is updated. ioc2rpz tracks if a file was updated by calculating MD5 check sum and updates zones only in case if there were any changes. ioc2rpz file source

The last step is to create an RPZ-feed based on the source.

Clone this wiki locally