High memory usage with V14 Metadata and Extrinsics

[insipx]

Running Archive off the current master (b5ee4d6) results in a high memory usage once blocks utilizing V14 metadata begin execution. The memory issue can be seen from this graph of memory gathered

At some point archive just crashes after oom error.

AFAICT from my debugging the reason for high memory usage as v14 is reached is because documentation is stored along with metadata. So when subxt decodes an extrinsic, it comes with all the associated metadata for those functions attached. This results in many extrinsics being held pre-postgres-insertion in memory with a high amount of extraneous text included.

Included is a query on the extrinsics table while running node-template. We see that each extrinsic has the same docs attached for 'Timestamp' -- this would be the same for every extrinsic ever executed V14+.

This is definitely redundant since the metadata docs can be stored once in a separate table

Even before V14 blocks are reached, memory is on a slow-uphill climb trending towards 10GB. This memory issue is more ambiguous and I haven't found one single cause. Things that I have tried investigating:

• Messages sent between actors use unbounded queues and could be a source of lots of data getting backed up somewhere
• Tasks in archive range in speed; EX: Loading blocks from rocksdb is much faster than executing them. These differences in indexing speed could cause memory issues somewhere in the messages sent between actors. This was partly solved through the use of rabbitmq which relieved some previous memory performance issues
• RocksDB might be a source that uses lots of memory; at some point it would be worth experimenting with paritydb to see if this relieves memory issues (whenever/if that is integrated into archive)