Open
Conversation
- Add parallel block processing for bounded ranges (arweave_block_workers) - Add header prefetch for auto-stop mode - Add block completion markers (block/<height>) to LMDB - Add cutover height for marker-aware auto-stop - Add per-block error isolation with failure summary events - Add arweave_marker_cutover_height opt for manual override
- Add configurable arweave_index_depth for multi-level bundle indexing - Recurse into nested ANS-104 bundles using header-only approach - Store achieved depth in block markers, reindex when depth increases - Skip Redstone bundles at L1 and L2+ (log events for observability) - Add read-ahead chunk cache scoped per recursion level - Clamp fetched data to ItemSize before parsing across item boundaries - Guard against inner bundle headers exceeding item data size - Retry with smaller fetch size when read-ahead overshoots available data - Track achieved depth as min() across all items/TXs per block
- Batched parallel_map for nested items - Bounded by arweave_nested_workers (default 10) - Disable 256KB read-ahead in parallel mode - take_batch streams items to bound memory
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@charmful0x this is an alternative for you, it's parallel block processing and Ln unbundling
it has redstone bundle detection and a few other optimizations.
a config that goes with it could be:
{ "ao-types": "generate_index=atom", "generate_index": false, "max_connections": 20000, "num_acceptors": 512, "conn_pool_read_size": 100, "arweave_index_ids": true, "arweave_index_workers": 32, "arweave_block_workers": 20, "arweave_index_depth": 2, "store": [ { "store-module": "hb_store_arweave", "ao-types": "store-module=atom,scope=atom", "scope": "remote", "index-store": [ { "ao-types": "store-module=atom", "store-module": "hb_store_lmdb", "name": "/tmp/indexer_alpha", "max-readers": 512 } ] } ], "routes": [ { "template": "/graphql", "nodes": [ { "prefix": "https://arweave.net", "opts": { "ao-types": "http_client=atom,protocol=atom", "http_client": "gun", "protocol": "http2" } } ] }, { "template": "^/arweave", "node": { "match": "^/arweave", "with": "https://arweave.net", "opts": { "ao-types": "http_client=atom,protocol=atom", "http_client": "gun", "protocol": "http2" } } } ] }it should run without overriding routes as well, but I wanted to bypass the node shuffling logic for my test
you can kick it off with
curl -v "http://localhost:PORTNUMBER/~copycat@1.0/arweave&from=-1&to=1867672"HB_PRINTS that could be useful
http_server_short,copycat_short,debug_copycat