Skip to content

Conversation

@walnut-the-cat
Copy link
Contributor

No description provided.

@bucanero bucanero temporarily deployed to walnut-the-cat-patch-1 - node-docs PR #130 September 22, 2025 15:39 — with Render Destroyed
@bucanero bucanero temporarily deployed to walnut-the-cat-patch-1 - node-docs PR #130 September 22, 2025 15:40 — with Render Destroyed
@bucanero bucanero temporarily deployed to walnut-the-cat-patch-1 - node-docs PR #130 September 22, 2025 15:49 — with Render Destroyed
@bucanero bucanero temporarily deployed to walnut-the-cat-patch-1 - node-docs PR #130 September 22, 2025 15:49 — with Render Destroyed
Copy link
Contributor

@VanBarbascu VanBarbascu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are still some mentions of s3://near-protocol-public/backups in this page.


With the 1.35.0 release we are rolling out split storage, a new option for storage management that is mainly targeted at archival nodes.. Split storage allows nodes to reduce storage costs and improve block production performance by splitting local storage into two separate areas: a hot and a cold database . The hot database is smaller in size and stores all of the data from the most recent epochs. The cold database is larger, and it stores all historical data for older epochs. Hot and cold databases can be independently placed on separate hardware storage, allowing configurations such as using hot storage on a fast NVMe drive, and placing cold storage on a cheaper HDD drive.

In split storage mode, the Near Client only works with a smaller hot database to produce blocks, which results in increased performance. Cold database is not accessed for reads during normal block production and is only read when historical data is specifically requested. Thus, we recommend keeping the cold database on cheaper and slower drives such as an HDD and only optimize speed for the hot database, which is about 10 times smaller.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In split storage mode, the Near Client only works with a smaller hot database to produce blocks, which results in increased performance. Cold database is not accessed for reads during normal block production and is only read when historical data is specifically requested. Thus, we recommend keeping the cold database on cheaper and slower drives such as an HDD and only optimize speed for the hot database, which is about 10 times smaller.
In split storage mode, the Near Client only works with a smaller hot database to produce blocks, which results in increased performance. Cold database is not accessed for reads during normal block production and is only read when historical data is specifically requested. Thus, we recommend keeping the cold database on cheaper and slower drives such as an HDD and only optimize speed for the hot database, which is significantly smaller.


In split storage mode, the Near Client only works with a smaller hot database to produce blocks, which results in increased performance. Cold database is not accessed for reads during normal block production and is only read when historical data is specifically requested. Thus, we recommend keeping the cold database on cheaper and slower drives such as an HDD and only optimize speed for the hot database, which is about 10 times smaller.

Split storage is disabled by default, and can be enabled with a config change and going through a migration process that requires manual steps. We have several choices for the migration:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, this section is about migrating from Legacy Archival (one disk), to split archival node. In this case, it is obsolete and can be removed.

FASTNEAR is now provider of Archival node snapshot. Please refer to their documentation [HERE](https://docs.fastnear.com/docs/snapshots/mainnet#archival-mainnet-snapshot)

### If your node is out of sync after restarting from the latest snapshot {#syncing node}
Downloading the cold database can take a long time (days). In this time your database can become very far behind the chain.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my understanding of the Archival implementation. This is not supported as it will generate gaps in the archival data if you put an RPC db with tail greater then the cold head.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, this procedure creates more problems than it solves

Copy link
Contributor

@wacban wacban left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

```

## Using Pagoda-provided S3/CloudFrontsplit-storage snapshots {#S3 migration}
## Using Snaphots {#S3 migration}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo snaphot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants