Ping tool exports prometheus metrics #8023
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
resolves #7004, see that issue for reasoning behind the change
See #7674 for more details.
This PR implements FlatStorageState, the struct that manages deltas in flat storage. After this PR, the main implementation of flat storage is mostly ready. The next step is to add more tests.
* Upgrade tracing and opentelemetry crates. * Avoid touching Cargo.toml's for minior version upgrade of `tracing`. * HTTP exporter is not the best but it: a) links b) doesn't require protoc * Undo opentelemtry upgrade, because it introduces an external dependency on protoc or link-time errors. Co-authored-by: near-bulldozer[bot] <73298989+near-bulldozer[bot]@users.noreply.github.com>
The bounded queue was intended to drop new requests on the sender side instead of blocking the main thread. The prefetching requests we send today are at least 100 times fewer than what fits in the channel, and only white listed accounts can even trigger it. So this is not a concern in practice, yet.
Removes a check in themis where we ensure that a given crate's rust version is not above the version mentioned in rust-toolchain.toml. This check is not needed because such a misconfiguration will prevent the repo from even being built in the first place.
To make it work also update rand_xorshift to 0.3. However, leave rand at 0.7 in core/crypto. This is because that crate depends on ed25519-dalek which expects rand 0.7 objects. Issue: #7674
Moves https://github.com/near/nearcore/wiki over to the book. I'll disable the wiki once this PR is merged.
Move all dependencies to root Cargo.toml file and refer to it using inheritance in all members of the workspace. This will hopefully help reduce noise when updating a package in the future. The only exception is `rand` dependency in core/crypto crate which needs to be held back at older version because of ed25519-dalek. Doing this, put union on cargo features in the root Cargo.toml so features don’t need to be enabled in member crates. The one exception to this rule is serde_json’s preserve_order feature which is only enabled in tools that need it. Fixes: #6301 Fixes: #7674
…7707) #7680 bumped the version of non-private crates to `0.15.0`. Unfortunately, the publish process failed, because `near-o11y` which is marked as a private package, is depended on by `near-vm-logic`, which is a non-private crate. This adds a check to `themis`, to ensure we never get this kind of discrepancy merged into master. Example Report: <img width="498" alt="CleanShot 2022-09-27 at 19 53 10@2x" src="https://user-images.githubusercontent.com/16881812/192575063-93c35ffb-e01e-41d9-8d4a-2d904ae85409.png">
Enabling or disabling tracing at runtime can be useful to debug issues with nodes running too slow. This allows us to avoid restarting a node, which is known to have a rather large impact on the node.
Tested that all combinations of logs verbosity and opentelemetry verbosity work well together.
Tested by reconfiguring logging as follows:
```
echo '{"verbose_module":"", "opentelemetry_level": "DEBUG"}' > ~/.near/betanet/log_config.json ; killall -SIGHUP neard
```
We create FlatStorageState on Chain creation, but it currently happens for any RuntimeAdapter impl. Because KeyValueRuntime doesn't and shouldn't support flat storage, it leads to tests failures. Here we move FSS creation inside RuntimeAdapter impl. ## Testing https://buildkite.com/nearprotocol/nearcore-flat-state/builds/60#018383c5-7838-4e5b-86f1-da2baa381789 - number of failing tests drops from 77 to 4 - which are failing due to other reasons
* Use the underlying reproducible random impls. * Improve testing * fix deny * fmt Co-authored-by: Michal Nazarewicz <[email protected]>
…ublishable crates depend on private crates (#7720) The original message was hard to understand at a first glance. #7707 (comment)
Explicitly stop and wait for prefetching background threads to terminate when the `ShardTriesInner` is dropped. This avoids that estimations are influenced by background threads left over from previous estimations, which we have observed since merging #7661.
Part of the public API of network package was placed in near-network-primitives and part was in near-network. Hence most of the other packages imported both anyway. Merging them back together allows us to: - decrease the public API surface (since within a crate visibility settings are more flexible) - put API next to the only implementation of that API - decrease the confusion when importing network stuff in other packages
Closes #7654. This PR adds `metadata.transfer_fee_type` field to the Operation object, which is only set for Gas prepayment (which happens on transaction processing) and Gas refunds (receipt that is sent by `system` user). Thus, there are just three states: * not set - it is either not a transfer operation (Construction API), or it is not a fee transfer * `GAS_PREPAYMENT` * `GAS_REFUND`
These are very clearly just API docs, they are much more discoverable in the code, rather than in the guide.
…#7695) * Draft for splitting chunks store * Fix a test * Unwrap ChunksLogic struct * Fix formatting
Fix: typos
Move logic of driving the database migration to StoreOpener. This is done by introducing StoreMigrator trait which is passed when creating StoreOpener through a new NodeStorage::opener_with_migrator method. Object implementing that trait contains code for performing migrations one level at a time and StoreOpener is what is now checking the database version and looping through all the migrations calling the migrator. The change is motivated by future plans of adding cold database to NodeStorage. With cold database, versions of two databases will need to be checked and it’s cleanest to encapsulate all that logic in StoreOpener. This makes rest of the code pretty much unaware of how versions are stored and only dedicated migration code needs to deal with performing single-version migrations.
The ping tool is not monitored by prometheus explicitly, therefore the metrics can be pushed to a metrics gateway.
nikurt
requested review from
akhi3030 and
marcelo-gonzalez
and removed request for
akhi3030
marcelo-gonzalez
approved these changes
Nov 11, 2022
There was a problem hiding this comment.
ah anyway i could have approved it even w the 2 comments above ^. if we wanna add labels later it wont be hard. Adding @akhi3030 again as a reviewer to approve
akhi3030
approved these changes
Nov 14, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Export prometheus metrics from the ping tool.
The ping tool is not monitored by prometheus explicitly, therefore the metrics can be pushed to a metrics gateway.