Data is lost in block 22633808 #47

telezhnaya · 2022-07-12T13:38:21Z

While investigating https://github.com/telezhnaya/indexer_balances/issues/1, I've found the issue
We have the receipt in this block, but we don't have execution_outcome

 select * from blocks where block_height = 22633808; – 1605411776828600726 68v7s5fg8inLQ2VVnFvZ375bbbx1akU4kbH8ezo6RyFg
 select * from account_changes where account_changes.changed_in_block_timestamp = 1605411776828600726;
 select * from receipts where receipt_id = '6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4';
 select * from execution_outcomes where receipt_id = '6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4';

Please check why receipt_execution_outcomes is empty for block 22633808

The text was updated successfully, but these errors were encountered:

khorolets · 2022-07-28T17:18:42Z

@telezhnaya could you check please, I've made near-lake-indexer to run against that block again.

frol · 2022-07-28T21:10:41Z

@khorolets ideally, we should have looked at what was the state of those files before rewriting and why near-lake-framework-rs passed incomplete StreamMessage down the line 🧐

telezhnaya · 2022-11-09T07:29:12Z

@khorolets we still have this problem

khorolets · 2022-12-12T10:29:15Z

OK, I've checked. I want to pass it on to you @telezhnaya I've found the ExecutionOutcome for that Receipt in the block 22635169

May I ask you to find out why the heck the ExecutionOutcome for Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 is stored as executed_in_block_hash of
68v7s5fg8inLQ2VVnFvZ375bbbx1akU4kbH8ezo6RyFg in Indexer for Explore Database if it was executed in 5WEpxdCAouEwxtbPrBuVmztX9X4cbnbfgWrnTHbvUWM8? O_o

22635169 `StreamerMessage`

StreamerMessage {
    block: BlockView {
        author: AccountId(
            "accomplice.poolv1.near",
        ),
        header: BlockHeaderView {
            height: 22635169,
            prev_height: None,
            epoch_id: `7Y4LN7GgyYVXMdsEuDV6dWfsML7A9sn6pwnYbY3whn5H`,
            next_epoch_id: `6NYcV1hbnNiW1VAWSkKDiBAvAFgDZqMC29y3vjPogoBF`,
            hash: `5WEpxdCAouEwxtbPrBuVmztX9X4cbnbfgWrnTHbvUWM8`,
            prev_hash: `BEzouvv6xJ9FWkGGgKN98bqEUde3ZKJx51KiWUZbtdQU`,
            prev_state_root: `CxgT68o6sQ2nDEBdrLYdWgYAC5YkiBykgVHTK6xL4J5C`,
            chunk_receipts_root: `98jPBYaQEMxGzDUJJHwoLoGHpi61LGfbCoebTG6LEH3v`,
            chunk_headers_root: `14DCgPqsMSZbLziSHbvjCZakweD2NVeeo5jUzRkKBf6c`,
            chunk_tx_root: `7tkzFg8RHBmMw1ncRJZCCZAizgq4rwCftTKYLce8RU8t`,
            outcome_root: `8zYQs2R2ENytfHVj5C2Y4CzPBqrenNPzfZ6Fx8z97vXW`,
            chunks_included: 1,
            challenges_root: `11111111111111111111111111111111`,
            timestamp: 1605412974411419407,
            timestamp_nanosec: 1605412974411419407,
            random_value: `RhWqciryRzrGxtg9WYeLRy1a8coh2SqkxpfSxNFerUB`,
            validator_proposals: [],
            chunk_mask: [
                true,
            ],
            gas_price: 100000000,
            block_ordinal: None,
            rent_paid: 0,
            validator_reward: 0,
            total_supply: 1004251838545238298658346767796377,
            challenges_result: [],
            last_final_block: `CXHmNbYaaf5N3fATTqTuC8Whb7GfvGpqkWxyM8t8hL43`,
            last_ds_final_block: `BEzouvv6xJ9FWkGGgKN98bqEUde3ZKJx51KiWUZbtdQU`,
            next_bp_hash: `5QmKcsZ7KwUg6579QNTwi9YV4EdWjw3F7F5fB2jiLCWq`,
            block_merkle_root: `8f924XiPysYmjofqBWTFypbm2Z6tkdda66zXWzoXHena`,
            epoch_sync_data_hash: None,
            approvals: [
                Some(
                    ed25519:4WnTJLRAcExdnzvP93ht3YskBmzsZo8mYJksRKsnMPGWL4TjvFyLqyrHUxbgvVukF5Vhh9wpXBozqxYWpetywyjF,
                ),
                Some(
                    ed25519:3JzkAAod6tAx2UZTj5i23JSCm9L4V8ZF1uamW5rX7fWdcrAZUMz8Sd56MA4rWoFUAup6ELZ3vpDZvH7KbKcsbTCF,
                ),
                Some(
                    ed25519:fQWxxjq81un7Yr2YhWA97FReH5kz35rtRXMTDbUJyZsB6Y33rQNCqxEFTGVdTeZNNZ9CBwLhfXbPLDdtxw3CJpg,
                ),
                Some(
                    ed25519:2cPEpz1a1k8Y5n41npygNcB79syJRRpqSkVrbcRANioqBMxDb6hsQBegXkkEAq3HKY2DtQ8TEggzAPbcKnL6EBEo,
                ),
                Some(
                    ed25519:2H22imgbVWiduzQ2GTBLhh4yBuqR6wDFPe3qbA6783oc6nmfWHUxhxzAGCQ9afwZcLPktUNwfhvZ7Vv55zRUioPu,
                ),
                Some(
                    ed25519:4fEGEdi82vMsgGpTQYGJQHC1xcwPq8eRsTaPKB9ZDyisSeW2CGb3ia6yy7FC1heUzX9hoEeUkETSmFk3Np6TjJTr,
                ),
                Some(
                    ed25519:2tMShKrnSwPuain8agTC19A5WWG1RxPzo8wXHJsdUMMCZrUjpwK2Pphg3Fu5Fnj1riEBFjmVaWdbmJuGVvkYobEC,
                ),
                Some(
                    ed25519:2znAzLdJDTnfdUT8p7YvL8UmLju3XnnoRULDdDdMXB4NwFvdPeE9EXEn63tsis2TkqFudScyVU849SbHDWHY8nLV,
                ),
                Some(
                    ed25519:2KxpsQGtHw7S5qYc5KjJ8DDVBo8PmUgak5h49jjTkGazgDFkmxRXrK26cVjUYsWEcS1g4UA2x4mcAWLE9U3SgAU3,
                ),
                Some(
                    ed25519:3vd7p9vtVjzjhgDvgZNLfBoH3CgwVo3xuNyz4WNrgCjoAzH5yrpsCKn2imr7K53uaXnV6LvEb9TXemAzQUGN94sH,
                ),
                None,
                None,
                Some(
                    ed25519:f6QcTVUsScxbYLCxGSyAonP4FPbcDAkW9n8WNBgKnVciouMYrfWUTCqiFHrDDKrpnmqUq1Yg4aXv7WDunSfgoKg,
                ),
                Some(
                    ed25519:4yKymfmAJQeDBFjCG4a6iE5uFzfiL7RGiWVARTwZnWKR7Qxse2VLq1RBneCrigCpPNAiKJ5tL1qv9xRe2xbcmxSM,
                ),
                Some(
                    ed25519:4paLHjhAj5GechpoeCpeFMvkjQ7yN5SSYAQKv8xz5xGmKSWtc8DJVm53DcC9Vo4fQ265pZe8Rky94D8iadyK598F,
                ),
                Some(
                    ed25519:3FKE1wRkiJrYyFdspspjCtgRcGvbaJnZ7duS1e1EuW5VUMo4iKKSpTDUpujwxrGcHo1ye7zsX3Ujtbj3r2nk34fT,
                ),
                Some(
                    ed25519:2p6M7FSvWFeJj9oy83i3de3CHFMWV7yhxj8cagtpPAoXLiwHxLwBKNQ2UGDxGG5GKKxU4n5V4NdZudkFBp18hwzT,
                ),
                Some(
                    ed25519:zrCSN3Z1Y2kyrNhuSz2FXUx6Vy5csJKG4mGRjWxi7MRXLA3dntfDExgFRGaYnvh44tjyt9crit75UVVXy8JB8YA,
                ),
                Some(
                    ed25519:26ZkT4d8S5TAYgAFbGyV5tnvam9myPQKqVm9nReE2eap1Z9AFrUnV2AgC1sarV1q7gkbL5rSMXyM7KGJEtythCbR,
                ),
                Some(
                    ed25519:3zijHgW7SEfhtroRq8Ghp6VpKrm612V5wkw5tkYQhFKJmDC5XaAYFrVsoh8ZUaZmAoFC2eqfgZMNgzuPZ91HfDsQ,
                ),
                None,
                Some(
                    ed25519:4tRifKr36e9FRzC1ffHdq37vEF61v7PeDkdjPFFNw72Gr8YtmS2RnDhWuQyUoD4cjjmEMCgLYiczTW3nw7guiH2A,
                ),
                None,
                Some(
                    ed25519:abEsVu1enV3DRPYvYkVNgVS8Gn6hdj9MuTEHBZ7fL2hgwhzf3JdfnVDKQtuLuxRTofhMncChPgd1uTVLfrm8uZ7,
                ),
                Some(
                    ed25519:5cZwog43zRrLJjL6mhuecPTCQ71iaBSGLhMKihk7DVDS6Uekr4ZijgLqtLEjLboNGhGYPPHpTQu7QSjJUXJNSH9e,
                ),
                Some(
                    ed25519:2NP8JyzTZWhEvysr4cbirzCeeXSeQfDjKsfj1v9L8HLgqfJ1hbjKWbB7r75ouqoos4fb3QGk6iuP82XFm6A2qhc6,
                ),
                Some(
                    ed25519:5sMFwQzDy4o38d9NcXhsX96XFtADb2nz2qRhY1QyZ1CHHTBP8FHc4SwxpjB8ua54zufoyDU6SiKfQUE659xd8Bkn,
                ),
                Some(
                    ed25519:5ahbyrmzKozgJ3LVHAk5NZz2EhwPeM78jU23PkXiNtcP3gRJXWw16YjXpZxE8cBVNFS5D3ikcZRWk2Kpio3bWTLb,
                ),
                Some(
                    ed25519:Ye4C6J3zdB87a5Uyb3vkfU4z4Bnj8ukaKhrxqQ8FxDZ6NvJMguB9UHU6ix1oUW1N5ZPFYCScVtXtW6HE4m8En39,
                ),
                Some(
                    ed25519:2X4oiBzVZSHdutDQCpqosZ88fxZP5xx4oKnocmSRRgH9wsTsWQJ8j4B2vsCnRZnwrPpdW9VnHGTnwb5bfUe9BWzY,
                ),
                Some(
                    ed25519:3hV7RhPb19co78TZBan5zXzRFszCwTosr39DpoarqiFm8gFNYhG8x8QCGBEygbrq9QJLuajQXwT6ogbTaA7vdXV5,
                ),
                Some(
                    ed25519:3Qg7LVWPiHpzADt22kcXAntwq5sKVbgcJDisT9WjYBa1kE9su84G7mE1VMXUnUQBwyfrYxPzbXg5w4L6p1TSDAez,
                ),
                None,
                Some(
                    ed25519:2JqXppXe231PaH5rJHQimSE3DDmsQk4vKbu2SwuabxUpxpiGHFa8puBvGHcE3zMD46f13j8y4hBT5bZGkZtBZmSm,
                ),
                Some(
                    ed25519:TiZkve8kHT8itVfpdPhvnYka3cdX7c6PrK8TuNfMEENtSSXAyAQRA9XtcBKYdEroKMqqfNs4BKds6FvDxxXEj5j,
                ),
                None,
                None,
                Some(
                    ed25519:4xWFYqiqJJ3fZbHE3NZPnRkSK2wzekH9Bhqc3GvQVk3fKjnjH6jvdjBYcUAGDLaubao3XHwS2Se1KGcNn8pubdDC,
                ),
                None,
                Some(
                    ed25519:DjFm9UNQPQPaAAGXkhxr5VjCzfqEH3iUMeNN1pwYhrHwhRWy1rFqKaxxYKBWqSoJBiSnv4evvrWdWND8osFdgST,
                ),
                Some(
                    ed25519:3VmECv2HQrX8Zp2mATVcaDuXgd2bhW9hU97NnACPPjJr63FoqLhL3mVC9JqzeLxaCRApK4ayKJbAMYxjMNmyNena,
                ),
                Some(
                    ed25519:2tBtz1kSag9vMrromqdNTj1sU3mC6RFAWdsmfrvJiBLLHnUr7NGssfWkZ3Pb2yEArguc3hgPXqKtozzp68PtiST4,
                ),
                None,
                Some(
                    ed25519:TPKthq5gv5Uy32C3YNEpXXahUNxN4AVW6WKXU1qC2iJrVMWfYrDcnc5fknRY59MRPb6ZteRaTqhAVQr2eK6YUK3,
                ),
                Some(
                    ed25519:2VbPs89GRUous4cSqbaeQqnTn2JWN4TuvuTiZatVWzvwPC9rNu8GjtXBi7Qz6zyvnGUaewMtWF7qu6TFJCYN9d4N,
                ),
                Some(
                    ed25519:4crAGqzng3GzxF4kVQzdDbzbsB3PfrvXXt4JNoQyyyjJsHiALkCMxV2yHe7u5Ch5oBr5xayQL9pSGW6juPa73MQv,
                ),
                Some(
                    ed25519:5CEr16Arxua7MCJjf17JsEWDGuv3WJPApRkCYs1tbouwZJSnE3JEsoNFWcj1fNykqyfNtvwbu3ztJ76C6uUAaEK9,
                ),
                Some(
                    ed25519:43UwrxBfn1H9GhayskoSnUEd5m8viCUixiU3cWQx8EgDA3FnbpbbDV67uJDkBHyKJK2FwfqBFoQE3w4at8CL2TJE,
                ),
                Some(
                    ed25519:52uPyxbceVRMNme1nacxUtqvEUNpAjacKRBUGJ7vh4irZ93fmdqj9k6vRAVA5hAvkZx8eG6VXfy2XcvKBF8hneiY,
                ),
                Some(
                    ed25519:3ZT1bK4gsDaavUyB9ozjqDjvu6gUtoNWCRXTWojqAWsBudaeuooL9EcXGEDAxQ6uv4jekTzZy8oCeNqH2GLBvQDH,
                ),
                Some(
                    ed25519:4eBCQceLonuJjmDG5cjDNds2bBo65DFRzETi7iN4t4QDy266j5QN2qoUbHhzRdwJtZYF2piSP74kmZdbTaBqxg2w,
                ),
                Some(
                    ed25519:4YA5HYQQeutE6p2G2PVFQd6ftC1CRCrgaFxAvQrxNt4N7HLik62WG5a9zp9hGEJ7NzoRThM6s9DYG4KYgTegwwBf,
                ),
                Some(
                    ed25519:4tXcpYYPZ6tVccenaGC29yk9aoRTknhwGwEB8yrVcb4qJK9S6FRKu8NhCfvGptyozQe4fYiY3vBAFGf4zvnbCSkH,
                ),
                Some(
                    ed25519:4s5BPRFRKyvGZhkAa7RowppK7s8r3YguN2XLicGxk3PrpfDTurFZxAJursEstD6QwBHFZGbNED2KxRM5JtbcHdDM,
                ),
            ],
            signature: ed25519:28F72kjLTn7B8hZqfF214y77uLunq6jWw7v7FbjDFoJ3WXrW9QpWwanaFNyKUZPf6YwvUcXJ9hb1hN7WbsUFKt93,
            latest_protocol_version: 36,
        },
        chunks: [
            ChunkHeaderView {
                chunk_hash: `HmDUWTVtHLf9uuuxiasU2ZCTwxEapGqVky9A26kmeTJT`,
                prev_block_hash: `BEzouvv6xJ9FWkGGgKN98bqEUde3ZKJx51KiWUZbtdQU`,
                outcome_root: `6MgWFeiCPzKRJPqF8FPMzkexXb1wGDLbFdmmZe5eKu6L`,
                prev_state_root: `41mGs9fEarHH8t6rgWSSatZ7xPvzhzfwwb91XyhMJMEP`,
                encoded_merkle_root: `Dg6eoP833WPmAtMpisU1oVDZWQDoV9cCTirz9Z8nKpkp`,
                encoded_length: 344,
                height_created: 22635169,
                height_included: 22635169,
                shard_id: 0,
                gas_used: 105525848812409,
                gas_limit: 1000000000000000,
                rent_paid: 0,
                validator_reward: 0,
                balance_burnt: 7477318296645900000000,
                outgoing_receipts_root: `76BH1tHiQ8UyBQMHY2xtxPfUby38Uf5UkePw4ZhGp2zy`,
                tx_root: `11111111111111111111111111111111`,
                validator_proposals: [],
                signature: ed25519:5zYQVo8oafyArE1xAgMBuf79D5XcnoaxZxc9TuhztvrG4hCuFnTmrzzTDWVxtWkgyGWajAqr2fSRYYpQvX89hRaL,
            },
        ],
    },
    shards: [
        IndexerShard {
            shard_id: 0,
            chunk: Some(
                IndexerChunkView {
                    author: AccountId(
                        "accomplice.poolv1.near",
                    ),
                    header: ChunkHeaderView {
                        chunk_hash: `HmDUWTVtHLf9uuuxiasU2ZCTwxEapGqVky9A26kmeTJT`,
                        prev_block_hash: `BEzouvv6xJ9FWkGGgKN98bqEUde3ZKJx51KiWUZbtdQU`,
                        outcome_root: `6MgWFeiCPzKRJPqF8FPMzkexXb1wGDLbFdmmZe5eKu6L`,
                        prev_state_root: `41mGs9fEarHH8t6rgWSSatZ7xPvzhzfwwb91XyhMJMEP`,
                        encoded_merkle_root: `Dg6eoP833WPmAtMpisU1oVDZWQDoV9cCTirz9Z8nKpkp`,
                        encoded_length: 344,
                        height_created: 22635169,
                        height_included: 22635169,
                        shard_id: 0,
                        gas_used: 105525848812409,
                        gas_limit: 1000000000000000,
                        rent_paid: 0,
                        validator_reward: 0,
                        balance_burnt: 7477318296645900000000,
                        outgoing_receipts_root: `76BH1tHiQ8UyBQMHY2xtxPfUby38Uf5UkePw4ZhGp2zy`,
                        tx_root: `11111111111111111111111111111111`,
                        validator_proposals: [],
                        signature: ed25519:5zYQVo8oafyArE1xAgMBuf79D5XcnoaxZxc9TuhztvrG4hCuFnTmrzzTDWVxtWkgyGWajAqr2fSRYYpQvX89hRaL,
                    },
                    transactions: [],
                    receipts: [
                        ReceiptView {
                            predecessor_id: AccountId(
                                "system",
                            ),
                            receiver_id: AccountId(
                                "aspaceodyssey2001.near",
                            ),
                            receipt_id: `6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4`,
                            receipt: Action {
                                signer_id: AccountId(
                                    "system",
                                ),
                                signer_public_key: ed25519:11111111111111111111111111111111,
                                gas_price: 0,
                                output_data_receivers: [],
                                input_data_ids: [],
                                actions: [
                                    Transfer {
                                        deposit: 3690308308890300000000000000,
                                    },
                                ],
                            },
                        },
                        ReceiptView {
                            predecessor_id: AccountId(
                                "system",
                            ),
                            receiver_id: AccountId(
                                "relay20201112.bridge.near",
                            ),
                            receipt_id: `7hvy4PwoSxWM3BsesEJCGUUsNnncruYCqbmpKtAG7qKu`,
                            receipt: Action {
                                signer_id: AccountId(
                                    "relay20201112.bridge.near",
                                ),
                                signer_public_key: ed25519:CmAGytdaDYfmrB85ArFfLdRepDTSTPdEnp8trUdh8U3L,
                                gas_price: 0,
                                output_data_receivers: [],
                                input_data_ids: [],
                                actions: [
                                    Transfer {
                                        deposit: 178576843726431094307760,
                                    },
                                ],
                            },
                        },
                    ],
                },
            ),
            receipt_execution_outcomes: [
                IndexerExecutionOutcomeWithReceipt {
                    execution_outcome: ExecutionOutcomeWithIdView {
                        proof: [
                            MerklePathItem {
                                hash: `5cA8GQnRkTdvvUf1KBZxJpH76XQkYbgaVDHudXKC5bn6`,
                                direction: Right,
                            },
                        ],
                        block_hash: `5WEpxdCAouEwxtbPrBuVmztX9X4cbnbfgWrnTHbvUWM8`,
                        id: `6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4`,
                        outcome: ExecutionOutcomeView {
                            logs: [],
                            receipt_ids: [],
                            gas_burnt: 0,
                            tokens_burnt: 0,
                            executor_id: AccountId(
                                "aspaceodyssey2001.near",
                            ),
                            status: SuccessValue(``),
                            metadata: ExecutionMetadataView {
                                version: 1,
                                gas_profile: None,
                            },
                        },
                    },
                    receipt: ReceiptView {
                        predecessor_id: AccountId(
                            "system",
                        ),
                        receiver_id: AccountId(
                            "aspaceodyssey2001.near",
                        ),
                        receipt_id: `6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4`,
                        receipt: Action {
                            signer_id: AccountId(
                                "system",
                            ),
                            signer_public_key: ed25519:11111111111111111111111111111111,
                            gas_price: 0,
                            output_data_receivers: [],
                            input_data_ids: [],
                            actions: [
                                Transfer {
                                    deposit: 3087754972079500200000000,
                                },
                            ],
                        },
                    },
                },
                IndexerExecutionOutcomeWithReceipt {
                    execution_outcome: ExecutionOutcomeWithIdView {
                        proof: [
                            MerklePathItem {
                                hash: `9ZJuzK1o7ob3tPs67z7iUk2fX99A6hVoX6T29q8A29t5`,
                                direction: Left,
                            },
                        ],
                        block_hash: `5WEpxdCAouEwxtbPrBuVmztX9X4cbnbfgWrnTHbvUWM8`,
                        id: `7hvy4PwoSxWM3BsesEJCGUUsNnncruYCqbmpKtAG7qKu`,
                        outcome: ExecutionOutcomeView {
                            logs: [],
                            receipt_ids: [],
                            gas_burnt: 0,
                            tokens_burnt: 0,
                            executor_id: AccountId(
                                "relay20201112.bridge.near",
                            ),
                            status: SuccessValue(``),
                            metadata: ExecutionMetadataView {
                                version: 1,
                                gas_profile: None,
                            },
                        },
                    },
                    receipt: ReceiptView {
                        predecessor_id: AccountId(
                            "system",
                        ),
                        receiver_id: AccountId(
                            "relay20201112.bridge.near",
                        ),
                        receipt_id: `7hvy4PwoSxWM3BsesEJCGUUsNnncruYCqbmpKtAG7qKu`,
                        receipt: Action {
                            signer_id: AccountId(
                                "relay20201112.bridge.near",
                            ),
                            signer_public_key: ed25519:CmAGytdaDYfmrB85ArFfLdRepDTSTPdEnp8trUdh8U3L,
                            gas_price: 0,
                            output_data_receivers: [],
                            input_data_ids: [],
                            actions: [
                                Transfer {
                                    deposit: 178576843726431094307760,
                                },
                            ],
                        },
                    },
                },
            ],
            state_changes: [
                StateChangeWithCauseView {
                    cause: ReceiptProcessing {
                        receipt_hash: `6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4`,
                    },
                    value: AccountUpdate {
                        account_id: AccountId(
                            "aspaceodyssey2001.near",
                        ),
                        account: AccountView {
                            amount: 3695396021799638242400000000,
                            locked: 0,
                            code_hash: `11111111111111111111111111111111`,
                            storage_usage: 333,
                            storage_paid_at: 0,
                        },
                    },
                },
                StateChangeWithCauseView {
                    cause: ReceiptProcessing {
                        receipt_hash: `7hvy4PwoSxWM3BsesEJCGUUsNnncruYCqbmpKtAG7qKu`,
                    },
                    value: AccountUpdate {
                        account_id: AccountId(
                            "relay20201112.bridge.near",
                        ),
                        account: AccountView {
                            amount: 439119056546033518936080338,
                            locked: 0,
                            code_hash: `11111111111111111111111111111111`,
                            storage_usage: 182,
                            storage_paid_at: 0,
                        },
                    },
                },
            ],
        },
    ],
}

telezhnaya · 2022-12-12T10:40:20Z

Are you sure it's for me? We just listen to the blockchain, we don't create this data from nowhere
Maybe we need to summon someone from the core team?

khorolets · 2022-12-12T10:47:11Z

@telezhnaya My main concern is why we have stored that piece of data in indexer for explorer db the way we have.
Let's not bother protocol team unless we have any evidence.

I am going to check the archival indexer data to narrow down the questions who's mistake is that and will update you.

khorolets · 2022-12-12T11:53:41Z

My checks on the archival node with Indexer Framework prove that ExecutionOutcome for that receipt is included in block 22635169.

22633808.log
22635169.log

Since the data reflected by Lake Indexer is correct I am closing this issue and I am going to open an issue on Indexer for Explorer project to figure out the reason the record in the database is incorrect.

telezhnaya · 2022-12-12T12:37:39Z

I feel it's still an issue for this project.
Indexer balances, where I've found this issue initially, has no dependency to indexer-for-explorer.
I mentioned indexer-for-explorer here just because it's the convenient way to look at the data.

Inside lake, we have json with block 22633808, where one field explicitly says "hey, we have the other useful data in the other field", and the second field is empty

khorolets · 2022-12-12T12:40:50Z

I disagree. Lake indexer doesn't say we have execution outcome for this receipt in that block. Have a look at my comment with attached logs from Indexer Framework. The block you refer to just doesn't contain ExecutionOutcome.

telezhnaya · 2022-12-12T13:03:36Z

At json about block 22633808, we see this

StateChangeWithCauseView {
                    cause: ReceiptProcessing {
                        receipt_hash: `6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4`,
                    },
                    value: AccountUpdate {
                        account_id: AccountId(
                            "aspaceodyssey2001.near",
                        ),
                        account: AccountView {
                            amount: 5087754972079500200000000,
                            locked: 0,
                            code_hash: `11111111111111111111111111111111`,
                            storage_usage: 182,
                            storage_paid_at: 0,
                        },
                    },
                },

The fee was taken. As I understand the blockchain logic, it means that the receipt should be executed in the same block.

telezhnaya · 2022-12-12T13:14:19Z

Wow, it looks like it's not the fee.
I'm naive and I still believe indexer-for-explorer data
According to that, 3.08 NEAR came to aspaceodyssey2001.near at block 22633808.
So I'm totally lost.
Do you have any idea why may we have such balance change?
I need to handle such situations at https://github.com/near/near-indexer-balances
The more confusing for me is why such a situation is so unique

telezhnaya · 2022-12-12T13:16:48Z

For future me:
3.08 is found by writing the query (the timestamps picked up randomly)

select * from account_changes 
where account_changes.affected_account_id = 'aspaceodyssey2001.near' and
account_changes.changed_in_block_timestamp > 1604411776828600726
and account_changes.changed_in_block_timestamp <= 1605411776828600726;

plus some calc (5087754972079500200000000 - 2000000000000000000000000) / 10**24

frol · 2022-12-12T15:06:53Z

@khorolets @telezhnaya This was an instance of duplicate transaction execution. Blocks #22633807 and #22635168 contain transaction J4CZZQrZK6kYPVLkrdbTEpcqhUNZiRxktbMzHviqeGgf. The implicit account was initially created by a transfer call at block #22633373, then it was deleted with a transaction at block #22633807 (unfortunately, Indexer for Explorer Postgres does not reflect it, so you will have to make a chunks RPC call, see below), and then the account was re-created at block #22634878 with a new transfer, and then the previous account deletion transaction got re-played (due to the nonce tracking issue; after the account was re-created, the nonce got reset to zero and now exactly the same transaction became valid again as its nonce is greater than the current one [zero]) and was executed again at block #22635168, which deleted the implicit account again and transferred all the tokens to aspaceodyssey2001.near (I can only hope that the tokens anyway were designated to be transferred to that account as otherwise it would lead to someone transferring more tokens than intended)

http post https://archival-rpc.mainnet.near.org jsonrpc=2.0 method=chunk params:='{"block_id": 22633807, "shard_id": 0}' id=dontcare

http post https://archival-rpc.mainnet.near.org jsonrpc=2.0 method=chunk params:='{"block_id": 22635168, "shard_id": 0}' id=dontcare

khorolets · 2022-12-12T16:51:24Z

then it was deleted with a transaction at block #22633807 (unfortunately, Indexer for Explorer Postgres does not reflect it, so you will have to make a chunks RPC call, see below)

Here are the transaction and the receipt you're referring to, I believe:
https://explorer.near.org/blocks/CVMvbRjcTJ4cbJsn2fNqWQgXmUffWsJs2pg2chRRnSDf

Thank you @frol for digging into it. Your input is very useful!

We're trying to understand the data flow behind these actions. Talking about blocks #22633807 and #22633808. We see the:

Transaction with DeleteAccount in #22633807
Transaction is converted into Receipt FA9zcm7WkWxdjkub7WFiKkQdnnQrcEmBht94VFzXfkm1
The Receipt FA9zcm7WkWxdjkub7WFiKkQdnnQrcEmBht94VFzXfkm1 is immediately executed in the same block. And it contains status: SuccessValue(``) and receipt_ids: [ '6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4' ]
This Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 is included in the very next block #22633808.

But receipt_execution_outcomes: [] is empty for the block #22633808, though StateChangeWithCauseView with cause ReceiptProcessing for the Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 is present.

At the same time, Indexer for Explorer says the Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 is executed in block #22633808. I've managed to find the ExecutionOutcome with Lake and Indexer Frameworks in block #22635169.

So my main assumption is that we might have introduced some changes to the Indexer Framework that we are no longer able to collect the Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 for block #22633808 while it should be there (it used to be there where we were indexing that block with Indexer for Explorer).

What do you think?

frol · 2022-12-13T10:15:11Z

So my main assumption is that we might have introduced some changes to the Indexer Framework that we are no longer able to collect the Receipt 6QKTEHsLihDgVrdtWYbURA28zF29Xdw5dnfWbDVWHPv4 for block #22633808 while it should be there (it used to be there where we were indexing that block with Indexer for Explorer).

This is an unpleasant mystery. I have no better guesses, but I don't see how that could have happened.

frol added the bug Something isn't working label Jul 14, 2022

exalate-issue-sync bot assigned telezhnaya Sep 6, 2022

khorolets self-assigned this Dec 12, 2022

khorolets closed this as completed Dec 12, 2022

khorolets mentioned this issue Dec 12, 2022

Misleading ExecutionOutcome record is stored in DB (mainnet) near/near-indexer-for-explorer#328

Open

telezhnaya reopened this Dec 12, 2022

telezhnaya mentioned this issue Dec 12, 2022

print error if indexer fails, drop unused cli args near/near-indexer-events#32

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data is lost in block 22633808 #47

Data is lost in block 22633808 #47

telezhnaya commented Jul 12, 2022 •

edited by exalate-issue-sync bot

Loading

khorolets commented Jul 28, 2022

frol commented Jul 28, 2022

telezhnaya commented Nov 9, 2022

khorolets commented Dec 12, 2022

telezhnaya commented Dec 12, 2022

khorolets commented Dec 12, 2022

khorolets commented Dec 12, 2022

telezhnaya commented Dec 12, 2022

khorolets commented Dec 12, 2022

telezhnaya commented Dec 12, 2022

telezhnaya commented Dec 12, 2022

telezhnaya commented Dec 12, 2022

frol commented Dec 12, 2022 •

edited

Loading

khorolets commented Dec 12, 2022

frol commented Dec 13, 2022

Data is lost in block 22633808 #47

Data is lost in block 22633808 #47

Comments

telezhnaya commented Jul 12, 2022 • edited by exalate-issue-sync bot Loading

khorolets commented Jul 28, 2022

frol commented Jul 28, 2022

telezhnaya commented Nov 9, 2022

khorolets commented Dec 12, 2022

telezhnaya commented Dec 12, 2022

khorolets commented Dec 12, 2022

khorolets commented Dec 12, 2022

telezhnaya commented Dec 12, 2022

khorolets commented Dec 12, 2022

telezhnaya commented Dec 12, 2022

telezhnaya commented Dec 12, 2022

telezhnaya commented Dec 12, 2022

frol commented Dec 12, 2022 • edited Loading

khorolets commented Dec 12, 2022

frol commented Dec 13, 2022

telezhnaya commented Jul 12, 2022 •

edited by exalate-issue-sync bot

Loading

frol commented Dec 12, 2022 •

edited

Loading