Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inaccurate partials proof marked as OG proof #42

Open
verifyfx opened this issue Jul 10, 2021 · 20 comments
Open

Inaccurate partials proof marked as OG proof #42

verifyfx opened this issue Jul 10, 2021 · 20 comments
Assignees
@stolk
Labels
bug Something isn't working

Comments

@verifyfx
Copy link

verifyfx commented Jul 10, 2021 •
edited
Loading

I suspected that there are issue where partials proof were parsed

image

2021-07-10T02:08:40.428 harvester chia.harvester.harvester: INFO     3 plots were eligible for farming 3ff73aafb4... Found 2 proofs. Time: 2.29660 s. Total 57 plots
2021-07-10T02:08:40.447 farmer chia.farmer.farmer         : INFO     Submitting partial for XXX to https://asia1.pool.space

from reddit
@stolk stolk self-assigned this Jul 10, 2021
@stolk stolk added the bug Something isn't working label Jul 10, 2021
@stolk
Copy link
Owner

stolk commented Jul 10, 2021

Thanks for reporting. Does the dot turn cyan if you resize the terminal (w tool running) to be shorter in height?

@kastermight
Copy link

kastermight commented Jul 10, 2021 •
edited
Loading

I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D

Screenshot from 2021-07-10 12-16-00

@stolk
Copy link
Owner

stolk commented Jul 10, 2021

Ok, I've been able to reproduce it now. And understand what is going on. This is triggered when it finds 2 proofs, it seems. And possibly only submits a partial for 1?

@stolk
Copy link
Owner

stolk commented Jul 10, 2021

I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D

Screenshot from 2021-07-10 12-16-00

Can you email your debug.log to b.stolk at gmail.com please? I would like to test a fix on your log.

@verifyfx
Copy link
Author

image
resizjng doesn't turn it cyan

@verifyfx
Copy link
Author

image

pulling latest ( 1dbc2c1 ) fixed this issue.

@kastermight
Copy link

kastermight commented Jul 11, 2021 •
edited
Loading

I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D
Screenshot from 2021-07-10 12-16-00

Can you email your debug.log to b.stolk at gmail.com please? I would like to test a fix on your log.

OK I did. My idea is there's some race condition going on since I have 48 full_nodes running (number of my cores) and sometimes they write in off order. I can see that you check for "Submitting partials" entries after "Found n proofs" entries to differentiate between OG proofs and pool proof. I noticed that sometimes "submitting" entries come earlier than the "found" entries. But I might be wrong, so good luck with debugging.

@stolk
Copy link
Owner

stolk commented Jul 11, 2021 via email

@verifyfx
Copy link
Author

verifyfx commented Jul 11, 2021 •
edited
Loading

Found another case:
It's a case where farmer logged submitting partials before harvester logged proof found
image

2021-07-11T22:01:16.087 farmer chia.farmer.farmer         : INFO     Submitting partial for XXX to https://XXX
2021-07-11T22:01:16.406 harvester chia.harvester.harvester: INFO     2 plots were eligible for farming 315babe1db... Found 1 proofs. Time: 1.08288 s. Total 138 plots
2021-07-11T22:01:23.747 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 315babe1db... Found 0 proofs. Time: 0.00621 s. Total 138 plots
2021-07-11T22:01:34.160 harvester chia.harvester.harvester: INFO     1 plots were eligible for farming 315babe1db... Found 1 proofs. Time: 2.18124 s. Total 138 plots
2021-07-11T22:01:34.203 farmer chia.farmer.farmer         : INFO     Submitting partial for XXX to https://asia1.pool.space
2021-07-11T22:01:35.091 farmer chia.farmer.farmer         : INFO     Pool response: {'new_difficulty': 4}
2021-07-11T22:01:41.452 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 315babe1db... Found 0 proofs. Time: 0.00669 s. Total 138 plots

@kastermight
Copy link

kastermight commented Jul 11, 2021 •
edited
Loading

Huh? that makes absolutely no sense, to run 48 full nodes on one machine. Are you sure you actually have that many running? How would you even start that many nodes?

afaik they start automatically when i do chia start farmer.
Screenshot from 2021-07-11 08-56-13

@stolk
Copy link
Owner

stolk commented Jul 11, 2021 via email

@kastermight
Copy link

kastermight commented Jul 12, 2021 •
edited
Loading

Those are threads of a single process. A thread per peer connection, probably. On Sun, Jul 11, 2021 at 8:56 AM Kaster Might @.> wrote:
Huh? that makes absolutely no sense, to run 48 full nodes on one machine. Are you sure you actually have that many running? How would you even start that many nodes? … <#m_-1223658502728578559_> On Sun, Jul 11, 2021 at 12:36 AM Kaster Might @. > wrote: I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D [image: Screenshot from 2021-07-10 12-16-00] https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png Can you email your debug.log to b.stolk at gmail.com http://gmail.com please? I would like to test a fix on your log. OK I did. My ideas is there's some race condition going on since I have 48 full_nodes running (number of my cores) and sometimes they write in off order. I can see that you check for "Submitting partials" entries after "Found n proofs" entries, and I noticed that sometimes "submitting" entries come earlier than the "found" entries. But I might be wrong, so good luck with debugging. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#42 (comment) <#42 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A . -- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @. afaik they start automatically when i do chia start farmer. [image: Screenshot from 2021-07-11 08-56-13] https://user-images.githubusercontent.com/2565943/125201878-e2b34980-e225-11eb-8862-e5bb5eb8b01d.png — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4AFQQMYTCBUAEXLTIDTXG5MHANCNFSM5AEPJL7A .
-- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.

Hi. Those are not threads as they have separate PIDs and TGIDs. I dont' think it matters though.

I saw your email, but I still can see blue squares. Maybe you had different branch or something. I'm on latest master. Recompiled and everything. Attaching both screenshots and the log file here just in case.

PS: I noticed that if the terminal window is short enough, blue squares disappear. Maybe that's what you're seeing.

debug.log
Screenshot from 2021-07-11 18-09-41
Screenshot from 2021-07-11 18-14-42

@xordos
Copy link

xordos commented Jul 13, 2021 •
edited
Loading

I want share my log as I have simplest (single machine run full farmer) case and I do see the problem too.

Basically, in the log, sometime, the "Submitting partial" log line doesn't immediate after the eligible line. I have 2 such cases that show blue dot but are actually pool proof. I think this is a race condition that these 2 lines are from different process. So the order is not guaranteed, see the logging process name (farmer vs harvester) below:

2021-07-12T16:42:04.397 harvester chia.harvester.harvester: INFO 2 plots were eligible for farming 686599092a... Found 0 proofs. Time: 0.35231 s. Total 798 plots
2021-07-12T16:42:14.216 farmer chia.farmer.farmer : INFO Submitting partial for 10768b8bd5ab299b85a2c3d8fc80f1a70e03b898acd8292aeb7eb959a2765c04 to https://XXX
2021-07-12T16:42:14.299 harvester chia.harvester.harvester: INFO 4 plots were eligible for farming 686599092a... Found 1 proofs. Time: 1.75216 s. Total 798 plots
2021-07-12T16:42:14.398 farmer chia.farmer.farmer : INFO Pool response: {'new_difficulty': 1}
2021-07-12T16:42:22.282 harvester chia.harvester.harvester: INFO 2 plots were eligible for farming 686599092a... Found 0 proofs. Time: 0.06132 s. Total 798 plots

2021-07-13T06:56:25.522 harvester chia.harvester.harvester: INFO 1 plots were eligible for farming 4c1214cc87... Found 0 proofs. Time: 0.20828 s. Total 799 plots
2021-07-13T06:56:34.317 farmer chia.farmer.farmer : INFO Submitting partial for 10768b8bd5ab299b85a2c3d8fc80f1a70e03b898acd8292aeb7eb959a2765c04 to https://XXX
2021-07-13T06:56:34.357 harvester chia.harvester.harvester: INFO 3 plots were eligible for farming 4c1214cc87... Found 1 proofs. Time: 1.09751 s. Total 799 plots
2021-07-13T06:56:34.523 farmer chia.farmer.farmer : INFO Pool response: {'new_difficulty': 1}
2021-07-13T06:56:43.013 harvester chia.harvester.harvester: INFO 3 plots were eligible for farming 4c1214cc87... Found 0 proofs. Time: 0.12265 s. Total 799 plots

BTW, your tool is really useful and thanks for your hard work!!

@kastermight
Copy link

Hi @stolk Can you give a little update on this? I just want to understand if a) it's not a bug but a feature, b) it's a bug but not fixable at the moment. Thanks.

@stolk
Copy link
Owner

stolk commented Aug 7, 2021 via email

stolk added a commit that referenced this issue Aug 7, 2021
@stolk
Address issue #42: showing pool proofs as solo proofs.
157e32f
@stolk
Copy link
Owner

stolk commented Aug 7, 2021 via email

@kastermight
Copy link

Seems to be working now :) I generally agree that soon pretty much no one will be doing both solo and pool. but at right this moment I know at least 5 people doing it. Not me though.

@stolk
Copy link
Owner

stolk commented Aug 8, 2021 via email

@kastermight
Copy link

kastermight commented Aug 9, 2021 •
edited
Loading

Are those people running multiple fullnodes and multiple farmers? Then it should be ok, they will run multiple graph tools.
On Sun, Aug 8, 2021 at 1:35 AM Kaster Might @.> wrote: Seems to be working now :) I generally agree that soon pretty much no one will be doing both solo and pool. but at right this moment I know at least 5 people doing it. Not me though. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4CW5F7TC4YUAX4SE6LT3Y6WHANCNFSM5AEPJL7A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .
-- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.

Not really. They still have their OG plots and nft plots on a single machine. They basically have a spare space to fill with nft plots and at the same time they keep solo farming OG plots. I was in the same boat but not anymore. I replaced all my OG plots with NFT plots already, and I'm sure soon everyone will be done with the same process. Depends on the farm size I guess.

@verifyfx
Copy link
Author

just another idea, is it possible to include full node for full proof detection? basically we should look for the following line for full block?

🍀 ️Farmed unfinished_block {block_hash}, SP: {block.reward_chain_block.signage_point_index},

sauce: https://github.com/Chia-Network/chia-blockchain/blob/7927a459456c1f79c3604eb2aa5cf238545d15e8/chia/full_node/full_node.py#L1360

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stolk stolk

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@stolk @xordos @kastermight @verifyfx