Provide an endpoints for TFT supply figures including both Stellar and Rivine

[scottyeager]

Currently we have the tft_statistics service running at https://statsdata.threefoldtoken.com/stellar_stats. This provides TFT supply info for the Stellar chain only.

We'd like to also have an endpoints that include any TFT on Rivine that are still eligible to be migrated to Stellar. This would be used, for example, to provide live supply figures to sites like Coinmarketcap.

There's already code in this repo to grab data from both Rivine and Stellar. What would be needed is to serve the output of certain calculations based on this data:

• Total TFT supply (total Stellar supply TFT and TFTA + total unmigrated Rivine TFT supply)
• Total liquid TFT supply (total supply - illiquid Stellar wallets - any locked Rivine TFT?)
• Total illiquid supply (total supply - liquid supply)

Update:

Per discussion with stakeholders today, we will cease conversion of Rivine TFT immediately and not count unmigrated tokens toward the supply.
Actually one thing remains to be done, and that's providing a simple endpoint that just adds TFT + TFTA on Stellar to give a total supply figure.

[sameh-farouk]

Update:
I started to work on this last Thursday and had this issue with installing Js-sdk but managed to sort it. see this comment.

Now I Have another error when I try to start the server. seems originated from an old configuration in my system. I tried to delete the sandbox dir, and also did threebot clean but the issue persists.

jumpscale.core.exceptions.exceptions.Runtime: Error happened during getting or installing auth package, the detailed error is [Errno 2] No such file or directory: '/home/sameh/tftshop/.venv/lib/python3.8/site-packages/jumpscale/packages/auth/package.toml'
Traceback (most recent call last):
  File "/home/sameh/.asdf/installs/python/3.8.5/lib/python3.8/site-packages/jumpscale/servers/threebot/threebot.py", line 872, in start
    self.packages.install(package)
    │    │                └ <jumpscale.servers.threebot.threebot.Package object at 0x7f7fb4218eb0>
    │    └ <property object at 0x7f7fb8b9e7c0>
    └ ThreebotServer(
        instance_name='default',
        _package_manager=StoredFactory(PackageManager),
        domain=None,
        email=None,
        a...
  File "/home/sameh/.asdf/installs/python/3.8.5/lib/python3.8/site-packages/jumpscale/servers/threebot/threebot.py", line 612, in install
    for static_dir in package.static_dirs:
                      │       └ <property object at 0x7f7fb8b9e220>
                      └ <jumpscale.servers.threebot.threebot.Package object at 0x7f7fb4218eb0>
  File "/home/sameh/.asdf/installs/python/3.8.5/lib/python3.8/site-packages/jumpscale/servers/threebot/threebot.py", line 315, in static_dirs
    return self.config.get("static_dirs", [])
           │    └ <property object at 0x7f7fb8b8e130>
           └ <jumpscale.servers.threebot.threebot.Package object at 0x7f7fb4218eb0>
  File "/home/sameh/.asdf/installs/python/3.8.5/lib/python3.8/site-packages/jumpscale/servers/threebot/threebot.py", line 288, in config
    self._config = self.load_config()
    │    │         │    └ <function Package.load_config at 0x7f7fb8b9d940>
    │    │         └ <jumpscale.servers.threebot.threebot.Package object at 0x7f7fb4218eb0>
    │    └ None
    └ <jumpscale.servers.threebot.threebot.Package object at 0x7f7fb4218eb0>
  File "/home/sameh/.asdf/installs/python/3.8.5/lib/python3.8/site-packages/jumpscale/servers/threebot/threebot.py", line 259, in load_config
    return toml.load(self.package_config_path)
           │    │    │    └ <property object at 0x7f7fb44ef4a0>
           │    │    └ <jumpscale.servers.threebot.threebot.Package object at 0x7f7fb4218eb0>
           │    └ <function load at 0x7f7fb5044280>
           └ <module 'toml' from '/home/sameh/.local/lib/python3.8/site-packages/toml/__init__.py'>
  File "/home/sameh/.local/lib/python3.8/site-packages/toml/decoder.py", line 133, in load
    with io.open(_getpath(f), encoding='utf-8') as ffile:
         │  │    │        └ '/home/sameh/tftshop/.venv/lib/python3.8/site-packages/jumpscale/packages/auth/package.toml'
         │  │    └ <function _getpath at 0x7f7fb502ee50>
         │  └ <built-in function open>
         └ <module 'io' from '/home/sameh/.asdf/installs/python/3.8.5/lib/python3.8/io.py'>

FileNotFoundError: [Errno 2] No such file or directory: '/home/sameh/tftshop/.venv/lib/python3.8/site-packages/jumpscale/packages/auth/package.toml'

Also, I want to mention that I don't have any prior experience with the Rivine project and I wasn't involved in setting up TFT on Stellar. So, some of the discussions might be a bit unclear for me. However, I'll come back to this once I've fixed my development environment.

[sameh-farouk]

Update:

• The previous issue was fixed. Now, I have the server and the package running locally.
• I am researching this issue's background to understand the migration process from TFT on Rivine to Stellar TFTA and later the conversion from TFTA to Stellar TFT. The information is scattered everywhere but I'm getting there.

In the meantime, can we get @robvanmieghem and @LeeSmet feedback on this issue to see if they can share any relevant information or opinions about this since they know the most about this area?

• Q: Did the decision to only provide supply information for stellar assets have a valid reason?
• Q: How can I check the total unmigrated TFT supply using Rivine Explorer? Can I verify it solely from the Rivine side, or do I need to cross-check with the Stellar side to confirm if the balance of an account on Rivine was migrated to Stellar?

[LeeSmet]

Q: Did the decision to only provide supply information for stellar assets have a valid reason?

To my knowledge, at the time this was created rivine was already deprecated (and the majority of tokens migrated), to the point where most unmigrated tokens can be considered "lost".

Q: How can I check the total unmigrated TFT supply using Rivine Explorer? Can I verify it solely from the Rivine side, or do I need to cross-check with the Stellar side to confirm if the balance of an account on Rivine was migrated to Stellar?

When we moved from rivine to stellar, all accounts on stellar were locked (except for the block creators but those don't really matter). The locking was done by a special "deauthorization" transaction on rivine. When tokens are/were migrated to stellar, the memo is set to type hash with the value the hash of the lock transaction on rivine, so the tokens which you have on stellar as result of a migration are linked to tokens on rivine (also, the stellar address can be verified to "match" the rivine address). Essentially this establishes a link from stellar to rivine so we can prove all tokens already existed, but this also means on rivine you can't see which tokens are migrated and which aren't.

In other words, to find out how much tokens have been migrated (and how many haven't), you'll need to list all transactions on stellar of both TFT and TFTA, and then filter based on transaction memo, where a migration tx is one which has a memo which also happens to be a rivine deauth tx ID. For the unmigtrated TFT, you'll need to figure out the total rivine TFT supply and subtract the migrated amount.

FWIW: I have some code on my personal github which finds all threefold mints across both networks (mint in this context being the result of minting code). It also checks if tokens are the result of a migration to find out if they should be counted or not. This might give some hints about what you want to achieve. https://github.com/LeeSmet/find_threefold_mints

[robvanmieghem]

https://github.com/threefoldfoundation/tft-stellar/tree/master/scripts/analysis has the scripts to put all this information in an sqlite database. The Stellar subfolder indexes all Stellar transactions as well.
https://github.com/threefoldfoundation/tft-stellar/tree/master/scripts/conversion has scripts that can help in the conversion investigation.

[sameh-farouk]

Update:
I back with an interesting finding, and a new question as well ..

Token burns should involve removing assets from circulation and It appears that we are not taking burned tokens on TFchain into consideration yet. As a result TFT asset on stellar is currently (inflated?).
For example, If I possess a certain amount of TFT on the Stellar network and bridge them to TFchain, then proceed to burn them on TFchain (simply because it is allowed), this action should be reflected in the current supply by burning the same quantity of tokens on Stellar.

Burning tokens isn’t solely voluntary; we actively burn 35% of tokens from billed contracts every 24 hours due to utilization. This deliberate reduction in token supply should, if correctly mirrored on Stellar, contribute to an increase in its value.

Now, could you shed some light on which specific wallets should considered illiquid supply?

[scottyeager]

Token burns should involve removing assets from circulation and It appears that we are not taking burned tokens on TFchain into consideration yet. As a result TFT asset on stellar is currently (inflated?).

This is a good point. Seems it was never implemented.

If we can query the total supply of TFT on both mainnet and testnet TF Chains, then compare them to the relevant Stellar vault addresses for each bridge, the discrepancy should be visible.

Now, could you shed some light on which specific wallets should considered illiquid supply?

As far as I know, the only illiquid tokens in the system right now are illiquid foundation wallets. Details about these wallets appear to be supplied through a config file that's not in the repo, but the info can be read from here.

Tokens from the vesting program are now eligible to be unlocked immediately, so I don't think they should be considered illiquid, even if they are still vested.

The remaining possibility that I'm aware of is that some TFT buyers had an arrangement where their purchase unlocked in chunks monthly. I don't know if any tokens covered by such arrangements are still locked.

I think only considering the foundation wallets when considering liquid vs illiquid is good enough.

[sameh-farouk]

FWIW: I have some code on my personal github which finds all threefold mints across both networks (mint in this context being the result of minting code). It also checks if tokens are the result of a migration to find out if they should be counted or not. This might give some hints about what you want to achieve. https://github.com/LeeSmet/find_threefold_mints

I had no luck with testing this code.

❯ go run .
go: downloading github.com/threefoldfoundation/tfchain v1.3.1-0.20200701080725-082a1bffda0f
go: downloading github.com/stellar/go v0.0.0-20211118170550-db1260757b4e
go: downloading github.com/threefoldtech/rivine v1.3.1
go: downloading github.com/threefoldtech/rivine-extension-erc20 v0.0.0-20200909163651-5ade1b4e7d45
go: downloading github.com/bgentry/speakeasy v0.1.0
go: downloading github.com/julienschmidt/httprouter v1.2.0
go: downloading github.com/spf13/pflag v1.0.3
go: downloading github.com/spf13/cobra v0.0.3
go: downloading github.com/rivine/bbolt v1.3.1-coreos.6.0.20180406082335-19c3af6fd3ce
go: downloading github.com/NebulousLabs/fastrand v0.0.0-20181203155948-6fb6489aac4e
go: downloading github.com/NebulousLabs/merkletree v0.0.0-20181203152040-08d5d54b07f5
go: downloading golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97
go: downloading github.com/ethereum/go-ethereum v1.10.12
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go: downloading golang.org/x/sys v0.0.0-20210816183151-1e6c022a8912
Processing block 699900
panic: interface conversion: error is *url.Error, not *horizonclient.Error

goroutine 1 [running]:
main.findAccPayments({0x770e93, 0x38})
	/home/sameh/projects/find_threefold_mints/main.go:244 +0x3d4
main.main()
	/home/sameh/projects/find_threefold_mints/main.go:142 +0x9b
exit status 2

[sameh-farouk]

To my knowledge, at the time this was created rivine was already deprecated (and the majority of tokens migrated), to the point where most unmigrated tokens can be considered "lost".

@LeeSmet @robvanmieghem Is it relevant to do this now. also, was there a planned/announced deadline for token migration? Will we need to monitor rivine unmigrated tokens indefinitely?

[scottyeager]

@LeeSmet Is it relevant to do this now. also, was there a planned/announced deadline for token migration? Will we need to monitor rivine unmigrated tokens indefinitely?

We still get occasional requests to migrate Rivine tokens via our support channels. I do think it's relevant to at least get a sense for how many tokens are left there and eligible to be migrated (would also be a natural question in any discussions around cutting off migrations).

I had no luck with testing this code.

I think all the same results can be obtained using the code in this repo under scripts. If I'm reading correctly, the necessary calls are like this:

# These steps are referencing now static Rivine data and thus only needs to run once
cd conversions
./lockedaddresses.py
./tfchainbalances.py
cd ../analysis
./rivineclosing.py

# These steps pull data from Stellar and would run periodically
./issuertxs.py
migrationaddresses.py

Once this is done, the result is a SQLite database with all necessary data to compute the desired figures.

I am testing it personally, and so far I've only managed to process about 30k of 708k blocks from Rivine via the lockedaddresses.py script, running for the last couple days now whenever my laptop was on for other reasons. Seems to be working one at a time in a serial fashion, so I guess this could be sped up with some threads.

[sameh-farouk]

In other words, to find out how much tokens have been migrated (and how many haven't), you'll need to list all transactions on stellar of both TFT and TFTA, and then filter based on transaction memo, where a migration tx is one which has a memo which also happens to be a rivine deauth tx ID. For the unmigtrated TFT, you'll need to figure out the total rivine TFT supply and subtract the migrated amount.

So to avoid doing redundant work , we can follow these steps:

1. Initially, we can find and store the deauth tx IDs in a database only once since this data won't change.
2. Start with all issuance balances on Rivine as unmigrated balances.
3. For TFT and TFTa, we can find all transactions on stellar issuer accounts with a memo that matches any of these deauth tx ID on the database.
   3.1. For every matched memo, we can remove the deauth tx IDs from the database and decrease the unmigrated balance with the same amount as this Rivine account balance.
   3.2. We should also store the TFT and TFTa issuer stellar sequence where we stopped.
4. Periodically check any new transactions on Stellar TFTA and TFT issuer accounts and repeat step 2 every xx hours.

Sounds good ?

[sameh-farouk]

Update:
After testing Rob scripts, I have the DB file ready, but the numbers appear incorrect.

Total issuance on Rivine

sqlite> select sum(locked + free) from Rivine;
611039330.762298

Migrated amount

sqlite> select sum(locked + free) from rivine where stellaraddress NOT NULL;
572134381.786762

Unmigrated amount

sqlite> select sum(locked + free) from rivine where stellaraddress IS NULL;
38904948.9755365

The minted TFT(A) amount on Stellar that has a lock transaction as a memo

sqlite> select sum(amount) from minted inner join rivine on minted.memo = rivine.locktransaction;
764641916.845019

I'll have to rerun everything and double-check for issues because the later number appears to be significantly larger than the total issuance on Rivine, even though it should be equal to the migrated amount.

Worth mentioning that I had a lot of these messages while running issuertxs.py

3066f15facf63818a82e051b6ce1026d54dc3c5f12c2eb4ed817f155c041da3a has no memo
Unable to parse transaction 723be93ca9c9fb2b5569aa7588b20483238f07d5db6f50bc0320172f0f740d90
Unable to parse transaction af4647db9b89ac0e9a87eddb9af016fb43e36ccfcedf3416706867873454ecae
Unable to parse transaction 62c4d0806df41cf6dddb5cf9e7df650f05d61ae44a2d22c777e449977c34c442
Unable to parse transaction f837195684a4ac87449dce5bd3d7f85f155600d436ff30551915f1c0faa941c2
Unable to parse transaction 43c608a349282d9a0aaf25d56e56027fcf52ed8cf5f13f84c9d1e742a6f6ee4f
Unable to parse transaction cca550fccc38ccd300218d91acaa7352f56e458e9300c3d7b0ba2fe1e83d60b1
Unable to parse transaction da94986b388bf3bc63d35cadef0d05c6f5160f5534eb899a9eaf5296c627994f
Unable to parse transaction 3eb951f8c30669b6df70e898ba298f9c78a1270889949d5ead2066f160614118
Unable to parse transaction 440e0d78a5ee43e17bb64ab57ec82829a464464cfe0e2daf9e05066f6286a877
Unable to parse transaction 2669020cee242288ffdd4094231cb789455dcfa41af6efadfe49577999fb6d35
...

Did you have better luck? @scottyeager

[scottyeager]

Sounds good ?

I think overall we have the same idea. My thinking would be to just continue matching Stellar mints that occur from Rivine migrations with their original Rivine accounts, rather than removing any items from the database. But overall the outcome is the same.

Worth mentioning that I had a lot of these messages while running issuertxs.py

I saw some of these too and spotted checked one. It seemed to be a Stellar transaction totally unrelated to TFT. Not sure why that's happening, but at least it doesn't seem to be a failure to parse any actually relevant transactions.

Did you have better luck? @scottyeager

I didn't manage to finish running lockedaddresses.py. It got stuck, probably when I closed the lid on my laptop. I'm going to try running it again in an always on system. Will report back when I have some data to compare.

[sameh-farouk]

I'm trying to make sense of the available data. But I keep get weird results.
Here is one example.

In the Rivine table, there’s an account with the identifier 033e7e20e80ceefb64c37c5f5453429aa450e9fea422888adcaf069a94d83b58dcc2bf89eb80f1. This account was deauthorized with tx 9e26ad97eee5fd308b9a4ef3d5124b57e80e4b348d27af34b5fcf94d32073d35 and had a balance of 0.0 tokens at the time it was locked. unexpectedly, it appears to be associated with a Stellar address, suggesting that these 0 tokens were somehow migrated to Stellar.

Upon investigating the lock transaction in the minted table, I discovered two transactions with the same memo 9e26ad97eee5fd308b9a4ef3d5124b57e80e4b348d27af34b5fcf94d32073d35:

Transaction 1:
Stellar TX hash: 6e7de7948873dde2b602dd3b36304d89daff61c1e75263b8e8c4233f35e128ad
Amount: 59,135,910.77 TFT
Destination: GDGDAGHYNMTXQLMYHAEA4WP6QMI4YV7JG4LAI7ATENEJKW7GXWA5MOOI
Transaction 2:
Stellar TX hash: 545ff3667ce266ce7ec3f994a250d5055be9766f87a60fd7faa38a4012a0e7fc
Amount: 110,000,000 TFT
Destination: GBJM6LV6B4ZEXZKS4WXYYJ4VNWLETHK2DLUGCHADPAOZGH4VGMCJ7SVX

Here’s where it gets intriguing: both transactions share the same memo but have different destinations. The total amount transferred to Stellar is 169,135,910.77 TFT. However, the issuance amount on Stellar doesn’t match the account balance in Rivine at the time of closing.

Now, the questions arise:

Why was the same Rivine lock transaction used as the memo to issue TFT for different addresses on Stellar?
Why does the issuance amount on Stellar differ from the account balance in Rivine during the time of closing?

These discrepancies warrant further investigation. Perhaps there’s a hidden layer of complexity not clear to me or an unexpected twist in the migration process.

[robvanmieghem]

What makes things even worse is that there was a bug in the original migration service when converting an unspent output where the input was not completely sent to the deauthorized address and multiple outputs were locked. No, I did not create the bug but noticed and fixed it in ca255dd

[scottyeager]

Why was the same Rivine lock transaction used as the memo to issue TFT for different addresses on Stellar?
Why does the issuance amount on Stellar differ from the account balance in Rivine during the time of closing?

For this particular instance, I took a look manually in the Rivine explorer (here's a link to the specific page).

Seems there were two amounts, locked and unlocked, which correspond to the two amounts you observed migrated to Stellar. I think the only question here is why weren't those balances included in the database.

[sameh-farouk]

Seems there were two amounts, locked and unlocked, which correspond to the two amounts you observed migrated to Stellar.

But Shouldn't both go to the same address? I read that locked token was migrated as TFTA and unlocked tokens were migrated as TFT. I see both were issues TFT for different addresses. Maybe a Normal account and a vesting account?

[sameh-farouk]

Also from what I read Migration service was later discontinued, and migration was processed manually through support channels.

Did we continue to follow the same migration logic and process as before?

[sameh-farouk]

Why was the same Rivine lock transaction used as the memo to issue TFT for different addresses on Stellar?
Why does the issuance amount on Stellar differ from the account balance in Rivine during the time of closing?

For this particular instance, I took a look manually in the Rivine explorer (here's a link to the specific page).

Seems there were two amounts, locked and unlocked, which correspond to the two amounts you observed migrated to Stellar. I think the only question here is why weren't those balances included in the database.

After reviewing the code, I observed that the Rivine explorer client handles balance queries exclusively for regular wallets. Unfortunately, querying balances for multi-sig wallets has not been implemented (the code returns just an empty balance). Consequently, the balance for this wallet (and any multisig wallets) becomes zero in the database.

tft-stellar/lib/tfchainexplorer.py

Lines 141 to 142 in a1c6745

	if self._unlockhash.type == UnlockHashType.MULTI_SIG:
	balance = WalletBalance() # ignore for now

[sameh-farouk]

I have scanned all the wallets and found that only a few of them are multi-sig wallets.

• 033e7e20e80ceefb64c37c5f5453429aa450e9fea422888adcaf069a94d83b58dcc2bf89eb80f1
• 030bdbd813dd9cc95efdced64af5747eb5f1cdd0f1d0f251af151692c8b5bfccbab2ea0b31150f

I can correct these entries manually in the database (since these data are static anyway).

[sameh-farouk]

Update:
I fixed the balances of the multi-sig wallets and calculated the total number of tokens that were migrated to Stellar and the total number of tokens that were issued on Rivine using a modified check script (I can push it if needed).

The analysis suggests that the number of tokens migrated to Stellar exceeds the number of tokens initially issued on Rivine with 1,725,373,122.3610737 TFT.
@robvanmieghem @LeeSmet @xmonader Can you advise where to go from here? Otherwise, I will set the issue status to blocked as there is no reason to implement this endpoint given that it won't serve reasonable/expected numbers anyway.

Update 2:
Since the numbers were too big, I double-checked everything, and found a bug in the counter.
Now, I am getting reasonable numbers!

764,831,278.8226932 TFT deauthorized balance done (total supply on Rivine before locking)
281,708,637.9640675 TFT was migrated (issued on Stellar)
483,122,640.8586257 TFT Remains unmigrated (deauthorized balance - migrated)

./check.py
/home/sameh/.asdf/lib/utils.bash: line 242: ASDF_.._VERSION: invalid variable name
1609 tfchain addresses are deauthorized
520 had 0 TFT and do not have to be migrated
total issued amount: 281708637.9640675 TFT
651 conversions done, 438 left
764831278.8226932 TFT deauthorized balance done, 281708637.9640675 TFT was migrated, 483122640.8586257 TFT Remains unmigrated
646 correct conversions
5 incorrect conversions:
deauthtx: 0db78d94f610c40c1a59b14ff23080b4262fe74eb8782476836aaa30d63af6bf tfchainaddress: 01b8d7b62281b65ea27d7f3a0cf9f3c68aa1a7a3cd9f51e5552a1df982cedd2e315518d6964e93 difference: 562500.0000000
deauthtx: eba019891ac41bc442c20b71d4f316231c0b196cff4535a010908d7c066afbd1 tfchainaddress: 01f58b2e91ebf19ef6f70d9fa787a470b3ba022c99cb1262f9fcb12e47f1e454f6043bfbf54ada difference: -459970.8333326
deauthtx: 59f80483eb570a0b8a094795d9cdf43ecadb702bded742c9cda909007c2e5214 tfchainaddress: 017b69b7468896cf0c62fb7d84149071db02036e51a752074a91aa61db25a9f293a0e5560d3f62 difference: -120789.3888898
deauthtx: 0f01f12381a669ef5bbac0966e1e8bf8505181f776cc6d88c093ce3aeb7bc67a tfchainaddress: 015fe50b9c596d8717e5e7ba79d5a7c9c8b82b1427a04d5c0771268197c90e99dccbcdf0ba9c90 difference: -128693.4156381
deauthtx: 3934a86918077bedfee3d6b90bdd2de752a409f24c5cc0b76d89c8dcc8164dc1 tfchainaddress: 01ecd6be4a9d4eb9052d6cbb2c0869622edab5e5b9622da2eb4fbf79d68431d9e8a1ed7d21a6f3 difference: -42405.3497939

[LeeSmet]

These numbers don't seem quite accurate to me, specifically the migrated tokens seem to be on the low side. From stellar.expert, we can see that at the time of writing there are 53.3M TFTA and 885.9M TFT, totaling roughly 939.3M TFT on stellar. If the provided numbers are correct and only 281.7M TFT have been converted, that would mean some 657.6M TFT have been minted on stellar since we started minting there in may 2020 with the start of V2 minting (minting for month of may -> payout actually happened in june). So that would mean 657.6M TFT / 44 months which gives an average of 15M TFT minted per month. From doing the actual minting, I know for a fact that this number is way too high. The average minted TFT per month is in the range of slightly above 3M per month, spiking to 5M with v3 for some time before now dropping back to slightly above 3M.

So I've updated my code to also print out all migrations (and conversions for TFTA->TFT though that is not really relevant right now). This gives me these numbers:

The total MINTED (as result of proof of capacity) TFT amount on both chains is 287M. The genesis TFT, which can be found in the tfchain code is slightly above 695M. If we only account for rivine based mints, the total TFT on rivine is 808M (including the genesis distribution). This means that minting on stellar accounts for 174.4M. Isolating the migrations (i.e. transactions issued by TFTA or TFT issuer with a memo hash which can be found as a deauthorization hash on rivine), shows a total of 764.9M TFT have been migrated. If we add the amount of TFT and TFTA on stellar as mentioned previously, we get 939,301,371 TFT. If we also add the numbers I pulled (migrated + minted on stellar), we get 939,320,654 TFT. I didn't sort out pure burns, but these numbers match except for some 19,283 TFT, which could be the result of a burn on stellar.

Considering the above, the remaining rivine TFT to be migrated would be 807,961,249 TFT issued - 764,897,352 TFT known to be migrated which is just north of 43M TFT left.

I haven't manually validated these numbers, however from running the minting the numbers presented do seem to line up (for the total issuance). Also the close match of stellar TFT+TFTA compared to the computed numbers do inspire some confidence.

Note, on rivine tfchain, there was a block creator reward of 1 TFT. I haven't included this (roughly 700K tft), since it was not the result of proof of capacity minting ,and it should never get migrated as to my knowledge, these accounts were never deauthorized.

[sameh-farouk]

Thanks, Lee, for the update. You are correct.
I just finished reviewing the code again and found another glitch.

Here are the final numbers I got after the fix.

./check.py
/home/sameh/.asdf/lib/utils.bash: line 242: ASDF_.._VERSION: invalid variable name
1609 tfchain addresses are deauthorized
520 had 0 TFT and do not have to be migrated
total issued amount: 1223396422.3971133 TFT
651 conversions done, 438 left
764831278.8226932 TFT deauthorized balance done, 1223396422.3971133 TFT was issued on stellar (any tx with hash memo), 764641919.8350195 TFT was migrated tp stellar, 189358.9876737 TFT Remains unmigrated
646 correct conversions
5 incorrect conversions:
deauthtx: 0db78d94f610c40c1a59b14ff23080b4262fe74eb8782476836aaa30d63af6bf tfchainaddress: 01b8d7b62281b65ea27d7f3a0cf9f3c68aa1a7a3cd9f51e5552a1df982cedd2e315518d6964e93 difference: 562500.0000000
deauthtx: eba019891ac41bc442c20b71d4f316231c0b196cff4535a010908d7c066afbd1 tfchainaddress: 01f58b2e91ebf19ef6f70d9fa787a470b3ba022c99cb1262f9fcb12e47f1e454f6043bfbf54ada difference: -459970.8333326
deauthtx: 59f80483eb570a0b8a094795d9cdf43ecadb702bded742c9cda909007c2e5214 tfchainaddress: 017b69b7468896cf0c62fb7d84149071db02036e51a752074a91aa61db25a9f293a0e5560d3f62 difference: -120789.3888898
deauthtx: 0f01f12381a669ef5bbac0966e1e8bf8505181f776cc6d88c093ce3aeb7bc67a tfchainaddress: 015fe50b9c596d8717e5e7ba79d5a7c9c8b82b1427a04d5c0771268197c90e99dccbcdf0ba9c90 difference: -128693.4156381
deauthtx: 3934a86918077bedfee3d6b90bdd2de752a409f24c5cc0b76d89c8dcc8164dc1 tfchainaddress: 01ecd6be4a9d4eb9052d6cbb2c0869622edab5e5b9622da2eb4fbf79d68431d9e8a1ed7d21a6f3 difference: -42405.3497939

The important part is here

764831278.8226932 TFT deauthorized balance done, 1223396422.3971133 TFT was issued on stellar (any tx with hash memo), 764641919.8350195 TFT was migrated tp stellar, 189358.9876737 TFT Remains unmigrated

so only 189,358.9876737 TFT Remains unmigrated!

But per your analysis :

the total TFT on rivine is 808M

Can we discuss the differences between the total issued amount on Rivine in your analysis and the results I obtained from modified Rob's script?
Total unauthorized balances on Rivine, based on my analysis, are 764,831278.8226932. Does this number seem incorrect, or is the difference due to burned balances that may have occurred on Rivine?

[sameh-farouk]

You can try check.py script here in my branch to see how I get these results (no magic I only sum the numbers collected by rob's scripts)
https://github.com/threefoldfoundation/tft-stellar/tree/master_update_conversion_scripts/scripts/conversion

[LeeSmet]

So here are some points to consider:

• The amount of TFT on rivine which I calculated is the result of genesis TFT + all TFT minted on tfchain (in what is known as a coincreation transaction). The code in this repo seems to first creates a list of all deauthorized transactions, and then looks up the balance on the accounts deauthorized by those transactions. So if we assume neither codebase has a bug, the obvious answer would be that not all accounts which have TFT have been deauthorized. It would be interesting to verify this.
• My code currently counts 652 conversions, 1 more than the code here (counted the unique transaction memos which are also a deauthorization hash). The conversions happened in 2353 stellar transactions.
• 1223396422.3971133 TFT was issued on stellar (any tx with hash memo): I don't really see where that originates from in the source, but according to the comment in parenthesis this includes all transactions with a hash memo. So this means that mints are also counted, and conversions from TFTA to TFT (since the TFT issuance has a memo hash which is the tx ID of the TFTA burn). My code suggests 279M TFTA has been converted to TFT which is effectively double counted then. If this value is detracted from the given number, 944M remains. That looks close to the 939M TFT currently available on stellar

I'll see if I can modify my code to also sort out the voluntary burns of TFT (on rivine and stellar TFT) if there are any.

If that doesn't work out, a list of all TFT addresses on rivine which have tokens might be needed, so we can see if indeed all have been deauthorized (which I suspect didn't happen at this point)

[LeeSmet]

As it stands I count 1604 address on tfchain which have ever appeared in a coinoutput (i.e. received tokens). So that suggests all of them got deauthorized.

Other than that there are some burns on TFT on stellar it seems (didn't check rivine as it seems the coindestructionTX never got registered there). There are roughly 2K burns, mostly for very small amounts (though some slightly larger). One which stands out is a burn of 5M tft. However these 5M tft were also credited from a non mint or migration tx. While this particular case does not really matter, it does invalidate the assumption that all TFT on stellar are the result of minting or conversion. https://stellar.expert/explorer/public/account/GBBYLWMOZOGJKV6XRLOVILJMXTT2MPA3GD5FAJJ2CK6N3OOKY3SXQCUO?asset[]=TFT-GBOVQKJYHXRR3DX6NOX2RRYFRCUMSADGDESTDNBDS6CDVLGVESRTAC47-1&cursor=151683972719280128&sort=id&type[]=payments

[scottyeager]

One which stands out is a burn of 5M tft.

@robvanmieghem mentions these TFT here. Something to do with initial setup of TFT on Stellar, it seems.

[scottyeager]

Per discussion with stakeholders today, we will cease conversion of Rivine TFT immediately and not count unmigrated tokens toward the supply.

Thanks for the work on this one @sameh-farouk and @LeeSmet.

[scottyeager]

Actually one thing remains to be done, and that's providing a simple endpoint that just adds TFT + TFTA on Stellar to give a total supply figure.

[sameh-farouk]

Update:
Pending deployment

https://github.com/threefoldtech/tf_operations/issues/2406

[sameh-farouk]

Update:
It's Deployed now.
Please feel free to verify and close the issue @scottyeager

[scottyeager]

Looks good! Thanks @sameh-farouk.