Add support for scanning additional token data

[taras-hk]

Add support for scanning token balances (and whatnot) for given addresses, supporting ERC-20 token contracts, and DAO membership tokens (might also be ERC20, or possibly NFTs or whatever).
Additional data to get to be defined (should start with token balances).

1. Token balance
2. Transaction metadata to understand where tokens go (a common transfer or purchasing something in particular)

[databu]

Please coordinate with @LeahN67 about required data points (to update "whatnot" above 🙂).

[taras-hk]

Please coordinate with @LeahN67 about required data points (to update "whatnot" above slightly_smiling_face).

Today we had a meeting with @LeahN67 regarding the metadata to get to proceed with that task.

[LeahN67]

@tarasherasymchuk , when can I schedule a meeting?Kindly communicate your availability. Thank you.

[LeahN67]

@tarasherasymchuk As described in the call, the data we are looking for would ideally be beyond the transactional data available on etherscan. Because we are trying to build a recommendation engine specifically tailored for Ocean, we need to ingest data points that would be relevant in providing a recommendation to Ocean Protocol users. But of course the transaction data found on etherscan is also highly useful as it helps us build a network of users and exchanges that can help map the Ocean network.

Further Data points we would like to gather:

1. Wallet addresses and what datasets have been purchased by these wallets
2. Wallet addresses that are providing liquidity on dataset pools
3. Wallet addresses that are staking on datasets in the Ocean market

Please let me know if you would like me to clarify anything and thanks again for your assistance. It is very much appreciated.

Let's talk tomorrow.

[databu]

Copying some Slack discussions around the data discussed in Leah's last comment here:

From me (slightly edited/re-ordered for clarity):

1. is already available in the graph:
   cypher (a:Address)-[:Consumption]-(d:Datatoken)-[:belongsTo]-(s:Asset)
   matches all addresses a which consumed asset s.

2. also:
   cypher (a:Address)-[j:join:exit]-(p:Pool)-[:belongsTo]-(s:Asset)
   where j is the join or exit relationship between a and p with the greatest Timestamp and PoolSharesBalance > 0

The core API provides an interactionSummary endpoint to get some of the data for 1. and 2. for the source user’s network.

See https://utu-apis-swagger.s3.eu-central-1.amazonaws.com/trustApi.html#get-/interactionSummary

But for your purposes more relevant might be the source code in utu-trust-api-core/src/graph/interactionSummary.ts

Look at getNetworkConsumers() and getNetworkLiquidityProviders()

3. What do you mean by staking in 3? I wasn’t aware that there’s staking on datasets in Ocean, at least in v3? Is this in Ocean v4?

Fom Rahul:

Hey Bastian, I suppose you have been talking to Leah about Ocean. But, I think I wrongly read that Ocean has a staking feature based on the documentation. But looking into it again, Ocean staking is just liquidity provision for ocean data pools. So the data point requested doesn't apply. Just wanted to clarify

Yes that clarifies it. So basically point 3. is covered by 2. Which means that the existing Ocean scanner covers all of this already.

So now I am thinking, we can also integrate transactional level data of Ocean tokens directly from Etherscan to better map ocean token holders and exchanges used by wallet holders. I believe these can be good signals to show a ocean protocol user. Yet I am unclear how this would fit into a collaborative filtering approach

As mentioned above, the scanner already scans user transactions with Ocean data tokens and pools and adds those tx as interaction-type relationships (with different action properties identifying the nature of the interaction; see interactionSummary source code linked above). It gets this from the Ocean sub graph API, which already servers all on-chain data. So there wouldn't be anything additional available from etherscan.

As for the collaborative filtering, I had outlined my idea for it in my comment on #6. The only data required for that which is not covered by the scanner yet is which tokens are held by a user (as already explained there).

[taras-hk]

@LeahN67 @rahulsrivatsa
Following our call yesterday, we agreed to gather additional data from datasets like tags and categories that can be used for data categorization.
That's a sample data from the scanner gathered to assets.json:

	{
		"name": "Internet Service Provider (ISP) Global Report",
		"description": "Report is generated on __all types of internet registries on each region__ on both versions **IPV4 and IPV6.**\n______________________\n\n**Value:** Very accurate and precise data on deployed amount of the allocation of internet service networks.\n_________\n**This data can help by judging based on which country has a good potential on where each service provider can expand to.**\nIt provides reflection on the scope of use of deployment of each associated resource. \n**Internet Service Providers (ISP) can benefit from this data by observing which countries/regions are lacking on reception and receiving below-average service.**\nBoth Internet Protocol Versions' data has been collected using ISO-3166 protocol, implying collection from geolocations of main interest.\n________\n__Disclaimer:__\n\nMany Internet service networks have international components, and their numbering resources are often deployed across multiple economies. \nFor this reason, the report presented here may not 100% accurately reflect the actual number resource distributions across these economies.\n___________\n__Database consists of:__\n\nFile 1: \n\nInternet Protocol Version 6 (IPV6) Deployment:\nUsing The help of ISO-3166 Code consists of \n1. The Total estimated number of internet Users in the whole country. \n2. How much ratio of people using the internet.\n3. Total estimated number of population\n4. ISO Code and country are both provided \n\nFile 2 + File 3: \n\nDeployment for each version of Internet Protocol amongst different calculations. All the assigned will be sorted highest to lower number of allocations.\n1. Assigned 64 bits to a single user to interface in total country to country.\n2. internet protocol space to that country.\n3. Regions involved in the recorded space.\n4. Calculated GDP per 64s deployed.\n5. Advertised and non-advertised prefixes.\n6. IPV6 users \n7. Population at the time recorded.\n8. GDP per capita and GDP recorded.\n\nFile 4: \n\nThis is an iso 3166 reference file for each country with an index number also included with a total GDP on affordability of internet use.\n1. Reference file of all the ISO numbers with the index of all countries with all calculated GDP based on the data.\n2. The more the GDP the more affordability to get access to the internet\n3. The less the GDP the less affordability to get access to the internet \n\nFile 5 + File 6: 2 registry files for IPV4 and IPV6\n\n1. Registry of all Internet protocols (AFRINIC,APNIC,ARIN,RIPENCC,LACNIC AND RESERVER OF IANA)\n2. Advertised autonomous system number on the internet protocol \n3. Unadvertised autonomous system number on the internet protocol\n4. Number of allocations of users by their country codes.\n5. Number of reserved IPs are available. \n6. Number of unreserved IPs are available.\n7. Number of allocations to to the users.\n8. Total number of allocations for IPV6 and IPV6, respectively.\n\nFile 7 + File 8: World region by region allocations of IPs (IPV4 and IPV6)\n1. Total data collected on region by region.\n2. Allocation of each version by file based on the region \n3. GDP calculations based on the region of the world.  \n_________\nCredits:\n1. I would like to extend my gratitude for the opportunity that Apnic Labs has provided the team with funds and resources for aggregation of this dataset.\n2. This data is estimated and accurately calculated as of 25th March 2020. \n3. Data will be updated on 25th March 2021.",
		"did": "did:op:5389Fb3f227c70aa8d292aFf54987f315afBC6bf",
		"pool": [
			{
				"address": "0x932e209a03fdd574a249b98251d8085e46298605",
				"controller": "0x2a585b6a2b51793c006cd53fec5a76a0e6a6293f",
				"total_swap_volume": 105.03661146680368,
				"ocean_reserve": 8.378504442508088,
				"datatoken_reserve": 96.91690367450946
			}
		],
		"datatoken": {
			"address": "0x5389fb3f227c70aa8d292aff54987f315afbc6bf",
			"name": "Inspired Dolphin Token",
			"symbol": "INSDOL-41",
			"order_count": 0,
			"publisher": "0x2a585b6a2b51793c006cd53fec5a76a0e6a6293f"
		},
		"published_by": "Asia Pacific Network Information Center",
		"published_by_address": "0x2a585b6a2b51793c006cd53fec5a76a0e6a6293f",
		"purgatory": true,
		"consumed": 0,
		"categories": [
			"isp",
			"internet-protocol-version",
			"internet",
			"network",
			"networking",
			"global",
			"report",
			"gdp",
			"allocations",
			"countries",
			"regions",
			"isp",
			"ip",
			"population",
			"demographics",
			"wi-fi",
			"analysis"
		]
	}

I scanned 387 items and none of them has tags but categories are available for every one of them though these two properties are optional.

Also, we discussed the possibility how stacking and one of the options is to filter by the transaction event which can have one of the next values:

• join
• swap
• exit

Going to investigate further in this direction.

[LeahN67]

Thank you @tarasherasymchuk ..Is it possible you include tags?

[databu]

Hi @tarasherasymchuk @LeahN67

Also, we discussed the possibility how stacking and one of the options is to filter by the transaction event which can have one of the next values:

Can you elaborate a bit more on that? Currently, the Ocean scanner already posts relationships of type interaction to the Trust API, having the nature of the integration as an action property. For join and exit, it also includes the resulting balance under PoolSharesBalance. E.g.

"properties": { 
  "action": "join", 
  "PoolSharesBalance": 10,
  <other properties...>
}

Let's make sure to not duplicate work, and only add what's needed and not already there.

[taras-hk]

Hello @databu
Thanks for pointing out this.
Basically, we have been discussing exactly what you mentioned.
Ok, as long as it's already available we can skip this and just add tags and caegories.

[taras-hk]

@LeahN67 we query tags too. However, datasets in this example just do not have them as this property is optional.

[taras-hk]

@LeahN67 Just noticed that it's categories property that is not available and tags is available, I must have mapped these properties incorrectly.
A sample response from assets.json:

	{
		"name": "📈Oceancap - Datapool Evaluation and Charting",
		"description": "With Oceancap we are able to track Ocean Market Movement of all Datapools. We want to create an interface, where every user can see the price movement, charting and liquidity change of each Datapool on the official Ocean Marketplace. Oceancap is a dynamic database, which will update daily.\n\nIMPORTANT NOTICE: Our Main focus is Oceancap. We have planned many more features, functions, and KPI’s to improve our set. This is an ALL-IN-ONE solution, you don’t have to buy any addons.\n\n### Who we are:\n\nWe are a team of developers and data analysts. We are fascinated about the technical solutions made of the blockchain principle in connection with demands and the challenges in the real world and the future. Our vision is to visualize the marketplace of ocean protocol, for everyone who is even fascinated as we are. To ensure this transparency we deliver updated pool figures every day and for every pool.\n\n### What you get:\n\nDaily updated record sets of the following key performance indicators (KPI):\n\nPool Name Author Data created DID Owner (ETH-Adress) Published Use VALUE (Ocean) Use VALUE (USD) POOL Statistics Liquidity (Ocean) POOL Statistics Liquidity (USD) Rate Ocean/USD Timestamp\n\n### What are your benefits\n\n- Oceancap is an ALL-IN-ONE solution, you will never have to buy any addons! Fast evaluation of pool-trends Minimize your own risk Evaluate possible scams Track pool creators liquidity Avoid scams by comparing pools and statistics\n\n- Liquidity withdraw schedule: \n-- 3% Liquidity withdrawal on release 16.11.20 to compensate development cost \n-- 1% Liquidity withdrawal every Sunday which equals pledged status!\n\n### What are our next Steps: \n- Live charting \n- Pool evaluation with new key matrix \n- Datatoken statistics - \n- User friendly interface\n\nIf you have any questions please contact us on Twitter: \n[Follow us on twitter](https://twitter.com/OCharting)\n\nWe encourage you to test our set and give us feedback.\n\nSpecial thanks to Robin for your support!\n\nYour Oceancap Team",
		"did": "did:op:C1e2dcCC25ed82AcF79e233780c0f613B1229F82",
		"pool": [
			{
				"address": "0x58d65bb61bdd4df764a4fba87d23d628a8a79b89",
				"controller": "0x31a7f70b8812daef0a6c7f19575db09f726f0dd1",
				"total_swap_volume": 52307.28783750857,
				"ocean_reserve": 1323.1941466292476,
				"datatoken_reserve": 659.301581043684
			}
		],
		"datatoken": {
			"address": "0xc1e2dccc25ed82acf79e233780c0f613b1229f82",
			"name": "Adamant Starfish Token",
			"symbol": "ADASTA-60",
			"order_count": 17,
			"publisher": "0x31a7f70b8812daef0a6c7f19575db09f726f0dd1"
		},
		"published_by": "Oceancap",
		"published_by_address": "0x31a7f70b8812daef0a6c7f19575db09f726f0dd1",
		"purgatory": false,
		"consumed": 17,
		"tags": [
			"analytics",
			"metadata",
			"charting",
			"evaluation",
			"trends",
			"analysis",
			"finance",
			"big-data"
		]
	},