Propose Advancing Web3.js Plugin for Polkadot

[Muhammad-Altabba]

Project Abstract

Advancing Web3.js Plugin for Polkadot

Propose to work more on the npm package: web3-plugin-polkadot and its website www.web3polka.com.
The project was initially developed for the Polkadot Hackathon Winter 2023. It passed the technical review and made it to the final stage. But did not get selected at the demo day that takes business into perspective.

In this grant request, I propose working on supporting subscribing/unsubscribing to blockchain events as the next functionality to be added.

Grant level

• Level 1: Up to $10,000, 2 approvals
• Level 2: Up to $30,000, 3 approvals
• Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

• The application template has been copied and aptly renamed (project_name.md).
• I have read the application guidelines.
• Payment details have been provided (bank details via email or Polkadot (USDC & USDT) address in the application).
• I am aware that, in order to receive a grant, I (and the entity I represent) have to successfully complete a KYC/KYB check.
• The software delivered for this grant will be released under an open-source license specified in the application.
• The initial PR contains only one commit (squash and force-push if needed).
• The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
• [UPDATED] I am open for the discussion of this application to take place here in public, or on private Element/Matrix channel. My username is: @maltabba:matrix.org

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[Muhammad-Altabba]

I have read and hereby sign the Contributor License Agreement.

[keeganquigley]

Thanks for the application @Muhammad-Altabba I have the following comments:

• The grants program focuses purely on software development, therefore we don't usually fund things like branding, website design, or CI/CD (we don't support production-ready deployments). Therefore I would argue that M1 is out of the scope of the program.
• M2 is more in line with the grants program, but deliverable 3. seems to be rather research-oriented? I wonder if perhaps you could reduce the scope to a single milestone with just the development improvements.

[Muhammad-Altabba]

Thanks for the application @Muhammad-Altabba I have the following comments:

• The grants program focuses purely on software development, therefore we don't usually fund things like branding, website design, or CI/CD (we don't support production-ready deployments). Therefore I would argue that M1 is out of the scope of the program.
• M2 is more in line with the grants program, but deliverable 3. seems to be rather research-oriented? I wonder if perhaps you could reduce the scope to a single milestone with just the development improvements.

Thanks @keeganquigley,
Sounds good, I now removed the non-programming tasks without adding new ones. So, there is now only 1 milestone. I want this grant request to be the simplest step to start with the process.

I would later open a request for research tasks according to https://github.com/w3f/Grants-Program/blob/master/docs/Support%20Docs/grant_guidelines_per_category.md#research. And request from the Treasury or the Decentralized Future for the no-programming tasks. And I would love to have your direction and recommendation, if you have any.

Thanks,

[semuelle]

Hey @Muhammad-Altabba, thanks for the application. While it is being reviewed, could you conduct the KYC check? Let me know if you have any questions or issues.

[keeganquigley]

Thanks for the changes @Muhammad-Altabba can you also change this in the PR itself? As CI/CD, branding, etc. are still listed above. Thanks!

[keeganquigley]

Thanks for the changes @Muhammad-Altabba I will mark the application as ready for review and ping the committee for comment.

[takahser]

@Muhammad-Altabba I have a few points/questions:

1. I was wondering if you're aware of PAPI and how your solution would differ from it.
2. Regarding the custom rpc types, do these have to be manually hard-coded or is there a way to (semi-) automate this? For example, maybe you could call the rpc_methods endpoint to generate the types, if that's possible.
3. Does your library support only Promises or also Observables?

[josepot]

👋 Hi everyone!

I'm Josep, the author of PAPI.

Recently, @takahser brought to my attention this project and I took some time to thoroughly review it. Initially, I was thrilled to discover another endeavor aimed at creating an alternative to @polkadot/api. However, upon closer examination, my enthusiasm waned somewhat.

I'd like to offer my insights on this project. Before I proceed, I want to emphasize that my intention isn't to oppose the grant for @Muhammad-Altabba. Despite the forthcoming critique, it's genuinely encouraging to see developers tackle these challenges. @Muhammad-Altabba, please view this as constructive feedback, not as a criticism of you or your efforts. Your experimentation and learning in this field are commendable and should certainly continue.

Let me start by clarifying what this project does, and more importantly, what it doesn't.

In essence, the project utilizes @polkadot/typegen to generate interfaces from three different sets of metadata (*1). However, of all these generated types, only the RPC ones are currently used in the Web3 plugin. Ironically, these are precisely the types which don't depend on the metadata 🤷.

Therefore, users of this library are unable to perform effective storage queries, as the library seems incapable of leveraging SCALE for encoding/decoding on-chain data. It's also unsuitable for accessing runtime constants, creating extrinsics, listening to events, etc.

Essentially, the project (ab)uses the Web3.js plugin system to provide a strongly-typed API for accessing "legacy" JSON-RPC methods. In other words, it's akin to using the WsProvider from @polkadot/rpc-provider through a Web3.js plugin, with typings for the RPC interactions, though.

Having clarified the project's functionality, I'd like to explain why I believe this approach is problematic:

1. Web3.js is the wrong tool for the job

   The substrate JSON-RPC API is built for a full-duplex, event-driven, and bidirectional connection (like WebSocket), whereas the Ethereum JSON-RPC API is designed for a "pull-based" REST API. Web3.js was created for Ethereum's JSON-RPC APIs, which significantly differ from Substrate's. Thus, adapting Web3.js for Substrate's low-level JSON-RPC APIs, while innovative, does not practically benefit DApp developers. It also leads to confusion, as Ethereum-specific methods and functions are accessible through the Web3.js API but won't ever be functional on Substrate. Additionally, the declining usage of Web3.js in favor of more developer-friendly tools like Ethers and Viem further questions its suitability for this purpose.

2. Reliance on "legacy" JSON-RPC API

   The current JSON-RPC API has significant limitations, as discussed in this forum post. With a new JSON-RPC API being stabilized, focusing on the older version may not be the most forward-thinking approach.

3. No dynamic codecs => No Events

   @Muhammad-Altabba stated that:

   In this grant request, I propose working on supporting subscribing/unsubscribing to blockchain events as the next functionality to be added.

   It appears there's a misconception about the simplicity of querying events on Substrate, akin to an eth_getLogs request (*2). In reality, accessing events in Substrate is far more complex. They are stored in the System.Events storage entry of each block, and decoding them requires being able to decode any of the events across all the pallets that are present in that runtime. Therefore, before tackling event subscriptions, it's crucial to first master querying and decoding storage entries, and being able to dynamically generate codecs using metadata V14 (or V15). Our metadata builders package might offer some insights into solving this issue.

   That's why I would like to suggest that before trying to support "subscribing/unsubscribing to blockchain events", that some progress is made towards providing APIs for accessing access to storage, while leveraging the metadata.

While I have other concerns, my primary observation is that a Web3.js plugin, falls short of providing an API that could viably alternate with the @polkadot/api package.

To reiterate, I am supportive of exploring alternatives to PJS and do not wish to discourage @Muhammad-Altabba from trying new approaches. Your efforts are valuable to our community, and I look forward to seeing how your project evolves.

---

*1) A pro-tip: Consider adding the src/interfaces folder, which contains generated code that changes with metadata updates, to your .gitignore file. Then, use the postInstall script for automatically generating those files.

*2) I also felt for that one in the past 🙂

[Muhammad-Altabba]

Hi @josepot
I it very interesting to see a perspective from an extremely biased point of view. I appreciate the time spent on writing this and the good language used. And I hope PAPI will get the attention and the care, if it could serve the good will of the developers community.

I totally understand that some people working with gold can sometime fail to evaluate how good a diamond is, and vice versa, and more importantly, how everything can server different audience or different use cases.

However, what is good about decentralization of tools is to provide multiple options to the community so everyone can select what is better for his/her use case.

[josepot]

However, what is good about decentralization of tools is to provide multiple options to the community so everyone can select what is better for his/her use case.

I couldn't agree more! However, the expectations the library is setting in its description is:

It is meant to be an alternative to @polkadot/api package

Therefore, I think that you will agree with me on the fact that: any library that aims to be an alternative to the @polkadot/api, must ensure that a minimum of functionalities are being covered, correct?

Which is why I would really appreciate it if you could please answer the following questions:

1. Can the library be used for accessing storage entries?

2. Can the library be used for reading runtime constants?

3. Can the library leverage the current metadata to dynamically generate the necessary codecs in order to encode/decode on-chain data?

If the answer to those questions is what I think it is, then: don't you think those functionalities should be added prior to:

adding supporting subscribing/unsubscribing to blockchain events as the next functionality to be added

?

The reason being that, as I already explained before, it's not technically possible to add the functionality that you want to add without having first solved these other problems.

On the other hand, I would like to reiterate the importance of stop building tools that leverage a set of JSON-RPC methods that are in the process of being deprecated in favor of the new JSON-RPC spec. Please @Muhammad-Altabba read this 🙏

Don't you think that it makes sense for new tools to leverage the new JSON-RPC API, rather than capitalizing on the old methods that are trying to be deprecated?

[Muhammad-Altabba]

Hi @josepot and @takahser,

It is very interesting to read your attempts to prevent developing a solution that you seem to see as a competitor. But I believe you do not need to see it like that.

And I have a personal advice which is to prepare yourself to be comfortable with having alternative solutions to what you are building, and to think of ways to collaborate instead of trying to discourage their development.

The library I am building will support any new standard that would be stable and finalized, and even probably at some earlier stages. As there will be multiple subsequent follow grant requests in addition to sponsorship. As written, this is just the first grant request that you are trying to prevent, and there will be much more. So, get yourself ready to accept the fact that there will be multiple solutions to the same problem that you are working on. And some of them might be much more helpful to a huge part of the community.

And for the technical questions which you are asking me about how I will do. Simply, my answer is: you will see the implementation later and can then understand how things will go. I will never be interested in proving something to someone who has a clash of interests.

However, sorry, I will stop replying to your new messages because it is just some destructive attempts to prevent developing a public good, just because you see it clashes with your own and your team interests.

So, I hope you change your mindset for looking for collaboration and never be afraid of competition. And to kindly stop spamming on grant requests that you think it is for your own benefits to get rejected.

[josepot]

Hi @Muhammad-Altabba ,

I respect the passion behind your project. It's clear we both aim to contribute positively to the community, albeit through different approaches.

I want to clarify my intent, as it seems there may have been a misunderstanding. My goal has never been to hinder your project or any other that seeks to innovate and serve the community. On the contrary, I firmly believe in the value of diversity in tools and solutions. It enriches the ecosystem and provides developers with choices that best suit their needs. My feedback was intended to highlight areas that could enhance the project's utility and ensure its alignment with the evolving landscape of our ecosystem.

The technical questions I raised were in the spirit of constructive dialogue, aiming to foster excellence and innovation. It's in this same spirit that I express eagerness to see your implementation strategies and how your project will adapt to and incorporate new standards. The vitality of our ecosystem is bolstered by such endeavors, and your commitment to embracing these standards is commendable.

Regarding the perception of competition, please allow me to reassure you that my interest lies in collaboration and mutual growth. The advancements we each contribute to are not just for our benefit but for the broader community. The richness of multiple solutions is a testament to the vibrancy and dynamism of our ecosystem.

It's disheartening to hear that my attempts at engagement have been perceived as oppositional. Please understand that my comments were driven by a desire to ensure the robustness and forward-compatibility of tools within our ecosystem. They were never meant to discourage or undermine your efforts. On the contrary, I welcome the innovation and diversity of solutions that projects like yours bring to the table.

In closing, I hope we can move past this misunderstanding and recognize our shared goal of contributing to the community's growth and success. I remain open to constructive dialogue and collaboration, and I sincerely wish you the best in your endeavors. Your work is a valuable addition to the ecosystem, and I look forward to witnessing its development and impact.

[takahser]

@Muhammad-Altabba thank you for your engagement in the grant proposal process and your contributions to the discussions around this proposal. The process of reviewing proposals is greatly enriched by the diverse perspectives and insights from all participants, including community members like @josepot, who bring valuable external viewpoints.

I it very interesting to see a perspective from an extremely biased point of view.
It is very interesting to read your attempts to prevent developing a solution that you seem to see as a competitor.
And I have a personal advice which is to prepare yourself to be comfortable with having alternative solutions to what you are building, and to think of ways to collaborate instead of trying to discourage their development.
this is just the first grant request that you are trying to prevent, and there will be much more
And to kindly stop spamming on grant requests that you think it is for your own benefits to get rejected.

To me, many of your points seem to imply that @josepot is trying to prevent this proposal from being passed. However, after reading the whole conversation, my impression is that he's argued objectively and brought up good technical questions and inputs, providing you with thoughts on how you can improve your proposal. Hence, I don't believe these allegations to be true.

A more comprehensive response to the technical questions would have been instrumental in understanding your project's architecture and approach. Unfortunately, the current replies offer limited insight:

The library I am building will support any new standard that would be stable and finalized, and even probably at some earlier stages.

and

you will see the implementation later and can then understand how things will go. I will never be interested in proving something to someone who has a clash of interests.

Please understand that as part of the grant proposal evaluation process, we're trying to understand the technical architecture of the solution proposed as detailed as possible. While W3F grants committee members are the most engaged in the discussion, our program is open to anybody who'd like to contribute, just as it's the case in this grant proposal. Likewise, you're also free to contribute to other people's grant proposals as long as you comply with our code of conduct.

For these reasons, I'll refrain from voting in favor of this grant application. Other W3F Grant committee members can have a different opinion on that. Thanks for your understanding!

[Muhammad-Altabba]

Hi @takahser ,
Sorry, it is my bad. I thought you are part of PAPI team.
@josepot has a clear clash of interests that seamed to me to led him to not see any of the grate benefits already mentioned 3 key benefits in the Ecosystem Fit. In addition to what I saw as irrelevant cretinism but mixed with few questions. So, sorry if I did not early see a value in answering his questions. But I am very happy to answer, since your found his points need clarifying. And I am sorry for any inconvenience.

First of all I want to point to what is stated in the project repo:

It is meant to be an alternative to @polkadot/api package.

Sorry if my language is not clear. And let me elaborate because this gives the context: 'it is meant to be an alternative' and so: 'it is not currently an alternative to @polkadot/api except for the on feature implemented that is calling RPC mehtods. However, it meant to be (planned to be) so by working on it till it is.'

So, for my reply:

The library I am building will support any new standard that would be stable and finalized, and even probably at some earlier stages.

This is regarding the new unstable RPC endpoints that @josepot referred to. So, I am here stating that web3-plugin-polkadot will support those new RPCs once they are finalized. And I am planning to open a grant request to implement them probably even before they are fully stable. As web3-plugin-polkadot should support both the current stable and the new RPC standard in the future. And I think it is more important to work on supporting the stable RPC methods as they will be there for long for many people. And the has to be in parallel on supporting the new usable RPCs. I would make the next grant request about the new unstable RPCs, if you think so.

And regarding your early questions, I am also happy to answer, and sorry again for the misunderstanding-

I was wondering if you're aware of PAPI and how your solution would differ from it.

The https://github.com/polkadot-api/polkadot-api is another alternative to polkadot.js. So, we are all providing alternatives to the developers which is great for the community. However, I found no documentation at there repo (even it is available since Oct 24, 2021) So it would not be easy to check and compare. But, for sure web3-plugin-polkadot is achieving the task differently. And the main advantage is are listed in its README.md file. And the library has a basic documentation in the readme file and also a project to play with.

And let me here speak about the number of downloads. Each PAPI sub-package has about 10,000 to 15.000 downloads per month which is good. But I expect web3-plugin-polkadot to bring more as web3.js has more than 1 million downloads per month. And this plugin will encourage some of those to try the plugin and benefit from its features (mainly the Development Experience and Learning Curve in addition to the Code Portability).

And as @josepot mentioned Ethers.js and Viem, I think it will be helpful to have plugin for them as well. However, web3.js v4 was designed to support plugins. So, even it is not easy at all to do a plugin for non-EVM nodes, it is still smoother I think to do a plugin for web3.js. But if anyone is interested in doing a plugin for viem or ethers.js, I would encourage him to give it a try.

Regarding the custom rpc types, do these have to be manually hard-coded or is there a way to (semi-) automate this? For example, maybe you could call the rpc_methods endpoint to generate the types, if that's possible.

Totally true, and it is in mind to make this possible. There are lots of features that this library will provide. It just started and I kept this grant request as an absolute minimal as this is my first grant request and I want it to be as simple and as smooth as possible.

Does your library support only Promises or also Observables?

The library currently support only Promises. I made it like this in order to be simple especially for entry-level developers in addition to be similar to web3.js. However, it could easily support also Observables similar to polkadot.js. Please let me know if you like it to support this as well.

I hope that I could clear the points. And this grant request is just a baby step. And upon finishing it, more grant requests will be made to work on the features. And I will be happy if you recommend which next feature to be implemented next.

Please, let me know if there are more points that needs clarifications.

[semuelle]

Thanks for the updates, @Muhammad-Altabba. Given that you have now mentioned multiple times that this is a minimal grant application and that there are more to follow, could you share any insights into what your roadmap looks like and how much time/effort/funds is required to build what you are aiming for?

[Muhammad-Altabba]

Thanks for the updates, @Muhammad-Altabba. Given that you have now mentioned multiple times that this is a minimal grant application and that there are more to follow, could you share any insights into what your roadmap looks like and how much time/effort/funds is required to build what you are aiming for?

Sure @semuelle,
And for this mater I would refer to the section written in this PR:

Future Plans

• Implement additional functionalities, such as accounts handling and smart contract interactions. There will be subsequent grant requests for these and other functionalities.
• Work on the complementary stuff like the plugin identity design and website content, the CD/CI integration in GitHub and an in-browser playground...
• Popularize the library through tutorials, illustrative videos, workshops, and sponsoring bounties at hackathons.
• Develop more plugins for the Polkadot ecosystem.

This project will rely on grants and sponsorships as it serves the entire ecosystem and its decentralized vision. There will be further grant requests, starting with this initial modest application to familiarize ourselves with the processes.

For the related plugin and libraries, additional applications for grants and sponsorships to the various Substrate networks and beneficiaries, for each different plugin serving different parts of the ecosystem, will be considered.

And I think that the most basic functionality to follow are: accounts management, sending transaction and smart contract interactions. After that I would depend on the community feedback to schedule and prioritize the next features to be implemented in order to be aligned with the community needs and preferences as possible.

Regarding the resource allocation for this plugin, it will be the best to have in the future at least 1 full-time Dev and 1 full-time/part-time DevRel. And we can therefore estimate the cost based on that. Those guys would work on the project on a continues basis to respond to community issues and implement new features in addition to help the community growth. However, starting small is always possible. And depending on the community acceptance and benefits more resources to be assigned...

A note on the side: This library is under the MIT licence. So it is a community project which can continue to be led by me or any other. And any company/person can take the lead of its future development. However, what I will try to do is to make it reach a maturity state and adoption and continue to propose more development and enhancements...

I hope this would help answering your question.

[semuelle]

Thanks again for your application, @Muhammad-Altabba. But given that the application has not attracted any approvals from the committee in the past four weeks, I am going to close it. Personally, I'd have preferred to see a more thorough roadmap. What you are asking for here is small, but it doesn't take us near what we would like to see, so I think the hurdle to commit to the project is much bigger than the amount you're asking for. Also, I believe Josep made some fair points.

I hope that you will still be able to continue working on it and bring it to life. Feel free to apply again in the future. In any case, best of luck!