From bbb1c44f3d625701e2d04d2f5fdf2bb572e69c5c Mon Sep 17 00:00:00 2001 From: Sabine Proll Date: Wed, 12 Jul 2023 15:27:44 +0200 Subject: [PATCH 1/8] proposal: SARP software design --- applications/sarp-2-swdesign.md | 154 ++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 applications/sarp-2-swdesign.md diff --git a/applications/sarp-2-swdesign.md b/applications/sarp-2-swdesign.md new file mode 100644 index 00000000000..5ee4da62efe --- /dev/null +++ b/applications/sarp-2-swdesign.md @@ -0,0 +1,154 @@ +# SARP - Software Design + +- **Team Name:** Supercomputing Systems AG (SCS) +- **Payment Address:** 0xd24622311a22470353bd21d9bcd9e02ba0cfebbe (USDC) +- **[Level](https://github.com/w3f/Grants-Program/tree/master#level_slider-levels):** 2 + +## Project Overview :page_facing_up: + +This is the follow up to our initial [research proposal](https://github.com/w3f/Grants-Program/blob/master/applications/sarp-basic-functionality.md), that we delivered [here](https://github.com/w3f/Grant-Milestone-Delivery/pull/880). The goal of this work package is to evaluate different software designs to implement a static code analysis on substrate pallets with MIRAI. Furthermore we want to investigate issues and bugs, we found in MIRAI in the previous work package. + +### Overview + +[Runtime Pallets](https://docs.substrate.io/learn/runtime-development/) are modules for writing the business logic of blockchains in [Substrate](https://github.com/paritytech/substrate) (a Rust framework for building blockchains). These are usually concise pieces of standalone code with relatively few dependencies and clear specifications, hence tractable targets for performing static analysis and verification. The code quality of a runtime pallet is crucial, as even minor defects can result in major exploits like DoS attacks or the stealing of funds by a malicious party. A static code analysis can help to automate the auditing processes and prevent introduction of defects throughout the software life-cycle. + +Therefore we would like to develop a tool - SARP (Static Analysis tool for Runtime Pallets) to perform static analysis with reasonable soundness guarantees. In particular, we would like to target vunerability classes that are detectable using dataflow analysis techniques like *tag analysis* and *taint analysis*. + +Our team has a good understanding of substrate and Rust. We are still getting started on the topic of static code analysis. + +### Project Details + +We will base our work on [MIRAI](https://github.com/facebookexperimental/MIRAI/) and extend it with checks on substrate pallets. For details see the [Development Roadmap](#development-roadmap-nut_and_bolt) + +### Ecosystem Fit + +The tool will help any team developing substrate pallets. It can further be integrated in the CI pipelines of the teams, providing a continuous quality check on the pallet code. + +In the long term it could be interesting to connect the work done here with the new emerging auditing DAOs (like [QRUCIAL DAO](https://github.com/w3f/Grants-Program/blob/master/applications/QRUCIAL_DAO.md)). + + +## Team :busts_in_silhouette: + +### Team members + +- Sabine Proll: Project Lead & Developer +- Thomas Niederberger: Developer +- Bigna Härdi: Developer +- Edith Chevrier: Developer + +### Contact + +- **Contact Name:** Sabine Proll +- **Contact Email:** Sabine.Proll@scs.ch | info@scs.ch +- **Website:** https://www.scs.ch + +### Legal Structure + +- **Registered Address:** Technoparkstrasse 1, 8005 Zürich, Switzerland +- **Registered Legal Entity:** Supercomputing Systems AG + +### Team's experience + +Supercomputing Systems AG is a contractor with 130 engineers, working in the fields of software, electronics and system design. Profound know-how, solid methodological competence as well as efficient project management are the foundation of our success. Within the company we have a team of 5 blockchain developers, who have experience in the Polkadot ecosystem. + +Our blockchain team has been a contributor to the ecoysystem since 2019. We started with grants from the Web3 Foundation to build the basis for [Integritee](https://github.com/integritee-network) (see our grants from waves [1](https://github.com/w3f/General-Grants-Program/blob/master/grants/speculative/substrate_sgx_proposal.md), [3](https://github.com/w3f/General-Grants-Program/blob/master/grants/speculative/substrate-api-client.md) and [5](https://github.com/w3f/General-Grants-Program/blob/master/grants/speculative/SubstraTEE-extension-pack1.md)). After that, our team has worked for Integritee and Encointer as a contractor. Recently the team received grants from the Kusama treasury for maintaining and improving the [substrate-api-client](https://github.com/scs/substrate-api-client), see our proposals for [Nov 22 - Jan 23](https://kusama.subsquare.io/referenda/referendum/26) and [Feb 23 - Apr 23](https://kusama.subsquare.io/referenda/referendum/88), [May 23 - Jul 23](https://kusama.polkassembly.io/referenda/182). Also, we successfully delivered the [first milestone for SARP](https://github.com/w3f/Grant-Milestone-Delivery/pull/880). + +### Team Code Repos + +The team has mainly worked on the following repositories + +- [SARP - Milestone 1 delivery](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples) +- [Substrate Api Client](https://github.com/scs/substrate-api-client) +- [Integritee Worker](https://github.com/integritee-network/worker) +- [Encointer Sidechain](https://github.com/encointer/community-sidechain) + +Github accounts of the team members + +- https://github.com/masapr +- https://github.com/haerdib +- https://github.com/echevrier +- https://github.com/Niederb + + +### Team LinkedIn Profiles + +- https://www.linkedin.com/in/sabine-proll-5a7118153 +- https://www.linkedin.com/in/bigna-h%C3%A4rdi-736bb21a9 +- https://www.linkedin.com/in/edith-chevrier-90233297 +- https://www.linkedin.com/in/thomas-niederberger-6057b71a7 + +## Development Status :open_book: + +In a first research project we investigated, if MIRAI can be used for static code analysis of substrate pallets. For this we did a proof of concept on two cases: +- Check of [incorrect origin](https://github.com/scs/MIRAI/blob/Milestone1_Research/substrate-examples/pallet_template/README.md) in the [substrate node template](https://github.com/substrate-developer-hub/substrate-node-template/tree/e0c480c0f322d0b0d1b310c93fa646fc0cfdd2df/pallets/template) +- Validation of [unsigned transactions](https://github.com/scs/MIRAI/blob/Milestone1_Research/substrate-examples/offchain-worker/README.md) for substrate's [offchain worker example](https://github.com/paritytech/substrate/tree/ea9ce4c0af36310c0b0db264ab12cf4766b83750/frame/examples/offchain-worker) + +The overall conclusion was, that it is best to run the analysis only on the newly written pallet code, but not on the code generated by substrate's macros. To facilitate this a detailed analysis of different software designs has to be evaluated. + + +The full documentation of our findings can be found [here](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples). + + +## Development Roadmap :nut_and_bolt: + + + +### Overview + +- **Estimated duration:** 1 month +- **FTE:** 1 +- **Costs:** 30,000 USD + +### Milestone 1 Software Design & Bug Fixes + +- **Estimated duration:** 1 month +- **FTE:** 1 +- **Costs:** 30,000 USD + +In our previous work, we found the following problems: + +1. **Crashes and timeouts of MIRAI** Certain pieces of substrate code lead to crashes of MIRAI. In other cases, parts of the code are not analyzed/do not produce warnings, because MIRAI runs into a timeout before reaching this code. Because of this, our examples are rather simple and we couldn't add and check tags at the locations we originally wanted to. + +2. **Complexity due to substrate macros** The main reason for crashes and timeouts in our examples, was caused by substrate macros, adding a lot of complexity to the code in the background. Ideally SARP only analyzes the newly written code of a pallet. + +3. **Invasiveness of tag analysis** The code we wrote in our PoC is very invasive and changes the code of the pallet. This is not practical for end-users. Ideally the user doesn't need to change anything on their code, or at least the changes should be very simple. + +To address 2. and 3. we plan to evaluate different software designs. These will be part of our deliverables and we plan to discuss these with Parity and/or Web3 Foundation. + +To address 1. we want to further analyze timeouts and crashes in MIRAI. Possibly they can be resolved by bugfixes in MIRAI. If not, we need to find workarounds. + + +#### Deliverables + +| Number | Deliverable | Specification | +|--------|------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that explains how to reproduce our examples. | +| 0c. | Testing and Testing Guide | We will provide a testing guide on how to run SARP with the different software designs. These might be on different branches of the repository. | +| 1. | Prototype Code | Prototype code to showcase the different software designs. | +| 2. | Documentation | Technical documentation describing the different variants of software design and its implications to pallet developers. | +| 3. | Engagement | We will discuss the different solutions and their implications with Web3 Foundation and/or Parity. | +| 4. | Bug reports and PRs in MIRAI | Any issues or bugfixes in MIRAI will be reported as an issue, resp. provided as a PR in the [MIRAI repository](https://github.com/facebookexperimental/MIRAI). | + + + + + + +## Future Plans + + +1. Decide on vulnerabilities for an MVP. +
For this we plan to engage with Web3 Foundation / Parity and auditing companies such as [OtterSec](https://osec.io/) or [FYEO](https://www.fyeo.io/). +2. Implement a first simple version of the tool, together with tests and documentation. +3. Improve the usability, by providing + * means to surpress warnings + * a comprehensive user tutorial, incl. documentation on the risks of each vulnerability +4. Add more features including checks on more vulnerability classes. + +Once we have a tool with a good feature set and basic usability features, we want to further promote it to auditors and developers. + + +## Additional Information :heavy_plus_sign: + +With our work in the previous grant, we deliberately invested into this project, as static code analysis was not our area of expertise. Our investment was two-fold: we used a lower hourly rate to calculate the cost and put in more effort than planned when implementing the project. With this package we increased the hourly rate and plan to stick closer to the estimated work effort. \ No newline at end of file From 65950227487dde8dd3cf60bf4f6758028c6d0f2d Mon Sep 17 00:00:00 2001 From: Sabine Proll Date: Wed, 12 Jul 2023 16:30:23 +0200 Subject: [PATCH 2/8] Fixed text in overview to satisfy checks --- applications/sarp-2-swdesign.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/applications/sarp-2-swdesign.md b/applications/sarp-2-swdesign.md index 5ee4da62efe..abdd1b2070b 100644 --- a/applications/sarp-2-swdesign.md +++ b/applications/sarp-2-swdesign.md @@ -95,9 +95,10 @@ The full documentation of our findings can be found [here](https://github.com/sc ### Overview -- **Estimated duration:** 1 month -- **FTE:** 1 -- **Costs:** 30,000 USD +- **Total Estimated Duration:** 1 month +- **Full-Time Equivalent (FTE):** 1 FTE +- **Total Costs:** 30,000 USD + ### Milestone 1 Software Design & Bug Fixes From bf0f18c4c18f6da20ed6e3ce96c2c0babac08bbe Mon Sep 17 00:00:00 2001 From: Sabine Proll Date: Thu, 13 Jul 2023 16:16:19 +0200 Subject: [PATCH 3/8] clarify deliverables --- applications/sarp-2-swdesign.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/applications/sarp-2-swdesign.md b/applications/sarp-2-swdesign.md index abdd1b2070b..f584102878b 100644 --- a/applications/sarp-2-swdesign.md +++ b/applications/sarp-2-swdesign.md @@ -114,22 +114,26 @@ In our previous work, we found the following problems: 3. **Invasiveness of tag analysis** The code we wrote in our PoC is very invasive and changes the code of the pallet. This is not practical for end-users. Ideally the user doesn't need to change anything on their code, or at least the changes should be very simple. -To address 2. and 3. we plan to evaluate different software designs. These will be part of our deliverables and we plan to discuss these with Parity and/or Web3 Foundation. +To address 2. and 3. we plan to evaluate different software designs. These will be part of our deliverables and we plan to discuss these with Parity and/or Web3 Foundation. The challenge here is the non-invasiveness of the solution. Specifically we plan to look into the following questions: +1. Is it possible to tweak MIRAI, so that we can run the analysis on pallets without modification? +2. Without tweaking MIRAI: What options do we have to separate the newly written pallet logic from the macro logic? Is it possible to do so, without changing the logic or adding artificial code (as the macros also connect the different functions of a pallet)? +3. Can we combine our findings of 1. and 2.? To address 1. we want to further analyze timeouts and crashes in MIRAI. Possibly they can be resolved by bugfixes in MIRAI. If not, we need to find workarounds. +Apart from the documentation of our analysis, we will deliver a first prototype version of our tool. #### Deliverables -| Number | Deliverable | Specification | -|--------|------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0a. | License | MIT | -| 0b. | User Documentation | We will provide a basic **tutorial** that explains how to reproduce our examples. | -| 0c. | Testing and Testing Guide | We will provide a testing guide on how to run SARP with the different software designs. These might be on different branches of the repository. | -| 1. | Prototype Code | Prototype code to showcase the different software designs. | -| 2. | Documentation | Technical documentation describing the different variants of software design and its implications to pallet developers. | -| 3. | Engagement | We will discuss the different solutions and their implications with Web3 Foundation and/or Parity. | -| 4. | Bug reports and PRs in MIRAI | Any issues or bugfixes in MIRAI will be reported as an issue, resp. provided as a PR in the [MIRAI repository](https://github.com/facebookexperimental/MIRAI). | +| Number | Deliverable | Specification | +|--------|-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on the examples provided. | +| 0c. | Testing and Testing Guide | We will *not* provide a test suite with this milestone, but documentation on how to run our examples as in 0b. | +| 1. | Tool | We will provide a prototype version of the tool. Following the approach, decided in 3. | +| 2. | Documentation | Technical documentation of the tool, incl. reasoning on the design decisions. | +| 3. | Engagement | We will discuss different solutions and their implications with Web3 Foundation and/or Parity. For this we will document each approach with
  • at least one example, incl. documentation on how to run it.
  • prototype code
  • analysis on the approach's invasiveness
| +| 4. | Analysis of errors in MIRAI | We will document each error we encounter in MIRAI (specifically crashes and faulty analyses) with information on:
  • how to reproduce it
  • reasons for its occurance and implications thereof
This analysis will include the problems we discovered in the previous work package, but haven't analyzed yet (see [1](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples/pallet_template#open-issues) and [2](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples/offchain-worker#open-issues)).

We will report each issue in the [MIRAI repository](https://github.com/facebookexperimental/MIRAI), resp. provide a PR there. | From fc5cad0814815342f69e60e12d7e0d99697620d0 Mon Sep 17 00:00:00 2001 From: Sabine Proll Date: Mon, 17 Jul 2023 11:28:59 +0200 Subject: [PATCH 4/8] more details on sw design approaches + deliverable with examples for vulnerability classes --- applications/sarp-2-swdesign.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/applications/sarp-2-swdesign.md b/applications/sarp-2-swdesign.md index f584102878b..a82d1d3ffe5 100644 --- a/applications/sarp-2-swdesign.md +++ b/applications/sarp-2-swdesign.md @@ -115,9 +115,10 @@ In our previous work, we found the following problems: 3. **Invasiveness of tag analysis** The code we wrote in our PoC is very invasive and changes the code of the pallet. This is not practical for end-users. Ideally the user doesn't need to change anything on their code, or at least the changes should be very simple. To address 2. and 3. we plan to evaluate different software designs. These will be part of our deliverables and we plan to discuss these with Parity and/or Web3 Foundation. The challenge here is the non-invasiveness of the solution. Specifically we plan to look into the following questions: -1. Is it possible to tweak MIRAI, so that we can run the analysis on pallets without modification? -2. Without tweaking MIRAI: What options do we have to separate the newly written pallet logic from the macro logic? Is it possible to do so, without changing the logic or adding artificial code (as the macros also connect the different functions of a pallet)? -3. Can we combine our findings of 1. and 2.? +1. Can we implement a preprocessing, that automatically annotates the pallet code for analysis in MIRAI? +2. Can MIRAI be tweaked to abstract out low-level details in the flow of function calls? +3. Is it possible to separate the newly written pallet logic from the macro logic? If so, is it possible, without changing the logic or adding artificial code (as the macros also connect the different functions of a pallet)? +4. Are there good combinations of the above approaches? To address 1. we want to further analyze timeouts and crashes in MIRAI. Possibly they can be resolved by bugfixes in MIRAI. If not, we need to find workarounds. @@ -125,15 +126,15 @@ Apart from the documentation of our analysis, we will deliver a first prototype #### Deliverables -| Number | Deliverable | Specification | -|--------|-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0a. | License | MIT | -| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on the examples provided. | -| 0c. | Testing and Testing Guide | We will *not* provide a test suite with this milestone, but documentation on how to run our examples as in 0b. | -| 1. | Tool | We will provide a prototype version of the tool. Following the approach, decided in 3. | -| 2. | Documentation | Technical documentation of the tool, incl. reasoning on the design decisions. | -| 3. | Engagement | We will discuss different solutions and their implications with Web3 Foundation and/or Parity. For this we will document each approach with
  • at least one example, incl. documentation on how to run it.
  • prototype code
  • analysis on the approach's invasiveness
| -| 4. | Analysis of errors in MIRAI | We will document each error we encounter in MIRAI (specifically crashes and faulty analyses) with information on:
  • how to reproduce it
  • reasons for its occurance and implications thereof
This analysis will include the problems we discovered in the previous work package, but haven't analyzed yet (see [1](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples/pallet_template#open-issues) and [2](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples/offchain-worker#open-issues)).

We will report each issue in the [MIRAI repository](https://github.com/facebookexperimental/MIRAI), resp. provide a PR there. | +| Number | Deliverable | Specification | +|--------|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on the examples provided. | +| 0c. | Testing and Testing Guide | We will *not* provide a test suite with this milestone, but documentation on how to run our examples as in 0b. | +| 1. | Tool | We will provide a prototype version of the tool. Following the approach, decided in 3. We will provide examples for applying the tool to 3 of the following 5 vulnerability classes:
  • [incorrect origin](https://github.com/bhargavbh/MIRAI/blob/main/substrate_examples/incorrect-origin/description.md) of dispatchable functions.
  • [unsigned transaction](https://github.com/bhargavbh/MIRAI/blob/main/substrate_examples/unsigned-transaction/description.md) validation.
  • tracking bad randomness: ensure bad randomness does not leak into sensitive functions.
  • detect panics statically to avoid potential DoS attacks: these include [unsafe arithmetic operations](https://github.com/bhargavbh/MIRAI/blob/main/substrate_examples/arithmetic-overflow/description.md), access outside bounds, assertion failures, etc.
  • tracking unsanitised input leakage for sensitive functions.
| +| 2. | Documentation | Technical documentation of the tool, incl. reasoning on the design decisions. | +| 3. | Engagement | We will discuss different solutions and their implications with Web3 Foundation and/or Parity. For this we will document each approach with
  • at least one example, incl. documentation on how to run it.
  • prototype code
  • analysis on the approach's invasiveness
| +| 4. | Analysis of errors in MIRAI | We will document each error we encounter in MIRAI (specifically crashes and faulty analyses) with information on:
  • how to reproduce it
  • reasons for its occurance and implications thereof
This analysis will include the problems we discovered in the previous work package, but haven't analyzed yet (see [1](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples/pallet_template#open-issues) and [2](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples/offchain-worker#open-issues)).

We will report each issue in the [MIRAI repository](https://github.com/facebookexperimental/MIRAI), resp. provide a PR there. | From 20635ad87eb83187f6eb4a50479d4cdec7d14987 Mon Sep 17 00:00:00 2001 From: Sabine Proll Date: Tue, 22 Aug 2023 09:18:38 +0200 Subject: [PATCH 5/8] New version with tool delivery and 3 milestones --- ...-swdesign.md => sarp-2-core-components.md} | 124 ++++++++++++------ 1 file changed, 85 insertions(+), 39 deletions(-) rename applications/{sarp-2-swdesign.md => sarp-2-core-components.md} (54%) diff --git a/applications/sarp-2-swdesign.md b/applications/sarp-2-core-components.md similarity index 54% rename from applications/sarp-2-swdesign.md rename to applications/sarp-2-core-components.md index a82d1d3ffe5..e9f511f704d 100644 --- a/applications/sarp-2-swdesign.md +++ b/applications/sarp-2-core-components.md @@ -1,12 +1,12 @@ -# SARP - Software Design +# SARP - Core Components - **Team Name:** Supercomputing Systems AG (SCS) - **Payment Address:** 0xd24622311a22470353bd21d9bcd9e02ba0cfebbe (USDC) -- **[Level](https://github.com/w3f/Grants-Program/tree/master#level_slider-levels):** 2 +- **[Level](https://github.com/w3f/Grants-Program/tree/master#level_slider-levels):** 3 ## Project Overview :page_facing_up: -This is the follow up to our initial [research proposal](https://github.com/w3f/Grants-Program/blob/master/applications/sarp-basic-functionality.md), that we delivered [here](https://github.com/w3f/Grant-Milestone-Delivery/pull/880). The goal of this work package is to evaluate different software designs to implement a static code analysis on substrate pallets with MIRAI. Furthermore we want to investigate issues and bugs, we found in MIRAI in the previous work package. +This is the follow up to our initial [research proposal](https://github.com/w3f/Grants-Program/blob/master/applications/sarp-basic-functionality.md), that we delivered [here](https://github.com/w3f/Grant-Milestone-Delivery/pull/880). The goal of this work package is to implement a first version of SARP (Static Analysis Tool for Runtime Pallets) with two core components: mocking of substrate functions and an annotations generator. ### Overview @@ -18,13 +18,13 @@ Our team has a good understanding of substrate and Rust. We are still getting st ### Project Details -We will base our work on [MIRAI](https://github.com/facebookexperimental/MIRAI/) and extend it with checks on substrate pallets. For details see the [Development Roadmap](#development-roadmap-nut_and_bolt) +Our work is based on [MIRAI](https://github.com/facebookexperimental/MIRAI/), it extends it with checks on substrate pallets. For details see the [Development Roadmap](#development-roadmap-nut_and_bolt) ### Ecosystem Fit The tool will help any team developing substrate pallets. It can further be integrated in the CI pipelines of the teams, providing a continuous quality check on the pallet code. -In the long term it could be interesting to connect the work done here with the new emerging auditing DAOs (like [QRUCIAL DAO](https://github.com/w3f/Grants-Program/blob/master/applications/QRUCIAL_DAO.md)). +In the long term it could be interesting to connect the work done here with new emerging auditing DAOs (like [QRUCIAL DAO](https://github.com/w3f/Grants-Program/blob/master/applications/QRUCIAL_DAO.md)). ## Team :busts_in_silhouette: @@ -83,10 +83,7 @@ In a first research project we investigated, if MIRAI can be used for static cod - Check of [incorrect origin](https://github.com/scs/MIRAI/blob/Milestone1_Research/substrate-examples/pallet_template/README.md) in the [substrate node template](https://github.com/substrate-developer-hub/substrate-node-template/tree/e0c480c0f322d0b0d1b310c93fa646fc0cfdd2df/pallets/template) - Validation of [unsigned transactions](https://github.com/scs/MIRAI/blob/Milestone1_Research/substrate-examples/offchain-worker/README.md) for substrate's [offchain worker example](https://github.com/paritytech/substrate/tree/ea9ce4c0af36310c0b0db264ab12cf4766b83750/frame/examples/offchain-worker) -The overall conclusion was, that it is best to run the analysis only on the newly written pallet code, but not on the code generated by substrate's macros. To facilitate this a detailed analysis of different software designs has to be evaluated. - - -The full documentation of our findings can be found [here](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples). +The overall conclusion was, that it is best to run the analysis only on the newly written pallet code, but not on the code generated by substrate's macros. The full documentation of our findings can be found [here](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples). ## Development Roadmap :nut_and_bolt: @@ -95,58 +92,107 @@ The full documentation of our findings can be found [here](https://github.com/sc ### Overview -- **Total Estimated Duration:** 1 month -- **Full-Time Equivalent (FTE):** 1 FTE -- **Total Costs:** 30,000 USD +- **Total Estimated Duration:** 4 months +- **Full-Time Equivalent (FTE):** 0.8 FTE +- **Total Costs:** 100'000 USD +In our previous work, we found the following problems: -### Milestone 1 Software Design & Bug Fixes +1. **Crashes and timeouts of MIRAI** Certain pieces of substrate code lead to crashes of MIRAI. In other cases, parts of the code are not analyzed/do not produce warnings, because MIRAI runs into a timeout before reaching this code. -- **Estimated duration:** 1 month -- **FTE:** 1 -- **Costs:** 30,000 USD +2. **Complexity due to substrate macros** The main reason for crashes and timeouts, was caused by substrate macros, adding a lot of complexity to the code in the background. Ideally SARP only analyzes the newly written code of a pallet. -In our previous work, we found the following problems: +3. **Invasiveness of tag analysis** The code we wrote in our PoC is very invasive and changes the code of the pallet. This is not practical for end-users. Ideally the user doesn't need to change anything on their code, or at least the changes should be very simple. -1. **Crashes and timeouts of MIRAI** Certain pieces of substrate code lead to crashes of MIRAI. In other cases, parts of the code are not analyzed/do not produce warnings, because MIRAI runs into a timeout before reaching this code. Because of this, our examples are rather simple and we couldn't add and check tags at the locations we originally wanted to. +To approach these issues, we plan to implement two core features in SARP: -2. **Complexity due to substrate macros** The main reason for crashes and timeouts in our examples, was caused by substrate macros, adding a lot of complexity to the code in the background. Ideally SARP only analyzes the newly written code of a pallet. +1. **Mock substrate functions**, that increase the complexity, to improve 1. and 2. +2. **Automatically generate annotations and checks**, that are needed for the control flow analysis, to improve 3. -3. **Invasiveness of tag analysis** The code we wrote in our PoC is very invasive and changes the code of the pallet. This is not practical for end-users. Ideally the user doesn't need to change anything on their code, or at least the changes should be very simple. +The planned workflow of the tool is: +1. Create a copy of the analyzed pallet. +2. For the copy: + 1. Adjust dependencies, so that the mocked substrate libraries are used + 2. Add annotations and checks, needed for the analysis +3. Run MIRAI on the adjusted copy. + +#### Vulnerabilities +The tool will cover the following cases of vulnerabilities: +- [incorrect origin](https://github.com/bhargavbh/MIRAI/blob/main/substrate_examples/incorrect-origin/description.md) of dispatchable functions +
We will provide documentation, example and tests for the described case of enforcing a custom origin type. +- [unsigned transaction](https://github.com/bhargavbh/MIRAI/blob/main/substrate_examples/unsigned-transaction/description.md) validation +
We will provide documentation, example and tests for enforcing checks on all input parameters into a function. Specifically for the case of validating unsigned transactions, but not limited to this. + +We plan to do a detailed analysis on interesting vulnerabilities in the [future](#future-plans). -To address 2. and 3. we plan to evaluate different software designs. These will be part of our deliverables and we plan to discuss these with Parity and/or Web3 Foundation. The challenge here is the non-invasiveness of the solution. Specifically we plan to look into the following questions: -1. Can we implement a preprocessing, that automatically annotates the pallet code for analysis in MIRAI? -2. Can MIRAI be tweaked to abstract out low-level details in the flow of function calls? -3. Is it possible to separate the newly written pallet logic from the macro logic? If so, is it possible, without changing the logic or adding artificial code (as the macros also connect the different functions of a pallet)? -4. Are there good combinations of the above approaches? -To address 1. we want to further analyze timeouts and crashes in MIRAI. Possibly they can be resolved by bugfixes in MIRAI. If not, we need to find workarounds. -Apart from the documentation of our analysis, we will deliver a first prototype version of our tool. +### Milestone 1 - Basic Setup & Mocking of Substrate Libraries +- **Estimated duration:** 2 months +- **FTE:** 0.7 +- **Costs:** 47'000 USD + +With the first milestone we want to do the basic setup of the project, integrate a first set of substrate mocks and deliver a first runnable tool. + +| Number | Deliverable | Specification | +|--------|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on a substrate pallet. | +| 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | +| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | +| 1. | Tool | A first version of SARP, that runs the above described workflow, without the annotations generator. The tool will cover the described [vulneraribilities](#Vulnerabilities). Annotations need to be added manually. | +| 2. | Engagement | Engage with teams at Web3 Foundation and Parity for prioritization. | + +### Milestone 2 - Annotations Generator +- **Estimated duration:** 1.5 month +- **FTE:** 0.8 +- **Costs:** 37'000 USD + +In the second milestone we implement the annotations generator (needed for step 2.ii. of the workflow process). + +| Number | Deliverable | Specification | +|--------|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on a substrate pallet. | +| 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | +| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | +| 1. | Tool | We will implement an annotations generator and integrate it in SARP. The annotations generator will cover annotations needed for the described [vulneraribilities](#Vulnerabilities). | +| 2. | Engagement | Engage with teams at Web3 Foundation and Parity for prioritization. | + +### Milestone 3 - Complete Tool with Documentation +- **Estimated duration:** 0.5 month +- **FTE:** 1 +- **Costs:** 16'000 USD + +In the final milestone we finetune the tool, making sure it runs on productive pallet code and write the final documentation. -#### Deliverables +| Number | Deliverable | Specification | +|--------|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on substrate pallets. | +| 0c. | Testing and Testing Guide | Each check, that the tool can perform, will be covered by comprehensive unit tests to ensure functionality and robustness. In the testing guide, we will describe how to run these tests.
The tests will further include running SARP on 5 pallets from the [frame pallets](https://github.com/paritytech/substrate/tree/master/frame) | +| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | +| 1. | Tool | We will finetune SARP, with the help of real world examples. The real world examples we will be taken from [substrate's frame pallets](https://github.com/paritytech/substrate/tree/master/frame). | +| 2. | Documentation | Technical documentation of the tool, incl. reasoning on the design decisions. | +| 3. | Engagement | Engage with teams at Web3 Foundation and Parity for prioritization. | -| Number | Deliverable | Specification | -|--------|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0a. | License | MIT | -| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on the examples provided. | -| 0c. | Testing and Testing Guide | We will *not* provide a test suite with this milestone, but documentation on how to run our examples as in 0b. | -| 1. | Tool | We will provide a prototype version of the tool. Following the approach, decided in 3. We will provide examples for applying the tool to 3 of the following 5 vulnerability classes:
  • [incorrect origin](https://github.com/bhargavbh/MIRAI/blob/main/substrate_examples/incorrect-origin/description.md) of dispatchable functions.
  • [unsigned transaction](https://github.com/bhargavbh/MIRAI/blob/main/substrate_examples/unsigned-transaction/description.md) validation.
  • tracking bad randomness: ensure bad randomness does not leak into sensitive functions.
  • detect panics statically to avoid potential DoS attacks: these include [unsafe arithmetic operations](https://github.com/bhargavbh/MIRAI/blob/main/substrate_examples/arithmetic-overflow/description.md), access outside bounds, assertion failures, etc.
  • tracking unsanitised input leakage for sensitive functions.
| -| 2. | Documentation | Technical documentation of the tool, incl. reasoning on the design decisions. | -| 3. | Engagement | We will discuss different solutions and their implications with Web3 Foundation and/or Parity. For this we will document each approach with
  • at least one example, incl. documentation on how to run it.
  • prototype code
  • analysis on the approach's invasiveness
| -| 4. | Analysis of errors in MIRAI | We will document each error we encounter in MIRAI (specifically crashes and faulty analyses) with information on:
  • how to reproduce it
  • reasons for its occurance and implications thereof
This analysis will include the problems we discovered in the previous work package, but haven't analyzed yet (see [1](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples/pallet_template#open-issues) and [2](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples/offchain-worker#open-issues)).

We will report each issue in the [MIRAI repository](https://github.com/facebookexperimental/MIRAI), resp. provide a PR there. | +### Limitations +#### Mocking +Static code analysis, in its nature, is a problem of exponential time complexity. As such, it needs good heuristics to deliver useful results. Our strategy is to mock the library code, that adds the most complexity and is used the most. For this delivery we plan to take an educated guess, on what to mock. Naturally this will not cover all cases. Especially with this first version, we expect that there will be many cases, where a vulnerability is not found or a false positive is returned. +Due to this limitation, we commit ourselves to 5 working real world examples, taken from the [frame pallets](https://github.com/paritytech/substrate/tree/master/frame). That means, the analysis will not crash and provide only few false positives. Note, that this is currently not possible, due to the induced complexity by the substrate macros. +#### Annotations Generator +The annotations generator will generate the needed annotations for the examples we provide for the described [vulnerabilities](#vulnerabilities). We expect, that there will be situations, where the user has to add annotations themselves. ## Future Plans -1. Decide on vulnerabilities for an MVP. +1. Specify and prioritize vulnerabilities for roadmap.
For this we plan to engage with Web3 Foundation / Parity and auditing companies such as [OtterSec](https://osec.io/) or [FYEO](https://www.fyeo.io/). -2. Implement a first simple version of the tool, together with tests and documentation. 3. Improve the usability, by providing * means to surpress warnings * a comprehensive user tutorial, incl. documentation on the risks of each vulnerability From 49628e4ccb24f4e016286a392bce3e973e3a9d4f Mon Sep 17 00:00:00 2001 From: Sabine Proll Date: Wed, 23 Aug 2023 16:23:53 +0200 Subject: [PATCH 6/8] Deliverables: add example in 0b. User Documentation + removed 2. Engagement --- applications/sarp-2-core-components.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/applications/sarp-2-core-components.md b/applications/sarp-2-core-components.md index e9f511f704d..cce5625754d 100644 --- a/applications/sarp-2-core-components.md +++ b/applications/sarp-2-core-components.md @@ -134,14 +134,14 @@ We plan to do a detailed analysis on interesting vulnerabilities in the [future] With the first milestone we want to do the basic setup of the project, integrate a first set of substrate mocks and deliver a first runnable tool. -| Number | Deliverable | Specification | -|--------|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0a. | License | MIT | -| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on a substrate pallet. | -| 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | -| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | +| Number | Deliverable | Specification | +|--------|-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that shows how the tool can be used on an example pallet, for which we will provide the code. | +| 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | +| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | | 1. | Tool | A first version of SARP, that runs the above described workflow, without the annotations generator. The tool will cover the described [vulneraribilities](#Vulnerabilities). Annotations need to be added manually. | -| 2. | Engagement | Engage with teams at Web3 Foundation and Parity for prioritization. | + ### Milestone 2 - Annotations Generator - **Estimated duration:** 1.5 month @@ -153,11 +153,11 @@ In the second milestone we implement the annotations generator (needed for step | Number | Deliverable | Specification | |--------|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 0a. | License | MIT | -| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on a substrate pallet. | +| 0b. | User Documentation | We will provide a basic **tutorial** that shows how the tool can be used on an example pallet, for which we will provide the code. | | 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | | 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | | 1. | Tool | We will implement an annotations generator and integrate it in SARP. The annotations generator will cover annotations needed for the described [vulneraribilities](#Vulnerabilities). | -| 2. | Engagement | Engage with teams at Web3 Foundation and Parity for prioritization. | + ### Milestone 3 - Complete Tool with Documentation - **Estimated duration:** 0.5 month @@ -169,12 +169,12 @@ In the final milestone we finetune the tool, making sure it runs on productive p | Number | Deliverable | Specification | |--------|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 0a. | License | MIT | -| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on substrate pallets. | +| 0b. | User Documentation | We will provide a basic **tutorial** that shows how the tool can be used on an example pallet, for which we will provide the code. | | 0c. | Testing and Testing Guide | Each check, that the tool can perform, will be covered by comprehensive unit tests to ensure functionality and robustness. In the testing guide, we will describe how to run these tests.
The tests will further include running SARP on 5 pallets from the [frame pallets](https://github.com/paritytech/substrate/tree/master/frame) | | 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | | 1. | Tool | We will finetune SARP, with the help of real world examples. The real world examples we will be taken from [substrate's frame pallets](https://github.com/paritytech/substrate/tree/master/frame). | | 2. | Documentation | Technical documentation of the tool, incl. reasoning on the design decisions. | -| 3. | Engagement | Engage with teams at Web3 Foundation and Parity for prioritization. | + ### Limitations From 510568c48790cb508ceac4240fb91ba9b5b925a1 Mon Sep 17 00:00:00 2001 From: Sabine Proll Date: Wed, 23 Aug 2023 16:32:14 +0200 Subject: [PATCH 7/8] add ink! smart contracts to future plans --- applications/sarp-2-core-components.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/applications/sarp-2-core-components.md b/applications/sarp-2-core-components.md index cce5625754d..4771925544f 100644 --- a/applications/sarp-2-core-components.md +++ b/applications/sarp-2-core-components.md @@ -200,6 +200,8 @@ The annotations generator will generate the needed annotations for the examples Once we have a tool with a good feature set and basic usability features, we want to further promote it to auditors and developers. +Alongside the roadmap for substrate pallets, we plan to investigate, if the same approach can be applied to ink! smart contracts. + ## Additional Information :heavy_plus_sign: From a99280589f1496a939edf804e3e3b635ae108a1a Mon Sep 17 00:00:00 2001 From: Sabine Proll Date: Mon, 28 Aug 2023 10:54:51 +0200 Subject: [PATCH 8/8] clarified deliverable --- applications/sarp-2-core-components.md | 30 ++++++++++++++------------ 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/applications/sarp-2-core-components.md b/applications/sarp-2-core-components.md index 4771925544f..c76b9432695 100644 --- a/applications/sarp-2-core-components.md +++ b/applications/sarp-2-core-components.md @@ -134,13 +134,14 @@ We plan to do a detailed analysis on interesting vulnerabilities in the [future] With the first milestone we want to do the basic setup of the project, integrate a first set of substrate mocks and deliver a first runnable tool. -| Number | Deliverable | Specification | -|--------|-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0a. | License | MIT | -| 0b. | User Documentation | We will provide a basic **tutorial** that shows how the tool can be used on an example pallet, for which we will provide the code. | -| 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | -| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | -| 1. | Tool | A first version of SARP, that runs the above described workflow, without the annotations generator. The tool will cover the described [vulneraribilities](#Vulnerabilities). Annotations need to be added manually. | +| Number | Deliverable | Specification | +|--------|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that shows how the tool can be used on an example pallet, for which we will provide the code. | +| 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | +| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | +| 1. | Tool | A first version of SARP, that runs the above described workflow, without the annotations generator. The tool will cover the described [vulneraribilities](#Vulnerabilities) and their provided examples. Annotations need to be added manually. | +| 2. | Documentation | We will evaluate the usage of SARP against at least 5 pallets from the [frame pallets](https://github.com/paritytech/substrate/tree/master/frame) and document our findings. | ### Milestone 2 - Annotations Generator @@ -150,13 +151,14 @@ With the first milestone we want to do the basic setup of the project, integrate In the second milestone we implement the annotations generator (needed for step 2.ii. of the workflow process). -| Number | Deliverable | Specification | -|--------|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0a. | License | MIT | -| 0b. | User Documentation | We will provide a basic **tutorial** that shows how the tool can be used on an example pallet, for which we will provide the code. | -| 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | -| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | -| 1. | Tool | We will implement an annotations generator and integrate it in SARP. The annotations generator will cover annotations needed for the described [vulneraribilities](#Vulnerabilities). | +| Number | Deliverable | Specification | +|--------|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that shows how the tool can be used on an example pallet, for which we will provide the code. | +| 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | +| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | +| 1. | Tool | We will implement an annotations generator and integrate it in SARP. The annotations generator will cover annotations needed for the described [vulneraribilities](#Vulnerabilities) and their provided examples. | +| 2. | Documentation | We will evaluate the usage of SARP against at least 5 pallets from the [frame pallets](https://github.com/paritytech/substrate/tree/master/frame) and document our findings. | ### Milestone 3 - Complete Tool with Documentation