[Release] v1.1.0 (#95)

* [Spike] Setup `gov-relayer` app (#53) * Add base setup for `gov-relayer` as CLI App * Rename to `proposal-sub` * Start putting things together for `proposal-pub` * Add `proposal-sub` that with new proposal handler * Add requeuing mechanic to the proposal-sub * Fix `lastKnownId` is not updated on the next iteration * Fix compiling error when running `gov-website` and `gov-relayer` on dev * Add basic README details for development * Fix build error * Slim down the `proposal/[pid].tsx` page temporarily to get the build working * Rename `getAMQClient` / `getMDBClient` to `getRabbitClient` and `getMongoClient` * [Spike] Update Discord DM & error messages (#52) * Slight improvement to error messaging * Update discord message * Prefer `response.text()`, fallback to `response.statusText` * [Chore] Tidy up the codebase (#56) * Rename the progress dialog steps * Add check to allow testing the form without having `DISCORD_BOT` defined * Rename to `signAndSend` * Use the new `signAndSend` impl across * Remove `node/models` in favor of `service/mongodb` * Migrate `node/types` to `node/utils` * Migrate `web/types` to `web/utils` * Consolidate all env vars into one service `service/env-vars` * [Hotfix] Wrap error message (#55) * Use `prose` so error text wraps nicely * Apply `prose` to `ProposalNewFormDialog` error message * [Spike] Add proposal activities monitor (#57) * Add `status` as part of the `fetchProposalInfo` * Rename to `gov-relayer:webpack` * Update `waitForBlock` to avoid ignoring the `no-async-promise-executor` rule * Remove the AbortController to make the code looks cleaner with async / await * Add `monitorProposalActivity` to publish `proposal-activity` message * Add `handleProposalActivityMessage` to update Deliberation votes * Add handler for `ReferendumDeliberation` status * [Spike] Add a placeholder for resolving justification content (#58) * Add function to resolve justification content from the Uri * Add comment block * Add `safeFetch` method * Remove the `npm-run-all` as dependency for runtime (#64) Remove the `npm-run-all` as dependency for for runtime * [Spike] Re-style Proposal Single page (#62) * Add `ProposalBody` component and add it to single page view * Add `ProposalSidebar` * Simplify the submit button label * Add `ProposalVoteForm` * Chang FormState.step to `Sign` * Add `useProposalVoteForm` * Add `ProposalVoteFormDialog` * Fix `AccountSelect` doesn't select the initial value * Ensure form is validated before submit the action * Remove `ProposalDetailsDisplay` * Add `votePercentage` and have it synced in the DB * Add `VotesInfo` to `ProposalSidebar` * Add `VetoInfo` * Round up the percentage number * Use `BLOCK_POLLING_INTERVAL` to control the refreshing rate * Make adjustment for smaller viewport * [Fix] Make `resolveProposalJustification` safer with white-listing domains (#66) * [Fix] Remove the `rejectVotes` out of `votePercentage` calculation (#67) * [Spike] Send proposals & referendums to Discord (#60) * Update `env.skel`, tidy up * Port over `getDiscordWebhook` from `gov-services` * Pass `discordWebhook` to `handleProposalActivityMessage` * Port over `getDiscordMessage` from `gov-services` * Update `getDiscordMessage` to work for both Proposals & Referendums * Fetch discord webhook for each channel * Fix logic to pick which `voteFields` are used * Update `DISCORD_RELAYER_BOT` const * Update `ProposalModel` * Fetch discord webhooks and pass to `handleProposalActivityMessage` * Update Discord with votes & status * Prettier format * Refactor * Use `resolveProposalJustification`, resolve circular dependencies * Fix build error * Avoid build fail from bad `justificationUri` * Improve typing for `getDiscordWebhooks` * Refactor sidebar colours, only display `proposalDetails` if they can be fetched successfully * Rename `service/relayer/utils` -> `service/relayer/src` * Update imports * Refactor `getProposalFields` * Prettier format * Temporary fix for `ipfs//` causing `HTTP/400` error and failing build * Fix pinata gateway * Ensure all necessary data is being passed to `getDiscordMessage` * Fetch veto threshold * Pass vote info in case of no change to avoid sending `undefined` to Discord * Display votes as a percentage * Avoid updating Discord message ids with empty string * Get `discordWebhooks` once * Revert to `while` loop * Fix typo * Get different discord message per channel * Prettier format * Refactor / tidy up * Refactor `getProposalEmbed`, `getVoteButton` * Regenerate `service-discord` package using `@nrwl/js` * Only log if update found * Regenerate `service-relayer` library with `@nrwl/js` * Move `DISCORD_` vars to `//DISCORD` partition * Move `assignDiscordRole` to `service-discord` * Check for & reject on `getDiscordBot` error * Update `getDiscordWebhooks` to accept arrays of channel & webhook ids * [Hotfix] Add `discord.js` to `gov-relayer#dependencies` (#69) Add `discord.js` to `gov-relayer#dependencies` * [Spike] Proposal updates (#70) * Extract `getHourInBlocks` to `node-utils` * Add `48 hours` option for `enactmentDelay` * Display `enactmentDelayInHours` on Discord * Add `github.com` into the `justificationUri` white list * Allow certain extrinsics, allow extrinsics that don't require args * Refactor `getProposalFields` * Update `service-relayer` tsconfig * Remove vote button if proposal is not in voting stage * Add `Pending` colour for `ApprovedWaitingEnactment` stage * Refactor `transformFormData` * Display `enactmentDelayInHours` for Proposal single view * [Spike] Omit `system.setCode` args (#75) * Catch error when decoding `setCode` proposal call and define `args` as omitted * Use `toFixed` to round `enactmentDelayInHours` for cases where proposal is submitted through portal * Refactor `call` catch block * Omit args if none, or if `omitted` i.e. for `system.setCode` * Display `omitted` args * Update libs/service/cennznet/src/fetchProposalInfo.ts Co-authored-by: Ken Vu <[email protected]> * Round `enactmentDelayInHours` with `toFixed` Co-authored-by: Ken Vu <[email protected]> * [Hotfix] Avoid property `justificationUri` is undefined error (#77) * Avoid property `justificationUri` is undefined error * Return early if no `justificationUri` * [Fix] Relayer performance issue (#78) * Avoid requeue messages for `proposal-activity` type Since we have to loop through all proposals and check for updates in every 2 blocks anyway * Always create a record first to avoid the same proposal is being processed again * Fix spacing for `update Discord` message * [Fix] Add extra guard to ignore proposals that doesn't have `status` (#80) Add extra guard to ignore building any proposals that doesn't have `status` * [Fix] Add `health check` log (#81) Add `health check` log * [Fix] Styling updates (#84) * Soften the shadow colors on Button and Dialog * Add `favicon` * Add the default meta description * [Fix] API `autoConnect` (#83) Add a regular check to make sure api is connected, if not throw error * [Spike] Proposal changes (#85) * Reduce revalidation time to 5 minutes * Update default function call * Hide `Arguments` section if function call has no arguments * [Spike] Throw `create type` error if not handled (#86) Throw `create type` error if not handled * [Fix] Inactive proposal causes app to crash (#88) * Prevent page crash on inactive proposal * Redirect user to `proposal/new` if their vote causes proposal to end * Fix build error * Add console warning on failure to format `call` * Inform user they will be redirected and why * Return early if `call` is empty * Revert "Redirect user to `proposal/new` if their vote causes proposal to end" This reverts commit 45a84fa. * Don't refetch proposal info if proposal is finalized * Revert "Inform user they will be redirected and why" * [Spike] Use `safeFetch` across code base (#79) Use `safeFetch` across the repo * Add a `proposal-patch <start> <end>` command to patch up DB (#91) * [Spike] Proposal copy & link updates (#92) * Update proposal submission copy * Link directly to appropriate channel after voting * Update env vars * Revert change to `DISCORD_RELAYER_CHANNEL_IDS` * Prettier format * [Spike] Refactor discord messaging (#93) Extract discord messaging to own function * [Chore] Setup GitHub actions (#94) * Add workflow when submit a PR * Fix lint issues * Reset version in `develop` branch * Add `nrwl/nx-set-shas` to make `yarn nx affected` work * Bump to version `v1.1.0` Co-authored-by: Aidan Starke <[email protected]>
cennznet · Jul 27, 2022 · 7ddebad · 7ddebad · vercel · Jul 27, 2022
1 parent bbea419
commit 7ddebad
Show file tree

Hide file tree

Showing 178 changed files with 3,695 additions and 920 deletions.
diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml
@@ -0,0 +1,47 @@
+name: PR
+
+on:
+  push:
+    branches:
+      - develop
+      - "prod/**"
+  pull_request:
+    branches:
+      - develop
+      - "prod/**"
+
+  workflow_dispatch:
+
+jobs:
+  lint:
+    name: Lint
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        node-version: [16.x]
+    steps:
+      - uses: actions/checkout@v1
+      - uses: styfle/[email protected]
+        with:
+          access_token: ${{ github.token }}
+      - name: Use Node.js ${{ matrix.node-version }}
+        uses: actions/setup-node@v1
+        with:
+          node-version: ${{ matrix.node-version }}
+      - name: Config yarn cache
+        id: yarn-cache-dir-path
+        run: echo "::set-output name=dir::$(yarn cache dir)"
+      - uses: actions/cache@v1
+        id: yarn-cache
+        with:
+          path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
+          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-yarn-
+      - uses: nrwl/nx-set-shas@v2
+        with:
+          main-branch-name: ${{ github.base_ref }}
+      - name: lint
+        run: |
+          yarn install --immutable
+          yarn nx affected --target=lint
diff --git a/.gitignore b/.gitignore
@@ -49,3 +49,5 @@ Thumbs.db
 
 .mongodb
 .rabbitmq
+
+logs
diff --git a/README.md b/README.md
@@ -1 +1,42 @@
 # App Gov
+
+- [Environment](#environments)
+  - [Development](#development)
+- [Deployment](#deployment)
+
+## Environments
+
+### Development
+
+##### Initial Setup
+
+- Run `yarn install` to install the dependencies
+
+##### Run `gov-website`
+
+> **URL:** http://localhost:4200
+
+```
+yarn gov-website:dev
+```
+
+##### Run `gov-relayer`
+
+```
+# start docker
+yarn gov-relayer:docker
+
+# start webpack
+yarn gov-relayer:webpack
+
+# start the CLI
+yarn relayer:run
+```
+
+## Deployment
+
+### `gov-relayer`
+
+- Ensure `apps/gov-relayer/package.json#dependencies` is up-to-date with the code base
+- Run `yarn gov-relayer:build`
+- The folder `dist/apps/gov-relayer` is ready to be copied over to a container
diff --git a/apps/gov-relayer/.env.skel b/apps/gov-relayer/.env.skel
@@ -0,0 +1,12 @@
+MONGODB_URI=
+RABBITMQ_URI=	
+
+NX_CENNZ_NETWORK=
+NX_PROPOSAL_URL=
+NX_PINATA_GATEWAY=
+
+DISCORD_BOT_TOKEN=
+DISCORD_PROPOSAL_WEBHOOK_ID=
+DISCORD_REFERENDUM_WEBHOOK_ID=
+NX_DISCORD_PROPOSAL_CHANNEL_ID=
+NX_DISCORD_REFERENDUM_CHANNEL_ID=
diff --git a/apps/gov-relayer/.eslintrc.json b/apps/gov-relayer/.eslintrc.json
@@ -0,0 +1,18 @@
+{
+  "extends": ["../../.eslintrc.json"],
+  "ignorePatterns": ["!**/*"],
+  "overrides": [
+    {
+      "files": ["*.ts", "*.tsx", "*.js", "*.jsx"],
+      "rules": {}
+    },
+    {
+      "files": ["*.ts", "*.tsx"],
+      "rules": {}
+    },
+    {
+      "files": ["*.js", "*.jsx"],
+      "rules": {}
+    }
+  ]
+}
diff --git a/apps/gov-relayer/commands/proposal-patch.ts b/apps/gov-relayer/commands/proposal-patch.ts
@@ -0,0 +1,62 @@
+import { AMQPError } from "@cloudamqp/amqp-client";
+import chalk from "chalk";
+
+import {
+	CENNZ_NETWORK,
+	MONGODB_URI,
+	PROPOSAL_QUEUE,
+	RABBITMQ_URI,
+} from "@app-gov/service/env-vars";
+import { getMongoClient, ProposalModel } from "@app-gov/service/mongodb";
+import { getQueueByName, getRabbitClient } from "@app-gov/service/rabbitmq";
+import { getLogger } from "@app-gov/service/relayer";
+
+module.exports = {
+	command: "proposal-patch <start> <end>",
+	desc: "Start a one-off [ProposalPatch] script to fill in missing proposals",
+	async handler(args) {
+		const logger = getLogger("ProposalPatch");
+		logger.info(
+			`Start process on ${chalk.magenta("%s")}...`,
+			CENNZ_NETWORK.ChainName
+		);
+
+		try {
+			const [amqClient, mdbClient] = await Promise.all([
+				getRabbitClient(RABBITMQ_URI),
+				getMongoClient(MONGODB_URI),
+			]);
+
+			const proposalQueue = await getQueueByName(
+				amqClient,
+				"Producer",
+				PROPOSAL_QUEUE
+			);
+
+			const { start, end } = args;
+			for (let i = start; i <= end; i++) {
+				const proposalId = i;
+				const proposal = await mdbClient
+					.model<ProposalModel>("Proposal")
+					.findOne({ proposalId });
+				if (proposal?.status) {
+					logger.info(
+						"Proposal #%d: 🔴 ignored, status `%s`",
+						proposalId,
+						proposal.status
+					);
+					continue;
+				}
+				await proposalQueue.publish(JSON.stringify({ proposalId }), {
+					type: "proposal-new",
+				});
+				logger.info("Proposal #%d: 🟢 added", proposalId);
+			}
+			process.exit(0);
+		} catch (error) {
+			if (error instanceof AMQPError) error?.connection?.close();
+			logger.error("%s", error);
+			process.exit(1);
+		}
+	},
+};
diff --git a/apps/gov-relayer/commands/proposal-pub.ts b/apps/gov-relayer/commands/proposal-pub.ts
@@ -0,0 +1,71 @@
+import { AMQPError } from "@cloudamqp/amqp-client";
+import chalk from "chalk";
+
+import { getApiInstance, waitForBlock } from "@app-gov/service/cennznet";
+import {
+	BLOCK_POLLING_INTERVAL,
+	CENNZ_NETWORK,
+	MONGODB_URI,
+	PROPOSAL_QUEUE,
+	RABBITMQ_URI,
+} from "@app-gov/service/env-vars";
+import { getMongoClient } from "@app-gov/service/mongodb";
+import { getQueueByName, getRabbitClient } from "@app-gov/service/rabbitmq";
+import {
+	getLogger,
+	monitorNewProposal,
+	monitorProposalActivity,
+} from "@app-gov/service/relayer";
+
+module.exports = {
+	command: "proposal-pub",
+	desc: "Start a [ProposalPub] process",
+	async handler() {
+		const logger = getLogger("ProposalPub");
+		logger.info(
+			`Start process on ${chalk.magenta("%s")}...`,
+			CENNZ_NETWORK.ChainName
+		);
+
+		try {
+			const [cennzApi, amqClient, mdbClient] = await Promise.all([
+				getApiInstance(CENNZ_NETWORK.ChainSlug),
+				getRabbitClient(RABBITMQ_URI),
+				getMongoClient(MONGODB_URI),
+			]);
+
+			const proposalQueue = await getQueueByName(
+				amqClient,
+				"Producer",
+				PROPOSAL_QUEUE
+			);
+
+			const polling = true;
+
+			do {
+				await monitorNewProposal(cennzApi, mdbClient, (proposalId) => {
+					proposalQueue.publish(JSON.stringify({ proposalId }), {
+						type: "proposal-new",
+					});
+				});
+
+				await monitorProposalActivity(mdbClient, (proposalId) => {
+					proposalQueue.publish(JSON.stringify({ proposalId }), {
+						type: "proposal-activity",
+					});
+				});
+
+				await waitForBlock(cennzApi, BLOCK_POLLING_INTERVAL, (blockNumber) => {
+					logger.info(
+						`Health check: 👌 ${chalk.green("ok")} @ ${chalk.gray("%s")}`,
+						blockNumber
+					);
+				});
+			} while (polling);
+		} catch (error) {
+			if (error instanceof AMQPError) error?.connection?.close();
+			logger.error("%s", error);
+			process.exit(1);
+		}
+	},
+};
diff --git a/apps/gov-relayer/commands/proposal-sub.ts b/apps/gov-relayer/commands/proposal-sub.ts
@@ -0,0 +1,104 @@
+import { AMQPError, AMQPMessage } from "@cloudamqp/amqp-client";
+import chalk from "chalk";
+
+import { getApiInstance } from "@app-gov/service/cennznet";
+import { getDiscordWebhooks } from "@app-gov/service/discord";
+import {
+	CENNZ_NETWORK,
+	DISCORD_CHANNEL_IDS,
+	DISCORD_RELAYER_BOT,
+	DISCORD_WEBHOOK_IDS,
+	MESSAGE_MAX_RETRY,
+	MONGODB_URI,
+	PROPOSAL_QUEUE,
+	RABBITMQ_URI,
+} from "@app-gov/service/env-vars";
+import { getMongoClient } from "@app-gov/service/mongodb";
+import {
+	getQueueByName,
+	getRabbitClient,
+	requeueMessage,
+} from "@app-gov/service/rabbitmq";
+import {
+	getLogger,
+	handleNewProposalMessage,
+	handleProposalActivityMessage,
+} from "@app-gov/service/relayer";
+
+module.exports = {
+	command: "proposal-sub",
+	desc: "Start a [ProposalSub] process",
+	async handler() {
+		const logger = getLogger("ProposalSub");
+		logger.info(
+			`Start process on ${chalk.magenta("%s")}...`,
+			CENNZ_NETWORK.ChainName
+		);
+
+		try {
+			const [cennzApi, amqClient, mdbClient] = await Promise.all([
+				getApiInstance(CENNZ_NETWORK.ChainSlug),
+				getRabbitClient(RABBITMQ_URI),
+				getMongoClient(MONGODB_URI),
+			]);
+
+			const proposalQueue = await getQueueByName(
+				amqClient,
+				"Consumer",
+				PROPOSAL_QUEUE
+			);
+
+			const discordWebhooks = await getDiscordWebhooks(
+				DISCORD_RELAYER_BOT.Token,
+				DISCORD_CHANNEL_IDS,
+				DISCORD_WEBHOOK_IDS
+			);
+
+			const onMessage = async (message: AMQPMessage) => {
+				const bodyString = message.bodyString();
+				const { type } = message.properties;
+				if (!bodyString || !type) return;
+				const body = JSON.parse(bodyString);
+
+				try {
+					switch (type) {
+						case "proposal-new":
+							await handleNewProposalMessage(cennzApi, mdbClient, body);
+							break;
+
+						case "proposal-activity":
+							await handleProposalActivityMessage(
+								cennzApi,
+								discordWebhooks,
+								mdbClient,
+								body
+							);
+							break;
+					}
+				} catch (error) {
+					logger.error("%s", error);
+					const { type } = message.properties;
+					if (type === "proposal-new") {
+						const result = await requeueMessage(
+							proposalQueue,
+							message,
+							MESSAGE_MAX_RETRY
+						);
+						logger.info(`Proposal #%d: ${result}`, body.proposalId);
+					}
+
+					if (type === "proposal-activity") {
+						logger.info(`Proposal #%d: ❌ skipped`, body.proposalId);
+					}
+				}
+				await message.ack();
+			};
+
+			proposalQueue.subscribe({ noAck: false }, onMessage);
+		} catch (error) {
+			if (error instanceof AMQPError) error?.connection?.close();
+			logger.error("%s", error);
+			process.exit(1);
+		}
+	},
+};
diff --git a/apps/gov-relayer/jest.config.ts b/apps/gov-relayer/jest.config.ts
@@ -0,0 +1,16 @@
+/* eslint-disable */
+export default {
+	displayName: "gov-relayer",
+	preset: "../../jest.preset.js",
+	globals: {
+		"ts-jest": {
+			tsconfig: "<rootDir>/tsconfig.spec.json",
+		},
+	},
+	testEnvironment: "node",
+	transform: {
+		"^.+\\.[tj]s$": "ts-jest",
+	},
+	moduleFileExtensions: ["ts", "js", "html"],
+	coverageDirectory: "../../coverage/apps/gov-relayer",
+};
diff --git a/apps/gov-relayer/main.ts b/apps/gov-relayer/main.ts
@@ -0,0 +1,9 @@
+/* eslint-disable @typescript-eslint/no-var-requires */
+require("dotenv").config({ path: `${__dirname}/.env` });
+
+require("yargs")
+	.scriptName("gov-relayer")
+	.command(require("./commands/proposal-pub"))
+	.command(require("./commands/proposal-sub"))
+	.command(require("./commands/proposal-patch"))
+	.help().argv;