Development 3botdeployer (#1441)

* Add location choice * Block the chatflow until the farms are ready * Add location policy * Fetch farm name early * Fix pool selection when forcing specific node * Use drop down menu instead of single choice * Create wallet on entry * working on adding 3bot status * Create a new wallet on 3bot creation * Rename the wallet to mainnet_wallet * Move wallet creation to threebot start * Log wallet creation errors and continue starting * Don't discriminate between self hosted and chatflow hosted 3Bots * update backup path related to restic server * Deploy on the new identity * add flavors for deployer * override the flavors * Create 3bot on new pool and new managed network * add destroy, delete and restart actions * add /root/.ssh to restic backup paths * add the deployer actions depending on the status * update methods with identity_name wip * separate 3bot deployers in separate component * refactor models dir * fix saved subdomain id * fix instance name * working on the frontend for listin the 3bots * use mainnet default * fixing and issue related to back /.ssh path * threebot listing wip * check for workload container type in building info * remove debugging prints * expired pools have empty_at at max int64 * add stop and delete threebot solutions * check if metadata exists before trying to decrypt it * fix typos * set owner key in networks metadata updated by the admin dashboard * move threebot config validation to a separate method * remove unused statement * address comments * deploy on mainnet * Development 3botdeployer escalation emails (#1478) * Add escalation emails button in developer options in fron-end and update the back-end * Add escalation email front-end and back-end * Handle the None stat in config manager related to escalating emails * fix invalid import * use management as name for networks 3bot deployment * pass identity to dry_run context * use new listing util in threebot chatflow * fix network name * return expiration * Update front-end and back end to use the new methods in listing the threebot states * add start method in utils * Restart 3Bot wip * Adds correct actions for stop / destroy threebot for threebot deployer Signed-off-by: Ashraf Fouda <[email protected]> * Fix steps missing comma * Add final step * Add some hardening for threebot deletion and stopping * clean up threebot identity * updated frontend view * delete only deployed workloads * add change size and location chatflows * receive 3bot name from params * clean up unused identity * fix log config * fix log config * fix message * remove extend * clear workloads in deployment_context before deleting threebot identity * show the correct empty at in blocked nodes * one table for all workloads * Add state filter in deployed 3bots UI * show bell icon and extend message and remove expiration column * update quick start guide * update quick start guide * change expiration icon * show href in info if the threebot is running * remove not needed inof when threebot is not running * fix error message * Create wallets when deploying new threebot * remove unused actor methods * override the flist with latest flist on redeployments * override the flist with latest flist on redeployments * fix api url and remove deprecated apis * return status 400 instead of raising exception * fix cancel solution for other identities * update branch in redeployments * open url in another tab * pass identity name to metadata encryption & decryption in delete_access * use root instead of homedir * change variable name * change ssh dir path based on the current user running 3bot server * fix message * revert ssh dir * respect over provisioning in farm capacity check * remove extend test * use identity name when decrypting metadata * Development 3botdeployer autoextend pool (#1526) auto-extend pools * Use std instead of default identity for the main wallet * update threebot tests * remove conflicts * remove recover and extend tests * Update email server config in admin dashboard vue to include tooltips * add threebot start & change size gedis patches * add threebot change location patch * add threebot change location patch * init test cases for threebot deployer * Make development as the default branch for threebot deployer Co-authored-by: Omar Elawady <[email protected]> Co-authored-by: Rafy Benjamin <[email protected]> Co-authored-by: m-motawea <[email protected]> Co-authored-by: RafyAmgadBenjamin <[email protected]> Co-authored-by: Ashraf Fouda <[email protected]> Co-authored-by: ranatrk <[email protected]> Co-authored-by: Waleed <[email protected]>
threefoldtech · Oct 29, 2020 · d3c4653 · d3c4653
1 parent fa0082e
commit d3c4653
Show file tree

Hide file tree

Showing 54 changed files with 2,810 additions and 687 deletions.
diff --git a/jumpscale/entry_points/threebot.py b/jumpscale/entry_points/threebot.py
@@ -11,6 +11,7 @@
 from jumpscale.loader import j
 from jumpscale.threesdk.identitymanager import IdentityManager
 from jumpscale.sals.nginx.nginx import PORTS
+from jumpscale.packages.admin.actors.wallet import Wallet
 
 SERVICES_PORTS = {"nginx": 8999, "nginx_http": 80, "nginx_https": 443, "gedis": 16000}
 
@@ -80,6 +81,9 @@ def start(identity=None, background=False, local=False, development=False, domai
         identity (str, optional): threebot name. Defaults to None.
         explorer (str, optional): which explorer network to use: mainnet, testnet, devnet. Defaults to None.
     """
+    if j.config.get("ANNOUNCED") is None:
+        j.config.set("ANNOUNCED", False)
+    create_wallets_if_not_exists()
     check_for_bins()
     PORTS.init_default_ports(local)
     SERVICES_PORTS["nginx_http"] = PORTS.HTTP
@@ -227,7 +231,7 @@ def clean(all=False):
                 print(f"exception was {e} for debugging")
 
             answer = j.tools.console.ask_yes_no(f"Do you want to remove {config_root} ? ")
-            if answer=="y":
+            if answer == "y":
                 j.sals.fs.rmtree(config_root)
                 print("Previous configuration is deleted.")
 
@@ -236,12 +240,44 @@ def clean(all=False):
             print(f"exception for debugging {e}")
 
 
-
 @click.group()
 def cli():
     pass
 
 
+def have_wallets():
+    wallets = j.clients.stellar.list_all()
+    test, main = False, False
+    for wallet_name in wallets:
+        wallet = j.clients.stellar.get(wallet_name)
+        if wallet.network.value == "TEST":
+            test = True
+        elif wallet.network.value == "STD":
+            main = True
+    return test, main
+
+
+def create_test_wallet(wallet_name):
+    try:
+        j.clients.stellar.create_testnet_funded_wallet(wallet_name)
+    except Exception as e:
+        j.logger.error(str(e))
+
+
+def create_main_wallet(wallet_name):
+    wallet_actor = Wallet()
+    try:
+        wallet_actor.create_wallet(wallet_name, "STD")
+    except Exception as e:
+        j.logger.error(str(e))
+
+
+def create_wallets_if_not_exists():
+    test, main = have_wallets()
+    if not test:
+        create_test_wallet("test")
+    if not main:
+        create_main_wallet("main")
 
 
 cli.add_command(start)

diff --git a/jumpscale/packages/admin/actors/admin.py b/jumpscale/packages/admin/actors/admin.py
@@ -185,32 +185,46 @@ def get_developer_options(self) -> str:
         test_cert = j.core.config.set_default("TEST_CERT", False)
         over_provision = j.core.config.set_default("OVER_PROVISIONING", False)
         explorer_logs = j.core.config.set_default("EXPLORER_LOGS", False)
+        escalation_emails = j.core.config.set_default("ESCALATION_EMAILS_ENABLED", False)
+        auto_extend_pools = j.core.config.set_default("AUTO_EXTEND_POOLS_ENABLED", False)
         sort_nodes_by_sru = j.core.config.set_default("SORT_NODES_BY_SRU", False)
         return j.data.serializers.json.dumps(
             {
                 "data": {
                     "test_cert": test_cert,
                     "over_provision": over_provision,
                     "explorer_logs": explorer_logs,
+                    "escalation_emails": escalation_emails,
+                    "auto_extend_pools": auto_extend_pools,
                     "sort_nodes_by_sru": sort_nodes_by_sru,
                 }
             }
         )
 
     @actor_method
     def set_developer_options(
-        self, test_cert: bool, over_provision: bool, explorer_logs: bool, sort_nodes_by_sru: bool
+        self,
+        test_cert: bool,
+        over_provision: bool,
+        explorer_logs: bool,
+        sort_nodes_by_sru: bool,
+        escalation_emails: bool,
+        auto_extend_pools: bool,
     ) -> str:
         j.core.config.set("TEST_CERT", test_cert)
         j.core.config.set("OVER_PROVISIONING", over_provision)
         j.core.config.set("EXPLORER_LOGS", explorer_logs)
+        j.core.config.set("ESCALATION_EMAILS_ENABLED", escalation_emails)
+        j.core.config.set("AUTO_EXTEND_POOLS_ENABLED", auto_extend_pools)
         j.core.config.set("SORT_NODES_BY_SRU", sort_nodes_by_sru)
         return j.data.serializers.json.dumps(
             {
                 "data": {
                     "test_cert": test_cert,
                     "over_provision": over_provision,
                     "explorer_logs": explorer_logs,
+                    "escalation_emails": escalation_emails,
+                    "auto_extend_pools": auto_extend_pools,
                     "sort_nodes_by_sru": sort_nodes_by_sru,
                 }
             }
@@ -222,6 +236,42 @@ def clear_blocked_nodes(self) -> str:
         return j.data.serializers.json.dumps({"data": "blocked nodes got cleared successfully."})
 
     @actor_method
+    def get_email_server_config(self) -> str:
+        email_server_config = j.core.config.get("EMAIL_SERVER_CONFIG", {})
+        email_server_config.setdefault("host", "")
+        email_server_config.setdefault("port", "")
+        email_server_config.setdefault("username", "")
+        email_server_config.setdefault("password", "")
+        return j.data.serializers.json.dumps({"data": email_server_config})
+
+    @actor_method
+    def set_email_server_config(self, host="", port="", username="", password="") -> str:
+        email_server_config = j.core.config.get("EMAIL_SERVER_CONFIG", {})
+        email_server_config = {"host": host, "port": port, "username": username, "password": password}
+        j.core.config.set("EMAIL_SERVER_CONFIG", email_server_config)
+        return j.data.serializers.json.dumps({"data": email_server_config})
+
+    @actor_method
+    def list_escalation_emails(self) -> str:
+        escalation_emails = j.core.config.get("ESCALATION_EMAILS", [])
+        return j.data.serializers.json.dumps({"data": escalation_emails})
+
+    @actor_method
+    def add_escalation_email(self, email) -> str:
+        escalation_emails = j.core.config.get("ESCALATION_EMAILS", [])
+        if email not in escalation_emails:
+            escalation_emails.append(email)
+            j.core.config.set("ESCALATION_EMAILS", escalation_emails)
+        return j.data.serializers.json.dumps({"data": escalation_emails})
+
+    @actor_method
+    def delete_escalation_email(self, email) -> str:
+        escalation_emails = j.core.config.get("ESCALATION_EMAILS", [])
+        if email in escalation_emails:
+            escalation_emails.remove(email)
+            j.core.config.set("ESCALATION_EMAILS", escalation_emails)
+        return j.data.serializers.json.dumps({"data": escalation_emails})
+
     def get_notifications(self) -> str:
         notifications = []
         if j.tools.notificationsqueue.count() >= 10:

diff --git a/jumpscale/packages/admin/actors/wallet.py b/jumpscale/packages/admin/actors/wallet.py
@@ -5,11 +5,13 @@
 
 class Wallet(BaseActor):
     @actor_method
-    def create_wallet(self, name: str) -> str:
+    def create_wallet(self, name: str, wallettype: str = None) -> str:
         explorer = j.core.identity.me.explorer
-        wallettype = "STD"
-        if "testnet" in explorer.url or "devnet" in explorer.url:
-            wallettype = "TEST"
+        if wallettype is None:
+            if "testnet" in explorer.url or "devnet" in explorer.url:
+                wallettype = "TEST"
+            else:
+                wallettype = "STD"
 
         if j.clients.stellar.find(name):
             raise j.exceptions.Value(f"Wallet {name} already exists")

diff --git a/jumpscale/packages/admin/bottle/admin.py b/jumpscale/packages/admin/bottle/admin.py
@@ -8,6 +8,7 @@
 
 app = Bottle()
 
+
 @app.route("/api/allowed", method="GET")
 @login_required
 def allowed():
@@ -44,4 +45,23 @@ def accept():
         )
 
 
+@app.route("/api/announced", method="GET")
+@login_required
+def announced():
+    result = bool(j.config.get("ANNOUNCED"))
+
+    return HTTPResponse(
+        j.data.serializers.json.dumps({"announced": result}), status=200, headers={"Content-Type": "application/json"}
+    )
+
+
+@app.route("/api/announce", method="GET")
+@login_required
+def announce():
+    j.config.set("ANNOUNCED", True)
+    return HTTPResponse(
+        j.data.serializers.json.dumps({"announced": True}), status=200, headers={"Content-Type": "application/json"}
+    )
+
+
 app = SessionMiddleware(app, SESSION_OPTS)
diff --git a/jumpscale/packages/admin/frontend/App.vue b/jumpscale/packages/admin/frontend/App.vue
@@ -136,6 +136,33 @@
       <identities v-model="dialogs.identity"></identities>
       <popup></popup>
     </v-main>
+     <v-dialog
+      v-model="announcement_dialog"
+      persistent
+      max-width="500"
+    >
+
+      <v-card>
+        <v-card-title class="headline">
+          Quick start guide
+        </v-card-title>
+        <v-card-text>
+        We've created two wallets for you; main and test. These are to be used for mainnet and testnet respectively. Make sure your wallets are funded to extend your 3Bots before they expire.
+        <br />
+        Please visit the <a href="https://manual.threefold.io">manual</a> for more information.
+        </v-card-text>
+        <v-card-actions>
+          <v-spacer></v-spacer>
+          <v-btn
+            color="green darken-1"
+            text
+            @click="announced = true"
+          >
+            Ok
+          </v-btn>
+        </v-card-actions>
+      </v-card>
+    </v-dialog>
   </v-app>
 </template>
 
@@ -171,14 +198,17 @@ module.exports = {
       dialogs: {
         identity: false,
       },
+      announced: true
     };
   },
   components: {
     identities: httpVueLoader("./Identity.vue"),
   },
-  computed: {},
   methods: {},
   computed: {
+    announcement_dialog() {
+      return !this.announced
+    },
     pages() {
       return this.$router.options.routes.filter((page) => {
         return page.meta.listed;
@@ -197,6 +227,13 @@ module.exports = {
         this.user = response.data;
       });
     },
+    getAnnouncementStatus() {
+      this.$api.announcement.announced().then((response) => {
+        console.log(response.data)
+        this.announced = response.data["announced"];
+        this.$api.announcement.announce();
+      });
+    },
     getIdentity() {
       this.$api.identity.get().then((response) => {
         this.identity = JSON.parse(response.data);
@@ -237,6 +274,7 @@ module.exports = {
     this.checkDarkMode();
     this.getIdentity();
     this.getCurrentUser();
+    this.getAnnouncementStatus();
     this.setTimeLocal();
     this.clockInterval = setInterval(() => {
       this.setTimeLocal();

diff --git a/jumpscale/packages/admin/frontend/api.js b/jumpscale/packages/admin/frontend/api.js
@@ -181,12 +181,12 @@ const apiClient = {
                 url: `${baseURL}/admin/get_developer_options`
             })
         },
-        setDeveloperOptions: (testCert, overProvision, explorerLogs, sortNodesBySRU) => {
+        setDeveloperOptions: (testCert, overProvision, explorerLogs, escalationEmails, autoExtendPools, sortNodesBySRU) => {
             return axios({
                 url: `${baseURL}/admin/set_developer_options`,
                 method: "post",
                 headers: { 'Content-Type': 'application/json' },
-                data: { test_cert: testCert, over_provision: overProvision, explorer_logs: explorerLogs, sort_nodes_by_sru: sortNodesBySRU }
+                data: { test_cert: testCert, over_provision: overProvision, explorer_logs: explorerLogs, sort_nodes_by_sru: sortNodesBySRU, escalation_emails: escalationEmails, auto_extend_pools: autoExtendPools }
             })
         },
         clearBlockedNodes: () => {
@@ -205,6 +205,50 @@ const apiClient = {
             })
         }
     },
+    emailServerConfig: {
+        get: () => {
+            return axios({
+                url: `${baseURL}/admin/get_email_server_config`
+            })
+        },
+        set: (host, port, username, password) => {
+            return axios({
+                url: `${baseURL}/admin/set_email_server_config`,
+                method: "post",
+                headers: { "Content-Type": "application/json" },
+                data: {
+                    host: host,
+                    port: port,
+                    username: username,
+                    password: password
+                }
+            })
+        },
+    },
+    escalationEmails: {
+        list: () => {
+            return axios({
+                url: `${baseURL}/admin/list_escalation_emails`
+            })
+        },
+        add: (email) => {
+            return axios({
+                url: `${baseURL}/admin/add_escalation_email`,
+                method: "post",
+                headers: { 'Content-Type': 'application/json' },
+                data: { email: email }
+            })
+        },
+        delete: (email) => {
+            return axios({
+                url: `${baseURL}/admin/delete_escalation_email`,
+                method: "post",
+                headers: { 'Content-Type': 'application/json' },
+                data: { email: email }
+            })
+        }
+
+    },
     explorers: {
         get: () => {
             return axios({
@@ -534,5 +578,19 @@ const apiClient = {
                 method: "get"
             })
         },
+    },
+    announcement: {
+        announced: () => {
+            return axios({
+                url: `/admin/api/announced`,
+                method: "get"
+            })
+        },
+        announce: () => {
+            return axios({
+                url: `/admin/api/announce`,
+                method: "get"
+            })
+        },
     }
 }