Merge pull request #249 from guydavis/integration

v0.5.3 release

Author: guydavis

.github/workflows/develop.yaml

@@ -39,7 +39,7 @@ jobs:
           platforms: linux/amd64
           push: true
           build-args: |
-            "CHIA_BRANCH=1.2.3"
+            "CHIA_BRANCH=1.2.4"
             "FLAX_BRANCH=main"
           tags: |
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris:develop

.github/workflows/main.yaml

@@ -40,7 +40,7 @@ jobs:
           platforms: linux/amd64,linux/arm64
           push: true
           build-args: |
-            "CHIA_BRANCH=1.2.3"
+            "CHIA_BRANCH=1.2.4"
             "FLAX_BRANCH=main"
           tags: |
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris:latest

.github/workflows/test.yaml

@@ -39,7 +39,7 @@ jobs:
           platforms: linux/amd64,linux/arm64
           push: true
           build-args: |
-            "CHIA_BRANCH=1.2.3"
+            "CHIA_BRANCH=1.2.4"
             "FLAX_BRANCH=main"
           tags: |
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris:test

CHANGELOG.md

@@ -2,6 +2,15 @@
 
 All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.3] - 2021-08-27
+
+- Worker page for each worker shows Warnings for certain configuraton issues.
+- Farming page now shows current disk usage for each drive (free and used)
+- Plotting page now shows recent disk usage for each drive (free and used)
+- Summary page now charts recent blockchain challenges and partial proofs
+- Fixes from feedback received by users on the Machinaris Discord.  Thanks all!
+- Chia - Update to version 1.2.4. See their [changelog for details](https://github.com/Chia-Network/chia-blockchain/releases/tag/1.2.4).
+
 ## [0.5.2] - 2021-08-13
 
 - Machinaris - Docker images now available for [Apple M1](https://github.com/guydavis/machinaris/issues/43) and [Raspberry Pi OS](https://github.com/guydavis/machinaris/issues/155) architectures. 

CREDITS.md

@@ -19,6 +19,7 @@ A huge thank-you to the great teams/devs behind these projects, being used by Ma
 * [Boostrap Sidebars](https://dev.to/codeply/bootstrap-5-sidebar-examples-38pb): Used for sidebar menu layout.
 * [Toplevel](https://github.com/and-semakin/marshmallow-toplevel): Used for list of objects sent to REST API.
 * [DataTables.js](https://datatables.net/): Filter/search/pagination of dynamic tables, as per [this tutorial](https://blog.miguelgrinberg.com/post/beautiful-interactive-tables-for-your-flask-templates).
+* [Charts.js](https://www.chartjs.org/): dynamic charting library.
 
 ## Blockchain Forks
 * [Flax](https://github.com/Flax-Network/flax-blockchain)
@@ -53,7 +54,7 @@ A big thanks to all that contributed with dev and test including:
 * dartec
 * elexx
 * danicraftscz
-
+* Kris Moore
 
 ## Trademark Notice
 CHIA NETWORK INC, CHIA™, the CHIA BLOCKCHAIN™, the CHIA PROTOCOL™, CHIALISP™ and the “leaf Logo” (including the leaf logo alone when it refers to or indicates Chia), are trademarks or registered trademarks of Chia Network, Inc., a Delaware corporation.  *There is no affliation between this Machinaris project and the main Chia Network project.*

README.md

@@ -1,6 +1,6 @@
 # machinaris
 
-A Docker image for plotting and farming the Chia™ cryptocurrency on [one computer](https://github.com/guydavis/machinaris/wiki/Docker) or across [many](https://github.com/guydavis/machinaris/wiki/Workers).  Try the easy install using [Launch Wizard](https://machinaris.app).
+A Docker image for plotting and farming the Chia™ cryptocurrency on [one computer](https://github.com/guydavis/machinaris/wiki/Docker) or across [many](https://github.com/guydavis/machinaris/wiki/Workers).  Try the easy install using the [Launch Wizard](https://machinaris.app).
 
 ![Home](https://raw.githubusercontent.com/guydavis/machinaris-unraid/master/docs/img/machinaris_home.png)
 

SECURITY.md

@@ -0,0 +1,9 @@
+# Security Policy
+
+## Supported Versions
+
+Machinaris security patches will be made for the [current release](https://github.com/guydavis/machinaris/wiki/Releases#release-streams), aka `:latest`.
+
+## Reporting a Vulnerability
+
+Please create a new [Discussion](https://github.com/guydavis/machinaris/discussions) with full details of the vulnerability.  I will provide a response asap.  Thanks in advance!

VERSION

@@ -1 +1 @@
-0.5.2
+0.5.3

api/commands/chia_cli.py

@@ -48,8 +48,7 @@ def load_farm_summary(blockchain):
             proc.communicate()
             abort(500, description="The timeout is expired!")
         if errs:
-            app.logger.info("Failed to load chia farm summary at.")
-            app.logger.info(traceback.format_exc())
+            app.logger.debug("Error from {0} farm summary because {1}".format(blockchain, outs.decode('utf-8')))
         farm_summary = chia.FarmSummary(cli_stdout=outs.decode('utf-8').splitlines())
     else:  # Just get plot count and size
         farm_summary = chia.FarmSummary(farm_plots=load_plots_farming())
@@ -98,7 +97,7 @@ def load_wallet_show(blockchain):
     child = pexpect.spawn("{0} wallet show".format(chia_binary))
     wallet_index = 1
     while True:
-        i = child.expect(["Wallet height:.*\r\n", "Choose wallet key:.*\r\n", "No online backup file found.*\r\n"])
+        i = child.expect(["Wallet height:.*\r\n", "Choose wallet key:.*\r\n", "No online backup file found.*\r\n"], timeout=120)
         if i == 0:
             app.logger.debug("wallet show returned 'Wallet height...' so collecting details.")
             wallet_show += child.after.decode("utf-8") + child.before.decode("utf-8") + child.read().decode("utf-8")
@@ -120,7 +119,7 @@ def load_plotnft_show(blockchain):
     child = pexpect.spawn("{0} plotnft show".format(chia_binary))
     wallet_index = 1
     while True:
-        i = child.expect(["Wallet height:.*\r\n", "Choose wallet key:.*\r\n", "No online backup file found.*\r\n"])
+        i = child.expect(["Wallet height:.*\r\n", "Choose wallet key:.*\r\n", "No online backup file found.*\r\n"], timeout=120)
         if i == 0:
             app.logger.debug("wallet show returned 'Wallet height...' so collecting details.")
             wallet_show += child.after.decode("utf-8") + child.before.decode("utf-8") + child.read().decode("utf-8")

api/commands/chiadog_cli.py

@@ -61,9 +61,12 @@ def dispatch_action(job):
     else:
         raise Exception("Unsupported action {0} for monitoring.".format(action))
 
-def start_chiadog():
+def start_chiadog(chain = None):
     #app.logger.info("Starting monitoring....")
-    blockchains = [ b.strip() for b in os.environ['blockchains'].split(',') ]
+    if chain:
+        blockchains = [ chain ]
+    else:
+        blockchains = [ b.strip() for b in os.environ['blockchains'].split(',') ]
     for blockchain in blockchains:
         try:
             workdir = "/{0}dog".format(blockchain)

api/commands/log_parser.py

@@ -24,8 +24,8 @@
 CHIA_LOG = '/root/.chia/mainnet/log/debug.log'
 FLAX_LOG = '/root/.flax/mainnet/log/debug.log'
 
-# Roughly 1 minutes worth of challenges
-CHALLENGES_TO_LOAD = 8
+# Roughly 2 minutes worth of challenges, sent 90 seconds, for overlap
+CHALLENGES_TO_LOAD = 16
 
 # Most recent partial proofs, actually double as 2 log lines per partial
 PARTIALS_TO_LOAD = 50

api/gunicorn.conf.py

@@ -24,20 +24,20 @@ def on_starting(server):
     #scheduler.add_job(func=stats_disk.collect, trigger='interval', seconds=10) # Test immediately
 
     # Status gathering - reported via API
-    scheduler.add_job(func=status_challenges.update, name="challenges", trigger='interval', seconds=5)
-    scheduler.add_job(func=status_worker.update, name="workers", trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_controller.update, name="controller", trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_farm.update, name="farms", trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_plotting.update, name="plottings", trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_plots.update, name="plots", trigger='interval', seconds=60, jitter=30)  
-    scheduler.add_job(func=status_wallets.update, name="wallets", trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_plotnfts.update, name="plotnfts", trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_blockchains.update, name="blockchains", trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_connections.update, name="connections", trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_keys.update, name="keys", trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_alerts.update, name="alerts", trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_pools.update, name="pools", trigger='interval', seconds=60, jitter=30)
-    scheduler.add_job(func=status_partials.update, name="partials", trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_challenges.update, name="challenges", trigger='interval', seconds=90, jitter=45)
+    scheduler.add_job(func=status_worker.update, name="workers", trigger='interval', seconds=120, jitter=60) 
+    scheduler.add_job(func=status_controller.update, name="controller", trigger='interval', seconds=120, jitter=60) 
+    scheduler.add_job(func=status_farm.update, name="farms", trigger='interval', seconds=120, jitter=60) 
+    scheduler.add_job(func=status_plotting.update, name="plottings", trigger='interval', seconds=120, jitter=60) 
+    scheduler.add_job(func=status_plots.update, name="plots", trigger='interval', seconds=300, jitter=150)  
+    scheduler.add_job(func=status_wallets.update, name="wallets", trigger='interval', seconds=600, jitter=300) 
+    scheduler.add_job(func=status_plotnfts.update, name="plotnfts", trigger='interval', seconds=600, jitter=300) 
+    scheduler.add_job(func=status_blockchains.update, name="blockchains", trigger='interval', seconds=300, jitter=150) 
+    scheduler.add_job(func=status_connections.update, name="connections", trigger='interval', seconds=300, jitter=150) 
+    scheduler.add_job(func=status_keys.update, name="keys", trigger='interval', seconds=600, jitter=300) 
+    scheduler.add_job(func=status_alerts.update, name="alerts", trigger='interval', seconds=300, jitter=150) 
+    scheduler.add_job(func=status_pools.update, name="pools", trigger='interval', seconds=300, jitter=150)
+    scheduler.add_job(func=status_partials.update, name="partials", trigger='interval', seconds=300, jitter=150) 
 
     #scheduler.add_job(func=status_points.update, name="points", trigger='interval', seconds=10, jitter=0) 
     app.logger.debug("Starting background scheduler...")

api/models/chia.py

@@ -16,7 +16,7 @@ def __init__(self, cli_stdout=None, farm_plots=None):
         if cli_stdout:
             next_line_local_harvester = False
             self.plot_count = 0
-            self.plots_size = None
+            self.plots_size = 0
             for line in cli_stdout:
                 if next_line_local_harvester:
                     self.plot_count = line.strip().split(' ')[0]
@@ -28,12 +28,8 @@ def __init__(self, cli_stdout=None, farm_plots=None):
                     self.total_chia = line.split(':')[1].strip()
                 elif "Total flax farmed" in line:
                     self.total_chia = line.split(':')[1].strip()
-                elif "Plot count:" in line:
-                    self.plot_count = line.split(':')[1].strip()
                 elif "Local Harvester" in line:
                     next_line_local_harvester = True
-                elif not self.plots_size and "Total size of plots" in line:
-                    self.plots_size = line.split(':')[1].strip()
                 elif "Estimated network space" in line:
                     self.calc_netspace_size(line.split(':')[1].strip())
                 elif "Expected time to win" in line:

api/models/log.py

@@ -36,21 +36,25 @@ class Partials:
     def __init__(self, cli_stdout):
         self.columns = [ 'challenge_id', 'plots_past_filter', 'proofs_found', 'time_taken', 'created_at']
         self.rows = []
+        launcher_id = None
         for line in cli_stdout:
             try:
                 if "Submitting partial" in line:
                     app.logger.debug(line)
                     created_at = line.split()[0].replace('T', ' ')
                     launcher_id = re.search('partial for (\w+) to', line, re.IGNORECASE).group(1)
                     pool_url = re.search('to (.*)$', line, re.IGNORECASE).group(1)
-                elif "Pool response" in line:
+                elif "Pool response" in line and launcher_id:
                     pool_response = line[line.index('{'):]
                     self.rows.append({
                         'launcher_id': launcher_id,
                         'pool_url': pool_url.strip(),
                         'pool_response': pool_response,
                         'created_at': created_at
                     })
+                    created_at = None
+                    launcher_id = None
+                    pool_url = None
             except:
                 app.logger.info("Failed to parse partial line: {0}".format(line))
                 app.logger.info(traceback.format_exc())

api/schedules/stats_disk.py

@@ -12,14 +12,15 @@
 from flask import g
 
 from common.config import globals
-from api import app
+from common.models import stats
+from api import app, utils
 
 DATABASE = '/root/.chia/machinaris/dbs/stats.db'
 
 TABLES = ['stat_plots_total_used', 'stat_plots_disk_used', 'stat_plots_disk_free',
           'stat_plotting_total_used', 'stat_plotting_disk_used', 'stat_plotting_disk_free']
 
-DELETE_OLD_STATS_AFTER_DAYS = 2
+DELETE_OLD_STATS_AFTER_DAYS = 1
 
 def get_db():
     db = getattr(g, '_stats_database', None)
@@ -89,3 +90,25 @@ def collect():
         store_disk_stats(db, current_datetime, 'plots')
         if gc['plotting_enabled']:
             store_disk_stats(db, current_datetime, 'plotting')
+            if not gc['is_controller']: 
+                send_stats(stats.StatPlottingDiskUsed, '/stats/plottingdiskused')
+                send_stats(stats.StatPlottingDiskFree, '/stats/plottingdiskfree')
+                send_stats(stats.StatPlotsDiskUsed, '/stats/plotsdiskused')
+                send_stats(stats.StatPlotsDiskFree, '/stats/plotsdiskfree')
+
+def send_stats(model, endpoint):
+    from api import db
+    try:
+        hostname = utils.get_displayname()
+        payload = []
+        for stat in db.session.query(model).all():
+            payload.append({
+                "hostname": stat.hostname,
+                "path": stat.path,
+                "value": stat.value,
+                "created_at": stat.created_at,
+            })
+        utils.send_post(endpoint, payload, debug=False)
+    except:
+        app.logger.info("Failed to load recent {0} stats and send.".format(endpoint))
+        app.logger.info(traceback.format_exc())

api/schedules/status_alerts.py

@@ -38,7 +38,7 @@ def update():
                 first_run = False
             else: # On subsequent schedules, load only last 5 minutes.
                 since = (datetime.datetime.now() - datetime.timedelta(minutes=5)).strftime("%Y-%m-%d %H:%M:%S.000")
-            alerts = db.session.query(a.Alert).filter(a.Alert.created_at >= since).order_by(a.Alert.created_at.desc()).limit(20).all()
+            alerts = db.session.query(a.Alert).filter(a.Alert.created_at >= since, a.Alert.hostname == hostname).order_by(a.Alert.created_at.desc()).limit(20).all()
             payload = []
             for alert in alerts:
                 payload.append({

api/schedules/status_challenges.py

@@ -2,22 +2,38 @@
 # Performs a REST call to controller (possibly localhost) of latest blockchain challenges.
 #
 
+import datetime
 import os
 import traceback
 
 from flask import g
 
 from common.config import globals
+from common.models import challenges as c
 from common.utils import converters
 from api import app
 from api.commands import log_parser
 from api import utils
 
+def delete_old_challenges(db):
+    try:
+        cutoff = datetime.datetime.now() - datetime.timedelta(hours=1)
+        cutoff_str = "{0}".format(cutoff.strftime("%Y-%m-%d %H:%M:%S.%f")[:-3])
+        #app.logger.info("Purging old challenges earlier than {0}".format(cutoff_str))
+        db.session.query(c.Challenge).filter(c.Challenge.created_at < cutoff_str).delete()
+        db.session.commit()
+    except:
+        app.logger.info("Failed to delete old challenges.")
+        app.logger.info(traceback.format_exc())
+
 def update():
     if not globals.farming_enabled() and not globals.harvesting_enabled():
         #app.logger.info("Skipping recent challenges collection on plotting-only instance.")
         return
     with app.app_context():
+        from api import db
+        if globals.load()['is_controller']:
+            delete_old_challenges(db)
         try:
             hostname = utils.get_displayname()
             blockchains = ['chia']

api/schedules/status_farm.py

@@ -13,6 +13,21 @@
 from api.commands import chia_cli
 from api import utils
 
+# On initialization Chia outputs 
+def safely_gather_plots_size_gibs(plots_size):
+    plots_size_gibs = 0
+    try:
+        plots_size_gibs = converters.str_to_gibs(plots_size)
+    except:
+        app.logger.info("Unconvertable plots size: {0}  Using zero.".format(plots_size))
+        plots_size_gibs = 0
+    try:
+        float(plots_size_gibs)
+    except: 
+        app.logger.info("Unfloatable plots size: {0}  Using zero.".format(plots_size))
+        plots_size_gibs = 0
+    return plots_size_gibs
+
 def update():
     if not globals.farming_enabled() and not globals.harvesting_enabled():
         #app.logger.info("Skipping farm summary status collection on plotting-only instance.")
@@ -26,7 +41,7 @@ def update():
                 "mode": os.environ['mode'],
                 "status": "" if not hasattr(farm_summary, 'status') else farm_summary.status,
                 "plot_count": farm_summary.plot_count,
-                "plots_size": converters.str_to_gibs(farm_summary.plots_size),
+                "plots_size": safely_gather_plots_size_gibs(farm_summary.plots_size),
                 "total_chia": 0 if not hasattr(farm_summary, 'total_chia') else farm_summary.total_chia,
                 "netspace_size": 0 if not hasattr(farm_summary, 'netspace_size') else converters.str_to_gibs(farm_summary.netspace_size),
                 "expected_time_to_win": "" if not hasattr(farm_summary, 'time_to_win') else farm_summary.time_to_win,