2.1.0 - CLI Tool, Benchmarks, Dynamic Methods, and more!

- Added some new methods to SteemAsync - `get_witness` - Get data for a given witness name - `get_witness_list` - Get the ordered witness list (i.e. top 20) including their metadata - `wrapped_call` - Easier way to call condenser_api/database_api prefixed methods. Primarily used by getattr to handle dynamic methods. - Added `__getattr__` to `SteemAsync` to allow for dynamically generated methods - e.g. calling `SteemAsync().lookup_account_names(["someguy123", true])` will be transparently converted into `.wrapped_call('lookup_account_names', ["someguy123", true])` - allowing users to call same-named RPC methods just like a fully implemented method, instead of having to use `json_call` or `api_call` - Added `privex.steem.cli` and `__main__` - which allows using steem-async straight from the CLI, which can be helpful for developers/node admins who need to check if a specific method is working properly, or to check what it outputs. It can also be used for shellscripting and other uses. - Added `privex.steem.benchmarks` - which contains `bench_async` (steem-async benchmark) and `bench_beem` (beem benchmark), to allow developers to see and verify the performance difference between beem and steem-async. - Added `rich` to setup.py + Pipfile, as well as the `bench` extra to setup.py - Added info to the README about using the new CLI tool, as well as the benchmarks - Probably some other stuff too :)
Privex · Sep 30, 2021 · 6ee69f3 · 6ee69f3
1 parent e6db58f
commit 6ee69f3
Show file tree

Hide file tree

Showing 12 changed files with 915 additions and 12 deletions.
diff --git a/Pipfile b/Pipfile
@@ -7,13 +7,15 @@ name = "pypi"
 httpx = {version = ">=0.7", extras = ["http2"]}
 privex-helpers = ">=3.0"
 async-property = ">=0.2.1"
+rich = "*"
 
 [dev-packages]
 jupyter = "*"
 pytest = "*"
 pytest-asyncio = "*"
 twine = "*"
 setuptools = "*"
+beem = "*"
 
 [requires]
 python_version = "3.9"
diff --git a/Pipfile.lock b/Pipfile.lock
diff --git a/README.md b/README.md
@@ -1,11 +1,11 @@
 # Steem Async
-### Async Steem library - A simple Python library for asynchronous interactions with Steem RPC nodes (and forks)
+## Async Steem library - A simple Python library for asynchronous interactions with Steem RPC nodes (and forks)
 
 [![Build Status](https://travis-ci.com/Privex/steem-async.svg?branch=master)](https://travis-ci.com/Privex/steem-async)
 
 **Official Repo:** https://github.com/privex/steem-async
 
-### Quick Install / Usage
+## Quick Install / Usage
 
 **WARNING:** Due to use of modern Python features such as dataclasses, you MUST use **Python 3.7** or newer. This
 library is **not compatible with Python 3.6 or older versions**.
@@ -76,6 +76,57 @@ a method's brackets (including the constructor brackets) to see the parameters y
 Alternatively, just view the files inside of `privex/steem/` - most methods and constructors
 are adequently commented with PyDoc.
 
+# Steem Async CLI tool
+
+You can use Steem Async to make RPC calls directly from the CLI, with pretty printed output :)
+
+```sh
+# Show the CLI help page
+python3 -m privex.steem -h
+
+# Get the current block and output it as JSON
+python3 -m privex.steem get_block
+
+# Get block 123456 but disable human friendly indentation (but not syntax highlighting)
+python3 -m privex.steem -r get_block 123456
+
+
+# Get block 123456 but disable BOTH human friendly indentation AND syntax highlighting
+# This can be important if you're using the output in a script
+python3 -m privex.steem -nr -r get_block 123456
+
+# Get block 1234567 - use the custom node list https://hived.privex.io + https://anyx.io
+python3 -m privex.steem -n https://hived.privex.io,https://anyx.io get_block 1234567
+
+# Get the current head block number and print it
+python3 -m privex.steem get_head_block
+
+# Get the account balances for someguy123 as JSON
+python3 -m privex.steem get_balances someguy123
+
+# Get the account balances for someguy123 as JSON, but cast the numbers to floats instead of strings
+python3 -m privex.steem -dc float get_balances someguy123
+
+# Make a custom RPC call - calling the method get_ticker with no params
+python3 -m privex.steem call get_ticker
+
+# Using '-I' will enable number casting, so that '10' is converted to an integer param instead of a string
+python3 -m privex.steem call -I get_order_book 10
+
+# For calls that require nested lists/dicts, you can use '-j' to parse parameters as JSON,
+# or use '-c' to parse parameters as CSV (CSV columns are auto-casted from numbers/bools)
+python3 -m privex.steem call -j lookup_account_names '["someguy123", true]'
+python3 -m privex.steem call -j lookup_account_names call -c lookup_account_names someguy123,true
+```
+
+# Benchmarks compared to Beem and other libraries
+
+Steem-Async is much faster than Beem and other synchronous libraries, due to fully supporting AsyncIO, as well 
+as the use of bulk/bundled calls (combining multiple calls into one request) - especially when it comes to tasks 
+which involve highly parallel queries, such as retrieving 1000's of blocks from the blockchain.
+
+Please see the [benchmarks folder](https://github.com/Privex/steem-async/tree/master/benchmarks) to see the results of our 
+benchmarks, as well as information on how you can run our benchmarks on your own system.
 
 # Information
 
@@ -234,4 +285,5 @@ Here's the important bits:
 
 # Thanks for reading!
 
-**If this project has helped you, consider [grabbing a VPS or Dedicated Server from Privex](https://www.privex.io) - prices start at as little as US$8/mo (we take cryptocurrency!)**
+**If this project has helped you, consider [grabbing a VPS or Dedicated Server from Privex](https://www.privex.io) - prices**
+**start at as little as US$0.99/mo (we take cryptocurrency!)**
diff --git a/benchmarks b/benchmarks
@@ -0,0 +1 @@
+privex/steem/benchmarks
diff --git a/privex/steem/SteemAsync.py b/privex/steem/SteemAsync.py
@@ -111,7 +111,20 @@ class SteemAsync(CacheHelper):
         >>> blocks = await s.get_blocks(10000, 20000)
         >>> print(blocks[100].number)
         10100
-
+    
+    You can also call undefined methods, so long as they match up to a condenser_api call (or database_api call for non-appbase),
+    as :meth:`.__getattr__` will dynamically generate a wrapped async method for the equivalent RPC call using :meth:`.wrapped_call`
+    
+    Example 1. - ``SteemAsync().get_ticker()`` would be translated to ``wrapped_call('get_ticker')``
+                 RPC Method: ``condenser_api.get_ticker``
+                 RPC Params: ``[]``
+     
+    Example 2. - ``SteemAsync().lookup_account_names(['someguy123', True])`` would be translated
+                 to ``wrapped_call('lookup_account_names', ['someguy123', True])``
+                 RPC Method: ``condenser_api.lookup_account_names``
+                 RPC Params: ``[['someguy123', True]]``
+    
+    
     If there isn't a wrapper function for what you need, you can use json_call and api_call directly:
 
         >>> # Appbase call
@@ -129,7 +142,7 @@ class SteemAsync(CacheHelper):
     Copyright::
 
         +===================================================+
-        |                 © 2019 Privex Inc.                |
+        |                 © 2021 Privex Inc.                |
         |               https://www.privex.io               |
         +===================================================+
         |                                                   |
@@ -875,6 +888,22 @@ async def get_balances(self, account: str) -> Dict[str, Amount]:
         accs = await self.get_accounts(account)
         return accs[account].balances
 
+    async def get_witness(self, account: str) -> Dict[str, Any]:
+        if is_true(self.config('use_appbase', True)):
+            a = await self.json_call('condenser_api.get_witness_by_account', [account])  # type: Dict[Any]
+            res = a['result']
+        else:
+            res = await self.api_call('database_api', 'get_witness_by_account', [account])
+        return res
+
+    async def get_witness_list(self, account: Optional[str] = None, limit: int = 21) -> List[Dict[str, Any]]:
+        if is_true(self.config('use_appbase', True)):
+            a = await self.json_call('condenser_api.get_witnesses_by_vote', [account, limit])  # type: Dict[Any]
+            res = a['result']
+        else:
+            res = await self.api_call('database_api', 'get_witnesses_by_vote', [account, limit])
+        return res
+
     async def get_accounts(self, *accounts: str) -> Dict[str, Account]:
         """
         Get the accounts ``accounts`` - returned as a dictionary which maps each account name to an :class:`.Account` object.
@@ -940,6 +969,24 @@ async def get_accounts(self, *accounts: str) -> Dict[str, Account]:
         await self.set_cache(cache_key, accs)
         return accs
 
+    async def wrapped_call(self, method: str, params: Union[list, dict, Any] = None, module: str = None) -> Union[dict, list, Any]:
+        """
+        This is a simple method which calls an RPC method under ``condenser_api`` (if appbase is enabled),
+        or ``database_api`` (if appbase is disabled), unless you set ``module`` to specify a base module.
+        
+        It's used in :meth:`.__getattr__` for dynamically generated methods, e.g. ``SteemAsync().get_ticker()``
+        would be translated to ``wrapped_call('get_ticker')`` - while ``SteemAsync().lookup_account_names(['someguy123', True])``
+        would be translated to ``wrapped_call('lookup_account_names', ['someguy123', True])``
+        """
+        if is_true(self.config('use_appbase', True)):
+            module = empty_if(module, 'condenser_api')
+            a = await self.json_call(f'{module}.{method}', params)  # type: Dict[Any]
+            res = a['result']
+        else:
+            module = empty_if(module, 'database_api')
+            res = await self.api_call(module, method, params)
+        return res
+
     async def __aenter__(self):
         SteemAsync.context_level += 1
         return self
@@ -949,5 +996,20 @@ async def __aexit__(self, exc_type, exc_val, exc_tb):
         if SteemAsync.context_level <= 0:
             await self.http.aclose()
 
-
-
+    def __getattr__(self, item):
+        """
+        This magic method is used to dynamically generate instance methods which translate directly to the same-named RPC call,
+        using :meth:`.wrapped_call`
+        
+        Example 1. - ``SteemAsync().get_ticker()`` would be translated to ``wrapped_call('get_ticker')``
+        
+        Example 2. - ``SteemAsync().lookup_account_names(['someguy123', True])`` would be translated
+                     to ``wrapped_call('lookup_account_names', ['someguy123', True])``
+        
+        """
+        try:
+            return object.__getattribute__(self, item)
+        except AttributeError:
+            async def _wrapped(*args, **kwargs):
+                return await self.wrapped_call(item, list(args), **kwargs)
+            return _wrapped
diff --git a/privex/steem/__main__.py b/privex/steem/__main__.py
@@ -0,0 +1,5 @@
+from privex.steem.cli import cli_main
+
+if __name__ == '__main__':
+    cli_main()
+
diff --git a/privex/steem/benchmarks/README.md b/privex/steem/benchmarks/README.md
@@ -0,0 +1,114 @@
+# Benchmark Comparison Tools
+
+This folder contains small scripts which benchmark `steem-async` as well as alternative steem/hive libraries, using simple
+benchmark tasks, such as loading 1000 blocks, reading account information, etc.
+
+This README contains results of some/all of the benchmarks that were ran on Someguy123's iMac Pro, using Python 3.9
+and his home internet connection.
+
+## Results
+
+### Loading 1000 blocks with `beem`
+
+```sh
+chris | ~/steem-async/benchmarks $ ./bench_beem.py
+
+[2021-09-30 06:01:27.600455] Loading last 1000 blocks using beem ...
+
+[2021-09-30 06:03:36.533510] Total blocks: 1001
+
+Start Time: 1632981687.6005 seconds
+End Time: 1632981816.5335 seconds
+
+Total Time: 128.9330 seconds
+```
+
+### Loading 1000 blocks with `steem-async`
+
+```sh
+chris | ~/steem-async/benchmarks $ ./bench_async.py
+
+[2021-09-30 06:07:52.741749] Loading last 1000 blocks using steem-async ... 
+
+[2021-09-30 06:08:10.053123] Total blocks: 1000 
+
+Start Time: 1632982072.7419 seconds
+End Time: 1632982090.0531 seconds
+
+Total Time: 17.3112 seconds
+```
+
+## How to run the benchmarks
+
+### Option 1. - Use the benchmarks via the PyPi package
+
+This is the easiest method, as it doesn't require cloning the repo or setting up a virtualenv.
+
+Simply install the package `steem-async[bench]` using pip - which will install the steem-async library,
+with the `bench` extra requirements - which are the optional extra packages you need, to be able to run
+all of the benchmarks.
+
+```sh
+python3.9 -m pip install -U 'steem-async[bench]'
+# Alternatively - if you can't use pip via python3.x -m pip, then you can use 'pip3' instead.
+pip3 install -U 'steem-async[bench]'
+```
+
+Now you should be able to call the benchmarks via the full module path:
+
+```sh
+# Run the steem-async 1000 block benchmark
+python3.9 -m privex.steem.benchmarks.bench_async
+# Run the beem 1000 block benchmark
+python3.9 -m privex.steem.benchmarks.bench_beem
+```
+
+### Option 2. - Clone the repo and setup a dev environment
+
+First, clone the repo:
+
+```sh
+git clone https://github.com/Privex/steem-async.git
+cd steem-async
+```
+
+Now install the dependencies + create a virtualenv using `pipenv` :
+
+```sh
+# If you don't already have pipenv installed - then you'll need to install it using pip
+python3.9 -m pip install -U pipenv
+
+# Install the main deps + create the virtualenv
+pipenv install
+
+# Now install the development deps, which should include the dependencies for running the benchmark
+pipenv install --dev
+```
+
+Finally, enter the virtualenv using `pipenv shell` , and run the benchmarks using either `python3.x -m` ,
+or cd into the folder and execute them `./bench_async.py`
+
+```sh
+# Activate the virtualenv
+pipenv shell
+
+###
+# Run the benchmarks using python's module runner:
+###
+
+# Run the steem-async 1000 block benchmark
+python3 -m benchmarks.bench_async
+# Run the beem 1000 block benchmark
+python3 -m benchmarks.bench_beem
+
+###
+# Alternatively, you can run the benchmarks as individual files
+###
+cd benchmarks
+# Run the steem-async 1000 block benchmark
+./bench_async.py
+# Run the beem 1000 block benchmark
+./bench_beem.py
+```
+
+
diff --git a/privex/steem/benchmarks/__init__.py b/privex/steem/benchmarks/__init__.py
diff --git a/privex/steem/benchmarks/bench_async.py b/privex/steem/benchmarks/bench_async.py
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+from datetime import datetime
+from privex.steem import SteemAsync
+from privex.helpers import dec_round
+from decimal import Decimal
+from privex.helpers import env_csv, env_int
+import time
+import asyncio
+import logging
+
+try:
+    from rich import print
+except ImportError:
+    pass
+
+log = logging.getLogger('privex.steem')
+log.setLevel(logging.ERROR)
+
+
+SECS_NS = Decimal('1000000000')
+
+HIVE_NODES = env_csv('HIVE_NODES', ['https://direct.hived.privex.io', 'https://anyx.io', 'https://api.deathwing.me'])
+BATCH_SIZE = env_int('BATCH_SIZE', 100)
+NUM_BLOCKS = env_int('NUM_BLOCKS', 1000)
+
+
+async def main():
+    ss = SteemAsync(HIVE_NODES)
+    ss.config_set('batch_size', BATCH_SIZE)
+    print(f"\n [{datetime.utcnow()!s}] Loading last {NUM_BLOCKS} blocks using steem-async ... \n\n")
+    start_time = time.time_ns()
+    blks = await ss.get_blocks(-NUM_BLOCKS)
+    end_time = time.time_ns()
+    print(f"\n [{datetime.utcnow()!s}] Total blocks:", len(blks), "\n")
+    start_time, end_time = Decimal(start_time), Decimal(end_time)
+    start_secs = start_time / SECS_NS
+    end_secs = end_time / SECS_NS
+    print("Start Time:", dec_round(start_secs, 4), "seconds")
+    print("End Time:", dec_round(end_secs, 4), "seconds\n")
+    print("Total Time:", dec_round(end_secs - start_secs, 4), "seconds\n")
+
+
+if __name__ == '__main__':
+    asyncio.run(main())