[RFC] 3.0: Scope, features and priorities

[niieani]

Continuing from #44 (comment).

@tterranigma Let's discuss the scope of 3.0.

I've added you as a collaborator so you have push rights to the repo.

We should decide on the features to focus on implementing and problems needing tackling first. I've opened a clean branch (3.0) so we can work with a clean slate.

We can keep the outcome of the discussion as documentation inside of the repo itself, e.g. in a docs folder.

Some things that come to mind while designing the future:

Module system

Any respecting language ecosystem has a module system for managing and sharing dependencies/libraries (JS has npm, Python has PyPi, Ruby has RubyGem, C# has NuGet, and so on). I think we should think about both a method of sharing internal and external "modules", which is especially difficult, since functions are global in bash.

For a non-polluting way of loading, I propose something in the lines of my experiment here.

For a shared repository of public (and private) libraries we could use GitHub, just as Go does it.

Another useful feature that I thought about is would be "bundling", i.e. replacing and inlining all imports, in order, with actual code. This would be great for distributing, since you could use all the features you want and just move/upload a single script file that has all of the functionality embedded.

Features from 2.0

Which are the most important to port first?

• I'm probably not that keen on doing OO, but we could provide a standard library of sorts for simplifying/unifying builtin features (like REGEXP).
• Error reporting is probably high on the list.
• New implementation of named parameters.
• Passing arrays/dictionaries as arguments (should go well with new named params).
• Things like colors/emojis/powerline, which are TERM specific. How to handle them safely and with high usability (we had bugs there in 2.0)?

Other ideas

• Compatibility with bashdb for debugging would be really useful.
• Integration with other popular tools.
• Good, composable, declarative parsing of CLI arguments/parameters. Perhaps something like DocOpt?
• Side effect functions: I've had this idea that we could have a format for the result of a function that could provide side-effects in the upper scope. This would enable things like modifying the current scope from within subshells.

Thoughts?

[AdeAttwood]

I think for this to work the framework will need to be installed global and have some kind of basher style package manager. In the include function it could look for a file called "main.sh" that would export the module to use in the installed packaged.

[AdeAttwood]

Having a little look around I think it would be a good idea to make it compatible with bpkg. https://github.com/bpkg/bpkg

[niieani]

Thanks for joining the discussion @AdeAttwood.

bpkg looks interesting, although as far as I see, it doesn't have a way to publish packages from the CLI to the global repository. Instead, you'd manually edit https://github.com/bpkg/bpkg/wiki/index, which isn't even documented. I'm also a bit worried about its dependencies, but it's possible we wouldn't be able to get by without them anyway. Maybe we can work with their maintainers to adapt it.

I'm also not entirely convinced about some of their decisions, like the one to use JSON to describe the package. It makes it easier for those coming from the NPM world, but on the other hand adds complexity (JSON parser) that could have been avoided.

It would be great for a potential package manager to support monorepo-style packages (many packages per repo). This makes it easier to create micro-libraries from individual helper utilities, and is generally what helped JavaScript become what it is now.

Some features would definitely have to be globally available (e.g. parametrized functions, module system), but that's a given in most languages anyway, but we could still keep many features in silos, if possible.

[AdeAttwood]

Basher was the other one I have looked into.
In the docks it has echo 'export PATH="$HOME/.basher/bin:$PATH"' >> ~/.bash_profile That would make scripts avalible in the global scope.

Not sure its what you looking for though. Maby create a merge of the two.

Form bpkg I like https://github.com/bpkg/bpkg#package-exports for exporting or running. And also the config file defining the bin files rather than an install script.

https://github.com/basherpm/basher

[nkakouros]

These are some really nice points. I will devote some time this weekend to lay my thoughts and feedback.

[dzove855]

Hii, i'm following this project for quite a time and it's a really project :D There is just one thing, i would even try to implement a web framework or classes into it.
I already wrote a web framework, but since i follow this repo i would like to rewrite my framework and implement this project into my framework, but why not directly put a framework into this project so it is out of the box.

https://github.com/yoctu/yosh
https://github.com/yoctu/yosh-starter-kit

I have some other projects wroten in my framework but they're still close, they will be probably open this mounth.

[nkakouros]

Before diving into the actual discussion, I believe it is important to acknowledge what bash is. This means that what we do should also be "bash-y", there is no reason to force OO if it feels unnatural/problematic or other project practices, eg configuration storing in json or yaml files. Also, I think that it would be beneficial to move in a 'set requirements first, implement after' fashion. Plus, whatever we decide to implement should be of coming from an actual need and practical use case, to avoid creating a framework that is of no use. Bash Infinity has already done a great job in that aspect, so let's keep the standard up!

What is a Bash Framework

There seem to be 3 types of bash frameworks.

One common type is a collection of functions, aliases, etc to make life in the terminal (and perhaps writing scripts) easier.

The second type is frameworks that provide a mechanism into which to hook to get your application running, like the go-script-bash framework.

Finally, there are frameworks that intervene in the way you write bash code. This is were we are. I mention this to make clear what should go into Bash Infinity and what not. For instance, it does not make sense to have eg a git module. To answer to @dzove855 , your project seems to me like sth that would be better written in Bash Infinity, than be part of its core. This means that parts of your project could be replaced by (or contributed to) Bash Infinity's core.

Module System

I think that there are three, though tightly related, things here. One is essentially a package/module manager, the second is a spec on how to interact with the packages/modules and third is our implementation of that spec. I think that we shouldn't try to do all three in the context of this single project, especially if we aim to bring interoperability into the bash ecosystem of projects. I propose the following action plan:

• Do a research on existing "package manager" projects and analyze them. Perhaps there is already sth that covers the needs? Maybe not. Let's make it sure and in the mean time build a list of useful features and things to avoid. This would be work towards a spec.

• After we reach a spec (for which we could also invite publicly interested people), provide a reference implementation, outside of bash Infinity.

• In the mean time, we can start working on Bash Infinity. We could go with a bare bone module system to get our initial work done and then move to support the new spec when it is completed.

Some questions that would need to be addressed by the spec would be:

• How will 3rd party libs be organized in a project? What will the actual file structure be?
• Will there be "autoloading" supported? Will this be based on function name? If not, how will modules be loaded? What kind of conflict resolution will be implemented?
• Will the embedding you mention be supported?

Also relating to the package manager implementation:

• What sources will supported? Github only? A brand new web repo?
• How will project dependencies be defined?
• How could versioning be supported? Eg choosing among dev/stable versions etc.
• How will the system keep track of module loading? Will there be support for different versions of the same library?

The module system you have posted looks great and we could start with this while developing version 3. But for instance this line:

 local path="${1}"

needs the above discussion to be settled first, IMO.

Features from 2.0

I agree on your shortlisted features to aim for initially.

For me, what bash programming is all about is basically commands and functions. We should focus on interacting with them as easy and intuitive as possible. In the mean time, we could start developing the framework library of helper functions (eg for string manipulation) as the needs we encounter come.

Functions

Let's start with functions, that practically means passing arguments and getting output.

For argument passing, the main pain points for me are:

• checking that required arguments are present
• default values
• argument typing (int, string, etc)
• the code "ambiguity" when calling function like get_hosts "project2" true 0 ''. What are these arguments exactly? Being able to pass named parameters would be great, like get_hosts cluster=project2 recursive=true tries=0.

The current notation of Bash Infinity is excellent, but maybe limits the implementation possibilities. Let's work on your code in #41 and research the possibility of backwards compatibility.

For getting output from a function there are 3 possibilities (that I know of) in bash:

• echoing values; this can be broken into two:
  • echoing text
  • echoing variable declarations
• return/exit codes
• global variables being set in the function.

I would like to have sth that makes this as simple as possible, without messing the scope of variables. I have some ideas to test out.

Error reporting

Bash Infinity currently prints a fancy call stack. This is helpful. But I often find myself wanting to:

• have more context (~10 previous lines)
• have debug output (set -x) context
• terminate code execution no matter what (right now, there are times that you have to hit Ctrl-C a couple of times to end execution)
• interfere with code execution (maybe incorporate bashdb directly?)

Moreover, when trying to debug a script, Bash Inifinity internals get a lot in the way. For instance, running with set -x produces output that is 80% Bash Infinity and the rest my code. We should try to be as transparent as possible, the end users will not want to debug Bash Infinity but their programs.

Working on this and porting parts of the existing stack trace will also be a good opportunity to work on the UI stuff.

Sth related is error logging. Contrary to error reporting which is useful for interactive sessions, logging is a second channel of error output. Bash Infinity has some kind of logging, but it feels a little incomplete, am I wrong?

---

These will keep us busy for a while. Then, we can look into variable typing, code structures, etc. I suggest we keep this discussion here. After we conclude on the requirements, we can break them into issues, set some milestones and keep track of the progress there.

The next 2-3 weeks will be really busy at work, so my presence here will be sub-optimal, but I will try my best to keep things moving.

[nicutor]

Hi guys,

I want just to come with an idea.
As @tterranigma told on other issues comments, I suggest to you to not reinvent some things but to use other which already are out there and they are pretty stable and maintained.

For example:
https://github.com/bats-core/bats-core for tests
https://github.com/kward/log4sh for logging

[niieani]

From @adudek #69 (comment)

If I might suggest any direction of #45, please focus first on data structures. Most used feature of all languages (and frameworks) is array/sets manipulation. Any project success is measured by comparing effort to savings ratio. If robust and easy(ier) data manipulation is not present as key feature, then any framework is but a gimmick. Also making data manipulation priority framework is setting toward stable long-term-support path acquiring bigger and more stable community.

[rugk]

BTW as for unit tests, I also like https://github.com/kward/shunit2/ very much. It is basically a more "traditional" version where not each line is an assertion.

[nkakouros]

I got drawn by other stuff, but here is a small draft I created on named parameters. I "stole" this wonderful macro snippet. What do you think?

[aadrian]

@niieani

Good, composable, declarative parsing of CLI arguments/parameters. Perhaps something like DocOpt?

That would be very valuable, since many non-trivial scripts require a form or another of CLI args handling.

[hyperupcall]

I've been doing a lot of Bash lately, brainstorming (and implementing) what hypermodern Bash scripts could look like, and stumbled upon this thread. I would like to share my thoughts as well as provide feedback on these great ideas. There are a lot of "necessities" that are absent from Bash and agreeing on / standardizing these (ex. error reporting, cli parsing) in the name of interoperability would be very useful and provide clarity to the ecosystem as a whole

Module / function loading

For a non-polluting way of loading, I propose something in the lines of my experiment here.

I like the idea of having a function just for loading a 'module' or 'package'. To me, this seems like a hard requirement if we want to be sure functions from modules are called with the correct context. For example, if we assume there exists some repository from GitHub is properly installed with a bash-json.parse function, we can do something like

call() {
	...
	PATH="$A_PARTICULAR_MODULES_PATH" "$@"
}

call bash-json.parse 'package.json'

It would follow that the call function must be included as a part of the Bash framework or package manager. $A_PARTICULAR_MODULES_PATH would best be calculated by the package manager, since it would know the PATH executable and function dependencies of that particular module or package, and can inject the path accordingly

I understand the link you send is a very quick prototype, but I'll mention that any eval is a no-go for me, since this can be done without it

It would be hard to provide some fallback for POSIX sh, since . is not specified to pass any positional parameters, rather than source. Additionally, . is a special built-in, so setting the PATH would clobber the PATH at the callsite.

Bundling

Another useful feature that I thought about is would be "bundling", i.e. replacing and inlining all imports, in order, with actual code. This would be great for distributing, since you could use all the features you want and just move/upload a single script file that has all of the functionality embedded.

I very much like the idea of bundling. Super useful if the user doesn't have some sort of Bash package manager installed, and needs to use the functionality. I haven't thought too hard about it lately, since I believe it would be wise to standardize entrypoints for packages/modules and certain variables

Error reporting

Error reporting is probably high on the list.

Agreed - it is kind of tricky due to errexit and -u. What about providing two ways to handle errors, and standardizing handling?

The first way is for quick scripts. On error, we terminate (exit) with a non-zero exit code, and provide a stacktrace

call bash-toml.parse 'file.sh'
# => Error: bash-toml.parse: Could not find file.sh

The second way, is that on error, we return with a non-zero exit code, and set the stack trace in some variable (e.g. ERR). (I'm pretty sure if ! call ... is used, then the error code won't be properly preserved)

if call bash-toml.parse 'file.sh'; then :; else
	printf '%s\n' "There appears to be an error with"
	printf '%s' "$ERR"
fi

Integration with other popular tools.

Like a linter to check for error reporting, correct module format, etc.

Good, composable, declarative parsing of CLI arguments/parameters. Perhaps something like DocOpt?

I like DocOpt in general, but for Bash, they do some weird Go codegen thing. I made a parser hyperupcall/bash-args that has similar functionality (but using an incompatible argument string spec) that supports a bit less features, but is very fast without a codegen step. I've been meaning to rewrite it to be compatible, but haven't gotten around to it

I also think it would be wise to separate argument parsing into two categories: for functions and executables. Full-blown argument parsing can get quite expensive if we are using it for many function calls. DocOpt-style stuff is good for executables, but for functions having a super basic stripped down version would really benefit speed of execution

data manipulation

If I might suggest any direction of #45, please focus first on data structures. Most used feature of all languages (and frameworks) is array/sets manipulation. Any project success is measured by comparing effort to savings ratio. If robust and easy(ier) data manipulation is not present as key feature, then any framework is but a gimmick. Also making data manipulation priority framework is setting toward stable long-term-support path acquiring bigger and more stable community

Method-based data manipulation compared to somewhat vague features at the syntax level may be the way to go, especially for people newer to Bash. That is something a standard library can provide. As for arbitrarily nested structures, I made bash-object, which may be of use. I made it because I had to for my fully-bash JSON and TOML parsers.

Package management

I think for this to work the framework will need to be installed global and have some kind of basher style package manager. In the include function it could look for a file called "main.sh" that would export the module to use in the installed packaged.

Agreed there would need to be some Basher-style package manager. Exiting projects like bats-assert and bats-support use a load.bash file, so I think we should go with something like that.

I actually forked Basher into a project called bpm. Basically everything has been rewritten, and automatically sourcing load.bash files for packages is on the roadmap. (you can already do it manually with bpm-load)

I'm also not entirely convinced about some of their decisions, like the one to use JSON to describe the package. It makes it easier for those coming from the NPM world, but on the other hand adds complexity (JSON parser) that could have been avoided.

Agreed. It clobbers NPM's package.json, and they automatically invoke make or something

It would be great for a potential package manager to support monorepo-style packages (many packages per repo). This makes it easier to create micro-libraries from individual helper utilities, and is generally what helped JavaScript become what it is now.

Agreed. Right now bpm doesn't inherently support multi-package monorepos, but it can be done by just setting different globals. For example, I'm working on a bash standard library (that isn't public yet), and naming functions std.print.array, std.is.* etc. You could just as easily remove the std. part. Either way, they would be loaded in the global scope with a load.bash file (which isn't automatically done yet)

Do a research on existing "package manager" projects and analyze them. Perhaps there is already sth that covers the needs? Maybe not. Let's make it sure and in the mean time build a list of useful features and things to avoid. This would be work towards a spec.

bpkg doesn't really fit in my opinion, which is why I forked Basher.

After we reach a spec (for which we could also invite publicly interested people), provide a reference implementation, outside of bash Infinity.

bpm could be that "reference implementation". I think it would be wise to specify what exactly what we want it to do. For example, the way bpm currently handles per project, local dependencies should probably change. Right now bpm behaves a certain way, but I am definitely open to changing that, there is some stuff that definitely needs to be changed

How will 3rd party libs be organized in a project?

Right now, with bpm, dependencies are Git cloned to bpm_packages/packages (bpm_modules gives me Vietnam flashbacks to npm/yarn), and packages bin, completion, man pages are symlinked to bpm_packages/bin and bpm_packages/completions, etc. Transitive dependencies are cloned in the bpm_packages directory of their respective consumers (like pre 4.0ish npm). I think this should be changed to having everything be placed at the top level bpm_packages with a version so less downloading is done ([email protected], [email protected], [email protected]). There should also be a config option so only release tarballs are downloaded, to enforce versioning). Thoughts?

Will there be "autoloading" supported? Will this be based on function name? If not, how will modules be loaded? What kind of conflict resolution will be implemented?

I think autoloading is a must. Something along the lines of the package manager providing a function that will read all of the load.bash files of dependencies. This could potentially conflict with any module or load/call functions which also call these functions, so more details will need to be fleshed out

What will the actual file structure be?

bpm automatically supports most standard Bash projects. bin, completions, man directories are automatically detected. Files and executables like shdoc and z.1 are automatically detected as well, and handled by bpm. If you want to change where bins, completions, etc. are found, that is specified in a bpm.toml file. I personally put everything in a pkg subdirectory to make things a bit cleaner, but that's just personal preference - bpm supports things like that

What sources will supported? Github only? A brand new web repo?

bpm supports Git repositories hosted on any website and arbitrary git ssh uris

Examples

Examples:
  bpm add tj/git-extras
  bpm add github.com/tj/git-extras
  bpm add https://github.com/tj/git-extras
  bpm add [email protected]:tj/git-extras
  bpm add hyperupcall/bash-args --branch=main
  bpm add hyperupcall/[email protected] # out of date

How will project dependencies be defined?

bpm reads from a bpm.toml file, which is at the project root. It doesn't clobber package.json like bpkg and doesn't allow for arbitrary code execution during the installation process like Basher's package.sh (bpm reads package.sh for backwards compatibility, but uses regex parsing)

How could versioning be supported? Eg choosing among dev/stable versions etc.

bpm supports specifying commits and tags after the package name hyperupcall/[email protected]. Global installations support a --branch parameter, but that has not been expanded to local per-project installations

How will the system keep track of module loading? Will there be support for different versions of the same library?

Yes, there will be support for multiple versions. This will be facilitated with the some call or module function as detailed above

I think a lot of the ideas mentioned here are important to get right as soon as possible, so we can update interfaces / iterate without worrying about a ton of breakage, at least until 1.0.0 RC's. If we can expose functionality with a modular library-approach rather than a framework approach, that would be good for composition and interoperability in the bash ecosystem as a whole. As far as bpm is concerned, I want to make sure we get the 'How will 3rd party libs be organized in a project?' question answered properly, and make cli usage as fluid as possible. For example right now bpm install/uninstall feels a bit clunky (especially with the --all and --global flags). And add/remove is not supported since that would require modifying the TOML file, which cannot be easily done right now. I'd love to hear people's thoughts

[adudek]

Bash is super easy language that glues things together in nice way. Only thing missing is proper data-structures support. If we want to get most milage from what is already in bash, framework should not affect regular script writing technique.

Lets try not to reinvent PowerShell or Perl

MVP:
datastructures (if possible might also be modularized)
liniting/validation
built-in testing (maybe via annotation support)
module support and autoloading - proper namespacing (simple and clean like terraform,)

[hyperupcall]

datastructures

Like data structure manipulation, as in common methods like .pop instead of hard to remember constructs like unset 'arr[${#arr[@]}-1]'?

liniting/validation

Am I correct that shellcheck doesn't seem to have a way for easily adding rules? Maybe tree-sitter-bash or mvdan/sh would be relevant here

built-in testing (maybe via annotation support)

Like integrating Bats or any particular test runner? Any examples of what annotation could look like? I would be good to be compatible with shdoc

[adudek]

Any examples of what annotation could look like?

I don't have any particular thing in mind, annotations could be used to refer to external testsets or implement/validate contract inline (so external module can lookup other modules annotation and perform validation of contract - microservice architecture)

[adudek]

Like data structure manipulation, as in common methods like .pop instead of hard to remember constructs like unset 'arr[${#arr[@]}-1]'?

This is trivial example, but implementing proper hashmap or dictionary of dictionaries is not as easy.
Even regular propertyfile format support needs to by done by 3-rdparties

[hyperupcall]

This is trivial example, but implementing proper hashmap or dictionary of dictionaries is not as easy.

I see. Yeah, it would be nice to have datastructures that are built over associative arrays and indexed arrays that give more guarantees about the containing data and how accesses and writes are performed.

As for 'dictionaries of dictionaries', a near complete rock-solid implementation does already exist - a library to provide arbitrary nesting of associative arrays and indexed arrays. You might be interested in it (it's still beta quality though)

[niieani]

Thanks for everyone's thoughts so far! @adudek @hyperupcall Since I'm rather tied up with other projects at the moment, I don't see myself working on 3.0 anytime soon. If any of you would like to give a POC/MVP a go, feel free to open a PR against the 3.0 branch, and we can all chime in with feedback/suggestions.

I think we're mostly in agreement as to what's missing in bash, perhaps the scope of what each of us wants differs, but if we start with a good module+package system, it won't really matter - anyone should be able to either use as much or as little of the framework as they want.

My suggestion for a module system is here, it's basically a simple implementation of node's CommonJS require. Granted, it's a little verbose in usage, cause you need to specify what module you're invoking from with every invocation, but I haven't found a way around it that wouldn't pollute the global scope.

[lafkpages]

bpkg looks interesting, although as far as I see, it doesn't have a way to publish packages from the CLI to the global repository.

Packages aren't published to a BPKG registry. You install packages in the format bpkg install username/repo and it installs the package at said repository on GitHub. That's how I've been using it, and how I've been told to do so.

[niieani]

As far as I can see, their docs say one should manually submit a PR with the package: https://bpkg.sh/submit/
Doesn't seem like that's easily automateable.

[lafkpages]

That's only to list the package on their website. But you can install any GitHub repo that has a bpkg.json file at the root of it.

[niieani]

That's good to know. But it's not documented on bpkg.sh :(