[RFC] 3.0: Scope, features and priorities

[niieani]

Continuing from #44 (comment).

@tterranigma Let's discuss the scope of 3.0.

I've added you as a collaborator so you have push rights to the repo.

We should decide on the features to focus on implementing and problems needing tackling first. I've opened a clean branch (3.0) so we can work with a clean slate.

We can keep the outcome of the discussion as documentation inside of the repo itself, e.g. in a docs folder.

Some things that come to mind while designing the future:

Module system

Any respecting language ecosystem has a module system for managing and sharing dependencies/libraries (JS has npm, Python has PyPi, Ruby has RubyGem, C# has NuGet, and so on). I think we should think about both a method of sharing internal and external "modules", which is especially difficult, since functions are global in bash.

For a non-polluting way of loading, I propose something in the lines of my experiment here.

For a shared repository of public (and private) libraries we could use GitHub, just as Go does it.

Another useful feature that I thought about is would be "bundling", i.e. replacing and inlining all imports, in order, with actual code. This would be great for distributing, since you could use all the features you want and just move/upload a single script file that has all of the functionality embedded.

Features from 2.0

Which are the most important to port first?

• I'm probably not that keen on doing OO, but we could provide a standard library of sorts for simplifying/unifying builtin features (like REGEXP).
• Error reporting is probably high on the list.
• New implementation of named parameters.
• Passing arrays/dictionaries as arguments (should go well with new named params).
• Things like colors/emojis/powerline, which are TERM specific. How to handle them safely and with high usability (we had bugs there in 2.0)?

Other ideas

• Compatibility with bashdb for debugging would be really useful.
• Integration with other popular tools.
• Good, composable, declarative parsing of CLI arguments/parameters. Perhaps something like DocOpt?
• Side effect functions: I've had this idea that we could have a format for the result of a function that could provide side-effects in the upper scope. This would enable things like modifying the current scope from within subshells.

Thoughts?

[AdeAttwood]

I think for this to work the framework will need to be installed global and have some kind of basher style package manager. In the include function it could look for a file called "main.sh" that would export the module to use in the installed packaged.

[AdeAttwood]

Having a little look around I think it would be a good idea to make it compatible with bpkg. https://github.com/bpkg/bpkg

[niieani]

Thanks for joining the discussion @AdeAttwood.

bpkg looks interesting, although as far as I see, it doesn't have a way to publish packages from the CLI to the global repository. Instead, you'd manually edit https://github.com/bpkg/bpkg/wiki/index, which isn't even documented. I'm also a bit worried about its dependencies, but it's possible we wouldn't be able to get by without them anyway. Maybe we can work with their maintainers to adapt it.

I'm also not entirely convinced about some of their decisions, like the one to use JSON to describe the package. It makes it easier for those coming from the NPM world, but on the other hand adds complexity (JSON parser) that could have been avoided.

It would be great for a potential package manager to support monorepo-style packages (many packages per repo). This makes it easier to create micro-libraries from individual helper utilities, and is generally what helped JavaScript become what it is now.

Some features would definitely have to be globally available (e.g. parametrized functions, module system), but that's a given in most languages anyway, but we could still keep many features in silos, if possible.

[AdeAttwood]

Basher was the other one I have looked into.
In the docks it has echo 'export PATH="$HOME/.basher/bin:$PATH"' >> ~/.bash_profile That would make scripts avalible in the global scope.

Not sure its what you looking for though. Maby create a merge of the two.

Form bpkg I like https://github.com/bpkg/bpkg#package-exports for exporting or running. And also the config file defining the bin files rather than an install script.

https://github.com/basherpm/basher

[nkakouros]

These are some really nice points. I will devote some time this weekend to lay my thoughts and feedback.

[dzove855]

Hii, i'm following this project for quite a time and it's a really project :D There is just one thing, i would even try to implement a web framework or classes into it.
I already wrote a web framework, but since i follow this repo i would like to rewrite my framework and implement this project into my framework, but why not directly put a framework into this project so it is out of the box.

https://github.com/yoctu/yosh
https://github.com/yoctu/yosh-starter-kit

I have some other projects wroten in my framework but they're still close, they will be probably open this mounth.

[nkakouros]

Before diving into the actual discussion, I believe it is important to acknowledge what bash is. This means that what we do should also be "bash-y", there is no reason to force OO if it feels unnatural/problematic or other project practices, eg configuration storing in json or yaml files. Also, I think that it would be beneficial to move in a 'set requirements first, implement after' fashion. Plus, whatever we decide to implement should be of coming from an actual need and practical use case, to avoid creating a framework that is of no use. Bash Infinity has already done a great job in that aspect, so let's keep the standard up!

What is a Bash Framework

There seem to be 3 types of bash frameworks.

One common type is a collection of functions, aliases, etc to make life in the terminal (and perhaps writing scripts) easier.

The second type is frameworks that provide a mechanism into which to hook to get your application running, like the go-script-bash framework.

Finally, there are frameworks that intervene in the way you write bash code. This is were we are. I mention this to make clear what should go into Bash Infinity and what not. For instance, it does not make sense to have eg a git module. To answer to @dzove855 , your project seems to me like sth that would be better written in Bash Infinity, than be part of its core. This means that parts of your project could be replaced by (or contributed to) Bash Infinity's core.

Module System

I think that there are three, though tightly related, things here. One is essentially a package/module manager, the second is a spec on how to interact with the packages/modules and third is our implementation of that spec. I think that we shouldn't try to do all three in the context of this single project, especially if we aim to bring interoperability into the bash ecosystem of projects. I propose the following action plan:

• Do a research on existing "package manager" projects and analyze them. Perhaps there is already sth that covers the needs? Maybe not. Let's make it sure and in the mean time build a list of useful features and things to avoid. This would be work towards a spec.

• After we reach a spec (for which we could also invite publicly interested people), provide a reference implementation, outside of bash Infinity.

• In the mean time, we can start working on Bash Infinity. We could go with a bare bone module system to get our initial work done and then move to support the new spec when it is completed.

Some questions that would need to be addressed by the spec would be:

• How will 3rd party libs be organized in a project? What will the actual file structure be?
• Will there be "autoloading" supported? Will this be based on function name? If not, how will modules be loaded? What kind of conflict resolution will be implemented?
• Will the embedding you mention be supported?

Also relating to the package manager implementation:

• What sources will supported? Github only? A brand new web repo?
• How will project dependencies be defined?
• How could versioning be supported? Eg choosing among dev/stable versions etc.
• How will the system keep track of module loading? Will there be support for different versions of the same library?

The module system you have posted looks great and we could start with this while developing version 3. But for instance this line:

 local path="${1}"

needs the above discussion to be settled first, IMO.

Features from 2.0

I agree on your shortlisted features to aim for initially.

For me, what bash programming is all about is basically commands and functions. We should focus on interacting with them as easy and intuitive as possible. In the mean time, we could start developing the framework library of helper functions (eg for string manipulation) as the needs we encounter come.

Functions

Let's start with functions, that practically means passing arguments and getting output.

For argument passing, the main pain points for me are:

• checking that required arguments are present
• default values
• argument typing (int, string, etc)
• the code "ambiguity" when calling function like get_hosts "project2" true 0 ''. What are these arguments exactly? Being able to pass named parameters would be great, like get_hosts cluster=project2 recursive=true tries=0.

The current notation of Bash Infinity is excellent, but maybe limits the implementation possibilities. Let's work on your code in #41 and research the possibility of backwards compatibility.

For getting output from a function there are 3 possibilities (that I know of) in bash:

• echoing values; this can be broken into two:
  • echoing text
  • echoing variable declarations
• return/exit codes
• global variables being set in the function.

I would like to have sth that makes this as simple as possible, without messing the scope of variables. I have some ideas to test out.

Error reporting

Bash Infinity currently prints a fancy call stack. This is helpful. But I often find myself wanting to:

• have more context (~10 previous lines)
• have debug output (set -x) context
• terminate code execution no matter what (right now, there are times that you have to hit Ctrl-C a couple of times to end execution)
• interfere with code execution (maybe incorporate bashdb directly?)

Moreover, when trying to debug a script, Bash Inifinity internals get a lot in the way. For instance, running with set -x produces output that is 80% Bash Infinity and the rest my code. We should try to be as transparent as possible, the end users will not want to debug Bash Infinity but their programs.

Working on this and porting parts of the existing stack trace will also be a good opportunity to work on the UI stuff.

Sth related is error logging. Contrary to error reporting which is useful for interactive sessions, logging is a second channel of error output. Bash Infinity has some kind of logging, but it feels a little incomplete, am I wrong?

---

These will keep us busy for a while. Then, we can look into variable typing, code structures, etc. I suggest we keep this discussion here. After we conclude on the requirements, we can break them into issues, set some milestones and keep track of the progress there.

The next 2-3 weeks will be really busy at work, so my presence here will be sub-optimal, but I will try my best to keep things moving.

[nicutor]

Hi guys,

I want just to come with an idea.
As @tterranigma told on other issues comments, I suggest to you to not reinvent some things but to use other which already are out there and they are pretty stable and maintained.

For example:
https://github.com/bats-core/bats-core for tests
https://github.com/kward/log4sh for logging

[niieani]

From @adudek #69 (comment)

If I might suggest any direction of #45, please focus first on data structures. Most used feature of all languages (and frameworks) is array/sets manipulation. Any project success is measured by comparing effort to savings ratio. If robust and easy(ier) data manipulation is not present as key feature, then any framework is but a gimmick. Also making data manipulation priority framework is setting toward stable long-term-support path acquiring bigger and more stable community.

[rugk]

BTW as for unit tests, I also like https://github.com/kward/shunit2/ very much. It is basically a more "traditional" version where not each line is an assertion.

[nkakouros]

I got drawn by other stuff, but here is a small draft I created on named parameters. I "stole" this wonderful macro snippet. What do you think?

[aadrian]

@niieani

Good, composable, declarative parsing of CLI arguments/parameters. Perhaps something like DocOpt?

That would be very valuable, since many non-trivial scripts require a form or another of CLI args handling.