Global variable support and Generate env JSON option #24

lovestaco · 2023-07-10T07:52:21Z

What type of MR is this?

Description

This MR adds the feature of supporting global variables.
Edit1:

L2 can now fetch variables declared in l2config.env which can be in the parent directory.
l2config.env is searched for, from the present directory to all its ancestors (upto /).
A directory may have both l2.env AND l2config.env

Edit2:

Added a new option for l2command l2 --env <l2FilePath>.
L2 will output a JSON of environment variables with metadata from l2config.env and l2.env.
Added this feature so frontend i.e. IntelliJ/ VScode suggesting variables can be done in a quicker and more detailed high-quality way.

{
  "KARMA_LOCAL_URL": {
    "src": "l2configenv",
    "val": "http://127.0.0.1:8000"
  },
  "PARSE_URL_LOCAL": {
    "src": "l2env",
    "val": "http://localhost:1337/parse"
  }
}

Important code file to start Code Review from

controller.go

Added tests?

👍 yes
🙅 no, because they aren't needed
🙋 no, because I need help

Added to documentation?

📓 l2format.md
📓 architecture.md
📜 preprocess.md
📜 examples.md

shrsv · 2023-07-10T07:55:11Z

After every documentation change - we are supposed to run the following and commit all resultant files:

mkdocs:
	cd docs/Lama2; ./build.sh

Anyway - wait till my overall review finishes, then you can do this.

shrsv · 2023-07-10T08:08:41Z

docs/Lama2/docs/tutorials/examples.md

@@ -69,8 +69,18 @@ Get [Source File](https://github.com/HexmosTech/Lama2/tree/main/examples/0003_co

 ## Environment Variables: Switch base URL



We can make the documentation more structured (something like this)

Case 1: l2.env adjacent to an API file

For any given .l2 file, one can place an l2.env file to store relevant variables. These variables will be available to be used within the API file

[example]

Case 2: Project-level variables

In Lama2, you can have a large number of API files stored in a hierarchical folder configuration. The root of such a project can be signified through l2config.env:

[API Hub folder screenshot with l2config.env highlighted]

Within such a structure, you can have an API file anywhere, which can use variables defined in the project-level variables:

[Example l2config.env and api.l2 with appropriate folder structure; the example must be hosted in Lama2 repo (see other examples)]

Case 3: Override Project-level variable with local variable

....

shrsv · 2023-07-10T08:10:10Z

docs/Lama2/docs/reference/preprocess.md

@@ -67,7 +67,7 @@ func GetLamaFileAsString(path string) string
 func LamaFile(inputFile string) (string, string)
 ```

-LamaFile takes in a path to an API file. It moves into the API file directory, reads the API contents, loads the \`l2.env\` file if available, and finally substitutes environment vars in the API contents Once done, it reverts back to the original directory, and returns the processed l2 file.
+LamaFile takes in a path to an API file. It moves into the API file directory, reads the API contents, loads the \`l2config.env\` global variables file if available in any of parent directory and \`l2.env\` file if available in present directory , and finally substitutes environment vars in the API contents Once done, it reverts back to the original directory, and returns the processed l2 file.


Comment can make the "overwriting" part more obvious. Sentence is too long now, maybe simplify a bit.

shrsv · 2023-07-10T08:11:42Z

docs/Lama2/docs/reference/architecture.md

@@ -48,7 +48,7 @@ From a high level, how does it work now?
 	2. Else if block is Requestor block
 		1. Replace variables with values in the following order
 			1. Try fetch variable from Javascript VM
-			2. If (1) fails, try fetch variable from `l2.env`
+			2. If (1) fails, try fetch variable from `l2config.env` and `l2.env`


"overwriting" is an implementation detail. When you talk to users - how does it work?

JS VM variable

Local env variable

l2config env variable.

Always explain in this way. The implementation detail doesn't matter at documentation/explanation level.

Okay got it

shrsv · 2023-07-10T08:12:48Z

docs/Lama2/docs/explanation/l2format.md

@@ -83,6 +83,17 @@ Cookies are specified in a `Cookie` header as follows:
 Cookie:'sessionid=foo;another-cookie=bar'
 ```

+### Global Environments variables/commands can be defined in `/l2config.env/<requests_dir>`
+
+The *l2* loads up variables from `l2config.env` and then `l2.env`.


Again - the overwrite explanation is more complex to the reader/user. Simply say what order in which variables are searched for (nearest first):

VM

local

global

Either:

The l2 binary loads...

or

l2 loads

shrsv · 2023-07-10T08:14:48Z

docs/Lama2/docs/explanation/l2format.md

@@ -83,6 +83,17 @@ Cookies are specified in a `Cookie` header as follows:
 Cookie:'sessionid=foo;another-cookie=bar'
 ```

+### Global Environments variables/commands can be defined in `/l2config.env/<requests_dir>`


Instead of calling it "global", I think we should call it something like:

Project-level variables

Root variables

Project-root variables

Let's pick one of the above and finalize (or make other suggestions).

Second, I think we should use git repo analogy to explain.

I have added the git repo analogy in example.md, should I link it here as well?

Analogy should be in the explanation section, not example section. When confused what belongs where, re-read this

shrsv · 2023-07-10T08:15:56Z

docs/Lama2/docs/explanation/l2format.md

@@ -83,6 +83,17 @@ Cookies are specified in a `Cookie` header as follows:
 Cookie:'sessionid=foo;another-cookie=bar'
 ```

+### Global Environments variables/commands can be defined in `/l2config.env/<requests_dir>`


Global Environments variables/commands can be defined in /l2config.env/<requests_dir>

Title is a bit clumsy.

API environment variables can be defined at project level using l2config.env

Is much simpler.

shrsv · 2023-07-10T08:17:20Z

Remaining comments - I think we still do not have functionality for obtaining the variable suggestions from VSCode (using some CLI switch - pls make proposals). That can be added to this MR itself.

shrsv · 2023-07-10T08:18:16Z

In general - nice first MR 👍

shrsv · 2023-07-10T08:21:56Z

Also - we must add test case(s) for this new logic.

- Command to get the envs `l2 --env ${l2FilePath}`. - All the envs of l2config.env is fetched and overwritten by l2.env. - Creates a JSON of all the envs with its value and source of origin.

shrsv · 2023-07-16T14:10:51Z

L2 can now fetch variables declared in l2config.env which can be in the parent directory.

L2 will first fetch the variables from the l2config.env which can be in the parent directory and
then fetch the variables from l2.env from the present dir.

The description has to be more precise.

l2config.env is searched for, from the present directory to all its ancestors (upto /). Yes, a directory may have both l2.env AND l2config.env (make sure this is so)

shrsv · 2023-07-16T14:14:27Z

docs/Lama2/docs/explanation/l2format.md

+
+### API environment variables can be defined at project level using `l2config.env`
+
+_l2_ loads local variables from `l2.env`.


This is confusing to the end user. You mean - at file level or at variable?

The fact is, we load variables from both files. If there is ever a conflict in a particular variable name, the way prioritize selection is:

VM vars

l2.env vars

l2config.env vars

shrsv · 2023-07-16T14:16:12Z

docs/Lama2/docs/explanation/l2format.md

+_l2_ loads local variables from `l2.env`.
+If not found then uses variables from `l2config.env`.
+Example `l2config.env`:
+


Screenshot needed to show project structure, and root.

shrsv · 2023-07-16T14:16:49Z

docs/Lama2/docs/explanation/l2format.md

+### If API environment variables are locally and at project level
+
+The local variable's value is taken into consideration
+


Must show full example here.

shrsv · 2023-07-16T14:19:21Z

preprocess/preprocess.go

@@ -96,11 +99,85 @@ func ExpandJSON(block *gabs.Container, vm *goja.Runtime) {
 	log.Debug().Str("Processed JSON block", block.String()).Msg("")
 }

-func LoadElfEnv(l2path string) {
+func SearchL2ConfigEnv(dir string) (string, error) {
+	parentDir := filepath.Dir(dir)


We should start from present folder itself. What if it is a project without any subdirs.

Yeah this case failed :(

shrsv · 2023-07-16T14:23:00Z

preprocess/preprocess.go

+		return
+	}
+	defer file.Close()
+	return godotenv.Parse(file)


One thing to verify is - does this properly handle back-tick commands?

Say I have an echo statement or something, is it going to do the right thing?

Are our test cases checking this case?

Like this in the l2.env file ?

export ECHO_CMD="`echo "base64 image.jpeg"`"

export PHOTO=`base64 image.jpeg`

?

yes, the photo case must work (check tests/examples).

in general, the backticks can have any arbitrary linux command - which runs first, and then the output is stored in the env variable.

No there is no test on that, will add

shrsv · 2023-07-16T14:24:22Z

preprocess/preprocess.go

+		return err
+	}
+	for key, value := range envs {
+		variable := make(map[string]interface{})


8 spaces? use the gofumpt or some other thing in the makefile to format.

You can run both of these and make any mods:

lint: golangci-lint run --disable-all -E revive ./... gofumpt: gofumpt -w .

shrsv · 2023-07-16T14:27:27Z

preprocess/preprocess.go

+	if err != nil {
+			log.Error().Str("Type", "Preprocess").Msg(err.Error())
+	} else {
+			err = populateEnvMap(finalEnvMap, l2ConfigPath, "l2configenv")


This function is confusing. Try to write a pure functions:

explicit inputs

no modification of inputs

clear outputs (new variables)

reassign outputs here

Yes removed the finalMap modifying part.

Removing populateFinalENv() Adding getEnvMap()

l2format.md architecture.md preperocess.md examples.md

shrsv · 2023-07-18T15:46:41Z

controller/controller.go

-		log.Info().Str("Type", "Preprocess").Msg("Env Variables:\n" + string(jsonEnvs))
-    return 
+		// Frontend can read the stdout for this command and get the JSON of all the env's
+		fmt.Println(string(jsonEnvs))


Just check that with verbose mode as well - this works. I think we have a -v and -vv option IIRC.

shrsv · 2023-07-18T15:47:57Z

docs/Lama2/docs/explanation/l2format.md


-The local variable's value is taken into consideration
+The local variable's value is taken into consideration regardless of both files reside in same directory


*residing

shrsv · 2023-07-18T15:48:43Z

preprocess/expandvar.go

@@ -114,8 +114,8 @@ func getShellName(s string) (string, int) {
 	}
 	// Scan alphanumerics.
 	var i int
-	for i = 0; i < len(s) && isAlphaNum(s[i]); i++ {
-	}
+	// for i = 0; i < len(s) && isAlphaNum(s[i]); i++ {


Why comment this? If lines not needed, should be deleted.

shrsv · 2023-07-18T15:50:48Z

preprocess/preprocess.go

+	return envMap, nil
+}
+
+func combineEnvMaps(envMaps ...map[string]map[string]interface{}) map[string]map[string]interface{} {


Add a comment regarding desired order for l2config.env and l2.env in this case. What overwrites what?

shrsv · 2023-07-18T15:51:31Z

Also - we must add test case(s) for this new logic.

This has not been addressed @lovestaco

- Making changes in the Documentation.

lovestaco · 2023-07-19T04:20:48Z

Also - we must add test case(s) for this new logic.

This has not been addressed @lovestaco

https://github.com/HexmosTech/Lama2/blob/globalvar-l2configenv/elfparser/ElfTestSuite/root_variable_override

shrsv · 2023-07-20T15:00:41Z

parser/jsonparser.go

@@ -25,7 +25,7 @@ func (p *Lama2Parser) PrimitiveType() (*gabs.Container, error) {

 // CustomPairMerge uses a gabs feature to deal with merge conflicts.
 // More here: https://github.com/HexmosTech/gabs/blob/master/gabs.go#L511
-func CustomPairMerge(destination, source interface{}) interface{} {
+func CustomPairMerge(_, source interface{}) interface{} {


Why rename this in jsonparser?

It was erroring out in the make lint

shrsv

LGTM 🥳

lovestaco added 4 commits July 9, 2023 20:41

search and load l2config.env file

cca4db6

consolidating LoadElfEnv and LoadConfigEnv

0c87717

Adding comments for LoadEnvironments function

84a4848

Updating documentation with global variable support

4dd0f23

shrsv reviewed Jul 10, 2023

View reviewed changes

lovestaco added 6 commits July 12, 2023 09:28

examples.md

655a236

Updating preprocess, documentation

548e43e

Updating architecture, documentation

48f3154

Updating l2format, documentation

d12b69b

Adding command to get JSON of envs

549affb

- Command to get the envs `l2 --env ${l2FilePath}`. - All the envs of l2config.env is fetched and overwritten by l2.env. - Creates a JSON of all the envs with its value and source of origin.

Updating l2format and examples, documentation

a66383b

lovestaco changed the title ~~Global variable support for Lama2~~ Global variable support and Generate env JSON option Jul 16, 2023

lovestaco mentioned this pull request Jul 16, 2023

Auto suggest environment variables using l2 --env <path> command HexmosTech/Lama2Code#10

Merged

13 tasks

shrsv reviewed Jul 16, 2023

View reviewed changes

lovestaco self-assigned this Jul 16, 2023

lovestaco added 5 commits July 17, 2023 20:27

Search l2configenv from the l2file level

be15f49

Pure functions for GetL2EnvVariables

a1b1b33

Removing populateFinalENv() Adding getEnvMap()

Updating documentation

caece11

l2format.md architecture.md preperocess.md examples.md

Make lint and gofumpt, lint errors

36d2d4f

Adding varjson variable example, documentation

ffc95e9

shrsv reviewed Jul 18, 2023

View reviewed changes

Adding root override and backtick echo var, test

82690f1

- Making changes in the Documentation.

lovestaco added 6 commits July 20, 2023 09:06

Adding env_command, test

c19afee

Updating log to os.Stderr, logging

ce1359c

Linting and adding verbose, test

251ddef

Adding env_command, test

cbe799f

Updating testapp.yml, build and test

58fbc92

Relative path for binary, test

2b505fc

shrsv reviewed Jul 20, 2023

View reviewed changes

Fixing all errors, linting

312d5bd

shrsv approved these changes Jul 20, 2023

View reviewed changes

shrsv merged commit 33858b3 into main Jul 20, 2023
1 check passed

		@@ -69,8 +69,18 @@ Get [Source File](https://github.com/HexmosTech/Lama2/tree/main/examples/0003_co

		## Environment Variables: Switch base URL


		### API environment variables can be defined at project level using `l2config.env`

		_l2_ loads local variables from `l2.env`.

		### If API environment variables are locally and at project level

		The local variable's value is taken into consideration

Global variable support and Generate env JSON option #24

Global variable support and Generate env JSON option #24

Conversation

lovestaco commented Jul 10, 2023 • edited Loading

What type of MR is this?

Description

Important code file to start Code Review from

Added tests?

Added to documentation?

shrsv commented Jul 10, 2023

shrsv Jul 10, 2023 • edited Loading

Choose a reason for hiding this comment

Case 1: l2.env adjacent to an API file

Case 2: Project-level variables

Case 3: Override Project-level variable with local variable

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shrsv Jul 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shrsv commented Jul 10, 2023

shrsv commented Jul 10, 2023

shrsv commented Jul 10, 2023

shrsv commented Jul 16, 2023

shrsv Jul 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shrsv Jul 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lovestaco Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

shrsv Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shrsv commented Jul 18, 2023

lovestaco commented Jul 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shrsv left a comment

Choose a reason for hiding this comment

lovestaco commented Jul 10, 2023 •

edited

Loading

shrsv Jul 10, 2023 •

edited

Loading

Case 1: `l2.env` adjacent to an API file

shrsv Jul 12, 2023 •

edited

Loading

shrsv Jul 16, 2023 •

edited

Loading

shrsv Jul 16, 2023 •

edited

Loading

lovestaco Jul 18, 2023 •

edited

Loading

shrsv Jul 18, 2023 •

edited

Loading