Ingest with shell expanded list of files throws "Error: accepts at most 1 arg(s), received x"

[lukasmalkmus]

The example of the ingest command says:

# Ingest the contents of all files inside /var/logs/nginx with
# extension ".log" into a dataset named "nginx-logs":
$ axiom ingest nginx-logs -f /var/logs/nginx/*.log

But it actually throws Error: accepts at most 1 arg(s), received x, with x being the amount of actual files the shell expanded to.

[lukasmalkmus]

This is actually harder than expected because values not associated with a flag are treated as "normal" arguments. And the ingest command already takes the dataset as an argument. This could make detection of dataset arguments and file arguments tricky.

Imagine

$ axiom ingest nginx-logs -f /var/logs/nginx/*.log

which will expand to

$ axiom ingest nginx-logs -f /var/logs/nginx/access.log /var/logs/nginx/error.log

We could get away with this by treating every argument after the -f flag as an argument to that very flag. But what if users swap flag and dataset argument?

$ axiom ingest -f /var/logs/nginx/access.log /var/logs/nginx/error.log nginx-logs

This would treat the nginx-logs argument as a file and not as the dataset to ingest into.

Feel free to propose solutions or other ideas.

/cc @cdeutsch

[cdeutsch]

@lukasmalkmus that's weird.

What's expanding "/var/logs/nginx/*.log" to an array? The OS?

[lukasmalkmus]

This is called shell expansion and is performed by your shell (most likely bash, zsh or fish) before the command is run. It is actually not that weird. Just harder to handle for us, because we take a flag argument.

Another, simple solution would be to make the dataset a flag argument and treat non-flag arguments as files:

$ axiom ingest -d test *.json

This would work.

[cdeutsch]

Yeah, not sure what's the best experience. How do most programs handle this?

Supporting wild cards isn't critical IMO, so can possibly drop this

[lukasmalkmus]

Good question tbh. I think I'll leave this open for discussion for now. I could argue for/against both approaches. Something else that came to my mind is, that this issue can currently be worked around:

$ cat *.json | axiom ingest test