What types of variables can Luau scripts create using qsv luau?

[derekmahar]

Can qsv lua or qsv luajit scripts

• Create list or array variables?
• Create map, dictionary, or associative array variables? (I'm not familiar with Lua, so I don't know whether Lua supports variables of these types or even what it calls them.)
• Create variables which are not columns in the output CSV file?
• Refer to columns in a previous row of a CSV file, including those which the script calculates?

[jqnatividad]

@derekmahar , your question inspired me to expand the Lua functionality of qsv:

• replaced Lua with Luau. Luau is an expanded version of Lua extended by Roblox with type checking, sandboxing, additional language operators, and improved performance.
• added --prologue and --epilogue options. The prologue allows you to run a Luau/LuaJIT script before the CSV is processed, typically for initializing variables. The epilogue is ran after the CSV is processed, typically for doing complex aggregations.
• added special variables _idx and _rowcount. _idx contains the current row number while processing the CSV. _rowcount is set to zero while the CSV is being processed, and to the row count during the epilogue.

With these changes, going to your questions:

• Create list or array variables - by using the --prologue to initialize variables, including arrays.
• Create map, dictionary or associative array variables - by using Lua Tables, and perhaps, taking advantage of the _idx variable to store columns in a Table, using _idx as index.
• Create variables which are not columns in the output CSV file - by using the --prologue option to initialize additional variables, and perhaps, even the --epilogue to create additional variables to facilitate computing complex aggregations that you can capture from stderr.
• Refer to columns in a previous row of a CSV file, including those which the script calculates - by making judicious use of the prologue to declare arrays, into which you can store each column selectively as each row is processed by using _idx.

To see an example of these extended capabilities, check out this test:

https://github.com/jqnatividad/qsv/blob/c7aa6e656d9bd7fa04339b7a28c11c5121728878/tests/test_luau.rs#L165-L204

[derekmahar]

@jqnatividad, thank you for so quickly and cleverly implementing these features so that you could answer "yes" to all of my questions! I'm pleased and impressed that I inspired you to make qsv more closely resemble AWK for CSV files! (Have you ever considered creating an AWK-like tool similar to Hawk that uses Lua or Luau as the embedded programming language?)

I must spend some time learning more about Lua tables before I can try these new features.

[derekmahar]

@jqnatividad, how difficult was it to replace the Lau interpreter with Luau?

[jqnatividad]

@derekmahar , I'm glad you liked it!

TBH, I got tired of fighting python's distribution and GIL limitations with the py command, and was really pleasantly surprised myself to find how performant and problem-free lua is in contrast.

I'm humbled by the "AWK for CSV" comparison. As for creating an AWK-like tool using Lua/Luau, I have enough on my plate with qsv and my CKAN-related projects 😄 - but I'm always open to new idears to make data-wrangling/data management better.

And it took me about a day to pull together the --prologue and --epilogue functionality. Actually, switching to luau was a late decision as I was putting together the CI tests for prologue/epilogue, as the test I wrote that was informed by your question was not working as I hoped.

As I was troubleshooting the test, I found out about luau's += operator and if-then-else expression, and decided to switch after learning more about luau.

Switching from Lua to Luau was not that big an effort thanks in large part to the wonderful mlua crate.

[derekmahar]

What is CKAN?

[jqnatividad]

CKAN stands for the "Comprehensive Knowledge Archive Network", and is the technology behind data portals like https://data.gov, https://humdata.org, http://wprdc.org, https://data.boston.gov, https://data.nhm.ac.uk/ and https://open.canada.ca/en/open-data.

You can find out more at https://ckan.org. We actually have a presentation this Wednesday that you may want to attend if you want to learn more about CKAN 3.0.

https://ckan.org/events/ckan-monthly-live-november-ckan-pathways-to-3-0

[derekmahar]

How does CKAN compare to Dataverse?

[jqnatividad]

CKAN, as you can see from the different example sites, is more of a Data Management System (DMS) framework with which you can build all kinds of DMS solutions. It has a very active community with hundreds of community-built extensions. Dataverse is more of a turn-key system targeted to academia.