feat!: implement new parsing into put command

Introduce pest parsing into put command.
This commit adds new features and breaking changes, but not limited to,

* Support binary types.
* Support single quotes for a string type.
* Support explicit set types.

However, it no longer supports the inference of set type when using `[elem]`
because we can now specify set type explicitly using `<<elem>>` literal.

BREKING CHANGE: put command no longer infers a set type.

README.md

@@ -553,7 +553,8 @@ dynein provides subcommands to write to DynamoDB tables as well.
 
 #### `dy put`
 
-`dy put` internally calls [PutItem API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html) and save an item to a target table. To save an item, you need to pass at least primary key that identifies an item among the table.
+`dy put` internally calls [PutItem API](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html) and save an item to a target table.
+To save an item, you need to pass at least primary key that identifies an item among the table.
 
 ```
 $ dy admin create table write_test --keys id,N
@@ -566,7 +567,8 @@ id  attributes
 123
 ```
 
-Additionally you can include item body (non-key attributes) by passing `--item` or `-i` option. The `--item` option takes JSON style syntax.
+Additionally, you can include an item body (non-key attributes) by passing `--item` or `-i` option.
+The `--item` option takes a JSON-style expression with extended syntax.
 
 ```
 $ dy put 456 --item '{"a": 9, "b": "str"}'
@@ -578,7 +580,9 @@ id  attributes
 456  {"a":9,"b":"str"}
 ```
 
-As dynein's `--item` option automatically transform standard JSON into DynamoDB style JSON syntax, writing items into a table would be simpler than AWS CLI. See following comparison:
+As the parameter of the `--item` option automatically transforms into DynamoDB-style JSON syntax,
+writing items into a table would be more straightforward than AWS CLI.
+See the following comparison:
 
 ```
 $ dy put 789 --item '{"a": 9, "b": "str"}'
@@ -587,10 +591,13 @@ $ dy put 789 --item '{"a": 9, "b": "str"}'
 $ aws dynamodb put-item --table-name write_test --item '{"id": {"N": "456"}, "a": {"N": "9"}, "b": {"S": "str"}}'
 ```
 
-Finally, in addition to the string ("S") and nubmer ("N"), dynein also supports other data types such as boolean ("BOOL"), null ("NULL"), string set ("SS"), number set ("NS"), list ("L"),  and nested object ("M").
+Please see the [dynein format](./docs/format.md) for details of JSON-style data.
+To summarize, in addition to the string ("S") and number ("N"), dynein also supports other data types such as boolean ("BOOL"),
+null ("NULL"), binary ("B"), string set ("SS"), number set ("NS"), binary set("BS"),
+list ("L"), and nested object ("M").
 
 ```
-$ dy put 999 --item '{"myfield": "is", "nested": {"can": true, "go": false, "deep": [1,2,{"this_is_set": ["x","y","z"]}]}}'
+$ dy put 999 --item '{"myfield": "is", "nested": {"can": true, "go": false, "deep": [1,2,{"this_is_set": <<"x","y","z">>}]}}'
 Successfully put an item to the table 'write_test'.
 $ dy get 999
 {

docs/format.md

@@ -0,0 +1,201 @@
+# Dynein format
+
+Dynein uses a JSON-like format called dynein format to express an item.
+Dynein format is not intended to be compatible with JSON; however, valid JSON should be parsed correctly as dynein format.
+Dynein format is designed to be easy to write and read its intention.
+This format is inspired by [PartiQL](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ql-reference.data-types.html), but we focus on more usability than compatibility for SQL.
+
+NOTE: The current implementation cannot read all valid JSON. This issue should be fixed in the future.
+
+## Supported types
+
+Dynein format supports all DynamoDB types. In other words, you can use the following types;
+
+* Null
+* Boolean
+* Number
+* String
+* Binary
+* List
+* Map
+* Number Set
+* String Set
+* Binary Set
+
+See [the documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html) to learn more data types of DynamoDB.
+
+### Null
+
+You can use `null` to express an attribute with an unknown or undefined state.
+
+```bash
+dy put 1 -i '{"null-field": null}'
+```
+
+### Boolean
+
+A boolean type attribute can store either `true` or `false` to express a boolean value.
+
+```bash
+dy put 5 -i '{"true-field": true, "false-field": false}'
+```
+
+### Number
+
+Numbers type express arbitrary numbers, including integers and fraction numbers up to 38 digits of precision.
+The number must be decimal. You can use exponential notation.
+
+```bash
+dy put 10 -i '{"integer": 1, "fraction": 0.1, "minus": -3, "exponential": -1.23e-3}'
+```
+
+### String
+
+Strings type represents an array of characters encoded with UTF-8.
+You can use both single quotes and double quotes to express a string value.
+
+```bash
+dy put 15 -i '{"date":"2022-02-22T22:22:22Z"}'
+dy put 16 -i "{'date':'2022-02-22T22:22:22Z'}"
+```
+
+You can use escape sequences if you use double quotes to express a string value.
+The complete list of escape sequences is the following;
+
+| Escape Sequence | Character Represented by Sequence |
+|-----------------|-----------------------------------|
+|       \0        | An ASCII NUL (X'00') character    |
+|       \r        | A carriage return character       |
+|       \n        | A newline (linefeed) character    |
+|       \t        | A tab character                   |
+|       \\\\      | A backslash (\\) character        |
+|       \\\"      | A double quote (") character      |
+|       \\\'      | A single quote (') character      |
+
+```bash
+dy put 17 -i '{"escape":"\"hello\",\tworld!\n"}'
+```
+
+On the other hand, you cannot use escape sequences if you use single quotes to express a string value.
+String values are evaluated as is.
+Because of shell behavior, you need special handling to input a single quote in the single-quoted argument.
+
+```bash
+dy put 18 -i '{"raw":'\''hello,\tworld!\n'\''}'
+```
+
+The above example creates an item with an attribute, `{"raw":{"S":"hello,\tworld!\n"}}`.
+Or, you can use a heredoc.
+
+```bash
+dy put 19 -i "$(cat <<EOS
+{
+  "escape":"hello,\tworld!\n",
+  "raw":'hello,\tworld!\n'
+}
+EOS
+)"
+```
+
+### Binary
+You can store any binary data as binary type. There are two type of literals.
+
+When you use `b"<binary-data>"` style, you can use the following escape sequences.
+
+| Escape Sequence | Character Represented by Sequence                    |
+|-----------------|------------------------------------------------------|
+| \0              | An ASCII NUL (X'00') character                       |
+| \r              | A carriage return character                          |
+| \n              | A newline (linefeed) character                       |
+| \t              | A tab character                                      |
+| \\\\            | A backslash (\\) character                           |
+| \\\"            | A double quote (") character                         |
+| \\\'            | A single quote (') character                         |
+| \x41            | 7-bit character code (exactly 2 digits, up to 0x7F)  |
+
+Additionally, you can skip leading spaces, including `\r`, `\n`, `\t` by putting a backslash at the end of a line.
+
+input.json
+```json
+{
+  "binary": b"Thi\x73 is a \
+              bin.\r\n"
+}
+```
+
+command
+```bash
+dy put 20 -i "$(cat input.json)"
+```
+
+When you use `b'<binary-data>'` style, binary data cannot span multiple lines.
+
+### List
+You can store an ordered collection of values using list type. Lists are enclosed in square brackets: `[ ... ]`.
+A list is similar to a JSON array. There are no restrictions on the data types that can be stored in a list element, and the elements in a list element do not have to be of the same type.
+
+The following example shows a list that contains two strings and a number.
+
+```bash
+dy put 25 -i '{"FavoriteThings": ["Cookies", "Coffee", 3.14159]}'
+```
+
+### Map
+You can use Map type to store an unordered collection of name-value pairs.
+Maps are enclosed in curly braces: `{ ... }`.
+A map is similar to a JSON object.
+There are no restrictions on the data types that can be stored in a map element,
+and the elements in a map do not have to be the same type.
+
+Maps are ideal for storing JSON documents in DynamoDB.
+The following example shows a map that contains a string, a number, and a nested list that contains another map.
+
+```bash
+dy put 30 -i '{
+    "Day": "Monday",
+    "UnreadEmails": 42,
+    "ItemsOnMyDesk": [
+        "Coffee Cup",
+        "Telephone",
+        {
+            "Pens": { "Quantity" : 3},
+            "Pencils": { "Quantity" : 2},
+            "Erasers": { "Quantity" : 1}
+        }
+    ]
+}'
+```
+
+### Set
+DynamoDB can represent sets of numbers, strings, or binary values.
+Sets are represented by double angle brackets in dynein: `<< ... >>`.
+All the elements within a set must be of the same type.
+For example, a number set can only contain numbers, and a string set can only contain strings.
+
+Dynein automatically inferences the type of set based on its elements.
+
+Each value within a set must be unique.
+The order of the values within a set is not preserved.
+Therefore, you must not rely on any particular order of elements within the set.
+DynamoDB does not support empty sets; however, empty string and binary values are allowed within a set.
+
+#### Number Set
+In the following example, put an item containing a number set.
+
+```bash
+dy put 35 -i '{"number-set": <<0, -1, 1, 2>>}'
+```
+
+#### String Set
+In the following example, put an item containing a string set.
+
+```bash
+dy put 36 -i '{"string-set": <<"0", "-1", "One", "Two">>}'
+```
+
+#### Binary Set
+In the following example, put an item containing a binary set.
+
+```bash
+dy put 37 -i '{"binary-set": <<b"\x00", b"0x01", b"0x02">>}'
+```

src/data.rs

@@ -315,16 +315,14 @@ pub async fn put_item(cx: app::Context, pval: String, sval: Option<String>, item
     match item {
         None => (),
         Some(_i) => {
-            match json_str_to_attributes(&_i) {
+            let parser = DyneinParser::new();
+            let result = parser.parse_put_content(full_item_image, &_i);
+            match result {
                 Ok(attrs) => {
-                    debug!(
-                        "Merging two HashMaps: {:?} <-- extend -- {:?}",
-                        full_item_image, attrs
-                    );
-                    full_item_image.extend(attrs);
+                    full_item_image = attrs;
                 }
                 Err(e) => {
-                    error!("ERROR: failed to load item. {}", e);
+                    error!("ERROR: failed to load item. {:?}", e);
                     std::process::exit(1);
                 }
             };
@@ -578,42 +576,6 @@ fn identify_target(
     target
 }
 
-/// this function converts a JSON string argument passed by `--item/-i` option into DynamoDB AttributeValue.
-/// for simplicity `--item` option does NOT force users to construct DynamoDB JSON. i.e.:
-/// - '{"field": { "S": "val"} }' ... this function doesn't support.
-/// - '{"field": "val"}' ... this is the format this function can convert.
-fn json_str_to_attributes(s: &str) -> Result<HashMap<String, AttributeValue>, serde_json::Error> {
-    debug!(
-        "Trying to convert from standard JSON received from command line into attributes: {}",
-        s
-    );
-
-    // In this line we assume that 1st element of JSON is not an array.
-    // i.e. this type annotation assumes single JSON element:
-    //     {"a": "val", "b": "yay"}
-    // on the other hand, cannot be used for JSON begins with array like:
-    //     [ {"a": "val"}, {"b": "yay"} ]
-    let hashmap: HashMap<String, JsonValue> = serde_json::from_str(s)?; // unknown JSON structure
-    debug!("JSON -> HashMap: {:?}", hashmap);
-
-    let result = convert_jsonval_to_attrvals_in_hashmap_val(hashmap);
-    debug!("JSON has been converted to AttributeValue(s): {:?}", result);
-
-    Ok(result)
-}
-
-/// Keeping key String as it is, this function converts HashMap value from serde_json::Value into DynamoDB AttributeValue.
-fn convert_jsonval_to_attrvals_in_hashmap_val(
-    hashmap: HashMap<String, JsonValue>,
-) -> HashMap<String, AttributeValue> {
-    let mut result = HashMap::<String, AttributeValue>::new();
-    for (k, v) in hashmap {
-        debug!("working on key '{}', and value '{:?}'", k, v);
-        result.insert(k, dispatch_jsonvalue_to_attrval(&v, true));
-    }
-    result
-}
-
 // top 3 scalar types that can be used for primary keys.
 //   ref: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html
 //        https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes

src/expression.pest

@@ -14,7 +14,7 @@
  * limitations under the License.
  */
 
-WHITESPACE = _{ SEPARATOR }
+WHITESPACE = _{ SEPARATOR | "\r\n" | "\n" | "\r" | "\t" }
 
 // EOI always produces token.
 // I use eoi instead of EIO to suppress generating token.