Standardize table schema format #75

asutula · 2023-09-27T18:23:26Z

Uses the Table.Schema type from the SDK. The schema builder component now uses that type to model its data, and when the table is saved to the store, a stringified version of that JSON object is stored.

Closes STU-116.

Signed-off-by: Aaron Sutula <[email protected]>

linear · 2023-09-27T18:23:56Z

STU-116 Standardize Table schema storage format

Currently, when a user creates a Table in Studio, we store the resulting create statement in a text column of the tables table. I started working on importing existing Tableland tables into Studio, and it seems clear that we will get a JSON representation of the columns and table constraints back from the validator. We can store that JSON in the text column of the tables table. The JSON format seems better because we can more easily use it to populate the UI for editing a Table schema, as well as for calculating migrations when editing a Table's schema.

carsonfarmer started on a PR that would give us access to the AST generated from a create statement, so we could easily implement the same logic the Go validator uses to construct the JSON representation of a Table:

tablelandnetwork/go-sqlparser#108

Here's the Go code that uses the create statement plus AST to create JSON:

https://github.com/tablelandnetwork/go-tableland/blob/main/internal/gateway/impl/gateway_store.go#L79

which creates JSON that looks like:

    {
      "columns": [
        {
          "name": "id",
          "type": "int",
          "constraints": [
            "primary key"
          ]
        },
        {
          "name": "name",
          "type": "text",
          "constraints": [
            "not null"
          ]
        },
        {
          "name": "age",
          "type": "int",
          "constraints": [
            "not null"
          ]
        },
        {
          "name": "city",
          "type": "text"
        }
      ],
      "tableConstraints": [
        "unique(name)"
      ]
    }

asutula · 2023-09-27T18:25:44Z

packages/store/src/api/tables.ts

@@ -44,7 +50,7 @@ export function initTables(
        .orderBy(tables.name)
        .all();
      const mapped = res.map((r) => r.tables);
-      return mapped;
+      return mapped as unknown as schema.Table[];


This is weird, see my comment in packages/store/src/schema/index.ts for an explanation.

asutula · 2023-09-27T18:32:32Z

packages/store/src/schema/index.ts

+export type Schema = TablelandTable["schema"];
+
+export type Table = Omit<InferSelectModel<typeof tables>, "schema"> & {
+  schema: Schema;
+};
 export type NewTable = InferInsertModel<typeof tables>;


There is an interesting/quirky behavior when reading data from a table that has a text column, but JSON strings are stored in that column: The validator marshals the data as JSON within the containing JSON response payload, not as a string property of the response payload. This is pretty convenient if you're simply querying the validator and using the returned JSON data because you don't have to do any further parsing of your JSON strings to convert them to objects. It's not so convenient in the context of using an ORM like Drizzle where types are inferred from your table schema definitions. Drizzle says "you defined this column as text, so when you query it, it will come back as text". So to get around that, I hacked this inferred drizzle type which represents the data returned from the tables table. Not ideal, but not sure what else to do short of adding support of an option to the validator to leave text as text no matter what.

What really isn't nice is the asymmetry between the table getting written to with string but returning a Schema

This isn't ideal, but seems like a worthy compromise.

I'm actually looking into Drizzle's support for custom column types. Have a feeling it will work.

asutula · 2023-09-27T18:34:03Z

...s/web/app/[team]/[project]/(project)/deployments/[[...slug]]/_components/exec-deployment.tsx

+
+        const stmt = generateCreateTableStatement(table.name, table.schema);
+        const res = await tbl.exec(stmt);


Convert the Schema object to a create table statement and execute it.

asutula · 2023-09-27T18:35:41Z

packages/web/app/[team]/[project]/new-table/_components/new-table-form.tsx

        values.name,
+        values.description,
+        cleanSchema(schema),


cleanSchema just removes any column definitions that don't have a name (this is possible in the UI to create an effectively empty column. We could change the ux to error out on those columns, but this seems easy enough)

asutula · 2023-09-27T18:36:12Z

packages/web/app/[team]/[project]/new-table/_components/new-table-form.tsx

-          <SchemaBuilder />
-          <pre>{createTableStatementFromObject(createTable, name)}</pre>
+          <SchemaBuilder schema={schema} setSchema={setSchema} />
+          <pre>{generateCreateTableStatement(name, schema)}</pre>


Display the equivalent create statement in real time.

asutula · 2023-09-27T18:38:53Z

packages/web/components/schema-builder.tsx

Nothing too crazy in here, just converting it to use the Schema type from the SDK as the backing state type. @carsonfarmer your readonly Table (and therefore readonly Schema type) forced me to get all functional in here. Lots of destructuring ahead! Actually feels way better then relying on shared references to objects.

asutula · 2023-09-27T18:40:19Z

packages/web/lib/schema.ts

+export type Constraint = "not null" | "primary key" | "unique";
+
+export function hasConstraint(
+  column: schema.Schema["columns"][number],
+  constraint: Constraint,
+) {
+  return column.constraints ? column.constraints.includes(constraint) : false;
+}
+
+export function setConstraint(
+  column: schema.Schema["columns"][number],
+  constraint: Constraint,
+  value: boolean,
+) {
+  const { constraints, ...rest } = column;
+  if (value) {
+    return { ...rest, constraints: [...(constraints || []), constraint] };
+  } else {
+    const res = [...(constraints || [])].filter((c) => c !== constraint);
+    return { ...rest, constraints: res.length ? res : undefined };
+  }
+}


Wrote this little lib to make it easy to check and set constraints on a Column. All functional readonly style as well.

asutula · 2023-09-27T18:41:11Z

packages/web/lib/schema.ts

+export function generateCreateTableStatement(
+  tableName: string,
+  schema: schema.Schema,
+) {
+  if (!tableName.length) {
+    return "";
+  }
+  const cleaned = cleanSchema(schema);
+  const columnDefinitions = cleaned.columns
+    .map((column) => {
+      const definition = `${column.name} ${column.type}`;
+      const columnConstraints = !!column.constraints?.length
+        ? " " + column.constraints.join(" ")
+        : "";
+      return `${definition}${columnConstraints.toLowerCase()}`;
+    })
+    .join(", ");
+
+  const tableConstraints = cleaned.table_constraints
+    ? cleaned.table_constraints.join(",")
+    : "";
+
+  return `create table ${tableName}(${columnDefinitions}${
+    tableConstraints ? `, ${tableConstraints}` : ""
+  });`;
+}


This was mostly taken from the tests that @carsonfarmer wrote in his PR, cleaned up a little to return and empty string if a valid create statement can't be created.

asutula · 2023-09-27T18:43:58Z

packages/web/components/schema-builder.tsx

-      <TableCell>
-        <Input
-          className="w-20"
-          type="text"
-          name="default"
-          placeholder="null"
-          defaultValue={column.default}
-          onChange={(e) => {
-            setCreateTable((prev) => {
-              prev.columns[columnIndex].default = e.target.value;
-              return {
-                ...prev,
-              };
-            });


I removed the option to set a column default constraint. While seemingly simple, default constraints are actually quite complex and translating them between input values and constraint string is tricky because of quoting strings, but not other things, supporting any expression, etc. Plus, if/when we do roll out a more sophisticated data model like in @carsonfarmer's PR, it seems like default constraints may not be supported, at least at first because they are tricky, so this will make us a little more future proof.

joewagner

This looks like some excellent improvements. I'll signoff, but might be good to have @carsonfarmer look it over too.

Signed-off-by: Aaron Sutula <[email protected]>

asutula · 2023-09-27T21:28:43Z

Ok @joewagner, I think I have it working with a custom Drizzle column type, and it is super clean now. It depends on this SDK pr being merged/released tablelandnetwork/tableland-js#39.

I'm having trouble running the web app locally to test it out at runtime. Something about npm linking to my local sdk branch is causing an error ../../node_modules/@tableland/sdk/dist/esm/helpers/chains.js:1:0 Module not found: Can't resolve '@tableland/evm/network.js'. Not sure what's up with that, but would rather not dig into that and just wait til we release a hotfix sdk with those changes.

joewagner · 2023-09-27T21:50:16Z

Ok @joewagner, I think I have it working with a custom Drizzle column type, and it is super clean now. It depends on this SDK pr being merged/released tablelandnetwork/tableland-js#39.

Awesome!

I'm having trouble running the web app locally to test it out at runtime. Something about npm linking to my local sdk branch is causing an error ../../node_modules/@tableland/sdk/dist/esm/helpers/chains.js:1:0 Module not found: Can't resolve '@tableland/evm/network.js'. Not sure what's up with that, but would rather not dig into that and just wait til we release a hotfix sdk with those changes.

Works for me. Without spending much (any) time on it, you might have to also link @tableland/evm?

Signed-off-by: Aaron Sutula <[email protected]>

asutula · 2023-09-27T22:30:36Z

packages/api/src/routers/tables.ts

+const schemaSchema: z.ZodType<Schema> = z.object({
+  columns: z.array(
+    z.object({
+      name: z.string().nonempty(),
+      type: z.string().nonempty(),
+      constraints: z.array(z.string().nonempty()).optional(),
+    }),
+  ),
+  table_constraints: z.array(z.string().nonempty()).optional(),
+});
+


This is the best way I could find to create an api input definition that conforms to the Schema type. At least if the Schema type changes, this will cause a TS error.

asutula · 2023-09-27T22:31:02Z

packages/api/src/routers/tables.ts

        const table = await store.tables.createTable(
          input.projectId,
          input.name,
          input.description,
-          JSON.stringify(tablelandTable.schema),
+          tablelandTable.schema,


Pretty cool, just send the Schema type into the db layer.

asutula · 2023-09-27T22:34:56Z

packages/store/src/custom-types/index.ts

+export type Schema = SDKSchema;
+
+export const schema = customType<{
+  data: Schema;
+}>({
+  dataType() {
+    return "text";
+  },
+  fromDriver(value: unknown): Schema {
+    return value as Schema;
+  },
+  toDriver(value: Schema): string {
+    return JSON.stringify(value);
+  },
+});


Our custom drizzle column type. I had to use unknown in the fromDriver implementation because we have an asymmetric mapping where we go from Schema -> string on the way into the db and Schema -> Schema on the way out of the db. This generic customType function for drizzle works better when it's a symmetric mapping like Schema <-> string.

asutula · 2023-09-27T22:35:53Z

packages/store/src/schema/index.ts

@@ -80,7 +81,7 @@ export const tables = sqliteTable("tables", {
  slug: text("slug").notNull(),
  name: text("name").notNull(),
  description: text("description").notNull(),
-  schema: text("schema").notNull(),
+  schema: schema("schema").notNull(),


Using the custom type in the table definition. I verified this doesn't effect the generated migration (create table statement) because it still just creates a text column to back this (according to our custom type definition).

Signed-off-by: Aaron Sutula <[email protected]>

asutula · 2023-09-27T22:38:16Z

Ok @joewagner and @carsonfarmer this is ready for review again.

joewagner

This is great, definitely looks better with the update to the sdk

asutula added 4 commits September 26, 2023 11:27

wip on new schema format

4b6a256

Signed-off-by: Aaron Sutula <[email protected]>

more wip

f94f48b

Signed-off-by: Aaron Sutula <[email protected]>

converted schema builder to use standard schema type and store that json

c548837

Signed-off-by: Aaron Sutula <[email protected]>

crumb key fix

e198e05

Signed-off-by: Aaron Sutula <[email protected]>

vercel bot deployed to Preview September 27, 2023 18:25 View deployment

asutula commented Sep 27, 2023

View reviewed changes

asutula self-assigned this Sep 27, 2023

asutula requested review from carsonfarmer and joewagner September 27, 2023 18:44

joewagner previously approved these changes Sep 27, 2023

View reviewed changes

custom type working

dc408e4

Signed-off-by: Aaron Sutula <[email protected]>

update to latest sdk, finish custom type

c0b5382

Signed-off-by: Aaron Sutula <[email protected]>

asutula dismissed joewagner’s stale review via c0b5382 September 27, 2023 22:28

asutula requested a review from joewagner September 27, 2023 22:29

vercel bot deployed to Preview September 27, 2023 22:31 View deployment

asutula commented Sep 27, 2023

View reviewed changes

couple last nits

8f45f90

Signed-off-by: Aaron Sutula <[email protected]>

vercel bot deployed to Preview September 27, 2023 22:40 View deployment

joewagner approved these changes Sep 27, 2023

View reviewed changes

asutula merged commit 7c3c4ce into main Sep 27, 2023
1 check passed

asutula deleted the table-schema branch September 27, 2023 23:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standardize table schema format #75

Standardize table schema format #75

asutula commented Sep 27, 2023 •

edited

Loading

linear bot commented Sep 27, 2023

asutula Sep 27, 2023

asutula Sep 27, 2023

asutula Sep 27, 2023

joewagner Sep 27, 2023

asutula Sep 27, 2023

asutula Sep 27, 2023

asutula Sep 27, 2023

asutula Sep 27, 2023

asutula Sep 27, 2023 •

edited

Loading

asutula Sep 27, 2023

asutula Sep 27, 2023

asutula Sep 27, 2023 •

edited

Loading

joewagner left a comment

asutula commented Sep 27, 2023

joewagner commented Sep 27, 2023

asutula Sep 27, 2023

asutula Sep 27, 2023

asutula Sep 27, 2023

asutula Sep 27, 2023

asutula commented Sep 27, 2023

joewagner left a comment


		const stmt = generateCreateTableStatement(table.name, table.schema);
		const res = await tbl.exec(stmt);

Standardize table schema format #75

Standardize table schema format #75

Conversation

asutula commented Sep 27, 2023 • edited Loading

linear bot commented Sep 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asutula Sep 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asutula Sep 27, 2023 • edited Loading

Choose a reason for hiding this comment

joewagner left a comment

Choose a reason for hiding this comment

asutula commented Sep 27, 2023

joewagner commented Sep 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asutula commented Sep 27, 2023

joewagner left a comment

Choose a reason for hiding this comment

asutula commented Sep 27, 2023 •

edited

Loading

asutula Sep 27, 2023 •

edited

Loading

asutula Sep 27, 2023 •

edited

Loading