feat: add createSearchTerms

.changeset/fruity-shoes-talk.md

@@ -0,0 +1,5 @@
+---
+"@cipherstash/protect": minor
+---
+
+Implemented createSearchTerms for a streamlined way of working with encrypted search terms.

docs/reference/supabase-sdk.md

@@ -14,18 +14,31 @@ Upvote this [issue](https://github.com/cipherstash/protectjs/issues/135) and fol
 When searching encrypted data, you need to convert the encrypted payload into a format that PostgreSQL and the Supabase SDK can understand. The encrypted payload needs to be converted to a raw composite type format by double stringifying the JSON:
 
 ```typescript
-const searchResult = await protectClient.encrypt('billy@example.com', {
-  column: users.email,
-  table: users,
-})
-
-const searchTerm = `(${JSON.stringify(JSON.stringify(searchResult.data))})`
+const searchTerms = await protectClient.createSearchTerms([
+  {
+    value: 'billy@example.com',
+    column: users.email,
+    table: users,
+    returnType: 'composite-literal'
+  }
+])
+
+const searchTerm = searchTerms.data[0]
 ```
 
-For certain queries, when including the encrypted search term with an operator that uses the string logic syntax, your need to triple stringify the payload.
+For certain queries, when including the encrypted search term with an operator that uses the string logic syntax, you need to use the 'escaped-composite-literal' return type:
 
 ```typescript
-const searchTerm = `${JSON.stringify(`(${JSON.stringify(JSON.stringify(searchResult.data))})`)}`
+const searchTerms = await protectClient.createSearchTerms([
+  {
+    value: 'billy@example.com',
+    column: users.email,
+    table: users,
+    returnType: 'escaped-composite-literal'
+  }
+])
+
+const searchTerm = searchTerms.data[0]
 ```
 
 ## Query Examples
@@ -35,33 +48,37 @@ Here are examples of different ways to search encrypted data using the Supabase
 ### Equality Search
 
 ```typescript
-const searchResult = await protectClient.encrypt('billy@example.com', {
-  column: users.email,
-  table: users,
-})
-
-const searchTerm = `(${JSON.stringify(JSON.stringify(searchResult.data))})`
+const searchTerms = await protectClient.createSearchTerms([
+  {
+    value: 'billy@example.com',
+    column: users.email,
+    table: users,
+    returnType: 'composite-literal'
+  }
+])
 
 const { data, error } = await supabase
   .from('users')
   .select('id, email::jsonb, name::jsonb')
-  .eq('email', searchTerm)
+  .eq('email', searchTerms.data[0])
 ```
 
 ### Pattern Matching Search
 
 ```typescript
-const searchResult = await protectClient.encrypt('example.com', {
-  column: users.email,
-  table: users,
-})
-
-const searchTerm = `(${JSON.stringify(JSON.stringify(searchResult.data))})`
+const searchTerms = await protectClient.createSearchTerms([
+  {
+    value: 'example.com',
+    column: users.email,
+    table: users,
+    returnType: 'composite-literal'
+  }
+])
 
 const { data, error } = await supabase
   .from('users')
   .select('id, email::jsonb, name::jsonb')
-  .like('email', searchTerm)
+  .like('email', searchTerms.data[0])
 ```
 
 ### IN Operator Search
@@ -70,25 +87,26 @@ When you need to search for multiple encrypted values, you can use the IN operat
 
 ```typescript
 // Encrypt multiple search terms
-const searchResult1 = await protectClient.encrypt('value1', {
-  column: users.name,
-  table: users,
-})
-
-const searchResult2 = await protectClient.encrypt('value2', {
-  column: users.name,
-  table: users,
-})
-
-// Format each search term
-const searchTerm = `${JSON.stringify(`(${JSON.stringify(JSON.stringify(searchResult.data))})`)}`
-const searchTerm2 = `${JSON.stringify(`(${JSON.stringify(JSON.stringify(searchResult2.data))})`)}`
+const searchTerms = await protectClient.createSearchTerms([
+  {
+    value: 'value1',
+    column: users.name,
+    table: users,
+    returnType: 'escaped-composite-literal'
+  },
+  {
+    value: 'value2',
+    column: users.name,
+    table: users,
+    returnType: 'escaped-composite-literal'
+  }
+])
 
 // Combine terms for IN operator
 const { data, error } = await supabase
   .from('users')
   .select('id, email::jsonb, name::jsonb')
-  .filter('name', 'in', `(${searchTerm1},${searchTerm2})`)
+  .filter('name', 'in', `(${searchTerms.data[0]},${searchTerms.data[1]})`)
 ```
 
 ### OR Condition Search
@@ -97,27 +115,32 @@ You can combine multiple encrypted search conditions using the `.or()` syntax. T
 
 ```typescript
 // Encrypt search terms for different columns
-const emailSearch = await protectClient.encrypt('user@example.com', {
-  column: users.email,
-  table: users,
-})
-
-const nameSearch = await protectClient.encrypt('John', {
-  column: users.name,
-  table: users,
-})
-
-// Format each search term
-const emailTerm = `${JSON.stringify(`(${JSON.stringify(JSON.stringify(emailSearch.data))})`)}`
-const nameTerm = `${JSON.stringify(`(${JSON.stringify(JSON.stringify(nameSearch.data))})`)}`
+const searchTerms = await protectClient.createSearchTerms([
+  {
+    value: 'user@example.com',
+    column: users.email,
+    table: users,
+    returnType: 'escaped-composite-literal'
+  },
+  {
+    value: 'John',
+    column: users.name,
+    table: users,
+    returnType: 'escaped-composite-literal'
+  }
+])
 
 // Combine conditions with OR
 const { data, error } = await supabase
   .from('users')
   .select('id, email::jsonb, name::jsonb')
-  .or(`email.ilike.${emailTerm}, name.ilike.${nameTerm}`)
+  .or(`email.ilike.${searchTerms.data[0]}, name.ilike.${searchTerms.data[1]}`)
 ```
 
 ## Conclusion
 
-The key is in the string formatting of the encrypted payload: `(${JSON.stringify(JSON.stringify(searchTerm))})`. This ensures the encrypted data is properly formatted for comparison in the database using the EQL custom type. You can use this pattern with any of Supabase's query methods like `.eq()`, `.like()`, `.ilike()`, etc.
+The key is in using the appropriate return type for your search terms:
+- Use `composite-literal` for simple equality and pattern matching queries
+- Use `escaped-composite-literal` when you need to include the search term in string-based operators like IN or OR conditions
+
+You can use these patterns with any of Supabase's query methods like `.eq()`, `.like()`, `.ilike()`, etc.

packages/protect/__tests__/search-terms.test.ts

@@ -0,0 +1,90 @@
+import 'dotenv/config'
+import { describe, expect, it } from 'vitest'
+
+import { protect, csTable, csColumn, type SearchTerm } from '../src'
+
+const users = csTable('users', {
+  email: csColumn('email').freeTextSearch().equality().orderAndRange(),
+  address: csColumn('address').freeTextSearch(),
+})
+
+describe('create search terms', () => {
+  it('should create search terms with default return type', async () => {
+    const protectClient = await protect(users)
+
+    const searchTerms = [
+      {
+        value: 'hello',
+        column: users.email,
+        table: users,
+      },
+      {
+        value: 'world',
+        column: users.address,
+        table: users,
+      },
+    ] as SearchTerm[]
+
+    const searchTermsResult = await protectClient.createSearchTerms(searchTerms)
+
+    if (searchTermsResult.failure) {
+      throw new Error(`[protect]: ${searchTermsResult.failure.message}`)
+    }
+
+    expect(searchTermsResult.data).toEqual(
+      expect.arrayContaining([
+        expect.objectContaining({
+          c: expect.any(String),
+        }),
+      ]),
+    )
+  }, 30000)
+
+  it('should create search terms with composite-literal return type', async () => {
+    const protectClient = await protect(users)
+
+    const searchTerms = [
+      {
+        value: 'hello',
+        column: users.email,
+        table: users,
+        returnType: 'composite-literal',
+      },
+    ] as SearchTerm[]
+
+    const searchTermsResult = await protectClient.createSearchTerms(searchTerms)
+
+    if (searchTermsResult.failure) {
+      throw new Error(`[protect]: ${searchTermsResult.failure.message}`)
+    }
+
+    const result = searchTermsResult.data[0] as string
+    expect(result).toMatch(/^\(.*\)$/)
+    expect(() => JSON.parse(result.slice(1, -1))).not.toThrow()
+  }, 30000)
+
+  it('should create search terms with escaped-composite-literal return type', async () => {
+    const protectClient = await protect(users)
+
+    const searchTerms = [
+      {
+        value: 'hello',
+        column: users.email,
+        table: users,
+        returnType: 'escaped-composite-literal',
+      },
+    ] as SearchTerm[]
+
+    const searchTermsResult = await protectClient.createSearchTerms(searchTerms)
+
+    if (searchTermsResult.failure) {
+      throw new Error(`[protect]: ${searchTermsResult.failure.message}`)
+    }
+
+    const result = searchTermsResult.data[0] as string
+    expect(result).toMatch(/^".*"$/)
+    const unescaped = JSON.parse(result)
+    expect(unescaped).toMatch(/^\(.*\)$/)
+    expect(() => JSON.parse(unescaped.slice(1, -1))).not.toThrow()
+  }, 30000)
+})

packages/protect/src/ffi/index.ts

@@ -7,15 +7,18 @@ import type {
   Client,
   Decrypted,
   EncryptedPayload,
+  EncryptedSearchTerm,
   EncryptOptions,
   EncryptPayload,
+  SearchTerm,
 } from '../types'
 import { EncryptModelOperation } from './operations/encrypt-model'
 import { DecryptModelOperation } from './operations/decrypt-model'
 import { BulkEncryptModelsOperation } from './operations/bulk-encrypt-models'
 import { BulkDecryptModelsOperation } from './operations/bulk-decrypt-models'
 import { EncryptOperation } from './operations/encrypt'
 import { DecryptOperation } from './operations/decrypt'
+import { SearchTermsOperation } from './operations/search-terms'
 import {
   type EncryptConfig,
   encryptConfigSchema,
@@ -136,6 +139,18 @@ export class ProtectClient {
     return new BulkDecryptModelsOperation(this.client, input)
   }
 
+  /**
+   * Create search terms to use in a query searching encrypted data
+   * Usage:
+   *    await eqlClient.createSearchTerms(searchTerms)
+   *    await eqlClient.createSearchTerms(searchTerms).withLockContext(lockContext)
+   */
+  createSearchTerms(
+    terms: SearchTerm[],
+  ): Promise<Result<EncryptedSearchTerm[], ProtectError>> {
+    return new SearchTermsOperation(this.client, terms).execute()
+  }
+
   /** e.g., debugging or environment info */
   clientInfo() {
     return {

packages/protect/src/ffi/operations/search-terms.ts

@@ -0,0 +1,55 @@
+import { encryptBulk } from '@cipherstash/protect-ffi'
+import { withResult, type Result } from '@byteslice/result'
+import { noClientError } from '../index'
+import { type ProtectError, ProtectErrorTypes } from '../..'
+import { logger } from '../../../../utils/logger'
+import type { Client, EncryptedSearchTerm, SearchTerm } from '../../types'
+
+export class SearchTermsOperation {
+  private client: Client
+  private terms: SearchTerm[]
+
+  constructor(client: Client, terms: SearchTerm[]) {
+    this.client = client
+    this.terms = terms
+  }
+
+  public async execute(): Promise<Result<EncryptedSearchTerm[], ProtectError>> {
+    logger.debug('Creating search terms', {
+      terms: this.terms,
+    })
+
+    return await withResult(
+      async () => {
+        if (!this.client) {
+          throw noClientError()
+        }
+
+        const encryptedSearchTerms = await encryptBulk(
+          this.client,
+          this.terms.map((term) => ({
+            plaintext: term.value,
+            column: term.column.getName(),
+            table: term.table.tableName,
+          })),
+        )
+
+        return this.terms.map((term, index) => {
+          if (term.returnType === 'composite-literal') {
+            return `(${JSON.stringify(JSON.stringify(encryptedSearchTerms[index]))})`
+          }
+
+          if (term.returnType === 'escaped-composite-literal') {
+            return `${JSON.stringify(`(${JSON.stringify(JSON.stringify(encryptedSearchTerms[index]))})`)}`
+          }
+
+          return encryptedSearchTerms[index]
+        })
+      },
+      (error) => ({
+        type: ProtectErrorTypes.EncryptionError,
+        message: error.message,
+      }),
+    )
+  }
+}

packages/protect/src/types.ts

@@ -13,6 +13,24 @@ export type Client = Awaited<ReturnType<typeof newClient>> | undefined
  */
 export type EncryptedPayload = Encrypted | null
 
+/**
+ * Represents a value that will be encrypted and used in a search
+ */
+export type SearchTerm = {
+  value: string
+  column: ProtectColumn
+  table: ProtectTable<ProtectTableColumn>
+  returnType?: 'eql' | 'composite-literal' | 'escaped-composite-literal'
+}
+
+/**
+ * The return type of the search term based on the return type specified in the `SearchTerm` type
+ * If the return type is `eql`, the return type is `EncryptedPayload`
+ * If the return type is `composite-literal`, the return type is `string` where the value is a composite literal
+ * If the return type is `escaped-composite-literal`, the return type is `string` where the value is an escaped composite literal
+ */
+export type EncryptedSearchTerm = EncryptedPayload | string
+
 /**
  * Represents a payload to be encrypted using the `encrypt` function
  * We currently only support the encryption of strings