Initial sync with polar best practices

[kurisu-dotto-komu]

I've implemented the webhook-based product syncing, but it appears that the products will only be added to convex when the product created webhook is triggered when it is created in polar after things are all setup.

This means polar products won't get synced if you connect a new convex project to an existing polar project.

This is problematic for local dev and for ensuring consistency between polar and convex (if say, a product is made while the database is offline for some reason).

So, unless I'm missing something, I'm wondering about what is the best approach to ensure that products are consistent without purely relying on the use of webhooks.

Two approaches I can think of:

• Restoring from a backup that includes the product details
• Some kind of process to pulls products from polar and upserts them to convex

Restoring from an export isn't great as it will need to be manually maintained as products are managed. A startup script or cron job that checks for existing products seems better.

So;

• Am I on the right path here or am I missing something?
• If a startup / cron script to sync is a good idea, where should this go?
• For the time being, I guess a convex cron job can do this within the project, but perhaps this common use case that should be part of this package/component - are PRs welcome to implement something like this?

Thank you.

[kurisu-dotto-komu]

For others, here's a convex action I'm using to sync products for now.

import { Polar as PolarSdk } from "@polar-sh/sdk";
import { type Product } from "@polar-sh/sdk/models/components/product";

import { components } from "./_generated/api";
import { internalAction } from "./_generated/server";

const polarSdk = new PolarSdk({
  accessToken: process.env.POLAR_ORGANIZATION_TOKEN!,
});

// instead of archiving, we should delete products that aren't remote (or flag them as deleted)
// but the polar component does't have a delete mutation yet

export const syncPolarProducts = internalAction({
  handler: async (ctx) => {
    const stats = { created: 0, archived: 0, synced: 0 };

    const polarProducts: Product[] = [];

    const pages = await polarSdk.products.list({ isArchived: false });
    for await (const page of pages) {
      if (page.result.items) {
        polarProducts.push(...page.result.items);
      }
    }

    const convexProducts = await ctx.runQuery(components.polar.lib.listProducts, {
      includeArchived: false,
    });

    // archive products that aren't in polar - TODO we should delete these...
    for (const convexProduct of convexProducts) {
      if (!polarProducts.some((p) => p.id === convexProduct.id)) {
        await ctx.runMutation(components.polar.lib.updateProduct, {
          product: {
            ...convexProduct,
            isArchived: true,
          },
        });
        stats.archived++;
      }
    }

    // upsert polar products
    for (const polarProduct of polarProducts) {
      const product = {
        id: polarProduct.id,
        name: polarProduct.name,
        description: polarProduct.description,
        createdAt: polarProduct.createdAt.toISOString(),
        modifiedAt: polarProduct.modifiedAt?.toISOString() ?? null,
        isArchived: polarProduct.isArchived,
        isRecurring: polarProduct.isRecurring,
        organizationId: polarProduct.organizationId,
        metadata: polarProduct.metadata,
        recurringInterval: polarProduct.recurringInterval,
        medias: polarProduct.medias.map((media) => ({
          ...media,
          createdAt: media.createdAt.toISOString(),
          lastModifiedAt: media.lastModifiedAt?.toISOString() ?? null,
        })),
        prices: polarProduct.prices.map((price) => ({
          ...price,
          createdAt: price.createdAt.toISOString(),
          modifiedAt: price.modifiedAt?.toISOString() ?? null,
        })),
      };

      const existingConvexProduct = convexProducts.find((p) => p.id === polarProduct.id);
      if (existingConvexProduct) {
        // TODO deep check and skip if the same?
        await ctx.runMutation(components.polar.lib.updateProduct, { product });
        stats.synced++;
      } else {
        await ctx.runMutation(components.polar.lib.createProduct, { product });
        stats.created++;
      }
    }
    const statsString = Object.entries(stats)
      .filter(([_, value]) => value > 0)
      .map(([key, value]) => `${key}: ${value}`)
      .join(", ");
    console.log(`Polar products - ${statsString}`);
  },
});

[erquhart]

Thanks for this - you're dead on, the component is built for greenfield projects. Would make sense to add a way to sync existing products, agree.