Use 🛻 pickup to handle pinning service requests #2309

olizilla · 2023-02-08T12:13:20Z

Review the api env, db schema, and cron jobs to find out what needs to be updated to have the api use pickup to handle pinning service requests.

TODO

update api/src/routes/pins-add.js to specify ElasticIpfs as the service when calling createUpload
update api/src/routes/pins-replace.js to specify ElasticIpfs as the service when calling createUpload
update cron/src/bin/pins.js to add a cluster client configured to hit pickup.
update cron/src/jobs/pins.js to add ElasticIpfs to the list of Clusters to check
update cron/src/bin/pins.js to add a check pin statuses on pickup for via status.all
add staging worker env var for CLUSTER_API_URL set to staging pickup url
remove staging worker env var for CLUSTER_SERVICE
add prod worker env var for CLUSTER_API_URL set to prod pickup url
remove prod worker env var for CLUSTER_SERVICE

note: the env var CLUSTER_SERVICE takes precedence over CLUSTER_API_URL, so we can add a value for CLUSTER_API_URL whenever we like, and then delete the CLUSTER_SERVICE env var when we are ready for to make the switch.

Parent: web3-storage/pickup#68

The text was updated successfully, but these errors were encountered:

olizilla · 2023-02-08T12:27:14Z

Either CLUSTER_SERVICE or CLUSTER_API_URL is set in the env.

nft.storage/packages/api/src/config.js

Lines 39 to 49 in 69a49ce

    
           export function serviceConfigFromVariables(vars) { 
        
             let clusterUrl 
        
             if (vars.CLUSTER_SERVICE) { 
        
               clusterUrl = CLUSTER_SERVICE_URLS[vars.CLUSTER_SERVICE] 
        
               if (!clusterUrl) { 
        
                 throw new Error(`unknown cluster service: ${vars.CLUSTER_SERVICE}`) 
        
               } 
        
             } 
        
             if (vars.CLUSTER_API_URL) { 
        
               clusterUrl = vars.CLUSTER_API_URL 
        
             }

CLUSTER_SERVICE takes precedence and is a magic string that is used to select a one of a hard-coded list of cluster urls

nft.storage/packages/api/src/config.js

Lines 16 to 20 in 69a49ce

    
           const CLUSTER_SERVICE_URLS = { 
        
             IpfsCluster: 'https://nft.storage.ipfscluster.io/api/', 
        
             IpfsCluster2: 'https://nft2.storage.ipfscluster.io/api/', 
        
             IpfsCluster3: 'https://nft3.storage.ipfscluster.io/api/', 
        
           }

olizilla · 2023-02-08T12:28:06Z

the cloudflare dashboard shows that only CLUSTER_SERVICE is set in the production env for the nft.storage api worker.

olizilla · 2023-02-08T14:38:08Z

The CLUSTER_API_URL is set as a pass through url in the api router. I understand this to mean if the api worker is invoked with a request url that matches the cluster api, it will leave it alone and let the request carry on to the upstream.

nft.storage/packages/api/src/utils/router.js

Lines 185 to 191 in 69a49ce

    
           const { DATABASE_URL, CLUSTER_API_URL } = getServiceConfig() 
        
           const passThrough = [DATABASE_URL, CLUSTER_API_URL] 
        
           // Ignore http requests from the passthrough list above 
        
           if (!passThrough.includes(`${url.protocol}//${url.host}`)) { 
        
             event.respondWith(this.route(event)) 
        
           }

I believe this is only relevant for testing, where multiple services are running on the same host.

olizilla · 2023-02-08T14:45:38Z

pins-add.js calls db.createUpload without specifying the service (the short name for cluster is it pinned on) or the pin status.

nft.storage/packages/api/src/routes/pins-add.js

Lines 53 to 54 in 69a49ce

    
           const upload = await db.createUpload({ 
        
             type: 'Remote',

therefore the service will be defaulted to IpfsCluster3 per:

nft.storage/packages/api/src/utils/db-client.js

Lines 289 to 301 in 69a49ce

    
           async createUpload(data) { 
        
             const defaultPins = [ 
        
               { 
        
                 status: 'PinQueued', 
        
                 service: 'IpfsCluster3', 
        
               }, 
        
             ] 
        
             const now = new Date().toISOString() 
        
             const rsp = await this.client.rpc('create_upload', { 
        
               data: { 
        
                 ...data, 
        
                 pins: data.pins || defaultPins,

and pins-replace.js does the same

https://github.com/nftstorage/nft.storage/blob/main/packages/api/src/routes/pins-replace.js#L75-L76

olizilla · 2023-02-08T15:03:05Z

toPinsResponse in db-transforms.js is called from all the pins-* operations, and includes delegates that are set from the cluster lib

nft.storage/packages/api/src/utils/db-transforms.js

Lines 48 to 60 in 69a49ce

    
           export function toPinsResponse(upload) { 
        
             /** @type {import('../bindings').PinsResponse} */ 
        
             const rsp = { 
        
               requestid: upload.source_cid, 
        
               status: transformPinStatus(upload.content.pin[0].status), 
        
               created: upload.inserted_at, 
        
               pin: { 
        
                 cid: upload.source_cid, 
        
                 meta: upload.meta, 
        
                 name: upload.name, 
        
                 origins: upload.origins, 
        
               }, 
        
               delegates: cluster.delegates(),

but this is currently stubbed out, so is not an issue. ~~We could optionally set that to be the multiaddr for e-ipfs.~~ nope, delegates are used to specify the ipfs node that will do the retrieval. If anything that should be a pickup kubo node, but that info is not exposed yet.

nft.storage/packages/api/src/cluster.js

Lines 101 to 103 in 69a49ce

    
           export function delegates() { 
        
             return [] 
        
           }

olizilla · 2023-02-08T15:38:08Z

Need to check ElasticIpfs pin statuses in cron job

nft.storage/packages/cron/src/jobs/pins.js

Line 11 in 69a49ce

const CLUSTERS = ['IpfsCluster', 'IpfsCluster2', 'IpfsCluster3']

olizilla · 2023-02-08T15:42:11Z

Update cron pins bin to pass in pickup cluster config

nft.storage/packages/cron/src/bin/pins.js

Lines 18 to 22 in 69a49ce

    
           const cluster1 = getCluster1(process.env) 
        
           const cluster2 = getCluster2(process.env) 
        
           const cluster3 = getCluster3(process.env) 
        
           await updatePendingPinStatuses({ pg, cluster1, cluster2, cluster3 })

...and query that from the job

nft.storage/packages/cron/src/jobs/pins.js

Lines 183 to 185 in 69a49ce

    
           const updatedPins = [] 
        
           const cids = pins.map((p) => p.source_cid) 
        
           const statuses = await cluster3.statusAll({ cids })

pickup pulls dags into E-IPFS, so this PR updates the api and cron jobs to reflect that in our db. - Update the pinning service add and update api routes to record `ElasticIpfs` as the service that the pin is queued on. - Update pins and pins-failed cron jobs to check pin status in pickup instead of cluster Fixes #2309 TODO - [ ] add `PICKUP_BASIC_AUTH_TOKEN` as a secret to the repo for cron jobs. - [ ] update `CLUSTER-*` api env vars. License: MIT Signed-off-by: Oli Evans <[email protected]>

pickup pulls dags into E-IPFS, so this PR updates the api and cron jobs to reflect that in our db. - Update pins and pins-failed cron jobs to check pin status in pickup instead of cluster Fixes #2309 TODO - [x] add `PICKUP_BASIC_AUTH_TOKEN` as a secret to the repo for cron jobs. - [ ] update `CLUSTER-*` api env vars. License: MIT Signed-off-by: Oli Evans <[email protected]>

pickup pulls dags into E-IPFS, so this PR updates the api to reflect that. - Update the pinning service add and update api routes to record `ElasticIpfs` as the service that the pin is queued on. - Update `GET /pins/:req-id` to check for a status change from pickup and update out DB, so we can inform users early when a pin request is pinned or failed. Cron changes are in #2339 Fixes #2309 License: MIT Signed-off-by: Oli Evans <[email protected]>

olizilla added kind/enhancement A net-new feature or improvement to an existing feature need/triage Needs initial labeling and prioritization labels Feb 8, 2023

olizilla mentioned this issue Feb 8, 2023

Rollout for dotStorage web3-storage/pickup#68

Closed

10 tasks

olizilla mentioned this issue Feb 9, 2023

feat: use 🛻 pickup from api #2310

Merged

4 tasks

olizilla mentioned this issue Mar 7, 2023

feat: use 🛻 pickup from cron pins job #2339

Open

2 tasks

olizilla closed this as completed in #2310 Mar 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use 🛻 pickup to handle pinning service requests #2309

Use 🛻 pickup to handle pinning service requests #2309

olizilla commented Feb 8, 2023 •

edited

Loading

olizilla commented Feb 8, 2023

olizilla commented Feb 8, 2023 •

edited

Loading

olizilla commented Feb 8, 2023

olizilla commented Feb 8, 2023 •

edited

Loading

olizilla commented Feb 8, 2023 •

edited

Loading

olizilla commented Feb 8, 2023

olizilla commented Feb 8, 2023

Use 🛻 pickup to handle pinning service requests #2309

Use 🛻 pickup to handle pinning service requests #2309

Comments

olizilla commented Feb 8, 2023 • edited Loading

TODO

olizilla commented Feb 8, 2023

olizilla commented Feb 8, 2023 • edited Loading

olizilla commented Feb 8, 2023

olizilla commented Feb 8, 2023 • edited Loading

olizilla commented Feb 8, 2023 • edited Loading

olizilla commented Feb 8, 2023

olizilla commented Feb 8, 2023

olizilla commented Feb 8, 2023 •

edited

Loading

olizilla commented Feb 8, 2023 •

edited

Loading

olizilla commented Feb 8, 2023 •

edited

Loading

olizilla commented Feb 8, 2023 •

edited

Loading