Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pageToken parameter is kept when extracting multiple channels. #3

Open
hideto0710 opened this issue Sep 11, 2021 · 0 comments · May be fixed by #4
Open

pageToken parameter is kept when extracting multiple channels. #3

hideto0710 opened this issue Sep 11, 2021 · 0 comments · May be fixed by #4

Comments

@hideto0710
Copy link

hideto0710 commented Sep 11, 2021 •
edited
Loading

version

% git status                                                                    
HEAD detached at v0.0.9

command

tap-youtube-analytics --config config.json --catalog catalog.json > state.json

summary

pageToken=CJYBEAA is still exist in next API call.

INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50&pageToken=CDIQAA
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50&pageToken=CGQQAA
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50&pageToken=CJYBEAA

...

INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UC7fk0CB07ly8oSl0aqKkqFg&order=date&type=video&maxResults=50&pageToken=CJYBEAA

detail

all logs

console 
INFO Authorized, token expires = 2021-09-11 23:18:56.751951
INFO last/currently syncing stream: None
INFO selected_streams: ['videos']
INFO STARTED Syncing: videos
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.4232058525085449, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 1, Results: 1-50 of Total: 168
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50&pageToken=CDIQAA
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.38460826873779297, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 2, Results: 51-100 of Total: 168
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50&pageToken=CGQQAA
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.3690910339355469, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 3, Results: 101-150 of Total: 168
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50&pageToken=CJYBEAA
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.34824299812316895, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 4, Results: 151-168 of Total: 168
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=TEmJCdjmUAY,6gDvJVeVNUQ,Sa9skWBlaqg,3r_qaOUjJMM,t7k0I_vBsAc,PORT-p0ex3E,tftbi551s8Q,jVWqvDyIEO8,5Aj-KX-0oLs,YUv2KBae8UM,WKWM_Hl0Un4,xfCUh2lQZt8,OhPS98-tE8E,u94b9HTMOCA,WtAw4WCSyb4,83XRzRkR23s,s2u3aHEg2OU,CP3KuFQoStA,zx3dSrleZ0Y,77F8EyJiCdc,Iq0mTJkJeJ0,nLmZ_cF2Fx4,KtcEnGjp5ew,cBOkKLqOyic,kA-Z5RTNDj8,rQrAVtvT5-c,BtYkLTsgfDo,7nQZfMXI7O8,Re6VLOXvAtU,MMyiCvxH2uA,zswO4CI72ls,mGRdhC5uo8k,TBWKi41S1ho,To2kOOARDLE,ZhCDaLGIuG8,Du_gWOCd2L4,xog1ydy2TWA,SmSOr42eJWc,W0szWI6e87A,may0lirp8Ws,gBCAoDuiTeU,461T8F4tDR0,_gZhFcrH5Ak,rf8Fz7BYLS0,fTIxn3wtOmI,SacbLXskakg,f7MQ2SN4uRc,2ZVUiVeKEK0,jWqOHAJ6l1g,-VkATJm-Xj4&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.3715939521789551, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-50 of Total: 50
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=gHbi2xhtZ00,oiZ5kBb3QjQ,A5K_TUEMZ-Y,64GtYwziNKM,8rUpQYW_ScI,vr5aGseITWU,eVQTfw8Kn0o,ba-fLjK6pdU,v8UmJ536Pjw,DpoVAL-Esbk,ZJyCg6kko64,_ebv7C0d8-Y,sSXtEGwhrWM,damyrfRkdAQ,G6d5E3-WZp0,B9bAiRXJh4A,-WTep9tOv60,psEh21WvXN8,blqykZwvInY,o12B4hZoKR8,Yn-itakV0wU,EWl0t1G-RBs,i3iXG91P3Dg,vpxF9eQYATk,ucLwtAAp__o,PVlIP732UBM,lTyMcl5Jd20,4cxvYk957vs,y_o8XKNHQxY,KkjhNxcNCx0,C4xvRNSNnAk,3-Ew_r1F3OE,wAH7Tu9gMgM,dK-dMorenBA,19bPLWNHYaA,iFRN48wMV64,9jj5Zehy2B4,XK1BKrA2pug,kT9yFgmoCb8,gwY1WkK07Zo,QXFstgVAoms,hDubQ7XAJxw,2EEnjESCFRs,R5jjmvEmRFA,_gBOkmVMLsc,OsPHUvr4-gI,3qPuwYhCUPY,iVgwEia_4zY,Nc2j8kI5JsY,g5gHZVMeUe4&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.31706809997558594, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-50 of Total: 50
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=joaPReqWYhk,Px5ZGz7RSoU,VpYLtASXaSI,sxRRqlioqM4,mudkdXfsOrE,uH3uknlca7Y,zwFSBWW2ziE,NeYHMZWhDsI,0Lekjajh1Eg,41E9_28Rtew,qidKnGqd36U,pxMerU_bhV8,AvFUS8DK9Bo,IoUq81uUshk,3-tfHXi46Ls,oj1ALVMGbfQ,YZvaq2evbK8,cjrhbGeon5M,omZ_pnsNBhQ,vkzEDZIpD0Q,L7fdt8NZCCQ,r5VlNwVceTw,sm5gXVgdKVs,1JYG3e3PWLk,2eNzcvW6WQY,WVQ11nZUVRw,J0p3Eve3OC0,hYtL1Xl-Mcc,mU9Ff9DY2Rg,fTEiRl2cSPE,M3CbiYUXGMw,KQMY1kLdFjw,ZPd3pXIo-xc,OXPaD9WMq7U,VmRD5sBzCCg,yi_y9xLolSU,vH3ymT6sIJ4,LdrTi3rRtyw,-st2zEWwTMU,eGPGrC3RCqw,TURDTCX9unI,gBNZxso04Zo,K30DfNmiPsI,pDcI7KRbtyg,HWMpjXSLWSo,h0UamDe2PyQ,8Opf25y39dM,eZoTBLmrlz4,Bhi00DgsOVY,7nME5zoHU1g&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.42879223823547363, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-50 of Total: 50
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=FzMDdA-mhOk,4guFnIDSAf4,BEvFye-y_KQ,ybUCpY_h9K8,vy85yx6qh-U,IbmLzpbpE68,T5D-6gzUY_A,78wBk1-6Qnc,j4p28FPJ7PY,4ugtIV1fhM0,C0KQZUyfZ3Q,vwSAImRnLoA,G9VbKTpaff0,ix3WUx25xII,KhrPiFZ4Sm8,jW4q0w6CCIo,Mmlj3WUBHbc,R4PEMkDb418&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.19980788230895996, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-18 of Total: 18
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UC7fk0CB07ly8oSl0aqKkqFg&order=date&type=video&maxResults=50&pageToken=CJYBEAA
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.47178101539611816, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 1, Results: 1-35 of Total: 189
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=ao0fpioDJOE,2bqcwqRWv3Q,ujx-czBUd_Y,Vwr0eUhAGHY,o_TfemFjME0,3GUg1uGTIF0,jCAF2e7V6_w,tgPOIvx5Ngk,RH4gW-1_4Qc,Y8syqWkoQXc,L2dmOGbE5n8,GyPd77ttGt4,cC8fs4iyATg,M52TUm11A1E,ajxsmz5L35g,R3dY8Ez5pA0,D-oDqG82FIg,vmEb0UtwU5s,aIG9o-S6cN4,0yExE1kXkto,4X4l58ChYDA,vhk1cD71rM0,t-ES9KW_Ij8,G2JfFK4q06Q,To-FnfaBCBo,XgBeGLwf4VE&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.30275678634643555, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-26 of Total: 26
INFO Write state for stream: videos, value: 2019-06-19T16:21:34Z
INFO Stream: videos, Processed 194 records
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 194, "tags": {"endpoint": "videos"}}
INFO FINISHED Syncing Stream: videos
INFO   Records Synced for Stream: 194

config

config.json 
{
  "client_id": "YOUR_CLIENT_ID",
  "client_secret": "YOUR_CLIENT_SECRET",
  "refresh_token": "YOUR_REFRESH_TOKEN",
  "channel_ids": "UCD8HOxPs4Xvsm8H0ZxXGiBw, UC7fk0CB07ly8oSl0aqKkqFg",
  "start_date": "2019-01-01T00:00:00Z",
  "user_agent": "tap-youtube-analytics <[email protected]>"
}

catalog

catalog.json 
{
  "streams": [
    {
      "tap_stream_id": "videos",
      "key_properties": [
        "id"
      ],
      "schema": {
        "selected": true,
        "properties": {
          "kind": {
            "type": [
              "null",
              "string"
            ]
          },
          "etag": {
            "type": [
              "null",
              "string"
            ]
          },
          "id": {
            "type": [
              "null",
              "string"
            ]
          },
          "published_at": {
            "format": "date-time",
            "type": [
              "null",
              "string"
            ]
          }
        },
        "type": "object",
        "additionalProperties": false
      },
      "stream": "videos",
      "metadata": [
        {
          "breadcrumb": [],
          "selected": true,
          "metadata": {
            "table-key-properties": [
              "id"
            ],
            "forced-replication-method": "INCREMENTAL",
            "valid-replication-keys": [
              "published_at"
            ],
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "kind"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "etag"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "id"
          ],
          "metadata": {
            "inclusion": "automatic"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "published_at"
          ],
          "metadata": {
            "inclusion": "available"
          }
        }
      ]
    }
  ]
}
@hideto0710 hideto0710 linked a pull request Sep 11, 2021 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

remove pageToken params at first mudazukai/tap-youtube-analytics
1 participant
@hideto0710