Website crawler iterator #2206
larry-lunaraspect
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi,
I would love some suggestions on my chatflow. Use case: Read a Google Sheet that stores the list of URLs. For each URL in the web scrape the data then run some basic extraction logic to see if the website includes or excludes certain text.
I used to have a python script that would easily read the Google sheet and store it in an array and then just loop and call ChatGPT API on each URL. This worked great, but it was a custom script.
Now using Flowise, I want to get rid of the script. I got the workflow working for a single URL.
TextSplitter --> Cheerio Web Scraper --> Pinecone ---> ChatOpenAI --> Conversational Retrieval QA Chain.
I can even get it working with 3 URLs by just creating more nodes of Cheerio Web Scraper. The problem is that the google sheet has 100 URLs. Is there a way I can have a loop to iterate across a number of webpage?
Thanks for your help in advance
Beta Was this translation helpful? Give feedback.
All reactions