Anyone managed to make ChatGPT extract data from more than 1 URL?

I'm trying but for some reason it only seems to visit 1 URL. My prompt is massive but this is the part about visiting the URLs:

"You will write an article based on content that you will extract from a list of URLs that I will give you, you will read the content of each URL and use them as your guideline:
URL1
URL2
URL3
URL4"

You'll produce the worst content possible with this framework. Develop a framework and use 10 prompts to write a single paragraph.
AI requires writing prompt frameworks / flows to be efficient.
It won't work.

Find a way to identify paragraphs on websites without AI or outsource scraping to someone, then there is content labeling and all that stuff. I'm not working with that. It's HELL.

spectrejoe said:

I'm trying but for some reason it only seems to visit 1 URL. My prompt is massive but this is the part about visiting the URLs:

"You will write an article based on content that you will extract from a list of URLs that I will give you, you will read the content of each URL and use them as your guideline:
URL1
URL2
URL3
URL4"

Click to expand...

It's better to scrape the urls content using a a scrapper and give your content as a input for your prompt.

There's a lot of stuff to get anyhow readable output from AI. It requires understanding that Context Window is the most important thing and letting it focus is crucial. Give it small tasks like to a human.

Pre-prompting (each task is a separate prompt):

• Give background info.
• Set the tone.
• Explain your goals.
• Tell it do to something like "write 3 - 7 words long sentences", don't use jargon and specialistic language; write for a beginner.

Prompting (you need a framework, each paragraph / each sentence / each unit of text repeats the process):

• Ask specific questions.
• Request desired output.
• Provide examples if helpful.
• Use corrections where each correction is a new prompt.
• After getting an output, tell it to point out own errors and look from a psychological standpoints at consequences of model's writings.
• Help it get better UX and refine its content.

It sounds like the issue might be with how the URLs are being processed.

If your prompt is too long or complex, it might cause issues. Break it into smaller steps. For example:
“Extract content from URL1 and summarize it.”
“Do the same for URL2, URL3, and URL4.”
“Write an article based on the summaries from all URLs.”

Ensure that all URLs are accessible and provide the content you expect. Sometimes, URLs might not load properly or might redirect, affecting the extraction process. Try testing with a smaller number of URLs to see if the issue persists. This can help determine if the problem is related to the number of URLs or the way they are processed. Clearly state in the prompt that each URL should be visited, and content should be extracted from all provided URLs. For example:

Check if there are any error messages or issues during the URL extraction process that could give clues about why only one URL is being visited.

Adjusting these aspects should help ensure that all provided URLs are processed correctly.

bavodo said:

“Extract content from URL1 and summarize it.”
“Do the same for URL2, URL3, and URL4.”
“Write an article based on the summaries from all URLs.”

Click to expand...

Will not get good UX. Unless someone is lucky, but anyways, rewriting articles is for kindergarten.

Write from scratch using database of knowledge and your own genuine ideas. Then you get the juice.

If you rewrite someone's content, 99% in my experiments it's garbage. Writers are really bad writers. You're shooting blindfolded.

You need standardization and your own standards. Don't let someone set them for you or you'll depend on someone's luck who depends on someone's else luck. Realize that content on google and on big websites to is written by amateurs.

When I write content, I commit a whole Investopedia within 6 seconds. Joke, I'm not writing content cause video is killing written content. Massive video edits, Mr Beast automated videos - that's where the FUN and fulfillment is!