Need to Automatically Extract Structured Data from Websites — AI Models Not Working Reliably

What I need:​

• Visits a target URL;
• Extracts specific fields (like license info, payment methods, supported languages, etc.);
• Outputs the data in a structured format (e.g., TSV or directly to Google Sheets);
• Works in an automated or semi-automated way.

What I’ve tried:​

• AI models work with cached data — they often return information from 2023, even when I specifically ask for current data.
• Even with browsing enabled, AI struggles to extract structured fields from websites. Most of the time, they:
  
  • Fail to access dynamically loaded content (JavaScript-heavy blocks);
  • Miss data inside tabs, modals, or expanding sections;
  • Can't parse full T&C documents or extract deep info (like withdrawal limits per user level);
  • Are inconsistent — sometimes they give good results, sometimes barely anything.

What else I explored:​

• Google Sheets + Apps Script — too limited; failed to parse structured data from raw input.
• Manual copy-paste + prompt to structure in AI — works, but it's slow and still requires site-by-site manual effort.
• Puppeteer / Playwright parsers — promising but require writing custom selectors per site (not scalable without templates).
• Thunderbit, Octoparse, ParseHub — useful, but either require credit-based plans or don't fully adapt to dynamic pages without tweaking.

The problem:​

• AI models are too shallow or outdated.
• Parsers require manual setup for each structure.
• Off-the-shelf tools are either limited or need too much handholding.

What I’m asking:​

• Tool recommendations;
• Working setups you’ve implemented;
• Scripts or APIs that do something similar;
• Or a smart combo of tools (even semi-automated).

OKKO said:

Hi all,

I'm looking for advice on how to automatically collect up-to-date structured data from websites based on specific parameters.

What I need:​

I want a solution that:

• Visits a target URL;
• Extracts specific fields (like license info, payment methods, supported languages, etc.);
• Outputs the data in a structured format (e.g., TSV or directly to Google Sheets);
• Works in an automated or semi-automated way.

What I’ve tried:​

I explored multiple approaches, including AI models like Gemini and ChatGPT (GPT-4 / GPT-4o with Browse). Here’s what I ran into:

• AI models work with cached data — they often return information from 2023, even when I specifically ask for current data.
• Even with browsing enabled, AI struggles to extract structured fields from websites. Most of the time, they:
  • Fail to access dynamically loaded content (JavaScript-heavy blocks);
  • Miss data inside tabs, modals, or expanding sections;
  • Can't parse full T&C documents or extract deep info (like withdrawal limits per user level);
  • Are inconsistent — sometimes they give good results, sometimes barely anything.

I tried adjusting model temperature, prompt structure, and formatting — no stable results so far.

What else I explored:​

Together with an assistant, we brainstormed and tested various technical options:

• Google Sheets + Apps Script — too limited; failed to parse structured data from raw input.
• Manual copy-paste + prompt to structure in AI — works, but it's slow and still requires site-by-site manual effort.
• Puppeteer / Playwright parsers — promising but require writing custom selectors per site (not scalable without templates).
• Thunderbit, Octoparse, ParseHub — useful, but either require credit-based plans or don't fully adapt to dynamic pages without tweaking.

The problem:​

What I really want is a hybrid solution — something that uses AI to adaptively read any website and extract structured fields, ideally in an automated fashion.

But no combination of tools seems to offer that fully:

• AI models are too shallow or outdated.
• Parsers require manual setup for each structure.
• Off-the-shelf tools are either limited or need too much handholding.

What I’m asking:​

Is anyone here using a working system for real-time, structured web data collection — especially one that can handle JS-heavy pages and output to Google Sheets or CSV?

I’d appreciate:

• Tool recommendations;
• Working setups you’ve implemented;
• Scripts or APIs that do something similar;
• Or a smart combo of tools (even semi-automated).

Thanks in advance to anyone who can share advice, links, or solutions!

Click to expand...

OKKO said:

I'm looking for advice on how to automatically collect up-to-date structured data from websites based on specific parameters.

Click to expand...