Journey to OpenScraper

Zepph

Junior Member
Joined
Apr 22, 2020
Messages
104
Reaction score
74
It's been a while since I've used this forum. I've learnt a lot, I've already made money for myself and I've increased my programming skills.

Now, I want to do something different, not necessarily to make money! I want to slowly replicate ScrapeBox, but make it open source :)

I'll be using Python, and will make it a CLI utility for now. After the main features are sorted out, I'll try to add a UI.

The current status is that I'm watching videos on the functionality of Scrapebox so I can replicate the main ideas. Working right now in the URL harvester! Will update whenever I have time (as I need to work to sustain myself lmao) and I've made progress! Will also provide the github repo as soon as I have something working...
 
looks like a fun project that can be useful. thanks for doing this ,looking forward to seeing your progress.
 
Nice, I've always been amazed that there isn't really a SaaS version of Scrapebox. Log into the app, set some scrapes going, dedupe some big lists, check the http responses of another big list, get an email or text when it's all finished.
 
Nice, I've always been amazed that there isn't really a SaaS version of Scrapebox. Log into the app, set some scrapes going, dedupe some big lists, check the http responses of another big list, get an email or text when it's all finished.
That's a really good idea actually. If all goes well I might implement it myself whilst keeping the local client open-source, would be lovely.

Update 1: Repo is here
Update 2: CLI usage for Google SERP harvesting is working now. Already supporting json extraction. It looks like this for the keyword "python web scraping":

JSON:
[
  {
    "keyword": "python web scraping",
    "url": "https://www.google.com/search?q=python+web+scraping&start=0&num=10&hl=en",
    "organic_results": [
      {
        "url": "https://www.geeksforgeeks.org/python/python-web-scraping-tutorial/",
        "title": "Python Web Scraping Tutorial",
        "description": "Jul 23, 2025 — Web scraping is the process of extracting data from websites automatically. Python is widely used for web scraping because of its easy syntax ...",
        "domain": "geeksforgeeks.org",
        "position": 1
      },
      {
        "url": "https://realpython.com/python-web-scraping-practical-introduction/",
        "title": "A Practical Introduction to Web Scraping in Python",
        "description": "Dec 21, 2024 — Python web scraping allows you to collect and parse data from websites programmatically. With powerful libraries like urllib , Beautiful Soup, ...",
        "domain": "realpython.com",
        "position": 2
      },
      {
        "url": "https://www.freecodecamp.org/espanol/news/como-hacer-scrape-de-una/",
        "title": "Cómo hacer scrape de sitios web con Python 3",
        "description": "Jan 29, 2021 — Hacer scrape es el proceso de extraer datos de sitios web. Antes de realizar la extracción de datos de una página web, debes asegurarte de que ...",
        "domain": "freecodecamp.org",
        "position": 3
      },
      {
        "url": "https://www.reddit.com/r/learnpython/comments/qzr8ir/how_to_start_web_scraping_with_python_w/",
        "title": "How to start Web scraping with python? : r/learnpython",
        "description": "Learn the basic html elements that build up a website. Inspect the element on the webpage that you're trying to get data from ... Use ... requests ...",
        "domain": "reddit.com",
        "position": 4
      },
      {
        "url": "https://scrapy.org/",
        "title": "Scrapy",
        "description": "The Scrapy framework, and especially its documentation, simplifies crawling and scraping for anyone with basic Python skills. I don't know, now there is this ...",
        "domain": "scrapy.org",
        "position": 5
      },
      {
        "url": "https://www.scrapingbee.com/blog/web-scraping-101-with-python/",
        "title": "Python Web Scraping: Full Tutorial With Examples (2025)",
        "description": "Jul 22, 2025 — In this tutorial, we'll explore the world of web scraping with Python, guiding you from the basics for beginners to advanced techniques for web scraping ...",
        "domain": "scrapingbee.com",
        "position": 6
      },
      {
        "url": "https://scrapfly.io/blog/posts/everything-to-know-about-web-scraping-python",
        "title": "Everything to Know to Start Web Scraping in Python Today",
        "description": "Jul 2, 2025 — Ultimate modern intro to web scraping using Python. How to scrape data using HTTP or headless browsers, parse it using AI and scale and ...",
        "domain": "scrapfly.io",
        "position": 7
      },
      {
        "url": "https://www.reddit.com/r/webscraping/comments/10lwpoa/best_language_for_web_scrapping/",
        "title": "Best language for web scrapping? : r/webscraping",
        "description": "",
        "domain": "reddit.com",
        "position": 8
      },
      {
        "url": "https://www.peerbits.com/blog/guide-for-python-web-scraping-libraries.html#:~:text=Web%20scraping%20is%20a%20Python,market%20research%2C%20or%20competitive%20intelligence.",
        "title": "Guide for Web Scraping: Python Libraries, Features and Techniques",
        "description": "",
        "domain": "peerbits.com",
        "position": 9
      },
      {
        "url": "https://medium.com/@datajournal/is-web-scraping-legal-covering-all-aspects-0df27c2e2ec6#:~:text=There%20are%20no%20specific%20laws,data%20can%20be%20a%20problem.",
        "title": "Is Web Scraping Legal? Covering All Aspects - Medium",
        "description": "",
        "domain": "medium.com",
        "position": 10
      },
      {
        "url": "https://crawlbase.com/blog/chatgpt-web-scraping/#:~:text=It%20can%20be%20trained%20to,to%20help%20you%20with%20that.",
        "title": "How to Scrape Websites with ChatGPT in 2025 - Crawlbase",
        "description": "",
        "domain": "crawlbase.com",
        "position": 11
      },
      {
        "url": "https://www.ionos.com/es-us/digitalguide/paginas-web/desarrollo-web/web-scraping-con-python/",
        "title": "Web scraping con Python: introducción y tutorial",
        "description": "Feb 22, 2023 — Aprende a usar Python para hacer web scraping. Comparamos Scrapy, Selenium y BeautifulSoup, tres de las herramientas más populares para el ...",
        "domain": "ionos.com",
        "position": 12
      }
    ],
    "related_keywords": [
      "Python web scraping library",
      "Python web scraping Selenium",
      "Python web scraping pdf",
      "Python web scraping examples",
      "Web scraping using Python BeautifulSoup",
      "Python web scraping w3schools",
      "Python web scraping projects",
      "Web scraping Python projects with source code"
    ],
    "people_also_ask": [
      "Is Python good for web scraping?",
      "What is Python web scraping?",
      "Is web scraping illegal?",
      "Can AutoGPT do web scraping?"
    ],
    "total_results": 12,
    "results_with_description": 8,
    "unique_domains": 11,
    "scraped_at": "2025-09-30T11:08:15.715836"
  },
  {
    "keyword": "data extraction",
    "url": "https://www.google.com/search?q=data+extraction&start=0&num=10&hl=en",
    "organic_results": [
      {
        "url": "https://en.wikipedia.org/wiki/Data_extraction",
        "title": "Data extraction",
        "description": "Data extraction is the act or process of retrieving data out of (usually unstructured or poorly structured) data sources for further data processing or data ...",
        "domain": "en.wikipedia.org",
        "position": 1
      },
      {
        "url": "https://en.wikipedia.org/wiki/Data_extraction#:~:text=Data%20extraction%20is%20the%20act,data%20storage%20(data%20migration).",
        "title": "Data extraction - Wikipedia",
        "description": "",
        "domain": "en.wikipedia.org",
        "position": 2
      },
      {
        "url": "https://zapier.com/blog/data-extraction/#:~:text=For%20example%2C%20a%20mortgage%20company,their%20services%20in%20the%20future.",
        "title": "What is data extraction? Examples + automation tips - Zapier",
        "description": "",
        "domain": "zapier.com",
        "position": 3
      },
      {
        "url": "https://www.atscale.com/glossary/data-extract/#:~:text=%E2%80%9CData%20extracts%20are%20typically%20a,into%20Tableau's%20own%20data%20store.%E2%80%9D",
        "title": "What is Data Extraction? Definition & Tools | AtScale",
        "description": "",
        "domain": "atscale.com",
        "position": 4
      },
      {
        "url": "https://www.promptcloud.com/blog/what-is-data-extraction-and-how-does-it-work/#:~:text=As%20data%20continues%20to%20play,data%20management%20and%20analysis%20domain.",
        "title": "What is Data Extraction and How Does it Work - PromptCloud",
        "description": "",
        "domain": "promptcloud.com",
        "position": 5
      },
      {
        "url": "https://www.talend.com/resources/data-extraction-defined/",
        "title": "What is Data Extraction? Definition and Examples",
        "description": "Data extraction is the process of collecting or retrieving disparate types of data from a variety of sources, many of which may be poorly organized or ...",
        "domain": "talend.com",
        "position": 6
      },
      {
        "url": "https://rivery.io/data-learning-center/data-extraction/",
        "title": "Data Extraction: Techniques, Tools, and Real-Time Benefits",
        "description": "Dec 14, 2024 — Data extraction is the process of retrieving relevant data from various sources such as databases, websites, or files for further processing or ...",
        "domain": "rivery.io",
        "position": 7
      },
      {
        "url": "https://www.matillion.com/blog/what-is-data-extraction-everything-you-need-to-know",
        "title": "Guide to Data Extraction: Definition, how it works & examples",
        "description": "Aug 2, 2024 — Data extraction is the process of retrieving data from various sources to prepare it for processing and analysis.",
        "domain": "matillion.com",
        "position": 8
      },
      {
        "url": "https://scispace.com/extract-data",
        "title": "AI-Powered Data Extraction from Research PDFs",
        "description": "Extract Data From Research Papers. SciSpace Data Extractor identifies tables, stats and citations in research PDFs, summarises key findings and exports ...",
        "domain": "scispace.com",
        "position": 9
      },
      {
        "url": "https://hevodata.com/learn/data-extraction/",
        "title": "What Is Data Extraction? Steps, Techniques & Use Cases",
        "description": "Mar 7, 2025 — Data extraction refers to the process of collecting information from various sources for transformation, storage, or feeding it to another system for further ...",
        "domain": "hevodata.com",
        "position": 10
      },
      {
        "url": "https://hslib.jabsom.hawaii.edu/systematicreview/dataextraction",
        "title": "Data Extraction - Systematic Review Toolbox",
        "description": "Oct 14, 2024 — Data extraction is the process of extracting the relevant pieces of information from the studies you have assessed for eligibility in your review.",
        "domain": "hslib.jabsom.hawaii.edu",
        "position": 11
      },
      {
        "url": "https://www.geeksforgeeks.org/data-analysis/what-is-data-extraction/",
        "title": "What is Data Extraction?",
        "description": "Jul 23, 2025 — Gathering data from various places, changing it so we can use it, and putting it where we need it for review is what data extraction is about.",
        "domain": "geeksforgeeks.org",
        "position": 12
      },
      {
        "url": "https://zapier.com/blog/data-extraction/",
        "title": "What is data extraction? Examples + automation tips",
        "description": "Oct 3, 2023 — Data extraction is the pulling of usable, targeted information from larger, unrefined sources. You start with massive, unstructured logs of data.",
        "domain": "zapier.com",
        "position": 13
      },
      {
        "url": "https://www.bmc.com/learn/data-extraction.html",
        "title": "What Is Data Extraction? Big Data Management",
        "description": "Data extraction is the process of identifying, retrieving, and replicating raw data from various sources into a target repository.",
        "domain": "bmc.com",
        "position": 14
      }
    ],
    "related_keywords": [
      "Data extraction tools",
      "Data extraction example",
      "Data extraction from image",
      "Data extraction pdf",
      "Data extraction - systematic review",
      "Data extraction in research",
      "Data extraction in Excel",
      "Data extraction AutoCAD"
    ],
    "people_also_ask": [
      "What do you mean by data extraction?",
      "What is an example of data extraction?",
      "What is a data extraction tool?",
      "Is data extraction a skill?"
    ],
    "total_results": 14,
    "results_with_description": 10,
    "unique_domains": 12,
    "scraped_at": "2025-09-30T11:08:20.784238"
  }
]

Update 3 & likely last of the day: Added basic keyword discovery, results look like this:

Code:
# Generated: 2025-09-30T11:29:37.991795
# Language: en, Country: US
# Total keywords: 99
#======================================================================

coffee near me (relevance: 1250, depth: 0)
coffee shops near me (relevance: 850, depth: 0)
coffee cake (relevance: 700, depth: 0)
coffee emporium (relevance: 602, depth: 0)
coffee grounds (relevance: 602, depth: 0)
coffee day (relevance: 601, depth: 0)
coffee and kareem (relevance: 601, depth: 0)
coffee bar (relevance: 601, depth: 0)
coffee emoji (relevance: 601, depth: 0)
coffee filters (relevance: 601, depth: 0)
coffee ground emesis (relevance: 601, depth: 0)
coffee shop (relevance: 600, depth: 0)
coffee alternative (relevance: 600, depth: 0)
coffee bean and tea leaf (relevance: 600, depth: 0)
coffee exchange (relevance: 600, depth: 0)
coffee festival (relevance: 600, depth: 0)
coffee grounds for plants (relevance: 600, depth: 0)
coffee and cream (relevance: 562, depth: 0)
coffee flights near me (relevance: 562, depth: 0)
coffee and breakfast near me (relevance: 561, depth: 0)
coffee bean menu (relevance: 561, depth: 0)
coffee cup (relevance: 561, depth: 0)
coffee espresso machine (relevance: 561, depth: 0)
coffee frother (relevance: 561, depth: 0)
coffee table (relevance: 560, depth: 0)
coffee shop near me (relevance: 560, depth: 0)
coffee bar cabinet (relevance: 560, depth: 0)
coffee caffeine content (relevance: 560, depth: 0)
coffee day deals 2025 (relevance: 560, depth: 0)
coffee expressions (relevance: 560, depth: 0)
coffee flavors (relevance: 560, depth: 0)
coffee bean (relevance: 559, depth: 0)
coffee and cigarettes (relevance: 559, depth: 0)
coffee bean near me (relevance: 559, depth: 0)
coffee cart (relevance: 559, depth: 0)
coffee day freebies (relevance: 559, depth: 0)
coffee enema near me (relevance: 559, depth: 0)
coffee flights (relevance: 559, depth: 0)
coffee day 2025 (relevance: 558, depth: 0)
coffee and vanilla (relevance: 558, depth: 0)
coffee bean grinder (relevance: 558, depth: 0)
coffee carafe (relevance: 558, depth: 0)
coffee drive thru near me (relevance: 558, depth: 0)
coffee enema kit (relevance: 558, depth: 0)
coffee for the soul (relevance: 558, depth: 0)
coffee maker (relevance: 557, depth: 0)
coffee and stone (relevance: 557, depth: 0)
coffee connection (relevance: 557, depth: 0)
coffee deals (relevance: 557, depth: 0)
coffee extract (relevance: 557, depth: 0)
coffee filter butterfly (relevance: 557, depth: 0)
coffee places near me (relevance: 556, depth: 0)
coffee and kareem cast (relevance: 556, depth: 0)
coffee bar ideas (relevance: 556, depth: 0)
coffee concentrate (relevance: 556, depth: 0)
coffee dose (relevance: 556, depth: 0)
coffee enema reddit (relevance: 556, depth: 0)
coffee fellows (relevance: 556, depth: 0)
coffee grinder (relevance: 555, depth: 0)
coffee around me (relevance: 555, depth: 0)
coffee brands (relevance: 555, depth: 0)
coffee candy (relevance: 555, depth: 0)
coffee drinks (relevance: 555, depth: 0)
coffee emporium cincinnati (relevance: 555, depth: 0)
coffee fruit (relevance: 555, depth: 0)
coffee machine (relevance: 554, depth: 0)
coffee and cars (relevance: 554, depth: 0)
coffee badging (relevance: 554, depth: 0)
coffee crisp (relevance: 554, depth: 0)
coffee delivery near me (relevance: 554, depth: 0)
coffee etc (relevance: 554, depth: 0)
coffee filter flowers (relevance: 554, depth: 0)
coffee cake recipe (relevance: 553, depth: 0)
coffee alcohol drink (relevance: 553, depth: 0)
coffee beanery (relevance: 553, depth: 0)
coffee cafe near me (relevance: 553, depth: 0)
coffee drunk (relevance: 553, depth: 0)
coffee emporium menu (relevance: 553, depth: 0)
coffee fix (relevance: 553, depth: 0)
coffee day deals (relevance: 552, depth: 0)
coffee break (relevance: 552, depth: 0)
coffee cake muffins (relevance: 552, depth: 0)
coffee day freebies 2025 (relevance: 552, depth: 0)
coffee energy drinks (relevance: 552, depth: 0)
coffee festival 2025 (relevance: 552, depth: 0)
coffee creamer (relevance: 551, depth: 0)
coffee and blood pressure (relevance: 551, depth: 0)
coffee benefits (relevance: 551, depth: 0)
coffee culture (relevance: 551, depth: 0)
coffee depot (relevance: 551, depth: 0)
coffee espresso (relevance: 551, depth: 0)
coffee flavor syrups (relevance: 551, depth: 0)
coffee deals today (relevance: 550, depth: 0)
coffee and food near me (relevance: 550, depth: 0)
coffee bucket (relevance: 550, depth: 0)
coffee caffeine (relevance: 550, depth: 0)
coffee day specials (relevance: 550, depth: 0)
coffee exchange providence (relevance: 550, depth: 0)
coffee flavored liqueur (relevance: 550, depth: 0)

I'm open to suggestions btw! Feel free to comment here.
 
Last edited:
Python CLI makes sense for now, but I’m already picturing the UI version being a game changer
 
Today's update is minor: I've worked on the storage pipeline so it natively supports Crawlee's methods. Will pay off later down the line :), tomorrow I'll probably keep working on improving the data export for SERP results!
Python CLI makes sense for now, but I’m already picturing the UI version being a game changer
I know! I first want to make sure that the code works, once that's done, calling it from an UI is a piece of cake. Making it look good and have good UX might be a challenge, though.
 
I've been pretty busy these last two weeks, however I've fixed some bugs including updated selectors from Google! And also added much better fake user agents.

Nonetheless it seems Google shadow updated their SERP results. Now even if I do a simple search with Crawlee, the library we're using, we get a captcha... Will think about a way to bypass it as captchas reduce result per second throughput.
 
https://codeberg.org/thermostatic/OpenScraper we're so back. I have much more experience this time around after scraping airlines at scale for a few clients! Now using the state of the art in anti-detect, the SERP scrapers are back at work with massive optimizations! With proxy support of course. I'll finish the core as CLI and then I'll start implementing a multi-platform UI and the installers too. Using codeberg instead of github as to not get taken down.
 
Back
Top