[Journey] Reverse Engineering Google with AI (Fine-tuning only. Advanced level)

Mrnewbie said:
What is the point of posting this? I don't understand some of you who are so negative and can't stand someone doing well in their life. Please, keep the abusive and negative comments to yourself, as this is one of the more interesting Journeys I have seen. Again, there is no reason to put this type of comment in this thread, as it's filled with anger.

Let's be a community and support one another as more people will post journeys like this, and if we don't, most will keep it to themselves, and we will not learn a thing from it.
Click to expand...
He says ChatGPT sucks! how come? It works great! at least it has worked well for me so far. When we do not understand something, we should not deny it!
 
Last edited:
Advertise on BHW
CryptoGuys said:
He says ChatGBT sucks! how come? It works great! at least it has worked well for me so far. When we do not understand something, we should not deny it!
Click to expand...
This is the last point I will make as I don't want to take up his journey. For @splishsplash , he has a better way so let him be instead of knocking him and putting him down. Why not ask a question like, why do you think ChatGBT is not good enough for your projects etc? That is a constructive way of asking a question without being so harsh. This is [his journey not everyone else's] and if you agree or disagree that is fine but the name calling and remarks are not what this forum is about. If you have questions to why you think he should do this or that than that is fine.
Off my soapbox and done commenting here.
 
CryptoGuys said:
He says ChatGPT sucks! how come? It works great! at least it has worked well for me so far. When we do not understand something, we should not deny it!
Click to expand...

It sucks for SEO because it's not made for SEO.

It's made for answering questions and chatting with humans. It has an understanding of natural language, not search algorithms.

In the same way that you wouldn't dream of asking chatgpt to predict when a stock is going to go up.

If you want to get meaningful data pertaining to SEO, then you need to fine tune models with real SEO data.
 
Mrnewbie said:
This is the last point I will make as I don't want to take up his journey. For @splishsplash , he has a better way so let him be instead of knocking him and putting him down. Why not ask a question like, why do you think ChatGBT is not good enough for your projects etc? That is a constructive way of asking a question without being so harsh. This is [his journey not everyone else's] and if you agree or disagree that is fine but the name calling and remarks are not what this forum is about. If you have questions to why you think he should do this or that than that is fine.
Off my soapbox and done commenting here.
Click to expand...
How old are you?
 
I don't know anything about coding or training AI but read all of this in total excitement. Agree on that the problem is not individuals doing this but when big corporations can implement it.
 
Ludwic said:
I don't know anything about coding or training AI but read all of this in total excitement. Agree on that the problem is not individuals doing this but when big corporations can implement it.
Click to expand...
I will keep sharing anyway :)



Im on holiday with the family in the algarve just now so updates will be sparse for 10 days but i will resume full on when back in the lab.
 
@splishsplash is doing a great job, If successful, it will definitely be a revolutionary product!

Today I found one site, that supports fine-tuning with their model, site name is "AI21 Labs", check if it's helpful.
 
hqin said:
@splishsplash is doing a great job, If successful, it will definitely be a revolutionary product!

Today I found one site, that supports fine-tuning with their model, site name is "AI21 Labs", check if it's helpful.
Click to expand...

Their models arent great. The best models are openai davinci and curie, and for open source lit-llama.

Mpt-7b should be good but i havent tested yet and its new.

Thanks for sharing though. Appreciate your input
 
Very interesting topic!

my simple question is: does fine-tuning allow for data training other than textual prompts?
for example, if I would like to give it numerical/mathematical data
 
Mrnewbie said:
What is the point of posting this? I don't understand some of you who are so negative and can't stand someone doing well in their life. Please, keep the abusive and negative comments to yourself, as this is one of the more interesting Journeys I have seen. Again, there is no reason to put this type of comment in this thread, as it's filled with anger.

Let's be a community and support one another as more people will post journeys like this, and if we don't, most will keep it to themselves, and we will not learn a thing from it.
Click to expand...
dude, let haters hate
haters gonna hate forever! :D
if you see haters hatin' on you, it means you are doing a god-damn good job! :D
 
Mrnewbie said:
What is the point of posting this? I don't understand some of you who are so negative and can't stand someone doing well in their life. Please, keep the abusive and negative comments to yourself, as this is one of the more interesting Journeys I have seen. Again, there is no reason to put this type of comment in this thread, as it's filled with anger.

Let's be a community and support one another as more people will post journeys like this, and if we don't, most will keep it to themselves, and we will not learn a thing from it.
Click to expand...
Great job man! We truly appreciate your efforts to keep us in a safe place!
 
How you doing :) ? I just came back from the holiday myself, relaxed a bit to work on complex problems.
 
matek697 said:
Very interesting topic!

my simple question is: does fine-tuning allow for data training other than textual prompts?
for example, if I would like to give it numerical/mathematical data
Click to expand...

LLMs aren't very good with numerical data. You would use other machine learning methods for this.


MichelangeSEO said:
Have you ever tried to fine tune GPT with the Google Raters Guidelines?
Click to expand...

You can't really fine tune with something like that :-)

Fine tuning needs very specific data as you're training it to recognize patterns.



tragicflaw said:
How you doing :) ? I just came back from the holiday myself, relaxed a bit to work on complex problems.
Click to expand...

I'm good. Need to get my brain back into AI mode though. Going to resume work just now and continue where I left off!
 
Update
I've updated the get_structure_for_classify_webpage.py to include a huge amount of detail which can be used for various fine tuning projects.
This is a powerful bit of code anyone can use as a basis for SEO fine tunes.
If you do anything with this, PLEASE do share. Don't be evil. Everyone does better if you share.
You don't have to share publicly, but at least share your results with me personally. Remember, 1+1=3.
https://github.com/tbelfort/ai-seo-tools/blob/main/get_structure_for_classify_webpage.pyHere's an example of running it on https://github.com/tbelfort/ai-seo-tools/blob/main/get_structure_for_classify_webpage.py

getting page: https://www.lumar.io/learn/seo/search-engines/how-do-search-engines-work/
https://www.lumar.io/learn/seo/search-engines/how-do-search-engines-work/{'all_counts': {'a': 187,
'article': 3,
'b': 1,
'button': 16,
'clippath': 8,
'defs': 16,
'div': 182,
'fecolormatrix': 8,
'fecomponenttransfer': 8,
'fecomposite': 8,
'fefunca': 8,
'fefuncb': 8,
'fefuncg': 8,
'fefuncr': 8,
'filter': 8,
'footer': 1,
'g': 8,
'h1': 1,
'h2': 6,
'h4': 3,
'h6': 6,
'header': 1,
'hr': 1,
'iframe': 1,
'img': 14,
'li': 106,
'link': 26,
'meta': 25,
'nav': 4,
'noscript': 1,
'p': 58,
'path': 105,
'rect': 16,
'script': 23,
'span': 37,
'strong': 24,
'style': 9,
'svg': 64,
'title': 2,
'ul': 22},
'external': defaultdict(<class 'int'>, {'external_link_count': 136}),
'external_links': [{'anchor': 'Read more.',
'link': '/blog/company-news/deepcrawl-is-now-lumar/'},
{'anchor': '詳細はこちら',
'link': '/ja/blog/company-news/deepcrawl-is-now-lumar/'},
{'anchor': 'Lumar', 'link': '/'},
{'anchor': 'The Full Lumar Platform', 'link': '/platform/'},
{'anchor': 'What is Website Intelligence?',
'link': '/website-intelligence/'},
{'anchor': 'Analyze', 'link': '/platform/analyze/'},
{'anchor': 'Monitor', 'link': '/platform/monitor/'},
{'anchor': 'Impact', 'link': '/platform/impact/'},
{'anchor': 'Protect', 'link': '/platform/protect/'},
{'anchor': 'Technical SEO Metrics',
'link': '/platform/technical-seo-metrics/'},
{'anchor': 'Accessibility Metrics',
'link': '/platform/accessibility-metrics/'},
{'anchor': 'Our Crawler',
'link': '/platform/website-crawler/'},
{'anchor': 'Integrations',
'link': '/platform/integrations/'},
{'anchor': 'Professional Services',
'link': '/professional-services/'},
{'anchor': 'Case Studies', 'link': '/case-studies/'},
{'anchor': 'Website Audits',
'link': '/use-case/website-audits/'},
{'anchor': 'Website Monitoring',
'link': '/use-case/website-monitoring/'},
{'anchor': 'SEO QA Testing',
'link': '/use-case/seo-qa-testing/'},
{'anchor': 'Site Migrations',
'link': '/use-case/site-migrations/'},
{'anchor': 'SEO Strategy',
'link': '/use-case/seo-organic-search-strategy/'},
{'anchor': 'Accessibility Audits',
'link': '/use-case/accessibility-audits/'},
{'anchor': 'SEO', 'link': '/by-team/for-seo-teams/'},
{'anchor': 'Marketing',
'link': '/by-team/for-marketing-teams/'},
{'anchor': 'Engineering',
'link': '/by-team/for-developers/'},
{'anchor': 'Digital Ops',
'link': '/by-team/for-digital-operations/'},
{'anchor': 'Accessibility Teams',
'link': '/by-team/for-accessibility-teams/'},
{'anchor': 'E-Commerce',
'link': '/industry/lumar-for-ecommerce/'},
{'anchor': 'Media & Publishing',
'link': '/industry/lumar-for-publishers-news-website/'},
{'anchor': 'Enterprise',
'link': '/industry/lumar-for-enterprise/'},
{'anchor': 'Agencies',
'link': '/industry/lumar-for-agencies/'},
{'anchor': 'FAQs', 'link': '/faq/'},
{'anchor': 'Academy', 'link': '/academy/'},
{'anchor': 'eBooks', 'link': '/ebooks/'},
{'anchor': 'Webinars & Events',
'link': '/webinars-events/'},
{'anchor': 'Blog', 'link': '/blog/'},
{'anchor': 'SEO Office Hours', 'link': '/office-hours/'},
{'anchor': 'All Product Guides',
'link': '/product-guides/'},
{'anchor': 'API', 'link': '/api/'},
{'anchor': 'Pricing', 'link': '/pricing/'},
{'anchor': 'About us', 'link': '/about/'},
{'anchor': 'About', 'link': '/about/'},
{'anchor': 'Careers', 'link': '/careers/'},
{'anchor': 'Partnerships', 'link': '/partnerships/'},
{'anchor': 'Our New Brand',
'link': '/meet-lumar-our-new-brand/'},
{'anchor': 'Press Room', 'link': '/press-room/'},
{'anchor': 'Contact', 'link': '/contact/'},
{'anchor': 'Analyze', 'link': 'https://analyze.lumar.io/'},
{'anchor': 'Monitor', 'link': 'https://monitor.lumar.io/'},
{'anchor': 'Protect', 'link': 'https://protect.lumar.io/'},
{'anchor': 'Impact', 'link': 'https://impact.lumar.io/'},
{'anchor': 'Book a demo', 'link': '/request-demo/'},
{'anchor': 'The Full Lumar Platform', 'link': '/platform/'},
{'anchor': 'Website intelligence',
'link': '/website-intelligence/'},
{'anchor': 'Analyze', 'link': '/platform/analyze/'},
{'anchor': 'Monitor', 'link': '/platform/monitor/'},
{'anchor': 'Impact', 'link': '/platform/impact/'},
{'anchor': 'Protect', 'link': '/platform/protect/'},
{'anchor': 'Technical SEO Metrics',
'link': '/platform/technical-seo-metrics/'},
{'anchor': 'Accessibility Metrics',
'link': '/platform/accessibility-metrics/'},
{'anchor': 'Our Crawler',
'link': '/platform/website-crawler/'},
{'anchor': 'Integrations',
'link': '/platform/integrations/'},
{'anchor': 'Professional Services',
'link': '/professional-services/'},
{'anchor': 'Case Studies', 'link': '/case-studies/'},
{'anchor': 'Website Audits',
'link': '/use-case/website-audits/'},
{'anchor': 'Website Monitoring',
'link': '/use-case/website-monitoring/'},
{'anchor': 'SEO QA Testing',
'link': '/use-case/seo-qa-testing/'},
{'anchor': 'Site Migrations',
'link': '/use-case/site-migrations/'},
{'anchor': 'SEO Strategy',
'link': '/use-case/seo-organic-search-strategy/'},
{'anchor': 'Accessibility Audits',
'link': '/use-case/accessibility-audits/'},
{'anchor': 'SEO', 'link': '/by-team/for-seo-teams/'},
{'anchor': 'Marketing',
'link': '/by-team/for-marketing-teams/'},
{'anchor': 'Engineering',
'link': '/by-team/for-developers/'},
{'anchor': 'Digital Ops',
'link': '/by-team/for-digital-operations/'},
{'anchor': 'Accessibility Teams',
'link': '/by-team/for-accessibility-teams/'},
{'anchor': 'E-Commerce',
'link': '/industry/lumar-for-ecommerce/'},
{'anchor': 'Media & Publishing',
'link': '/industry/lumar-for-publishers-news-website/'},
{'anchor': 'Enterprise',
'link': '/industry/lumar-for-enterprise/'},
{'anchor': 'Agencies',
'link': '/industry/lumar-for-agencies/'},
{'anchor': 'FAQs', 'link': '/faq/'},
{'anchor': 'Academy', 'link': '/academy/'},
{'anchor': 'eBooks', 'link': '/ebooks/'},
{'anchor': 'Seo Office Hours', 'link': '/office-hours/'},
{'anchor': 'Blog', 'link': '/blog/'},
{'anchor': 'Webinars & Events',
'link': '/webinars-events/'},
{'anchor': 'All Product Guides',
'link': '/product-guides/'},
{'anchor': 'API', 'link': '/api/'},
{'anchor': 'Pricing', 'link': '/pricing/'},
{'anchor': 'About us', 'link': '/about/'},
{'anchor': 'Careers', 'link': '/careers/'},
{'anchor': 'Partnerships', 'link': '/partnerships/'},
{'anchor': 'Our New Brand',
'link': '/meet-lumar-our-new-brand/'},
{'anchor': 'Press Room', 'link': '/press-room/'},
{'anchor': 'Contact', 'link': '/contact/'},
{'anchor': 'Book a demo', 'link': '/request-demo/'},
{'anchor': 'Analyze', 'link': 'https://analyze.lumar.io/'},
{'anchor': 'Monitor', 'link': 'https://monitor.lumar.io/'},
{'anchor': 'Protect', 'link': 'https://protect.lumar.io/'},
{'anchor': 'Impact', 'link': 'https://impact.lumar.io/'},
{'anchor': '',
'link': 'https://twitter.com/share?url=https://www.lumar.io/learn/seo/search-engines/how-do-search-engines-work/'},
{'anchor': '',
'link': 'https://www.facebook.com/sharer/sharer.php?u=https://www.lumar.io/learn/seo/search-engines/how-do-search-engines-work/'},
{'anchor': '',
'link': 'https://www.linkedin.com/sharing/share-offsite/?url=https://www.lumar.io/learn/seo/search-engines/how-do-search-engines-work/'},
{'anchor': 'Search engines work by crawling hundreds of '
'billions of pages',
'link': 'https://support.google.com/webmasters/answer/70897'},
{'anchor': 'downloading web pages and following links on '
'these pages to discover new pages',
'link': 'https://www.google.com/search/howsearchworks/crawling-indexing/'},
{'anchor': 'called an index.',
'link': 'https://en.wikipedia.org/wiki/Search_engine_indexing'},
{'anchor': 'to present a relevant set of high-quality '
'search results',
'link': 'https://static.googleusercontent.co...s/assets/searchqualityevaluatorguidelines.pdf'},
{'anchor': 'is used to hierarchically rank the relevant '
'pages into a set of results.',
'link': 'https://moz.com/beginners-guide-to-seo/how-search-engines-operate'},
{'anchor': 'Google',
'link': 'https://support.google.com/webmasters/answer/35769?hl=en'},
{'anchor': 'Bing',
'link': 'https://www.bing.com/webmaster/help/webmaster-guidelines-30fba23a'},
{'anchor': 'Some search queries are location-dependent',
'link': 'https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37522.pdf'},
{'anchor': 'Search engines will return results in the '
'language of the user',
'link': 'https://en.wikipedia.org/wiki/Google_Personalized_Search#Data_collection'},
{'anchor': 'dependent on what user has previously searched '
'for.',
'link': 'https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/32708.pdf'},
{'anchor': 'A different set of results may be returned '
'based on the device',
'link': 'https://searchengineland.com/google-divide-index-giving-mobile-users-better-fresher-content-261037'},
{'anchor': 'Robots.txt file exclusions',
'link': 'https://www.robotstxt.org/orig.html'},
{'anchor': 'low quality',
'link': 'https://support.google.com/webmasters/answer/66361?hl=en'},
{'anchor': 'Get started with Lumar',
'link': '/request-demo/'},
{'anchor': 'Full Platform', 'link': '/platform/'},
{'anchor': 'Monitor', 'link': '/platform/monitor/'},
{'anchor': 'Analyze', 'link': '/platform/analyze/'},
{'anchor': 'Protect', 'link': '/platform/protect/'},
{'anchor': 'Impact', 'link': '/platform/impact/'},
{'anchor': 'Service Status',
'link': 'https://status.lumar.io/'},
{'anchor': 'Academy', 'link': '/academy/'},
{'anchor': 'Blog', 'link': '/blog/'},
{'anchor': 'E-Books', 'link': '/ebooks/'},
{'anchor': 'Webinars', 'link': '/webinars-events/'},
{'anchor': 'Product Guides', 'link': '/product-guides/'},
{'anchor': 'About', 'link': '/about/'},
{'anchor': 'Careers', 'link': '/careers/'},
{'anchor': 'Contact', 'link': '/contact/'},
{'anchor': '', 'link': 'https://twitter.com/lumarhq'},
{'anchor': '', 'link': 'https://www.facebook.com/lumarhq/'},
{'anchor': '',
'link': 'https://www.linkedin.com/company/lumarhq/'},
{'anchor': 'Privacy', 'link': '/privacy-policy/'},
{'anchor': 'Terms of Use', 'link': '/terms-of-use/'},
{'anchor': 'Cookies', 'link': '/cookie-policy/'},
{'anchor': 'Accessibility Statement',
'link': '/accessibility-statement/'}],
'general': defaultdict(<class 'int'>, {}),
'header_outline': ['title:How Do Search Engines Work? –\xa0Lumar Website '
'Intelligence',
'h1:How Do Search Engines Work?',
'h2:The search engine index',
'h2:What is the aim of a search engine algorithm?',
'h2:What happens when a search is performed?',
'h2:Why might a page not be indexed?',
'h2:The Full Guide to How Search Engines Work:',
'h2:Additional Learning Resources',
'h6:The Ultimate Guide to Website Health',
'h6:How to Make Your Site a Performance Multiplier for '
'Demand Generation',
'h4:Lumar is the intelligence & automation platform behind '
'revenue-driving websites',
'h6:Sam Marsden',
'h4:Related Articles',
'h6:pagination SEO: A Best Practice Guide',
'h6:What are Redirects? (URL Redirection Guide)',
'h6:A Guide to Redirect Types – 301, 302, JavaScript, Meta '
'Redirects & More',
'h4:Get the best digital marketing & SEO insights, '
'straight to your inbox'],
'internal': defaultdict(<class 'int'>,
{'bookmark_count': 17,
'internal_link_count': 28}),
'internal_links': [{'anchor': 'Home', 'link': 'https://www.lumar.io/'},
{'anchor': 'Technical SEO',
'link': 'https://www.lumar.io/learn/seo/'},
{'anchor': 'Search Engine Fundamentals',
'link': 'https://www.lumar.io/learn/seo/search-engines/'},
{'anchor': 'Sam Marsden',
'link': 'https://www.lumar.io/blog/author/sam-marsden/'},
{'anchor': 'crawling',
'link': 'https://www.lumar.io/learn/seo/crawlability/search-engine-crawling/'},
{'anchor': 'indexing',
'link': 'https://www.lumar.io/learn/seo/indexability/search-engine-indexing/'},
{'anchor': 'crawl budget',
'link': 'https://www.lumar.io/learn/seo/crawlability/crawl-budget/'},
{'anchor': 'telling search engines not to index that page',
'link': 'https://www.lumar.io/blog/best-practice/noindex-disallow-nofollow/'},
{'anchor': 'canonical tag',
'link': 'https://www.lumar.io/blog/best-practice/why-canonical-tags-are-essential/'},
{'anchor': 'thin content',
'link': 'https://www.lumar.io/blog/best-prac...-thin-or-poor-quality-content-with-deepcrawl/'},
{'anchor': 'duplicate content',
'link': 'https://www.lumar.io/blog/best-practice/advanced-duplicate-content/'},
{'anchor': '404 Not Found',
'link': 'https://www.lumar.io/blog/best-prac...ors-on-your-website-and-what-to-do-with-them/'},
{'anchor': 'Next Chapter: Search Engine Crawling',
'link': 'https://www.lumar.io/learn/seo/crawlability/search-engine-crawling/'},
{'anchor': 'How Search Engines Crawl Websites',
'link': 'https://www.lumar.io/learn/seo/crawlability/search-engine-crawling/'},
{'anchor': 'How Does Search Engine Indexing Work?',
'link': 'https://www.lumar.io/learn/seo/indexability/search-engine-indexing/'},
{'anchor': 'What are the Differences Between Search '
'Engines?',
'link': 'https://www.lumar.io/learn/seo/search-engines/search-engine-differences/'},
{'anchor': 'What is Crawl Budget?',
'link': 'https://www.lumar.io/learn/seo/crawlability/crawl-budget/'},
{'anchor': 'What is Robots.txt? How is Robots.txt Used by '
'Search Engines?',
'link': 'https://www.lumar.io/learn/seo/crawlability/robots-txt/'},
{'anchor': 'A Guide to Robots.txt Directives',
'link': 'https://www.lumar.io/learn/seo/crawlability/url-level-robots-directives/'},
{'anchor': 'The Ultimate Guide to Website HealthHere’s how '
'to succeed in a search-first future by '
'incorporating website health and SEO into your '
'wider marketing strategies.',
'link': 'https://www.lumar.io/ebooks/ultimat...th-planning-for-your-2022-marketing-strategy/'},
{'anchor': 'How to Make Your Site a Performance Multiplier '
'for Demand GenerationLearn how to utilize '
'website health and SEO as performance '
'multipliers for growth marketing teams’ demand '
'generation efforts.',
'link': 'https://www.lumar.io/ebooks/digital-ops-demand-gen-and-organic-search/'},
{'anchor': 'View all',
'link': 'https://www.lumar.io/learn/seo/search-engines/'},
{'anchor': '',
'link': 'https://www.lumar.io/learn/seo/site-architecture/pagination-seo-guide/'},
{'anchor': 'Pagination SEO: A Best Practice Guide',
'link': 'https://www.lumar.io/learn/seo/site-architecture/pagination-seo-guide/'},
{'anchor': '',
'link': 'https://www.lumar.io/learn/seo/http-availability/url-redirection/'},
{'anchor': 'What are Redirects? (URL Redirection Guide)',
'link': 'https://www.lumar.io/learn/seo/http-availability/url-redirection/'},
{'anchor': '',
'link': 'https://www.lumar.io/learn/seo/http-availability/redirect-types/'},
{'anchor': 'A Guide to Redirect Types – 301, 302, '
'JavaScript, Meta Redirects & More',
'link': 'https://www.lumar.io/learn/seo/http-availability/redirect-types/'}],
'li_contents': ['PlatformPlatformA full suite of website intelligence '
'solutionsThe Full Lumar PlatformWhat is Website '
'Intelligence?FeaturesAnalyzeComprehensive technical SEO & '
'website health analyticsMonitorTrack site health trends - '
'even across multiple domainsImpactShow the business impact '
'of your website projectsProtectProtect your website traffic '
'with automated SEO QATechnical SEO MetricsIdentify, '
'prioritize and fix technical issues negatively impacting '
'search performance.Accessibility MetricsAnalytics and '
'insights to drive compliance to WCAG 2.1 A to AAA.Our '
'CrawlerUltra-fast website crawling for enterprise '
'scaleIntegrationsConnect your website intelligence data with '
'other platformsProfessional ServicesBespoke SEO guidance & '
'monthly engagements',
'AnalyzeComprehensive technical SEO & website health '
'analytics',
'MonitorTrack site health trends - even across multiple '
'domains',
'ImpactShow the business impact of your website projects',
'ProtectProtect your website traffic with automated SEO QA',
'Technical SEO MetricsIdentify, prioritize and fix technical '
'issues negatively impacting search performance.',
'Accessibility MetricsAnalytics and insights to drive '
'compliance to WCAG 2.1 A to AAA.',
'Our CrawlerUltra-fast website crawling for enterprise scale',
'IntegrationsConnect your website intelligence data with '
'other platforms',
'Professional ServicesBespoke SEO guidance & monthly '
'engagements',
'SolutionsSolutionsHow businesses like yours use Lumar to '
'optimize their websites.Case StudiesUse casesWebsite '
'AuditsWebsite MonitoringSEO QA TestingSite MigrationsSEO '
'StrategyAccessibility AuditsBy '
'TeamSEOMarketingEngineeringDigital OpsAccessibility TeamsBy '
'IndustryE-CommerceMedia & PublishingEnterpriseAgencies',
'Website Audits',
'Website Monitoring',
'SEO QA Testing',
'Site Migrations',
'SEO Strategy',
'Accessibility Audits',
'SEO',
'Marketing',
'Engineering',
'Digital Ops',
'Accessibility Teams',
'E-Commerce',
'Media & Publishing',
'Enterprise',
'Agencies',
'ResourcesResourcesComprehensive educational resources for '
'digital teams.FAQsLearnAcademyExplore by topiceBooksIn-depth '
'guides for digital teamsWebinars & EventsWebinars on-demand '
'& upcoming eventsBlogThe latest website intelligence '
'insightsSEO Office HoursVideo library & key takewaysProduct '
'GuidesAll Product GuidesBecome an expert Lumar '
'userAPIIntegrate with or build on top of Lumar',
'AcademyExplore by topic',
'eBooksIn-depth guides for digital teams',
'Webinars & EventsWebinars on-demand & upcoming events',
'BlogThe latest website intelligence insights',
'SEO Office HoursVideo library & key takeways',
'All Product GuidesBecome an expert Lumar user',
'APIIntegrate with or build on top of Lumar',
'Pricing',
'CompanyAbout LumarLearn about Lumar, get in touch, or join '
"our global teamAbout usAboutGet to know Lumar's mission, "
'vision, & leadership teamCareersJoin the Lumar '
"teamPartnershipsLet's build something great togetherOur New "
'BrandDeepcrawl is now Lumar — learn all about our brand new '
'brandPress RoomMedia resources & speaking '
'engagementsContactGet in touch',
'PlatformThe Full Lumar PlatformWebsite '
'intelligenceFeaturesAnalyzeMonitorImpactProtectTechnical SEO '
'MetricsAccessibility MetricsOur '
'CrawlerIntegrationsProfessional Services',
'The Full Lumar Platform',
'Website intelligence',
'FeaturesAnalyzeMonitorImpactProtectTechnical SEO '
'MetricsAccessibility MetricsOur '
'CrawlerIntegrationsProfessional Services',
'Analyze',
'Monitor',
'Impact',
'Protect',
'Technical SEO Metrics',
'Accessibility Metrics',
'Our Crawler',
'Integrations',
'Professional Services',
'SolutionsCase StudiesUse casesWebsite AuditsWebsite '
'MonitoringSEO QA TestingSite MigrationsSEO '
'StrategyAccessibility AuditsBy '
'TeamSEOMarketingEngineeringDigital OpsAccessibility TeamsBy '
'IndustryE-CommerceMedia & PublishingEnterpriseAgencies',
'Case Studies',
'Use casesWebsite AuditsWebsite MonitoringSEO QA TestingSite '
'MigrationsSEO StrategyAccessibility Audits',
'Website Audits',
'Website Monitoring',
'SEO QA Testing',
'Site Migrations',
'SEO Strategy',
'Accessibility Audits',
'By TeamSEOMarketingEngineeringDigital OpsAccessibility Teams',
'SEO',
'Marketing',
'Engineering',
'Digital Ops',
'Accessibility Teams',
'By IndustryE-CommerceMedia & PublishingEnterpriseAgencies',
'E-Commerce',
'Media & Publishing',
'Enterprise',
'Agencies',
'ResourcesFAQsLearnAcademyeBooksSeo Office HoursBlogWebinars '
'& EventsProduct GuidesAll Product GuidesAPI',
'FAQs',
'LearnAcademyeBooksSeo Office HoursBlogWebinars & Events',
'Academy',
'eBooks',
'Seo Office Hours',
'Blog',
'Webinars & Events',
'Product GuidesAll Product GuidesAPI',
'All Product Guides',
'API',
'Pricing',
'CompanyAbout usCareersPartnershipsOur New BrandPress '
'RoomContact',
'About us',
'Careers',
'Partnerships',
'Our New Brand',
'Press Room',
'Contact',
'The search engine index',
'What is the aim of a search engine algorithm?',
'What happens when a search is performed?',
'Why might a page not be indexed?',
'The Full Guide to How Search Engines Work:',
'Additional Learning Resources',
'Thekeywordsdiscovered within the page’s content – what '
'topics does the page cover?',
'The type ofcontentthat is being crawled (using microdata '
'called Schema) – what is included on the page?',
'Thefreshnessof the page – how recently was it updated?',
'The previoususer engagementof the page and/or domain – how '
'do people interact with the page?',
'Location–Some search queries are location-dependente.g. '
'‘cafes near me’ or ‘movie times’.',
'Language detected–Search engines will return results in the '
'language of the user, if it can be detected.',
'Previous search history– Search engines will return '
'different results for a querydependent on what user has '
'previously searched for.',
'Device–A different set of results may be returned based on '
'the devicefrom which the query was made.',
'Robots.txt file exclusions– a file which tells search '
'engines what they shouldn’t visit on your site.',
'Directives on the webpagetelling search engines not to index '
'that page(noindex tag) or to index another similar page '
'(canonical tag).',
'Search engine algorithms judging the page to be oflow '
'quality, havethin contentor containduplicate content.',
'The URL returningan error page(e.g. a404 Not FoundHTTP '
'response code).'],
'li_count': 106,
'ol_count': 0,
'p_count': 57,
'sample_words': 'Platform A full suite of website intelligence solutions '
'Features Solutions',
'template_string': 'a,a,a,ul(li(p,p,a,a,p,ul(li(a),li(a),li(a),li(a),li(a),li(a),li(a),li(a),li(a))),li(p,p,a,p,ul(li(a),li(a),li(a),li(a),li(a),li(a)),p,ul(li(a),li(a),li(a),li(a),li(a)),p,ul(li(a),li(a),li(a),li(a))),li(p,p,a,p,ul(li(a),li(a),li(a),li(a),li(a)),p,ul(li(a),li(a))),li(a),li(p,p,a,a,a,a,a,a,a)),a(p),a(p),a(p),a(p),a,ul(li(a,ul(li(a),li(a),li(a,ul(li(a),li(a),li(a),li(a),li(a),li(a),li(a),li(a),li(a))))),li(a,ul(li(a),li(a,ul(li(a),li(a),li(a),li(a),li(a),li(a))),li(a,ul(li(a),li(a),li(a),li(a),li(a))),li(a,ul(li(a),li(a),li(a),li(a))))),li(a,ul(li(a),li(a,ul(li(a),li(a),li(a),li(a),li(a))),li(a,ul(li(a),li(a))))),li(a),li(a,ul(li(a),li(a),li(a),li(a),li(a),li(a)))),a,a(p),a(p),a(p),a(p),p(a,a,a),h1,p,img,img,p(a),p,p,a,a,a,p,ul(li(a),li(a),li(a),li(a),li(a),li(a)),p(a,a,a),p(a,a),p,a,h2,p(a),p,ul(li,li,li,li),a,h2,p(a),p,a,h2,p(a),p(a,a),p,ul(li(a),li(a),li(a),li(a)),a,h2,p,ul(li(a),li(a,a),li(a,a,a),li(a)),p(a),a,h2,a(img(p)),a(img,p),a(img,p),a(img,p),a(img,p),a(img,p),a,h2,a(img,h6,p),a(img,h6,p),h4,a,img,a(h6,p),p,h4,a,a(img),p,h6(a),a(img),p,h6(a),a(img),p,h6(a),h4,p,a,a,a,a,a,a,p,a,a,a,a,a,p,a,a,a,a,a,a,p,a,a,a,a,p,iframe',
'ul_count': 22,
'url': 'https://www.lumar.io/learn/seo/search-engines/how-do-search-engines-work/',
'word_count': 558}
Click to expand...


To summarize what you have here :-

"all_counts" - counts for all tags on the page.
"external" - Number of external links
"external_links" All the external links. url+anchor
"general" - What is this lol. I started this before going on holiday. I can't remember what this was going to be for.. Oh well :)
"header_outline" - Outline with just header tags and title.
"internal" and "internal_links" same as external, but for internal.
"li_contents" - The contents of all <li> tags
"li_count" - Number of <li> tags
"ol_count" - Number of <ol> tags
"p_count" number of <p> tags
"sample_words": When calling you specify how many sample words you want from the first <p>
"template_string" - This is the structure of the page in a shorthand format. You can specify only the tags you want, so I went for
html_tags_for_template = ["h1", "h2", "h3", "h4", "h5", "h6", "p", "ul", "ol", "li", "a", "iframe", "img", "a", "code", "blockquote", "cite", "q", "pre", "object"]

But you can change that. Things like divs/spans/script tags aren't useful to me for the outline finetune, but they might be useful for you, so change it as you see fit.

"ul_count" - Number of <ul> tags
"url" - the url
"word_count - the number of words on the page. It only counts words inside <p>.
 
I'm running the training data gathering program now with the new data, feeding it to gpt4 to get classifications.

It's quite expensive, but I need good data. Should end up being about $400-$500 for the training data for this one, which should give me about 5000 samples. That should be more than enough.

I'll then fine-tune curie and babbage. I want to see if babbage can give me accurate results on gpt4's level so I can get really cheap gpt4 level classifications for $0.0024 per 1k tokens.

Curie definitely will, so that'll at least be $0.012 per 1k tokens, which is 2.5x cheaper.

This is actually one of the hardest to gather good data for.

Going to bed now and leaving this running. Let's hope there's no bug in my code and I wake up bankrupt ;-) (I have a $5k monthly limit just now on my openai account)
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top