Congratulations for the baby news. I am sure the baby will light up your home with joy. Thanks for your invaluable contribution to this forum. I have learned a great deal from your posts over the years. Stay blessed.
[Journey] Reverse Engineering Google with AI (Fine-tuning only. Advanced level)
- Thread starter splishsplash
- Start date
- Oct 9, 2013
- 3,471
- 14,478
Thank you so much!
Soul_Surfer
Registered Member
- Feb 2, 2018
- 50
- 9
Congratulations my friend on your new baby! ;-)
What a blessing this is...
What a blessing this is...
noellarkin
Senior Member
- Mar 14, 2021
- 1,006
- 1,492
Congratulations, @splishsplash 
Rosath
Regular Member
- Nov 22, 2014
- 216
- 117
There are so many trolls in this thread who do not realize GPT is spewing hallucinated shit at them. It must really hurt to know they've done all that "work" for nothing. They'll be the first taken out of the game if they even manage to get in it at all.
Thanks for sharing your fine-tuning process OP. There really isn't a lot of data about this right now.
I'm excited to learn which model you end up with.
Thanks for sharing your fine-tuning process OP. There really isn't a lot of data about this right now.
I'm excited to learn which model you end up with.
- Oct 9, 2013
- 3,471
- 14,478
I'm just busy with other things just now, but I will resume this thread with more stuff, don't worry
I'm focusing on building some production level tools just now, but I'll get back to more research/fine-tuning SEO tests soon
I'm focusing on building some production level tools just now, but I'll get back to more research/fine-tuning SEO tests soon
wimiphp
Regular Member
- Dec 16, 2023
- 310
- 159
Take your time, we will be here to listen to any updates regardless when that might happens. I hope all turns out good with your porduction level tools!I'm just busy with other things just now, but I will resume this thread with more stuff, don't worry
I'm focusing on building some production level tools just now, but I'll get back to more research/fine-tuning SEO tests soon
Dark Hat 007
Senior Member
- Dec 2, 2016
- 1,164
- 773
I'm just busy with other things just now, but I will resume this thread with more stuff, don't worry
I'm focusing on building some production level tools just now, but I'll get back to more research/fine-tuning SEO tests soon
Hey man, wishing you a happy new year, just asking, do you know Koray and his topical authority course, if you do and have probably seen it, what do you think of it?
- Jan 1, 2024
- 507
- 44
"Incredible journey so far! Thrilled to be following along now, eager to see where it takes you. Best of luck on the next chapters!"
- Oct 9, 2013
- 3,471
- 14,478
Happy New Year to you too buddy.
I've written about him before. I just didn't open a full "Koray is a scammer" thread because I didn't want the extra drama. I just buried it in other places.
https://www.blackhatworld.com/seo/g...os-new-technique.1514798/page-2#post-16744499
This is also an interesting post - https://www.blackhatworld.com/seo/it-is-so-easy-to-scam-people-in-bhw.1541895/page-4#post-16960420
Executive Summary
----------------------------
Koray isn't a "scammer" per-say. I'm being harsh because I don't like him. He's not a good person though. He doesn't treat his staff well and he responds very aggressively to anyone who questions him. I know I'm bad sometimes and get a bit defensive when people disagree with me, but at least I recognize that part of myself, admit it and work on giving everyone more respect no matter their opinion. Even people who say Python isn't a real language.. (A little in-joke for anyone following me in the Python thread) -- But Koray is like me on my worst day x 10. Hyper aggressive.
But his course is mostly BS.
I'm not a psychologist, but my analysis of him is he's somewhere on the NPD spectrum and he feeds off having a fan-base that think he's god-like. He almost definitely drinks his own coolade. He is a clever enough guy, and he does have good SEO knowledge, and this is what makes him dangerous.
He takes some truths, and adds a bunch of made-up and mis-used language that he gets away with because English isn't his native language. He takes things further by taking the view that his way is the ONLY way and he's basically reverse engineered Google.
You'll get more from my guides which are free and deliberately made to not be confusing so you buy a $997 course, then find out that to get the real secrets you need the $5k private mentoring package.(Yes, he really did this)
And my guides don't have a hidden agenda. I'm not seeking fame because I like fame. It's just the best way to do business. The most valuable thing in business is your brand/authority. If you can build that up and gain people's trust, you make money easily, so I put most of my energy into just helping and giving value, because in the long run it pays off for me, giving me a very easy/relaxed life because I have an edge over competitors.
The ultimate example of this is Alex Hormozi. Hes' built his brand to such an extreme he could do almost anything and make money because he has such personal brand power. In fact his personal brand is SO strong, that people will pay HIM to email their list. The rest of us have to pay to use other people's email lists.
Think about it.. Imagine you have a bodybuilding list. Would you pay Arnold Schwarzenegger to email your list, or would you charge him money to send an email?
When your value becomes net positive, someone's list actually STRENGTHENS because of you, compared with most, where their list weakens. Ie, if you let some random guy email your list with his supplement offer, you lose trust on your list, so you need to charge them enough to justify this loss in your list strength. Branding is so powerful. People don't realize how powerful it is. I talk about skills being the most valuable thing in business, but really, your personal brand is top, followed by your skill. If you invest in your brand, you'll always be successful.
I've gone a bit off topic, but hey.. It's my journey
Dark Hat 007
Senior Member
- Dec 2, 2016
- 1,164
- 773
Wow!! Thanks for taking time to give this answer and the restrain you used is not that easily found on reddit, they seem to hate him so much there and as much as I try to see him for his video titles, support from other SEO gurus, and the charts he uploads, I also know there's no smoke without fire.. Thanks once again for the time, I found the course here on BHW and would just see it for myself.. Thanks.Happy New Year to you too buddy.
I've written about him before. I just didn't open a full "Koray is a scammer" thread because I didn't want the extra drama. I just buried it in other places.
https://www.blackhatworld.com/seo/g...os-new-technique.1514798/page-2#post-16744499
This is also an interesting post - https://www.blackhatworld.com/seo/it-is-so-easy-to-scam-people-in-bhw.1541895/page-4#post-16960420
Executive Summary
----------------------------
Koray isn't a "scammer" per-say. I'm being harsh because I don't like him. He's not a good person though. He doesn't treat his staff well and he responds very aggressively to anyone who questions him. I know I'm bad sometimes and get a bit defensive when people disagree with me, but at least I recognize that part of myself, admit it and work on giving everyone more respect no matter their opinion. Even people who say Python isn't a real language.. (A little in-joke for anyone following me in the Python thread) -- But Koray is like me on my worst day x 10. Hyper aggressive.
But his course is mostly BS.
I'm not a psychologist, but my analysis of him is he's somewhere on the NPD spectrum and he feeds off having a fan-base that think he's god-like. He almost definitely drinks his own coolade. He is a clever enough guy, and he does have good SEO knowledge, and this is what makes him dangerous.
He takes some truths, and adds a bunch of made-up and mis-used language that he gets away with because English isn't his native language. He takes things further by taking the view that his way is the ONLY way and he's basically reverse engineered Google.
You'll get more from my guides which are free and deliberately made to not be confusing so you buy a $997 course, then find out that to get the real secrets you need the $5k private mentoring package.(Yes, he really did this)
And my guides don't have a hidden agenda. I'm not seeking fame because I like fame. It's just the best way to do business. The most valuable thing in business is your brand/authority. If you can build that up and gain people's trust, you make money easily, so I put most of my energy into just helping and giving value, because in the long run it pays off for me, giving me a very easy/relaxed life because I have an edge over competitors.
The ultimate example of this is Alex Hormozi. Hes' built his brand to such an extreme he could do almost anything and make money because he has such personal brand power. In fact his personal brand is SO strong, that people will pay HIM to email their list. The rest of us have to pay to use other people's email lists.
Think about it.. Imagine you have a bodybuilding list. Would you pay Arnold Schwarzenegger to email your list, or would you charge him money to send an email?
When your value becomes net positive, someone's list actually STRENGTHENS because of you, compared with most, where their list weakens. Ie, if you let some random guy email your list with his supplement offer, you lose trust on your list, so you need to charge them enough to justify this loss in your list strength. Branding is so powerful. People don't realize how powerful it is. I talk about skills being the most valuable thing in business, but really, your personal brand is top, followed by your skill. If you invest in your brand, you'll always be successful.
I've gone a bit off topic, but hey.. It's my journey
- Oct 9, 2013
- 3,471
- 14,478
llama3 is INSANELY powerful when fine tuned.
As many of you will know gpt 4o has been released, and it's WAY smarter than gpt4.
It really blows it out the water imo. I've given it some really hard programming problems that gpt4 couldn't even begin to tackle, and it's given me a complete working solution.
But..
I just fine-tuned a llama3-8b and it's beating chatgpt 4o on its downstream task, which is one of my fine-tunes above.. Classifying a webpage.
Here, look at this.
This is giving the full instruction prompt to gpt 4o
It's response?
gallery:9
If I give it to llama-3-8b-instruct (ie, the baseline 8b) we get "gallery:10"
If we give it to llama-3-70b-instruct (ie, the baseline 70b) we get "best X: 10"
lol.
If we give it to my fine-tuned llama-3-8b..
And by give I mean just the outline, no instruction: -
Cost to fine-tune? $2.83
Time to fine-tune - 45 mins
Cost to use? With predibase $0.2 per 1M tokens.
25x cheaper than gpt 4o and more accurate than gpt 4o
I've tested it on difficult ones, and it gets them right every time.
Notice my instruction for gpt 4o and the other instruct models. I even modified the definition of gallery to try to help them by saying it should have no text, and I'm giving them the word count. Heck, it has 'recipe' slapped all over it. They really should get it, but alas, they don't.. They also sometimes make mistakes with blog category pages and single product reviews where the fine tuned llama-3-8b gets it right every time.
EDIT: Just to make clear to anyone trying to replicate this. I'm using llama-3-8b BASE. Not llama-3-8b-instruct. You want the base one for stuff where you're feeding it heavy patterns like this with no instructions.
As many of you will know gpt 4o has been released, and it's WAY smarter than gpt4.
It really blows it out the water imo. I've given it some really hard programming problems that gpt4 couldn't even begin to tackle, and it's given me a complete working solution.
But..
I just fine-tuned a llama3-8b and it's beating chatgpt 4o on its downstream task, which is one of my fine-tunes above.. Classifying a webpage.
Here, look at this.
This is giving the full instruction prompt to gpt 4o
Code:
I will give you the outline of a webpage.
I want them classified into one of the following categories:
info - informational style article
tutorial - guide/tutorial. A more in depth article that teaches something rather than just informs.
ecom cat - ecommerce category page
ecom product - single ecommerce product page
best X - a best X type page. These pages have less than 5 products.
reviews top 10 - X reviews, top 10 X. Different than best which is more best 2-4 products. The key thing here is they review more than 5 products.
single product review - a review of just 1 product
news - news article. Reporting on current events in the world.
faq - a faq. Generally in the sense of the old school faqs as opposed to a people also ask style PAA page.
forum - a forum post
service - a service being sold. It can be physical or digital, but it's a service, not a product.
recipe - a cooking recipe.
blog cat - a blog category/silo/tag page. The key with this is it's probably going to have a LOT of internal links compared with external.
directory - a directory page/list of links
profile - a profile link, business or person
gallery - If the page doesn't fall into any of the other categories, and has a high amount of images, it's a gallery. Generally a gallery is going to have almost no text.
contact - A contact page
legal - legal documents like privacy policy etc
listacle - This is a blog post that contains lists of things, usually with info and an external link. Ie it's lists of information. Usually it'll have a lot of external links.
Also write out your confidence score out of 10 that scores how confident you are that the category is correct. If you are ABSOLUTELY CERTAIN, then score 10, if you are certain, and there's a tiny chance you might be wrong, score 9, if you are confident it's correct, but there's a slight chance you're wrong, score 7 or 8. If you are fairly confident, but there's a not insignificant chance you are wrong, then score it 5 to 6. If you are not quite sure and making a guess you feel is a good guess, score it 3 to 4. If you have no confidence in your guess and feel it's essentially like rolling a dice, then score it 1 to 2.
Don't explain, just give a category and confidence. Do not make up your own categories, only use the ones I have given you. Examples of results(ALWAYS give the result in this format):
info:8
blog cat:9
best X:10
news:8
Here's the outline:
page:https://www.allrecipes.com/gallery/best-authentic-mexican-recipes/
externals:1
internals:251
a:257
em:1
form:5
h2:25
h3:0
h4:0
h5:0
h6:0
iframe:0
img:0
input:2
li:216
p:23
svg:149
ul:34
picture:0
figure:23
figcaption:23
g:0
sample:"In this true Mexican carne asada recipe, skirt steak is marinated in a cilantro-beer sauce and grilled (asada) to perfection," says Crema. "Serve with a side of drunken beans and corn tortillas. If preferred, substitute flank steak for the skirt steak."
word_count:1018
title:Authentic Mexican Recipes
h1:Our 21 Best Authentic Mexican Recipes
h2:Homemade Mexican Chorizo
h2:Authentic Enchiladas Verdes
h2:Carne en su Jugo (Meat in its Juices)
h2:Authentic Mole Sauce
h2:Menudo Rojo (Red Menudo)
h2:Birria de Res Tacos (Beef Birria Tacos)
h2:Churros
h2:Carne Asada al Cilantro
h2:Mexican Mango and White Fish Ceviche
h2:Sweet Orange Tamales
h2:Mexican Enchiladas Suizas
h2:Migas
h2:Guacamole with Corn
h2:Tamales Oaxaqueños (Oaxacan-Style Tamales)
h2:Authentic Mexican Chili Rellenos
h2:Cochinita Pibil (Mexican Pulled Pork in Annatto Sauce)
h2:Chiles en Nogada (Mexican Stuffed Poblano Peppers in Walnut Sauce)
h2:Authentic Mexican Hot Chocolate with Chile
h2:Green Rice with Cheese
h2:Pescado en Achiote (Mexican Fish in Annatto Sauce)
h2:Authentic Tacos al Pastor
h2:More Mexican Recipes and Inspiraton
h2:You’ll Also Love
h2:Add to collections
h2:New CollectionIt's response?
gallery:9
If I give it to llama-3-8b-instruct (ie, the baseline 8b) we get "gallery:10"
If we give it to llama-3-70b-instruct (ie, the baseline 70b) we get "best X: 10"
lol.
If we give it to my fine-tuned llama-3-8b..
And by give I mean just the outline, no instruction: -
Cost to fine-tune? $2.83
Time to fine-tune - 45 mins
Cost to use? With predibase $0.2 per 1M tokens.
25x cheaper than gpt 4o and more accurate than gpt 4o
I've tested it on difficult ones, and it gets them right every time.
Notice my instruction for gpt 4o and the other instruct models. I even modified the definition of gallery to try to help them by saying it should have no text, and I'm giving them the word count. Heck, it has 'recipe' slapped all over it. They really should get it, but alas, they don't.. They also sometimes make mistakes with blog category pages and single product reviews where the fine tuned llama-3-8b gets it right every time.
EDIT: Just to make clear to anyone trying to replicate this. I'm using llama-3-8b BASE. Not llama-3-8b-instruct. You want the base one for stuff where you're feeding it heavy patterns like this with no instructions.
TheCrow
Junior Member
- Aug 23, 2016
- 199
- 128
This is one of my favorite threads here on BHW, glad you're picking it up again!
I've been interested in learning how to fine-tune models for a while.
Are you aware of any "getting started" guide or resources that you can recommend? For context, I've been programming on and off for a couple of years now.
Thanks!
I've been interested in learning how to fine-tune models for a while.
Are you aware of any "getting started" guide or resources that you can recommend? For context, I've been programming on and off for a couple of years now.
Thanks!
- Jun 23, 2021
- 1,510
- 1,557
why not feed the whole text instead of samples? llama 3 8b's 8k context window can handle most articles. Is it to keep a low cost per classification?
How large is your dataset for this? I mean, do you manually collect some of the data? and how many examples are sufficient?
I was using LLM as a classfier for a long time without fine-tuning but having seen the amazing performance of llama3 8b and the ease of predibase, I might finally train some of 8b for specific-task fine-tunes.
Thanks
- Oct 9, 2013
- 3,471
- 14,478
When i originally created it we didnt have large context windows, but it feels redundant having to do that.
It shouldnt need the entire page to classify. A human can classify from just the title and h tags.
I wonder if it can do it from just the html tag counts..
It would drive the cost up passing more and plus 8k isnt enough for all pages.
This model is one component of a SaaS im working on so thats one reason i want to keep the costs down. They all add up and i need to be able to classify 1000s of pages.
The original data cost me about $300 to generate with gpt4 back before prices dropped. It is 10,000 samples and about 2.5 million tokens.
I first asked gpt4 to generate search terms that had a high probability of giving me the classification i wanted. Did that for all 20-25 of them then passed those search terms into a google scraper which searched google and put the results in a json with original kw.
Next i passed that data into a program to website scraper which parsed out the structure you see above for each page then added back to the json.
Next i passed to a program to pass each template and prompt to gpt4 and saved the classification it gave with confidence level back into the json.
Then i manually checked certain groups that i knew it struggled with and edited them. Some but 10k is a lot. I am actually going to generate new training data doing the following:
Classify with my l3 classifier
Classify with gpt4o
where they agree i will mark as good and when they disagree i will mark for manual human review.
Generate 20k samples this time and then train a new l3-8b with that.
Then ill try training another with the same data but with only the title and h tags, and another with only the tag counts then validate them against the main one and compare(just for the experiment)
- Jun 23, 2021
- 1,510
- 1,557
Sounds fair.
I'm surprised you need 10K samples for fine-tuning, I thought 100-ish would suffice.
I agree feeding the whole visible text or HTML is overkill, I will find a way to balance the cost and classify efficiency.
My workflow was pretty similar to yours for preparing dataset, thanks for the insight. I learned something new!
I'll be waiting for your experiment and follow this thread, if you will update them later in the post.