- Oct 9, 2013
- 3,471
- 14,478
When i originally created it we didnt have large context windows, but it feels redundant having to do that.
It shouldnt need the entire page to classify. A human can classify from just the title and h tags.
I wonder if it can do it from just the html tag counts..
It would drive the cost up passing more and plus 8k isnt enough for all pages.
This model is one component of a SaaS im working on so thats one reason i want to keep the costs down. They all add up and i need to be able to classify 1000s of pages.
The original data cost me about $300 to generate with gpt4 back before prices dropped. It is 10,000 samples and about 2.5 million tokens.
I first asked gpt4 to generate search terms that had a high probability of giving me the classification i wanted. Did that for all 20-25 of them then passed those search terms into a google scraper which searched google and put the results in a json with original kw.
Next i passed that data into a program to website scraper which parsed out the structure you see above for each page then added back to the json.
Next i passed to a program to pass each template and prompt to gpt4 and saved the classification it gave with confidence level back into the json.
Then i manually checked certain groups that i knew it struggled with and edited them. Some but 10k is a lot. I am actually going to generate new training data doing the following:
Classify with my l3 classifier
Classify with gpt4o
where they agree i will mark as good and when they disagree i will mark for manual human review.
Generate 20k samples this time and then train a new l3-8b with that.
Then ill try training another with the same data but with only the title and h tags, and another with only the tag counts then validate them against the main one and compare(just for the experiment)
Yeah html is near impossible. You would need a special htm tokenizer otherwise the prompt size can be like 50k to 100k tokens.
Wow 100 samples, never
Fine tuning should have at least a million tokens.
There are 20+ categories so 100 would be like 5 examples per category.
Remember we are fine tuning base models. If done right you will always get higher performance vs an instruct model.
The confusion comes from doing simple fine tunes with instruct models.
Ie a fine tune of an instruct model vs doing say a 2 to 5 shot will give you an improvement. Thats for simple tasks where the instruct model only needs a few examples to get the hang of it.
There is a LOT of data and variation with the classification task so the model needs many examples to start to differentiate between the classes. Its harder than things like sentiment classification. Ie BERT couldnt do this. With normal classification BERT understands both the text and the semtiment, ie positive, as they are natural language but what the heck is a “single product ecom page” thats technical vocabulary that needs to be learned and understood. Even a “service page” vs a “single product ecom” isnt a distinction a normal human could make without teaching them first. Try asking your gran to do that for example
she would need you to give her examples and explain the differences.Hence why we need at least 10k examples. Preferably 20k. The more data the better. Overfitting occurs if you have too little data, too many epochs and a small batch size. With more data you need less epochs and at the same time you can use larger batch sizes which helps the model converge in general patterns and not micro patterns within the training data
