Removing Domain Duplicates From 1gb file.

1

12121231

Registered Member
Joined
Nov 26, 2010
Messages
83
Reaction score
18
Hi guys. I got 1gb worth of auto-approve blogs and I need to filter out the domains, and it'll take a long time on scrapebox, so i was wondering if their was an easier way to do it?

Any advice/help appreciated and if someone can help me I'll be sure to email you the completed 100's of thousands of auto-approve blogs.
Thanks :)
 
Advertise on BHW
Use the scrapebox dupe remove add-on , You can't really do it faster i think.
 
I can do this for you VERY quickly. Please PM me and let me know how you want them filtered specifically.
 
I am creating my own duplicates remover and can run your file through it if you want.

It was needed because there is no software I could find that can do that the way I want it. All compare the string between the beginning and the first "/" but this means that abf.yahoo.com and fgr.yahoo com will come out as different domains while they are not.

1 GB uncompressed file should be about 10 million URLs. But unless your list is really comprehensive you will not get very many different domains, something like 200k or so. I routenly monitor urls submitted to ping services and last time came up with 1 456 183 out of 99 236 306.
 
Dude, I just got your PM. This is the file of URLs posted here on the forum already. It's not yours, and it's not original auto-approved blogs right?
 
Hey, yeah they are from the forum, and I've been checking them with scrapebox and majority are auto-approve. So yes they are good quality
 
12121231 said:
Hey, yeah they are from the forum, and I've been checking them with scrapebox and majority are auto-approve. So yes they are good quality
Click to expand...

They aren't auto approve. Something isn't right on your end, because those are not even 5% auto approve domains if they are the 926mb file posted the other day. That guy was just posting a random list of urls. How are you checking them with scrapebox? It would take you weeks to post to all of those URLs, so I don't see how it is possible you could have "checked the majority of them." In fact, since the time since you downloaded the file, you wouldn't have even been able to post to 10% of the list. And also, if you already posted to even 10% of the list, then you would easily already have the file broken up into manageable packages which you can load into scrapebox to post with. If that was the case, you would just use the dup remover button in scrapebox.

Response?
 
Kickflip said:
They aren't auto approve. Something isn't right on your end, because those are not even 5% auto approve domains if they are the 926mb file posted the other day. That guy was just posting a random list of urls. How are you checking them with scrapebox? It would take you weeks to post to all of those URLs, so I don't see how it is possible you could have "checked the majority of them." In fact, since the time since you downloaded the file, you wouldn't have even been able to post to 10% of the list. And also, if you already posted to even 10% of the list, then you would easily already have the file broken up into manageable packages which you can load into scrapebox to post with. If that was the case, you would just use the dup remover button in scrapebox.

Response?
Click to expand...

Valid points my friend, I wasn't meaning I scanned them all and the majority were auto approved. I scanned about 10, 000 of them and they were mainly auto approve, so I assumed it was consistant. On top of this the guy who posted it said they were auto-approved, and your right I wouldn't have been even able to scan 10% of them. Thanks for the good idea about the dup remover as well:)
 
12121231 said:
Valid points my friend, I wasn't meaning I scanned them all and the majority were auto approved. I scanned about 10, 000 of them and they were mainly auto approve, so I assumed it was consistant. On top of this the guy who posted it said they were auto-approved, and your right I wouldn't have been even able to scan 10% of them. Thanks for the good idea about the dup remover as well:)
Click to expand...

[GET] Over 14 Million Unique WordPress Blog Links

If that is the post you are talking about, the guy did not say they are auto approve. And what do you mean you "scanned 10,000 of them and they were mainly auto approve". I am telling you man, they are not auto approve urls. I think you are mistaken about what you found. I ran over 200,000 of the urls and only got 900 unique auto approves.
 
I dont know mate, I remember seeing heaps of auto-approve, it doesnt matter anyway, Ill use the ones I do get and get more.
 
12121231 said:
Hi guys. I got 1gb worth of auto-approve blogs and I need to filter out the domains, and it'll take a long time on scrapebox, so i was wondering if their was an easier way to do it?

Any advice/help appreciated and if someone can help me I'll be sure to email you the completed 100's of thousands of auto-approve blogs.
Thanks :)
Click to expand...

you can split file with soft TextFileSplitter soft then delete dupes for each splited file and then merge again with LinksList Merger soft
 
mazgalici said:
"unique" command from Linux
Click to expand...
uniq requires you to presort so I find it much easier to just do. This works far better than scrapebox and can actually dedupe lists that your computer can't fully hold in memory.

Code: 
sort -u -o file.txt file.txt
 
Last edited:
Crazyflx's guide is pretty good if SB dupe addon is crashing. http://crazyflx.com/scrapebox-tips/remove-duplicate-domains-urls-from-huge-txt-files-with-ease/ - can still freeze if your list is too big though.

btw there's no way in hell you have 1gb of auto approve blogs. if you do i applaud you.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top