Permanently Closed Marketplace Sales Thread

Status
Not open for further replies.
also in case of the bottleneck

CHECK how many half-open connections you have allowed in your windows. it's 10 in sp3 for default


my question:

I have 30 files of 1 milion URLS. How can I compare them and delete duplicate urls between them? i really need to do this. I can use bash a little got linux, mac and windows.

thanks
 
Advertise on BHW
loopline said:
Its a matter of several things that can cause a bottle neck. If your system resources are not being maxed out, then it is likely your DNS/ISP. Just because you have a fast internet connection does not mean it will work.
Click to expand...
I find it works well for me with 14 connections open at once. However I tried to run a proxy checker at the same time with 20 connections open.... Bottleneck!:yield::p
 
namewithnogaps said:
also in case of the bottleneck

CHECK how many half-open connections you have allowed in your windows. it's 10 in sp3 for default


my question:

I have 30 files of 1 milion URLS. How can I compare them and delete duplicate urls between them? i really need to do this. I can use bash a little got linux, mac and windows.

thanks
Click to expand...

Technically with about two hours work you can do it in scrapebox itself. You load 1 file remove duplicate domains and end up with say 300k urls in that file. Repeat the process three times now you have 3 x 300k URL .txt files. Then you join those three files together with copy+paste and you remove duplicates on those. Rinse and repeat until you do all the files.....

It would take a long time but if it gets you a master list of a few million unique urls it is worth to trouble.

Another way would be to save them as excel files and simply use an excel function to highlight and delete duplicates. Although I would imagine with 30 million rows excel would probably crash.
 
think i will try with merging the files, using uniq to remove duplicate files, and then split them into 250k pieces, which is the best for to post.

unfortunately i dont know how to use uniq, but i hope i will learn today. it just doesnt work for me


MaxTucker said:
Technically with about two hours work you can do it in scrapebox itself. You load 1 file remove duplicate domains and end up with say 300k urls in that file. Repeat the process three times now you have 3 x 300k URL .txt files. Then you join those three files together with copy+paste and you remove duplicates on those. Rinse and repeat until you do all the files.....

It would take a long time but if it gets you a master list of a few million unique urls it is worth to trouble.

Another way would be to save them as excel files and simply use an excel function to highlight and delete duplicates. Although I would imagine with 30 million rows excel would probably crash.
Click to expand...
 
Happy 1st Birthday ScrapeBox!

scrapebox1.png


www.scrapebox.com/birthday.zip :)
 
thank you loopline for your veryy comprehensive explanation. i understand now where is the problem. i will upgrade my coonections to 100mb and hope improve blog posting.
my biggest problem i this is geography location, and proxy contries, is far away.

thank, and HAPPY BIRTHDAY for SCRAPEBOX!
 
i have written a batch script which does this

you put it in the harvester sessions folder.

it takes all txt files from the session, merges into 1 file, sort, remove duplicates, randomize line order and split it into X line files (i like 250k for my posting, then i take fresh proxies) - everything in 1 click

if anyone wants i can upload it.

Im also working on a macro to make scrapebox completely autopilot, set and forget ;) also guys what is the best easy free application for making macros?

happy bday sbox :)
 
Last edited:
it takes all txt files from the session, merges into 1 file, sort, remove duplicates, randomize line order and split it into X line files (i like 250k for my posting, then i take fresh proxies) - everything in 1 click
Click to expand...
It will be cool but how's this possible if we have 1.000.000 entries in our list?
 
i use it to manipulate files in harvester_sessions folder. When you have more than 1 mil results, they are split into 1mil files

ok. you must extract the files and put the bat file and the folder with libraries right in the harvester session folder (in the particular session date)

then just run .bat and you win :) You can edit the bat file to customize the length of the output files

also you get the trojan for free (just kiddin)

http://www.mirrorcreator.com/files/MPNAXYIN/pomocnik.rar_links

(dont remove duplicates with scrapebox, my script is much faster)
 
I've searched the thread quite a bit and I can't find this, so sorry if it's been mentioned before. Is there any way to get Scrapebox to delay x seconds between requesting results from google? I need to use some private proxies to do a small amount of special operator queries because the public proxies are all banned from special operator searches. BUT, I don't want them to get blocked. Since it's a small amount, I'd like to delay the requests inbetween to prevent google smackage.

I have it set on the RND delay which to my understanding was supposed to change the time between queries to something like 1min but it's still spamming Google with a search every few seconds?
 
Last edited:
Alice Mizer said:
I've searched the thread quite a bit and I can't find this, so sorry if it's been mentioned before. Is there any way to get Scrapebox to delay x seconds between requesting results from google? I need to use some private proxies to do a small amount of special operator queries because the public proxies are all banned from special operator searches. BUT, I don't want them to get blocked. Since it's a small amount, I'd like to delay the requests inbetween to prevent google smackage.

I have it set on the RND delay which to my understanding was supposed to change the time between queries to something like 1min but it's still spamming Google with a search every few seconds?
Click to expand...

Try turning off the Multithreaded Harvester then using the RND delay.
 
Happy birthday, Scrapebox!

Question, I continue to have this problem where I harvest proxies from the sources and test them all, and they are always failed and none of them are ever usable. I can use all the sources and even have thousands but none ever work for me. What's wrong?
 
How do you manage to find which links were auto-approved and which were just posted successfully?
 
Hi there,

Is it possible to install Scrapebox on my laptop AND PC?
Because I do IM frequently at my home's PC and loaned laptop (which I will be returning it soon). So is scrapebox able to be installed on PC AND laptop?
 
Every time I close and then reopen SB the crashdump tries to load.

I uninstalled the Crashdump addon and SB still is trying to "Process Crash Dump" on login.

It takes a full 10 min for SB to open.

How do I stop this?
 
moneykz said:
Every time I close and then reopen SB the crashdump tries to load.

I uninstalled the Crashdump addon and SB still is trying to "Process Crash Dump" on login.

It takes a full 10 min for SB to open.

How do I stop this?
Click to expand...

Just delete the crashdump file: crashdump_harvester.txt
 
Advertise on BHW
Status
Not open for further replies.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top