How do I clean a scraped list of sites effectively?

My usual process is to:

1. use scrapebox to scrape a list of sites for one of my seo tools
2. trim to root
3. remove duplicate domains
4. remove domains with bad keywords in them
5. I'd say run alive check here, but lately I haven't even done that. I've noticed alive check works only with high quality proxies, in large numbers, which I don't have. When I run it without proxies or with a small number of proxies, I get thousands of "DEAD" results for sites that are really "ALIVE" and its killing my lists, so I stopped doing alive checks.
6. At this step I plug my list into my software, do a run, and then use the successful submissions for my list.

The problem is, I'm scrapping lists of 100k+ and it's taking my bots and tools forever to run through the whole list. Are there some better ways to trim down the results before doing the test run? Am I using alive check wrong and it does actually work?

Any help is greatly appreciate, thanks guys.

Try using Ultraedit and learn all the special regex functions there in combination with search and replace

why would alive checker need proxies at all?

RBseatown said:

My usual process is to:

2. trim to root

Click to expand...

You have to be getting terrible results trimming to root. Most home pages are not able to accept a comment.

Also I would not waste my time with alive checker.

If you are going after auto approve I usually just do this.

1-scrape urls
2-fast post to all without any cleanup "use either a throwaway url or just put all your tier 1's here"
3-run link checker
4-if you want to check pagerank and filter out the bad ones do it here.
5-use link extractor tool to extract all internal links from links found from step 3
6-run fast poster again
7-check links and add found to auto approve list.

Like ThreadKiller, I use UltraEdit to clean a list via tons of regular expressions.

Beny

kokoloko75 said:

Like ThreadKiller, I use UltraEdit to clean a list via tons of regular expressions.

Beny

Click to expand...

Beny, like what?
can you give an example please?

thanks

I use the sick submitter platform sorter to help out with this. I need large numbers of bookmarking sites for what I do, and I don't have a year to wait for my bot to go through and determine which sites are actually useful. I don't even use sicksubmitter anymore, but the platform sorter is extremely useful.

It is a free download, and it will go through a large list of sites and organize them based on the platform they are using. For example, I can load a 100k+ sites into it, and it will determine which ones are phpdug, pligg etc. You might want to check it out. It covers all the major platforms and it isnt just limited to bookmarking platforms either.

If OP (or anyone) shows a couple of lines from a typical proxy list, I will post an example of how it can be cleaned in UltraEdit using regular expressions.

metalice said:

Beny, like what?
can you give an example please?

thanks

Click to expand...

what do you mean a couple of lines from a typical proxy list?

a few proxy servers!?

Just an example of the type of raw list that the OP wants to trim down.

metalice said:

what do you mean a couple of lines from a typical proxy list?

a few proxy servers!?

Click to expand...

I personally find trimming to root and then removing duplicate domains takes longer than removing duplicate domains then trimming to root, but it might just be me.

Tenshisendo said:

You have to be getting terrible results trimming to root. Most home pages are not able to accept a comment.

Also I would not waste my time with alive checker.

If you are going after auto approve I usually just do this.

1-scrape urls
2-fast post to all without any cleanup "use either a throwaway url or just put all your tier 1's here"
3-run link checker
4-if you want to check pagerank and filter out the bad ones do it here.
5-use link extractor tool to extract all internal links from links found from step 3
6-run fast poster again
7-check links and add found to auto approve list.

Click to expand...

I don't trim to root when going for blog comments

sixalarm said:

I use the sick submitter platform sorter to help out with this. I need large numbers of bookmarking sites for what I do, and I don't have a year to wait for my bot to go through and determine which sites are actually useful. I don't even use sicksubmitter anymore, but the platform sorter is extremely useful.

It is a free download, and it will go through a large list of sites and organize them based on the platform they are using. For example, I can load a 100k+ sites into it, and it will determine which ones are phpdug, pligg etc. You might want to check it out. It covers all the major platforms and it isnt just limited to bookmarking platforms either.

Click to expand...

BINGO! Thanks man.