What kind of features would you like to see in a proxy crawler? I am writing a small one for a project. So far I have settings for websites to crawl, keywords to search google,yahoo,bing regex to find ip port. What other kind of things should I try? Also anyone have a good website/keyword list?
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
Proxy Crawler Ideas
- Thread starter civilwar
- Start date