Making a program that scans websites?

1. copy and paste each page (only has 20 per page) to word
2. delete all the cells except address and port
3. copy and paste to notepad to remove cell format and prepare for transfer
4. i then delete the spaces between address and port and replace with ':'
5. save file and import to scrapebox.

DearSanta said:

I would suggest using Zennoposter. It's easy to start with and you don't need programming skills for easy jobs. Try the demo.

Click to expand...

package testing;

import java.util.List;
import java.util.logging.Logger;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.phantomjs.PhantomJSDriver;

public class HideMyAssProxyListGetter {
    private final static Logger LOGGER = Logger.getLogger(HideMyAssProxyListGetter.class .getName());
    
    
    public static void main(String[] args) {
        WebDriver driver = new PhantomJSDriver();
        
        String url = null;
        try{
            for (int i = 5; i > 0; i--){
                url = "http://proxylist.hidemyass.com/"+i;
                driver.get(url);
                
                WebElement table = driver.findElement(By.id("listable"));
                
                for (WebElement tr : table.findElements(By.tagName("tr"))){
                    List<WebElement> tds = tr.findElements(By.tagName("td"));
                
                    if (tds.size()<8){
                        continue;
                    }
                    String ipStr = tds.get(1).getText();
                    if (ipStr.equals("IP address")){
                        continue; //ignore the header line
                    }
                    
                    String portStr = tds.get(2).getText();
                    String location = tds.get(3).getText();
                    
                    
                    String speed = tds.get(4).findElement(By.cssSelector("div.progress-indicator>div")).getAttribute("style").substring(7);
                    String connectionTime = tds.get(5).findElement(By.cssSelector("div.progress-indicator>div")).getAttribute("style").substring(7);
                    String protocol = tds.get(6).getText();
                    String anonymity = tds.get(7).getText();
    
                    LOGGER.info(ipStr + ":" +portStr
                            + " ("+location+")" 
                            + " speed="+speed 
                            + " connectionTime="+connectionTime
                            + " protocol="+ protocol
                            + " anonymity="+ anonymity);

                }
            }
        }
        catch (org.openqa.selenium.NoSuchElementException nse){
            LOGGER.warning("can't access "+url +" try next service..., error was: "+nse.getMessage());
        }
        driver.quit();

    }
}

bluehatface said:

Can you not just grab the page with CURL, and parse the DOM with a XML parser, or regex?

Click to expand...

bluehatface said:

Can you not just grab the page with CURL, and parse the DOM with a XML parser, or regex?

Click to expand...

Arbvestor said:

It depends a bit on the website you want to scrape.

1) If if does not use JavaScript and or AJAX to render the table within the browser you can use a simple approach with a library like JSoup. It has a connect method to load content from the web directly. then you parse the content using JSoup and can easily select (JSoup uses css selectors) the parts of teh site you need. You can print out the results in the format you need. Or directly write to a database or whatever you need.

2) If the site uses AJAX calls to get the actual content or if it renders the page using JavaScript you probably need an approach that uses a real browser. For that, I find the best tool to be Selenium webdriver. Together with PhantomJS, which is a headless browser this is an easy to manage system to scrape otherwise hard to scrape websites.


Genaral note: I assume you use Java for this, but any other language will prbably do, where there is a library like Jsoup and/or selenium bindngs

Click to expand...

Arbvestor said:

It depends a bit on the website you want to scrape.

1) If if does not use JavaScript and or AJAX to render the table within the browser you can use a simple approach with a library like JSoup. It has a connect method to load content from the web directly. then you parse the content using JSoup and can easily select (JSoup uses css selectors) the parts of teh site you need. You can print out the results in the format you need. Or directly write to a database or whatever you need.

2) If the site uses AJAX calls to get the actual content or if it renders the page using JavaScript you probably need an approach that uses a real browser. For that, I find the best tool to be Selenium webdriver. Together with PhantomJS, which is a headless browser this is an easy to manage system to scrape otherwise hard to scrape websites.


Genaral note: I assume you use Java for this, but any other language will prbably do, where there is a library like Jsoup and/or selenium bindngs

Click to expand...

bubbubber said:

Yes, if the site uses AJAX and/or just in general generates a lot of table and form elements on the fly or in nested layers, then it makes it more difficult to scrape the site. Rather, it makes it more difficult to interact with the site (that is, for sites where interaction is necessary).

Click to expand...

• Making a program that scans websites?