1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Google search experts please help me out

Discussion in 'Black Hat SEO' started by IMShane, Apr 20, 2012.

  1. IMShane

    IMShane Junior Member

    Joined:
    Sep 20, 2011
    Messages:
    131
    Likes Received:
    23
    Let's take domain.com as an example, if I google "site:domain.com" all pages such as domain.com/about.html, domain.com/example/whats-up.html, etc. would appear as search results. If I only want results such as domain.com/xxxx and keep deeper urls like domain.com/xxxx/xxxx.html out, what parameters should I use?

    Thanks

    P.S.: I'm trying to scrape indexed pages of a site to a certain level (to domain.com/xxxx in this case).
     
    Last edited: Apr 20, 2012
  2. Nordajgt

    Nordajgt Regular Member Premium Member

    Joined:
    May 12, 2011
    Messages:
    464
    Likes Received:
    195
    Occupation:
    Director of Marketing
    Location:
    Brooklyn
    Put a no index on each page you don't want indexed and disallow pages on your robots.txt
     
  3. IMShane

    IMShane Junior Member

    Joined:
    Sep 20, 2011
    Messages:
    131
    Likes Received:
    23
    Well, I think you misunderstood. I was just trying to scrape indexed results of a domain (only to domain.com/xxxx level). domain.com is not my site.