Google search experts please help me out

Discussion in 'Black Hat SEO' started by IMShane, Apr 20, 2012.

  1. IMShane

    IMShane Junior Member

    Joined:
    Sep 20, 2011
    Messages:
    131
    Likes Received:
    23
    Let's take domain.com as an example, if I google "site:domain.com" all pages such as domain.com/about.html, domain.com/example/whats-up.html, etc. would appear as search results. If I only want results such as domain.com/xxxx and keep deeper urls like domain.com/xxxx/xxxx.html out, what parameters should I use?

    Thanks

    P.S.: I'm trying to scrape indexed pages of a site to a certain level (to domain.com/xxxx in this case).
     
    Last edited: Apr 20, 2012
  2. Nordajgt

    Nordajgt Regular Member Premium Member

    Joined:
    May 12, 2011
    Messages:
    466
    Likes Received:
    195
    Occupation:
    Director of Marketing
    Location:
    Brooklyn
    Put a no index on each page you don't want indexed and disallow pages on your robots.txt
     
  3. IMShane

    IMShane Junior Member

    Joined:
    Sep 20, 2011
    Messages:
    131
    Likes Received:
    23
    Well, I think you misunderstood. I was just trying to scrape indexed results of a domain (only to domain.com/xxxx level). domain.com is not my site.