1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Indexing thousands of urls at once?

Discussion in 'Black Hat SEO' started by batiatus, Jul 31, 2017.

  1. batiatus

    batiatus Junior Member

    Joined:
    May 8, 2011
    Messages:
    194
    Likes Received:
    188
    Hey all,
    I have 2 websites with large database, once is on WP + Yoast has almost 250k posts and 15k categories. Once I imported my database, Google started to index my website without any submitting or SEO effort. Thought it will continue like that. Right after I did set up basic SEO stuffs like Yoast, adding G webmaster and analytics tool and adding my website to Bing + Yandex, indexing has stopped at 44k like website is not even exist on WWW anymore. It has been a month and not a single url has been indexed in 3 weeks. Webmaster tool shows no error, seems there is no problem about my website content, speed or accessibility at all. Is there any way to get all my urls indexed, except add sitemap to webmaster tool and cross your fingers for G to get them indexed or not? I'm also looking for an indexing tool/service with free or affordable price.

    Another website is in much terrible situation. It has 900k urls in sitemap, only have 60k indexed in like 4 months. It is DLE based website with a cheap hosting. It is really frustrating when you work on your website and with huge database to fit it in, giving your hours to make it happen but do not get back your effort because Big G doesn't index your urls. I'm here to listen your advices guys. Thanks.
     
  2. kickthat

    kickthat Jr. VIP Jr. VIP

    Joined:
    Sep 18, 2014
    Messages:
    471
    Likes Received:
    592
    Gender:
    Male
    Location:
    UK
    How much content does each of your individual pages have? Is that content unique or copied from elsewhere?

    If there is not much content or it's copied or poor quality then pages probably won't get indexed at all.

    Aside from that, make sure your pages are well interlinked. Link all categories from the homepage and make sure posts are easily accessible from the category pages. If there are too many steps for google to take to see a URL, or if it's difficult to find on the site, crawlers will struggle or not bother.
     
  3. batiatus

    batiatus Junior Member

    Joined:
    May 8, 2011
    Messages:
    194
    Likes Received:
    188
    I can't say content is top quality, scraped and spinned. But all the pages are unique, nothing duplicated, well categorized etc.
    When you have a big website, it is always hussle to make every page accessible from index, you know. I tried to put a categories dropdown menu to sidebar, only the dropdown html code was about 400kb so I decided to give it up.
     
  4. kickthat

    kickthat Jr. VIP Jr. VIP

    Joined:
    Sep 18, 2014
    Messages:
    471
    Likes Received:
    592
    Gender:
    Male
    Location:
    UK
    I get that. Is there any way you could make broader top level category pages, e.g 15 top level pages each with 1000 categories on them, or 30 with 500 etc, then link them from the index?
    They don't all need to be accessible straight from your homepage, just ideally as few steps as possible away from it.

    Or 'feature' say 10-20 categories each month, linking them from the homepage to give Google a chance to see them?

    Edit: one other thing - I work on a few sites with huge amount of pages. We link to a user site map in the footer with all the categories. It's not really a user site map, it's there for search engines. Not sure whether we have as many categories as you, and it takes a while to load, by it seems to have worked in improving indexing.
     
  5. batiatus

    batiatus Junior Member

    Joined:
    May 8, 2011
    Messages:
    194
    Likes Received:
    188
    That might work. I have posted randomly my category pages to blogspot blogs (around 100 different blogs), this trick was helped me before but it is not as effective as past. Categories are indexed, but pages are invisible.
     
  6. kickthat

    kickthat Jr. VIP Jr. VIP

    Joined:
    Sep 18, 2014
    Messages:
    471
    Likes Received:
    592
    Gender:
    Male
    Location:
    UK
    Try using fetch as google in search console on a few of the specific categories or pages. If they get indexed and stick, it's likely just that google is struggling to see them. If they drop back out, I would suggest it is probably your content.
     
  7. djp371

    djp371 Elite Member

    Joined:
    Apr 20, 2009
    Messages:
    2,082
    Likes Received:
    350
    Occupation:
    Internet marketing
    Location:
    United States
    Home Page:
    Check your internal content for duplicates. Trailing slashes at the end of your URL may cause that. Then you can also use the Fetch as Google. Usually works like charm.
     
  8. Unavailable

    Unavailable Junior Member

    Joined:
    Mar 24, 2015
    Messages:
    116
    Likes Received:
    16
    You can use indexing services. There arent effective as before but it will help
     
  9. redrays

    redrays Regular Member

    Joined:
    Apr 3, 2015
    Messages:
    388
    Likes Received:
    130
    It's simply gotten much harder to index these sort of sites over the past few years, at least on no / low authority domains. The people I know who still build them have responded with some combination of building smaller sites (i.e. break up your 250k posts into 10 sites of 25k each), using a higher % of unique content on each post, and using indexers that submit urls to Google Webmaster Console. Two examples of this style of indexer are Index Inject and Authority Indexer, though there are at least a half dozen out there. They work, but they're not free and tend to be expensive relative to other indexers.