BHW's Growth Visualized

  • Thread starter Thread starter Deleted member 969102
  • Start date Start date
Advertise on BHW
I've been active on forum recently, that might be the reason for the growth :P
 
Conor said:
Yeah but Proxygo is a member here, so.
Click to expand...
I actually also have the graphs for each member as well as a whole host of other stuff like the average number of thanks per post, average thread replies within one month, average message length, post rate per sub-forum etc. etc. etc. But I felt that this thread made more sense with just two (now three) graphs.
 
Interesting. What's the source of this data?
 
bartosimpsonio said:
Interesting. What's the source of this data?
Click to expand...
jimbobo2779 said:
I'm interested in knowing this myself.
Click to expand...

I am.
Code: 
import urllib2
from bs4 import BeautifulSoup
import csv

n=100
m=108259 # Total number of posts / n. For example, 10825900/100 = 108259.

id_list = range(0, (m+1)*n, n)[1:]

i = 99080
while i < len(id_list):
   
    current_id = id_list[99080]
    web_page = "https://blackhatworld.com/posts/"
    post_id = current_id
    full_url = web_page + str(post_id)
   
    print full_url
   
    #Try to open page
    try:
        page = urllib2.urlopen(full_url)
        soup = BeautifulSoup(page, 'html.parser')
       
        li_post_id = "post-" + str(post_id)
        li_soup = soup.find('li', attrs={'id':li_post_id})
       
        div1_soup = li_soup.find(attrs={'class':'DateTime'})
       
        s1 = div1_soup
        l = s1['title']
       
        print(current_id)
        print("")
        print(l + str(current_id))
       
        with open('date_and_id.csv', 'a') as csv_file:
            writer = csv.writer(csv_file)
            writer.writerow([l, current_id])
       
        del id_list[99079]
       
       
        print len(id_list)
   
    # Response for 404 or 403
    except urllib2.HTTPError as e:
            if e.getcode() == 404: # check return code
                div1_soup = "missing"
               
                print(current_id)
                print("")
                print(div1_soup + str(current_id))
   
                with open('date_and_id.csv', 'a') as csv_file:
                    writer = csv.writer(csv_file)
                    writer.writerow([div1_soup, current_id])
               
                del id_list[0]
               
                print len(id_list)
               
            if e.getcode() == 403: # check return code
                div1_soup = "403"
               
                print(current_id)
                print("")
                print(div1_soup + str(current_id))
                with open('date_and_id.csv', 'a') as csv_file:
                    writer = csv.writer(csv_file)
                    writer.writerow([div1_soup, current_id])
               
                del id_list[99079]
               
                print len(id_list)
 
elavmunretea said:
I am.
Code: 
import urllib2
from bs4 import BeautifulSoup
import csv

n=100
m=108259 # Total number of posts / n. For example, 10825900/100 = 108259.

id_list = range(0, (m+1)*n, n)[1:]

i = 99080
while i < len(id_list):
  
    current_id = id_list[99080]
    web_page = "https://blackhatworld.com/posts/"
    post_id = current_id
    full_url = web_page + str(post_id)
  
    print full_url
  
    #Try to open page
    try:
        page = urllib2.urlopen(full_url)
        soup = BeautifulSoup(page, 'html.parser')
      
        li_post_id = "post-" + str(post_id)
        li_soup = soup.find('li', attrs={'id':li_post_id})
      
        div1_soup = li_soup.find(attrs={'class':'DateTime'})
      
        s1 = div1_soup
        l = s1['title']
      
        print(current_id)
        print("")
        print(l + str(current_id))
      
        with open('date_and_id.csv', 'a') as csv_file:
            writer = csv.writer(csv_file)
            writer.writerow([l, current_id])
      
        del id_list[99079]
      
      
        print len(id_list)
  
    # Response for 404 or 403
    except urllib2.HTTPError as e:
            if e.getcode() == 404: # check return code
                div1_soup = "missing"
              
                print(current_id)
                print("")
                print(div1_soup + str(current_id))
  
                with open('date_and_id.csv', 'a') as csv_file:
                    writer = csv.writer(csv_file)
                    writer.writerow([div1_soup, current_id])
              
                del id_list[0]
              
                print len(id_list)
              
            if e.getcode() == 403: # check return code
                div1_soup = "403"
              
                print(current_id)
                print("")
                print(div1_soup + str(current_id))
                with open('date_and_id.csv', 'a') as csv_file:
                    writer = csv.writer(csv_file)
                    writer.writerow([div1_soup, current_id])
              
                del id_list[99079]
              
                print len(id_list)
Click to expand...

You crawled the whole site? I bet that took ages or have I read this wrong and it is just scraping a few stats pages?

Either way I am impressed and relieved it is not from a hacked DB.
 
I had a feeling this place was a little less populated recently, nice to see the actual data.
 
elavmunretea said:
I am.
Code: 
import urllib2
from bs4 import BeautifulSoup
import csv

n=100
m=108259 # Total number of posts / n. For example, 10825900/100 = 108259.

id_list = range(0, (m+1)*n, n)[1:]

i = 99080
while i < len(id_list):
  
    current_id = id_list[99080]
    web_page = "https://blackhatworld.com/posts/"
    post_id = current_id
    full_url = web_page + str(post_id)
  
    print full_url
  
    #Try to open page
    try:
        page = urllib2.urlopen(full_url)
        soup = BeautifulSoup(page, 'html.parser')
      
        li_post_id = "post-" + str(post_id)
        li_soup = soup.find('li', attrs={'id':li_post_id})
      
        div1_soup = li_soup.find(attrs={'class':'DateTime'})
      
        s1 = div1_soup
        l = s1['title']
      
        print(current_id)
        print("")
        print(l + str(current_id))
      
        with open('date_and_id.csv', 'a') as csv_file:
            writer = csv.writer(csv_file)
            writer.writerow([l, current_id])
      
        del id_list[99079]
      
      
        print len(id_list)
  
    # Response for 404 or 403
    except urllib2.HTTPError as e:
            if e.getcode() == 404: # check return code
                div1_soup = "missing"
              
                print(current_id)
                print("")
                print(div1_soup + str(current_id))
  
                with open('date_and_id.csv', 'a') as csv_file:
                    writer = csv.writer(csv_file)
                    writer.writerow([div1_soup, current_id])
              
                del id_list[0]
              
                print len(id_list)
              
            if e.getcode() == 403: # check return code
                div1_soup = "403"
              
                print(current_id)
                print("")
                print(div1_soup + str(current_id))
                with open('date_and_id.csv', 'a') as csv_file:
                    writer = csv.writer(csv_file)
                    writer.writerow([div1_soup, current_id])
              
                del id_list[99079]
              
                print len(id_list)
Click to expand...

Stealed
 
jimbobo2779 said:
You crawled the whole site? I bet that took ages or have I read this wrong and it is just scraping a few stats pages?

Either way I am impressed and relieved it is not from a hacked DB.
Click to expand...
One in every 100 posts, didn’t take that long. No need for any hacked stuff, it’s all publicly available information, if you know where to look.

MatthewGraham said:
Stealed
Click to expand...
That’s one of my first drafts - just something I had on the closest laptop to me. It starts scraping from some random post near the end because I was testing the scripts’s ability to restart where it left off.
 
Asif WILSON Khan said:
Did you ask permission to crawl the site OP?
Click to expand...
Permission? It’s part of a large-scale SE project I’m working on. Last time I checked web crawling was allowed, as @MatthewGraham pointed out.
MatthewGraham said:
Seems to be implicitly allowed via the site's robots.txt file.

https://www.blackhatworld.com/robots.txt
Code: 
User-agent: *
Disallow: /members/

User-agent: Fasterfox
Disallow: /
Sitemap: https://www.blackhatworld.com/sitemap.php
Click to expand...
 
elavmunretea said:
Permission? It’s part of a large-scale SE project I’m working on. Last time I checked web crawling was allowed, as @MatthewGraham pointed out.
Click to expand...

It was more of a courtesy thing, as some sites don't like being crawled like that (as we all know), it may very well be permitted but I would have checked with @Diamond Damien or @Apricot first.
 
Asif WILSON Khan said:
It was more of a courtesy thin
Click to expand...

I don’t have time to ask permission for all x,xxx,xxx sites I’m scraping as part of the project. I thought that some members would find this useful, so used BHW as one of the sites I tested my system on.

If it’s allowed in their robots.txt file that’s all the permission I need, imo.

I did reach out to notify a mod that they may see multiple requests from my IP. I also only initially gathered the dates from every 100th post to minimise the load on the server, and also staggered the gathering to not overwhelm anything:

elavmunretea said:
I was testing the scripts’s ability to restart where it left off.
Click to expand...

Although, to be honest, I think we all know the reason for your “concern”.
 
That's some interesting data. Perhaps the fact that IM/DM is getting harder and harder has left people dropping out of the game? Who knows. Would be interesting to find out the possible reasons for the drop in post.

That being said quality over quantity any day of the week. BHW has some amazing members who out a lot of effort into the site and helping people.

May BHW continue to be the great forum that it is today.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top