BHW's Growth Visualized

  • Thread starter Thread starter Deleted member 969102
  • Start date Start date
I don’t have time to ask permission for all x,xxx,xxx sites I’m scraping as part of the project. I thought that some members would find this useful, so used BHW as one of the sites I tested my system on.

If it’s allowed in their robots.txt file that’s all the permission I need, imo.

I did reach out to notify a mod that they may see multiple requests from my IP. I also only initially gathered the dates from every 100th post to minimise the load on the server, and also staggered the gathering to not overwhelm anything:



Although, to be honest, I think we all know the reason for your “concern”.


Well if you reached out to a Mod informing them of what you were doing, and they had no objection, then it shouldn't be a problem. Although you might want to share the dates of your scraping with Admin just to see if it had any negative effects on the server etc, to see if anything can be improved.
 
My money would be on people leaving due to the infestation of Crypto Clowns.
Or maybe the increase in posts around April was due to the popularity of the Crypto forum...
 
Or maybe the increase in posts around April was due to the popularity of the Crypto forum...

Yeah. Morons from all over the place with their dog shit talk. Now look what is left. Shit loads of regulars have left, and the Crypto tards have all gone quiet.

Scammers coming from left right and centre from Facebook and Redit, with their moronic predictions. All because they thought they could double their paper round money with their "inside information". Yeah, tell us all about the popularity of the Crypto Forum and its benefits for BHW. Did you manage to scrape that data?
 
Yes I am interested on seeing the crypto data also, I need to know when btc will go back up so I can double down
 
Yes I am interested on seeing the crypto data also, I need to know when btc will go back up so I can double down

Jokes aside I bet you could use this kind of data for trend prediction. Analyze frequency of mentions of particular currencies and their mention-velocity to look for early signs of pumps/dumps bulls/bears.
 
Yeah. Morons from all over the place with their dog shit talk. Now look what is left. Shit loads of regulars have left, and the Crypto tards have all gone quiet.

Scammers coming from left right and centre from Facebook and Redit, with their moronic predictions. All because they thought they could double their paper round money with their "inside information". Yeah, tell us all about the popularity of the Crypto Forum and its benefits for BHW. Did you manage to scrape that data?

100% agree
 
LOL what's the old saying. ”lies damn lies and statistics” and ”always look at the source”.

We undoubtedly had an increase due to crypto. I don't think that scared off any blackhatters as has been suggested. More likely they looked on and gamed yet another bubble. Plus most of our engaged members know where to manage their feed so they only get to see what they want to.

We’ve been working hard recently on quality on the forum engaging more mods and Clamping down on thread bumpers from within the marketplace and removing spam (hence the ddos attempts from bummed people trying to game the system). We appreciate the comments from the op. But looking at a top level, without detail from the trenches, is never a good idea.

As @Asif WILSON Khan mentions we always appreciate a heads up when someone wants to use us for a case study or to test their new product. When this doesn't happen and they rely on antiquated things like a robots file as perceived acceptance they may find they are provided with limited data.

Don't worry. BHW is here to stay. Forum stats are always available at the bottom of this page. Must be logged in. https://www.blackhatworld.com/forums/ Note 2 tabs
 
always look at the source
The data can be verified by anyone using the script that I shared or their own.

It would be interesting to see what the graph that the admins see looks like and how close (or far) mine is to it.

antiquated
I’m not sure what you mean by this. It’s a web scraper and the robots.txt file is created for this exact reason.

If you want to choose when you allow SE projects to access this publicly available information then you should be more explicit.

they may find they are provided with limited data.
Yes, this obviously doesn’t account for PMs, removed posts or hidden threads, but let’s not pretend that it’s wildly inaccurate.

The data is less reliable at the beginning, as I was scraping every 100th post - meaning that if the 100th post was 403/4’d but the other 99 weren’t, it’s not a fair representation. However, after the numbers reached 1000s/ day it should be almost identical to the real numbers. The last 3 months are more exact numbers, being every post instead of 100th.

I also gathered the thread views but that’s wildly misleading, so I left that out.
 
Nice compilation, if the mods and the owner had no objection.

Well, can you tell me from that data, how many were actually buying, and what services were they buying? Does your program cover this aspect -- of getting down to where it actually becomes Golden?

If they came only to buy, did they bought the things after deliberations (visiting competitors' pages, then which competitor of which services they visited once/many times or they liked what they saw and bought immediately)...which are the top 20 services, and the top 50 buyers, and are those services periodic selling items or are evergreen?

Who sold how many items and for what prices, and how frequently he sold the service/item? His stats...I want to try to duplicate his service, may be taking his own, or some similar creators' help.

Who bought how many items and for what prices, and how frequently he bought the service/item? His stats...I want to befriend those members... :)

There can be many such aspects apart from being a commercial one. To me, such data can mean pure GOLD!

Otherwise, the entire compilation was great, but powerless in a way imho.

Congrats again. Make the data work now and make bank. You should share with the Owner because it is HIS data, So think of that angle too if the data has some or much commercial value.

I'm just spreading my thoughts out. :)
 
Well, can you tell me from that data, how many were actually buying, and what services were they buying?
Kind of hard to tell with just publicly available data.

Replies to threads =/= sales.
 
We’ve been working hard recently on quality on the forum engaging more mods and Clamping down on thread bumpers from within the marketplace and removing spam

Yes, this obviously doesn’t account for PMs, removed posts or hidden threads, but let’s not pretend that it’s wildly inaccurate.

At the risk of being accused of jumping down your throat, @elavumnretea, as you have some kind of issue with me for some reason, I
l'll address this and should you ever be unbanned AGAIN you can respond if you want. I'd say your data is probably pretty inaccurate.

There is a discussion in a area of the forum where you do not have access that is germane to this specific issue.

In fact I am pretty sure it came up, in part, because of the use of some of the following reasons for deleting posts:

nqF2CJb.png

nubJHU0.png

Gwl2GtD.png

X5ysGdc.png

otLxEoZ.png


Those are just some of the reasons I delete posts and there are a bunch of other active staff member that delete many others for the same and similar reasons.

There are numerous other reasons posts get deleted by me and others for other reasons and that does not take into account posts that get deleted because of warnings.

Tools are good... most of us that are successful use them but they do not take into account everything that actually goes on and can't possibly do that either.
 
At the risk of being accused of jumping down your throat, @elavumnretea, as you have some kind of issue with me for some reason, I
l'll address this and should you ever be unbanned AGAIN you can respond if you want. I'd say your data is probably pretty inaccurate.

There is a discussion in a area of the forum where you do not have access that is germane to this specific issue.

In fact I am pretty sure it came up, in part, because of the use of some of the following reasons for deleting posts:

nqF2CJb.png

nubJHU0.png

Gwl2GtD.png

X5ysGdc.png

otLxEoZ.png


Those are just some of the reasons I delete posts and there are a bunch of other active staff member that delete many others for the same and similar reasons.

There are numerous other reasons posts get deleted by me and others for other reasons and that does not take into account posts that get deleted because of warnings.

Tools are good... most of us that are successful use them but they do not take into account everything that actually goes on and can't possibly do that either.
I’m genuinely not sure what your point is. Not being a dick, just genuinely don’t know.

Maybe I wasn’t clear.

All posts, threads, and user profiles have a permanent URL, no matter if they are live, deleted, or hidden.

For example, https://www.blackhatworld.com/members/BHW.100373/ should still show your profile, as the number is still the same.

This principle is how I generated the graph, as shown by the script.

It visits every 100th post (/posts/100 ... /posts/200 .... /posts/1000000 ... /posts/1000100) and scrapes the post time/date.

If a post is deleted, it returns an error and is not counted. However, that will not change the next result, as /posts/x can be deleted but /posts/x+100 is still there.

My earlier comment about inaccuracy was aimed at the initial stage. I’m talking about when the forum got <100 posts/month - due to the fact that if, say, /posts/200 is hidden, it would show no post increase for that month, and then the next month it would be higher.

That was only an issue for 2006. Every point after that, there are 1000s per day so missing 100 max in a month is nothing, and it’s fixed by the next month.

If you’re insinuating that the post count is going down because you are removing more spam (again, not trying to be a dick but that’s all I could think your comment was talking about) then, besides the fact that it still counts deleted posts somewhere (as explained) I don’t think the fact that spam is increasing (but being removed) disproves the trend.

If you want to reply, that’s fine, but please close the thread after doing so (or if you don’t reply). It wasn’t my intention to argue with anyone or justify my right to gather this information. I don’t plan to post again.
 
Back
Top