1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Cryptocurrency analysis and predictions using AI and big data

Discussion in 'CryptoCurrency' started by healzer, Jan 3, 2018.

  1. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,879
    Likes Received:
    2,845
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    The world of the blockchain technology is rapidly transforming our digital space, and it will not take long before it manifests into the physical realm at large scale.

    The future of cryptocurrencies:
    There is a huge debate among powerful people and everyone has their own opinion. Some say cryptocurrencies are just a bubble that will explode and fade away, while others see it as a replacement for our entire monetary system. The reality is that bitcoin will not disappear any time soon, definitely not overnight.

    [​IMG]

    My findings:
    In 2017 I have put a lot of thought in developing a system that could help me analyze the value of cryptocurrencies (Bitcoin; BTC) in particular. Unfortunately there are not many resources out there (at least not for free) which can make serious predictions. Most are just "dumb" statistical extrapolations used for marketing purposes (trying to sell you some guide, book or a service).

    About a week ago I finally set out on a journey to realize such a system. As I have been working day and night on it, I have made great progress and found a lot of interesting information. In this thread I will be posting my findings and opinions. You may use this information to make decisions but don't be reckless.

    The strategy

    I have been talking to quite some people who are actively trading cryptos. I realized their decisions are not primarily based on the price of a certain coin/token, but they follow the trends (social media, news, forums, ...).

    We all know that trading is all about supply & demand: crypto is in a very "baby" phase compared to Wall Street and everything related. It is hard to predict how Coca Cola's stock price will behave, but with crypto it is simpler. Let me briefly explain why. I think we can all agree that +99% of the people who buy/sell bitcoins have the following:
    • Access to the internet
    • Have at least one social media profile
    • Are daily active on social media in some way
    • Read forums, blogs and news sites
    • Rarely watch TV compared to 10 years ago.
    So their decisions must be based on "whatever" is happening on the internet. The reason they/you bought something is because you did your research and found out that coin "XYZ" is a worthwhile investment; so you read information somewhere about "XYZ", your action of "buying/selling XYZ" had an impact on its price (just a very small one).

    In this thread I will regularly write updates about my strategy and keep you updated on my findings. I also invite fellow developers and data experts to contribute :)

    Framework

    [​IMG]

    Wednesday 27, 2017
    I started developing a set of tools that would allow me to track, monitor and record everything about a specific coin, I chose Bitcoin (BTC) for simplicity since it's the biggest and causes most hype worldwide.
    1. The idea is to gather everything I can about Bitcoin, everything from social channels (Twitter, Reddit, Facebook, ...) and all articles found on various news sites (CNN, CNBC, BBC, ...) and blogs/forums.

    2. Once I had some data I would put it on a chart so I would had a graphical representation.

    Since I have been a developer/programmer for almost a decade I could pretty easily put all the pieces together. Thanks to Python and MySQL I put together a basic prototype. While my system was gathering data, I was working on making a basic interface (a simple graph using ChartJS open source software). A several hours later I already found something remarkable:

    [​IMG]

    Let me briefly explain this graph:
    • The left Y-axis is the price of BTC (obtained from https://www.bitstamp.net/)
    • The right Y-axis is the "hype". In this case it is the total number of social media mentions related to Bitcoin.
    • The X-axis is an hourly time line.

    • The data is aggregated by the hour, this means that data generated between [01:00, 02:00] is shown under 01:00. And the same is true for every X-value, including the last one 08:00 means data for the period of [08:00, 09:00]. It's important to keep this decision/notation in mind. I may change this in the future but for now it's good enough.
    * On the chart's legend you can see that the blue/yellow/red graphs are the prices of BTC (min, avg and max) during a certain period (e.g. between 03:00 and 04:00). [left Y-axis]
    * The dotted line is my "hype", it is the sum of all mentions during a certain period [right Y-axis].

    You may realize that this graph has two Y-axes and their data is completely unrelated to each other.
    However I found something interesting! Using Paint I drew a black box to indicate an interesting region. And I also drew two black lines to indicate the slope for both sets of data. It appeared to me that they are somewhat related after all. This means that "hype" (from social media/news) could be related to the price of BTC.

    Also note that the "hype" goes up before the price goes up. Maybe we can use this to predict the near-future? Let's find out soon.

    Thursday 28 - Friday 29, 2017
    To verify whether my findings are right or wrong, I had to accumulate more data.
    But it was not an easy task, as I started gathering more data I noticed it was getting very "intensive" for my server -- there was A LOT of data so I had to re-design my super basic scraper and use something special for big data. In addition to using plain Python and PHP, I had to incorporate Apache Spark and Apache Kafka for processing the data. Right now I am storing it in a MongoDB database.

    And since it's already 2am in Belgium ...
    [​IMG]

    I hope you enjoyed the introduction. :)
    Cheers all!

    - healzer
     
    • Thanks Thanks x 77
  2. Frenzied

    Frenzied Junior Member

    Joined:
    Feb 16, 2016
    Messages:
    121
    Likes Received:
    44
    Holy shit that was an awesome read. Congrats man, can't wait to read more.
     
    • Thanks Thanks x 2
  3. Wolston

    Wolston Regular Member

    Joined:
    Jan 17, 2014
    Messages:
    235
    Likes Received:
    89
    Subbed. This was a good read. Can't wait to see whats next. Btw, I've seen a correlation between Google trends and the prices too.
    Are you incorporating search trends too?
     
    • Thanks Thanks x 1
  4. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,879
    Likes Received:
    2,845
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    We have not (yet) incorporated Google trends.
    Right now we are gathering our own "trend" data, however, in the near future I may add data from Google trends so we can compare it against our own.
     
  5. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,792
    Likes Received:
    12,245
    Occupation:
    Potentate
    Location:
    Asuncion
    [​IMG]

    There is a reason why probability and statistics are their own fields in mathematics. That said, have fun! It's great to work with data, getting your hands dirty with playing around, testing, getting excited etc.
     
    • Thanks Thanks x 12
  6. cnick79

    cnick79 Jr. VIP Jr. VIP

    Joined:
    Jun 10, 2010
    Messages:
    802
    Likes Received:
    409
    Location:
    Wandering
    How do you know if the “hype” is positive or negative? I think at some point you might want to analyze the hype and determine if it’s positive or negative because I would imagine negative hype would cause people to sell.
     
    • Thanks Thanks x 3
  7. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,879
    Likes Received:
    2,845
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    That's exactly what I am doing right now, and it's called "sentiment analysis".
    As posted above, "hype" is just the number of mentions gathered from social platforms. But it gets really interesting when we can make an algorithm to determine if hype is positive or negative, then we can make better predictions.

    But that's a topic for another day :) Stay tuned!
     
  8. Moto801

    Moto801 Senior Member

    Joined:
    Apr 25, 2009
    Messages:
    861
    Likes Received:
    428
    Location:
    Far away
    This website https://solume.io does kinda what you are talking about.

    It tracks the social media volume of every coin so at any given time you can check which coins are getting the most social media talk currently. It can even determine if there is positive chat or not.

    [​IMG]
     
    • Thanks Thanks x 15
  9. Heiko

    Heiko Senior Member

    Joined:
    Jul 26, 2016
    Messages:
    937
    Likes Received:
    739
    Gender:
    Male
    Location:
    Fuckin' Germany
    @healzer I have not found one any post from you that is useless! This is exactly what i was looking for since yesterday!
    Will be following this for sure!
     
    • Thanks Thanks x 1
  10. sandyfor80

    sandyfor80 Junior Member

    Joined:
    Aug 31, 2013
    Messages:
    113
    Likes Received:
    11
    Gender:
    Male
    @healzer You are the real Reverse Engineer of BHW.
     
    • Thanks Thanks x 1
  11. Steve Waller

    Steve Waller Junior Member

    Joined:
    Sep 1, 2014
    Messages:
    107
    Likes Received:
    28
    One of the best Threads to watch out for in 2018! This is going to be Huge!
     
    • Thanks Thanks x 1
  12. DeltaX

    DeltaX Registered Member

    Joined:
    Sep 15, 2017
    Messages:
    52
    Likes Received:
    16
    Gender:
    Male
    This is a different approach for mining crypto, smart ppl's way.. keep updated :mad:
     
    • Thanks Thanks x 1
    Last edited by a moderator: Jan 28, 2018
  13. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,879
    Likes Received:
    2,845
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Hey guys :)

    Thank you for all the great comments and support, let's make this big!!!

    Quick update, a couple of hours ago my laptop's HDD failed.
    It's 7am right now and ordered a new one, if all goes well I should have it by this evening.
    Until then I'm working on a memory-based Linux OS, so I'll try to be as productive as I can. #whenlifesucks
    [​IMG]
     
    • Thanks Thanks x 1
  14. wokaka

    wokaka Senior Member

    Joined:
    Apr 1, 2010
    Messages:
    904
    Likes Received:
    234
    you will never get anywhere if you try to do it alone. singularityNet is trying to make open source AI marketplace where you might find something there after they launch. check it out later. i happen to have some investment there as well
     
  15. Luk3

    Luk3 Newbie

    Joined:
    Oct 27, 2016
    Messages:
    41
    Likes Received:
    10
    Gender:
    Male
    Wish you all the best!
    Great opportunity to make some money.
    Following
     
    • Thanks Thanks x 1
  16. tankybank

    tankybank Junior Member

    Joined:
    May 12, 2015
    Messages:
    149
    Likes Received:
    12
    Great work OP! For someone thats knows nothing about coding, you made this easy to read. Looking forward to more updates on this, good luck!
     
    • Thanks Thanks x 1
  17. laur.laurix

    laur.laurix Senior Member

    Joined:
    May 8, 2013
    Messages:
    804
    Likes Received:
    322
    Occupation:
    Reverse Engineering Maniac
    Location:
    Mars
    7 a.m and u repairing ur laptop while browsing BHW? What drugs are u using OP? Because I want some.


     
    • Thanks Thanks x 3
  18. ContentExpert

    ContentExpert Power Member

    Joined:
    Jan 16, 2017
    Messages:
    792
    Likes Received:
    499
    Occupation:
    Your Content Writer
    Location:
    My Desk
    Home Page:
    You deserved more than just a "like" for crafting together such a brilliant perspective on Crypto value predictions. So, wanted to chime in and let you know that you've done a great job on this!

    Keep up the great work! :)
     
    • Thanks Thanks x 2
  19. TheDivine85

    TheDivine85 Newbie

    Joined:
    Jul 10, 2016
    Messages:
    26
    Likes Received:
    3
    Gender:
    Male
    Great Read, but eventually some of the biggest cryptos are getting that much hype, the small ones don't get that much hype, and there is certain altcoins that rise 100% in a day or even more, those are more like penny stockings and include pump and dump, you find whales putting lot's of money in certain altcoin and cause price changes in their favor. Also, you can use basics of technical analysis, due to the fact that most of the people join just on hype, and the guys that know how to trade use this against them, you need to follow the volume of trades, acceleration and deaceleration, and the orderbook for the coins of your choosing.

    Regards
     
    • Thanks Thanks x 2
  20. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,879
    Likes Received:
    2,845
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    It's a secret recipe mate, very experimental product. Maybe one day I'll share it with you ;)
     
    • Thanks Thanks x 1