Cryptocurrency analysis and predictions using AI and big data

Discussion in 'CryptoCurrency' started by healzer, Jan 3, 2018.

  1. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    ========
    ======== Jan 16, 2018
    ========

    I just realized I was writing "2017" in my posts instead of 2018 *facepalm*.
    Today and Yesterday I kept analyzing Spark, trying to find the bug.
    I got in touch with an expert who pointed out a few things that I'm testing out now.

    I have also spent time "normalizing" different social sources and comparing facebook vs twitter vs reddit.
    Until now all social hype was just a sum of all mentions, but facebook =/= twitter =/= reddit.

    Cheers! :)
     
    • Thanks Thanks x 2
  2. allphase

    allphase Junior Member

    Joined:
    Feb 6, 2010
    Messages:
    142
    Likes Received:
    30
    Occupation:
    self unemployed
    Keep up the good work healzer, you'll get it done. Interesting thread, I'm in when you need beta testers. Go crypto!
     
    • Thanks Thanks x 1
  3. cnick79

    cnick79 Jr. VIP Jr. VIP

    Joined:
    Jun 10, 2010
    Messages:
    815
    Likes Received:
    417
    Location:
    Wandering
    With all of the coins dropping it would be interesting to see what kind of data you produced.
     
    • Thanks Thanks x 1
  4. allphase

    allphase Junior Member

    Joined:
    Feb 6, 2010
    Messages:
    142
    Likes Received:
    30
    Occupation:
    self unemployed
    Riding the bitcoin wave down... Be prepared to buy ;)
     
  5. seoz87

    seoz87 Power Member

    Joined:
    Oct 31, 2008
    Messages:
    561
    Likes Received:
    307
    Gender:
    Male
    dude you should have created a backlink checker tool for us instead of this :(
     
  6. IamNRE

    IamNRE Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 18, 2010
    Messages:
    5,027
    Likes Received:
    7,413
    Occupation:
    Helping Small Businesses Get More Calls
    Home Page:
    So... did you predict the crash.... whats going to happen next? :) #SoapOpera
     
  7. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    To be honest, I have been too busy trying to predicting a crash in my technology.
    Trying to keep the system up and running 24/7, but it still keeps crashing after several hours, but every time I'm getting closer to the core.

    Regarding the crypto crash, here's a graph of BTC:
    [​IMG]

    And the same one for ETH:
    [​IMG]
    * 30 min intervals, and 8-point trendlines

    Both prices are going down (duh!)
    The hype of both cryptos has a similar trend.
    But what's interesting is that most of BTC's sentiment is negative, while ETH has only positive sentiments.
    Beware that my current sentiment algorithm is flawed, but this is still a huge difference.

    --Edit--
    It's crazy how similar the price trends are of ETH and BTC, isn't it?
     
    • Thanks Thanks x 3
  8. waterworks

    waterworks Senior Member

    Joined:
    Sep 28, 2013
    Messages:
    1,019
    Likes Received:
    359
    very interesting stuff. Thanks for sharing.

    Does the prediction model carry to the end of the month?
     
  9. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Right now the prediction model is very early stage, but works for predicting the next 1-5 time intervals (e.g. hours).
    The further we try to predict, the less and less accurate it gets.
    Once we have a better model and more stable system, we'll add long-term forecasting (e.g. one month ahead). :)
     
    • Thanks Thanks x 2
  10. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Super good news guys!
    After almost 2 weeks of debugging our system, I have finally been able to resolve the crashes :D !!
    If anyone wants to learn more about the problem, then you can read my post here: https://stackoverflow.com/questions...gs-forever-during-real-time-analysis/48309452

    I also noticed that BTC's price is going up. I think it will continue to rise as more people are making use of its drop in price.
    Since now I can finally have consistent 24/7 data, so I can finally shift my focus on developing the prediction framework and expanding to my coins.
    [​IMG]

    Cheers!
     
    • Thanks Thanks x 6
  11. waterworks

    waterworks Senior Member

    Joined:
    Sep 28, 2013
    Messages:
    1,019
    Likes Received:
    359
    Well done man! Exciting to see this come together. If you need beta testers, let me know. :)
     
    • Thanks Thanks x 1
  12. mindmaster

    mindmaster Jr. VIP Jr. VIP

    Joined:
    Sep 16, 2010
    Messages:
    3,540
    Likes Received:
    1,608
    Home Page:
    Good to hear that.

    Well done.
     
    • Thanks Thanks x 2
  13. elavmunretea

    elavmunretea BANNED BANNED

    Joined:
    May 14, 2016
    Messages:
    2,220
    Likes Received:
    2,970
    I was tinkering around with this idea while travelling into the City today. So far I have plotted the price of a bunch of coins & their release dates onto new exchanges. How are you measuring social media posts? All tweets with Bitcoin, BTC etc. in or is there an easier way to do it?
     
  14. elavmunretea

    elavmunretea BANNED BANNED

    Joined:
    May 14, 2016
    Messages:
    2,220
    Likes Received:
    2,970
    Here are some interesting posts I have been reading:

    https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4988639/ - Nice Supporting Information

    https://dashee87.github.io/deep%20learning/python/predicting-cryptocurrency-prices-with-deep-learning/

    https://blog.patricktriest.com/analyzing-cryptocurrencies-python/

    Starting to piece it all together now.
     
    • Thanks Thanks x 1
  15. BigManTing

    BigManTing Junior Member

    Joined:
    Feb 18, 2012
    Messages:
    168
    Likes Received:
    93
    Gender:
    Male
    Occupation:
    Amazon/Etsy
    Location:
    United Kingdom
    Beautiful work! this is what dreams are made off! *drools*

    [​IMG]
     
    • Thanks Thanks x 1
  16. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    I don't think there's a way around this.
    I did encounter a few service providers who "track keyword/brand mentions as a service", but in the end it all comes down to the same system.

    ====
    ==== 18 Jan. 2018
    ====
    Yet another all-nighter, yet another productive day.
    I have spent some time playing with the prediction algorithm, trying to understand it better.
    There are exactly 4 parameters that I can play with, so for each parameter in a range ( i --> j ) , I let it generate a prediction model. So the end-result is just a hell lot of images:
    [​IMG]
    However, I decided to do something more meaningful: scaling the system to more coins and more data.
    I have successfully installed a distributed system (currently 2 servers). There is one broker that receives scraped data and sends it to workers (currently only one worker available).
    The hardest part was making Spark work with Hadoop HDFS, as I couldn't find any well-documented tutorials. But it all worked out just fine ^phew.
    I've also redesigned the API (which took many hours) due to complex array transformations.

    The next step is to refine the system and prepare it for more crypto currencies.
    Once I have at least one week's data then we can do some prediction stunts.

    Have a great day !! :D
     
    • Thanks Thanks x 1
  17. mindmaster

    mindmaster Jr. VIP Jr. VIP

    Joined:
    Sep 16, 2010
    Messages:
    3,540
    Likes Received:
    1,608
    Home Page:
    Are you taking into account other data than mentions in the media, like number of offers (sell/buy) and their prices?
     
  18. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Not yet. Will add that soon :) There's so much I want to do but so little time.
     
    • Thanks Thanks x 1
  19. elavmunretea

    elavmunretea BANNED BANNED

    Joined:
    May 14, 2016
    Messages:
    2,220
    Likes Received:
    2,970
    Are you weighting the mentions?

    Like say an account with 100,000 followers is more influential than an account with 10.
     
  20. PaulLAllen

    PaulLAllen Newbie

    Joined:
    Aug 26, 2016
    Messages:
    9
    Likes Received:
    0
    Gender:
    Male
    I personally hate crypto. it's always too much risky.