Cryptocurrency analysis and predictions using AI and big data

Discussion in 'CryptoCurrency' started by healzer, Jan 3, 2018.

  1. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Nope, at this stage I only analyze the influx of mentions per minute.
     
  2. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    ========
    ======== 19 Jan. 2018
    ========

    Today I have been fine-tuning my system and getting bug-free.
    The next task is to add more data sources, more features per data and more cryptocurrencies.

    The recent crypto crash got a lot of people worried, so I have made a few predictions as requested by some of you.
    These predictions were generated on data at 12-hour intervals since 29 Dec. 2017, including what was available today.
    Each prediction was generated using different parameters, so I could analyze its impact on the output:

    * Red line: prediction of hype
    * Dark black line: prediction of BTC's value (in USD)

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    As you can see, it predicts that BTC's USD-value will go up in the coming 1-2 weeks.
    However, let me warn you that these are just predictions generated by a machine.

    Stay tuned for the next update.
    Have a great day all :)

    -- Edit --

    Here's a graph generated on 3-hour intervals:
    [​IMG]
    * On the graph above we have 115 x 3hours of known-data = 345 hours; this equals 14.375 days. But it should be more than that. The reason it's only 14 days is that some days our system was down and no data was collected, these points are removed from the set.
     
    • Thanks Thanks x 5
    Last edited: Jan 19, 2018
  3. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Update (continued)

    In preparation for a public version I have implemented a caching system.
    Caching allows me to store the results of a database query, so that on the next same request it won't have to re-calculate everything.
    This decreases response time and increases browsing speed.

    As I was thinking about the sentiment analysis part, I noticed that my list of words was not tuned for social media mentions.
    So it did not include emojis❤ and slang words -- so I found a new list and extended my own. This should give better and more sentiment results.

    Have a good night all :)
     
    • Thanks Thanks x 5
  4. elavmunretea

    elavmunretea BANNED BANNED

    Joined:
    May 14, 2016
    Messages:
    2,220
    Likes Received:
    2,970
    I was also reading about this too.

    Things like "?!?!" can turn a positive sentence into a negative one.

    Sarcasm is going to be much harder to detect, though, and I haven't come up with a solution, as of right now.
     
    • Thanks Thanks x 3
  5. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    That is true. Sentiment analysis is a complex thing, however, a few sarcastic mentions won't make a huge difference when analyzing over 20k mentions per hour.
    The outcome of sentiment will lean towards negative or positive by a bunch of other strings such as, but not limited to: *sadface* :(, sh!t, f*ck, ... and *happyface* :), awesome!, great!, ...
    We don't need a 100% accurate outcome of every single mention but a general trend.

    I also have another strategy for sentiment analysis (based on machine learning) instead of dictionary matching -- but that's something I'll test in the near future.

    Cheers!
     
    • Thanks Thanks x 2
  6. Christopher87

    Christopher87 Junior Member

    Joined:
    Aug 27, 2015
    Messages:
    171
    Likes Received:
    33
    I'm an hooked! Just found this this morning and just finished reading it all now. Amazing stuff!

    Being able to predict small market trends is a definite win and for those playing around smaller alt coins would be able to make relatively good trades.. I think you said 3 hours accuracy which is amazing.


    Anyways - please let me know if you do release this even as a beta, as I would love to test holding multiple Alt coins myself and alot of the hype/decisions comes from a personally view, multiple Reddit posts/threads and some basic research into the whitepaper as well as any underlying business partners that face the coin credibility.

    I wish you all the success to follow and will be monitoring this closely for your next update. :)
     
    • Thanks Thanks x 2
  7. cnick79

    cnick79 Jr. VIP Jr. VIP

    Joined:
    Jun 10, 2010
    Messages:
    815
    Likes Received:
    417
    Location:
    Wandering
    Awesome stuff. Interesting to see that hype goes down over time but the predicted btc is going up. Is the decrease in hype meaning more negativity?

    Hopefully your bot isn’t programmed to feel pain. I’d hate to see it kill itself should the price of btc get depressingly low.
     
    • Thanks Thanks x 1
  8. mesbahuddin

    mesbahuddin Regular Member

    Joined:
    Nov 12, 2013
    Messages:
    244
    Likes Received:
    16
    Gender:
    Male
    Location:
    UK
    More and more people are getting familiar with Bitcoin and crypto world day by day. I don't think it's a bubble which will be banished any moment. If the traffic is increasing then I don't believe it's heading to be "fades away". Yes, it's changing the technology already. ICO is introduced to the world. Our mobile phones gonna be different pretty soon. I think it's gonna make a huge difference in Crypto currency.
     
    • Thanks Thanks x 1
  9. liamlio

    liamlio Newbie

    Joined:
    Aug 28, 2017
    Messages:
    37
    Likes Received:
    20
    Gender:
    Male
    This is amazing! I'm really interested in Machine learning, data visualization and data analysis, but I'm a complete newbie other than knowing Python and R :p.

    Any recommendation to get started/gain experience in this field? Books/courses:eek:
     
    • Thanks Thanks x 1
  10. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Hi guys, thanks for the great comments! :D

    At this stage it's hard to say "why" it's like that.
    However, by looking at historic data I do notice that hype starts to go down when the price is stable (or increasing), then people start to calm down.
    But when there's a huge drop in price, the media goes crazy.

    I have never programmed Python before this project. So in just a couple of days it became my language of preference, I couldn't believe I was messing around with php/java/C# before -- Python allows me to be 2x more productive. I would encourage you to "just do it". I learn from experience not books. If you don't have any specific project where you can apply Python/R to, then just make something up. Someone previously complained that he/she doesn't have a good backlink checker, well that's already a demand you can fill. By helping that guy you will gain experiences (and maybe some money).

    Have a great weekend all!
     
    • Thanks Thanks x 1
  11. elavmunretea

    elavmunretea BANNED BANNED

    Joined:
    May 14, 2016
    Messages:
    2,220
    Likes Received:
    2,970
    Very interesting Reddit account I just stumbled on:

    https://www.reddit.com/user/cryptochecker

    https://www.reddit.com/user/cryptochecker/comments/7q3jum/faq_about_ucryptochecker/
     
    • Thanks Thanks x 1
  12. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    BTC is on the rise, looks like the predictions were spot-on this time.
    Let's hope it doesn't crash again soon.

    [​IMG]
     
    • Thanks Thanks x 3
  13. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    ======
    ====== 20 Jan. 2018
    ======

    Today I've researched new charting libraries for javascript and found some much better ones than ChartJS.
    ChartJS is free and easy to create simple/basic charts as I have shown until now.
    But for financial and more complex data (especially some 3D stuff) I will changing to PlotlyJS soon :)

    A few moments ago I've added 8 new currencies to track (as seen on screenshot below).
    I am testing the stability and load on various APIs and scrapers. So if all goes well, I'll be adding another batch in a day or two.

    [​IMG]

    --edit--
    Forgot to mention that I also added a server status viewer.
    It shows a summary of the system, so I know when something fails.
    [​IMG]

    Have a nice weekend all :)
     
    • Thanks Thanks x 5
  14. jamie3000

    jamie3000 Elite Member Premium Member

    Joined:
    Jun 30, 2014
    Messages:
    2,064
    Likes Received:
    953
    Occupation:
    Owner of BigGuestPosting.com
    Location:
    uk
    Home Page:
    sentiment analysis is notoriously hard, hedge funds get the full twitter stream and still struggle with it. Interested how your implementation progresses man.
     
    • Thanks Thanks x 2
  15. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    ======
    ====== 21 Jan. 2018
    ======

    Today I've spent most of my time changing the charting library (from ChartJS to PlotlyJS).
    I must admit that Plotly is much richer, but also a bit harder to customize and lacks a few elements.

    The screenshot below is the new chart. It looks almost identical to the previous ones.
    However, it's not very convenient to set interval ticks on X-axis for the date/time (e.g. hourly, 1minute, 5 minutes, ...). In ChartJS this was automatically generated, here I have to mess around with code to get it right. I do like the fact that PlotlyJS comes with a toolbox which allows me to zoom in, select, drag, scroll, ... This was not possible in ChartJS.
    [​IMG]

    Once I was done with the above, I immediately tackled a new challenge.
    For quite some time I wanted to cluster data visually and this is one of the reasons I changed to PlotlyJS -- it has 3D charts.

    The idea is to use clustering to categorize coins by their "stage", as suggested by @Ensili (on page 2). I haven't gotten to this part yet, so that's still work in progress.

    My first attempt was to visualize 3 variables: time, hype and price. But this didn't appear to be very meaningful (a 2D chart was better for this).
    My second attempt was: hype, price, trading volume (past 24hrs) -- yet again not much to conclude for this plot.

    But the following did have something to say:
    [​IMG]
    I have plotted (for the past 24hours, with 2-min intervals): time, BTC/USD price and trade volume (24hrs).
    Here are a few things we can conclude:
    • The cluster where most trades were done ==> the price was lowest (so at midnight UTC Jan. 22): buy low.
    • There is a specific cluster between volumes 115k and 120k, where the price was highest: sell high.
    • All volumes in between the highest price and lowest price are people who were too late to the selling/buying party -- or don't know what the heck is going on.
    Have a good night all :D !

    --edit--
    here's a slightly different angle of the above 3D plot:
    [​IMG]
     
    • Thanks Thanks x 3
  16. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Quick update regarding the Time-Price-Volume plot:

    I've been playing a bit more with the 3D plot today (this time with 8 hours more data than yesterday).
    First I have visualized the Volume on a regular 2D chart:
    [​IMG]
    The yellow area is the price (at hourly intervals).
    The dark red line is the traded volume (for the past 24 hours) at each hourly interval.
    However, we are more interested in the purple line, which is the difference (delta) of the currently traded volume and the traded volume of the previous interval.
    This allows us to see if at some time 't' there were more or less trades compared to the previous interval.
    Notice that it can be negative if there were less trades, and positive if there were more ( zero if the volumes were equal ).

    From this chart we can conclude that most trades were done when the price dropped. Each volume-peak corresponds to a price-valley.
    But as soon as the price starts going up, the trading volume drops (people don't buy/sell).

    Here's another way to look at this situation.
    In the past ±30 hours the price has dropped from about $12.5k to just about $10.5k :
    [​IMG]

    If we rotate the plot a bit, we see that the volume went up as people started to sell/buy.
    They sold either out of fear or bought out of opportunity.
    [​IMG]


    Here's a rotating gif animation I've made:
    [​IMG]

    Cheers! :)
     
    • Thanks Thanks x 3
  17. IamNRE

    IamNRE Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 18, 2010
    Messages:
    5,027
    Likes Received:
    7,413
    Occupation:
    Helping Small Businesses Get More Calls
    Home Page:
    Honesty.... I think the 3rd thing is gimmicky. Doesn't add much value. I say cut that fat and focus on the 2d version and making that awesome.
     
    • Thanks Thanks x 1
  18. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    I had that thought at first as well.
    But as I kept looking more and more into the 3D shapes, it started making more and more sense.
    Thanks to the 3D volume-price-time plot I could see the pattern of supply & demand much better than on a 2D plot.
    So I'm looking for patterns in 3D shapes that could help me visualize things in 2D better and/or maybe even help me make better prediction algorithms.

    For instance, I would personally use the 3D to determine when to buy or sell.
    When I look at the plot for the past ±36 hours, I see when most trades were done.
    And so if we could have such a plot across many days/weeks -- we could identify whether we are on the far left or the far right of the plot (relative to the volume). So for instance, right now is a good time to buy because a lot of people are selling.
    We see a small cluster evolving "between 12:00 and 18:00 GMT" where people are buying/selling like crazy.
    [​IMG]

    But that's more of a philosophical thing right now :D
    You are right, the 2D data is much more important at this stage.

    Thanks!
     
    • Thanks Thanks x 1
    Last edited: Jan 22, 2018
  19. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,949
    Likes Received:
    2,932
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Quick update (can't edit my previous post):

    A couple of days ago I've added a whole range of new words and emoji's to the sentiment analysis algorithm.
    Now I have plotted the price, volume and sentiments at 10min intervals for the past 24 hours:

    [​IMG]
    It's quite stunning to me how much the sentiments (green) and volume (purple) are related.
    I could go over the entire plot and discuss every hour, but I'll limit myself to the blue rectangle.

    At some point the price dropped drastically (from 11.8k to 11.4k) and simultaneously the sentiments dropped (more negativity) on social media.
    But then at its lowest the trade volume peaked very high and sentiment started going up (people started buying -- buy at a lower price).
    So right now the sentiment is steadily increasing, so I wonder whether that will make the price go up or down in the next few hours.

    PS: Once I have more volume-data I will add it as a new parameter to the machine learning algorithm (& make some predictions).
     
    • Thanks Thanks x 4
  20. Safff

    Safff Junior Member

    Joined:
    Apr 11, 2016
    Messages:
    153
    Likes Received:
    30
    Gender:
    Male
    Location:
    Poland
    well done!
     
    • Thanks Thanks x 2