Cryptocurrency analysis and predictions using AI and big data

Discussion in 'CryptoCurrency' started by healzer, Jan 3, 2018.

  1. Dave Dubauskas

    Dave Dubauskas Newbie

    Joined:
    Jan 28, 2018
    Messages:
    3
    Likes Received:
    3
    Ahh... very cool.
     
    • Thanks Thanks x 1
  2. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,940
    Likes Received:
    2,919
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Hi guys :)

    Here's a useful update for the predictions chart.
    Until now I had to spend a lot of time using photoshop overlaying actual data with the predictions.
    So to be a lot more productive I have extended the predictions tool with "extend actual data".
    This allows us to keep as many known prices on the plot (max limit is the number of predictions, 20 in our case):

    [​IMG]

    You can simply enable that checkbox and hit the "prev" button to look at the predicted accuracy.
    The screenshot above shows how it works: we have a datetime set to "... 19:00" so at that interval our server made 20-interval predictions starting at "... 19:30" (indicated by the red circle).

    You will notice that some predictions are a far cry from the actual data (a.k.a. reality), while others are more or less "okay".

    PS: @ses this may make your life a bit easier as well.

    PSS: when you enable this checkbox it will not use caching, so it may take up to a second (depending on your connection speed) to return the data (instead of 100-200ms).

    Have a good night all! :)
     
    • Thanks Thanks x 3
  3. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,940
    Likes Received:
    2,919
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    I hope you are all doing very well today :)
    Yesterday night and today's entire day I've been working on an improved version of the prediction chart.
    The previous version generates predictions based on one set of pre-defined parameters.
    But that's not a good system at all. We should instead generate (A) many predictions using different parameters and settings, (B) compare them and (C) use common sense to make trading decisions.

    (A): The previous version of the prediction tool lacked "multiple predictions". So I spent many hours tweaking and expanding it.
    (B): Right now comparing the "multiple predictions" is a manual process, but in the near-future I want to use AI/data mining to detect patterns in the multiple predictions.
    (C): Idem ditto as (B).

    The redesigned prediction tool's page looks like the screenshot below.

    [​IMG]

    Here is a detailed description of every new option/setting:
    • features type:
      The old tool's version used one combination consisting of 5 features to make predictions (price, volume, social hype, social & news sentiments).
      In the new version you can choose from 6 different feature combinations:
      hopkins: price and volume24h
      leonard: price, social and news hype
      maggy: price, social and news sentiments
      davinci: price, volume24h, social and news hype, social and news sentiments
      jack: price, volume24h, social and news sentiments
      zeus: price, volume24h, social and news hype

      * you will notice that some feature types yield more accurate results than other depending on the interval size. Sometimes just the price and volume appear to be quite accurate, but in other cases it needs hype and/or sentiment data.

    • Batch size:
      Choose from [100, 200, 500] -- the batch size indicates how many data points were used in a single training session. It said that a higher batch size results in less accurate predictions, so it is recommended to use 100 or 200. A few minutes ago I've added 500 in there to analyze its predictions.

    • neurons:
      The number of neurons used in the LSTM network. At this stage I will not make any claims on which value you should use.

    • sequence size:
      Making a prediction is all about taking a sequence of historical values [x, x+1, x+2, ..., x+n-1] and teaching the neural network that the outcome is [x+n] (where n is the sequence size). In some cases we use small sequences to relate it to the next outcome and in other cases we use larger sequences. I believe a sequence size of 20 is the best value right now (but I may be wrong, need to investigate it further).

    • epochs:
      This is the last learning parameter -- it is the number of forward+backward passes throughout the learning process. I am testing 500 vs 1000 passes at this stage.
    You also see that the plot has three graphs:
    1. The dark-yellow is the actual (average) price of the selected crypto at every interval.
    2. The green line is something I have explained a few weeks ago, but let me rephrase it. We use the actual/historical data to train neural network, and then use the same training data and ask the network to make predictions. And if the network is more or less well-trained, it should return a graph that resembles the actual/training data. You may toggle this graph off, but for now I have included it to make sure there is (A) no over-fitting and (B) no under-fitting.
    3. The red line is the predicted future (20 intervals).
    If you are going to use this 2nd version of the prediction tool, keep it mind that it has not made many predictions at this moment.
    However there are 3 intervals it predicts for: 1hr, 12hrs and 24hrs.
    So every one of these intervals it makes a prediction for the associated interval.
    • 1hr: the prediction starts at the start of every hour, and usually takes 15-30 minutes to complete.
    • 12hrs: this starts at 00:00 and 12:00 (GMT) and takes 15-30 minutes.
    • 24hrs: this starts at 23:00 (GMT) and takes 15-30 minutes to complete.
    Take care everybody!
    One more screenshot :)
    [​IMG]

    -- edit --
    I forgot to mention that in v2.0 only Bitcoin (BTC) is enabled.
    The reason is that the system makes 216 predictions per interval.
    So for the hourly interval it comes down to 216 x 24 = 5184 predictions per day.
    Then the 12hrs + 24hrs = 648 predictions per day.
    In total this system generates 5832 predictions in a single day.
    Every predictions takes several seconds to complete, so it's quite expensive.
    The plan is to find a small set of useful parameters which we can generalize for other cryptocurrencies.
     
    • Thanks Thanks x 3
    Last edited: Jan 29, 2018
  4. Dave Dubauskas

    Dave Dubauskas Newbie

    Joined:
    Jan 28, 2018
    Messages:
    3
    Likes Received:
    3
    Well I’m an idiot...ha ha. After my post I dwelled into your product and of course discovered how much it utilizes social input. If I could suggest a feature...an automatic ranking of “most mentioned” coins would be useful.
     
    • Thanks Thanks x 1
  5. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,940
    Likes Received:
    2,919
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Hey :)
    Don't sweat it mate.

    Regarding the automatic ranking, I think our 3D chart should give you the information you need.
    It shows the number of social mentions, news mentions, price (as well as traded volume = size of the ball) on a single graph:
    And clearly BTC is the winner followed by ETH and LTC.
    [​IMG]

    =====================
    ===== Jan 30, 2018 =====
    =====================

    Most of today I have spent tweaking the predictions, analyzing results and more...
    I am making some good progress and figured out some interesting parameters that yield good predictions.

    But also developed some extras. I have added a candlestick chart which shows the open/close and low/high per interval.
    I also want to add additional graphs (e.g. the Price trend, and traded volume), but there is a bug in Plotly JS which I've reported: it doesn't hide graphs by clicking on the label -- it should be taken care of in the coming few days. :)

    Screenshot of the Candlestick chart:
    [​IMG]

    Lastly but definitely an important update is that the intervals on the x-axis no longer indicate the result of the previous time.
    Here's what I mean by it: Previously you would see something like "19:00" and that would mean all results from 18:00 until 18:59 included.
    But now "19:00" means from 19:00 until 19:59 included. This is the correct way and much more intuitive, not sure why I didn't do it from the start.

    Cheers all!
    - Ilya
     
    • Thanks Thanks x 1
  6. Sheepp

    Sheepp Jr. VIP Jr. VIP

    Joined:
    Jan 15, 2018
    Messages:
    246
    Likes Received:
    182
    Gender:
    Male
    Occupation:
    Finding time for business
    Home Page:
    Awesome work ! Keep it up !
     
    • Thanks Thanks x 1
  7. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,940
    Likes Received:
    2,919
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Today has been less productive than usual.
    Was out visiting a friend who graduated and some other stuff that I had to take care of.

    A few hours ago I noticed that Apache Spark has been down for the past 24 hours (crashed after 5 days of up-time).
    I haven't found out the reason why, but I think it's because the history logs were not cleaned up.
    I did some tweaking and hopefully this won't happen again.

    I've also received a nice PM from @waterworks with a cool suggestion for the prediction system -- will see what I can do with it.
    Stay tuned for the next update, take care all ! :)
     
  8. elavmunretea

    elavmunretea Jr. VIP Jr. VIP

    Joined:
    May 14, 2016
    Messages:
    2,220
    Likes Received:
    2,987
    Occupation:
    bUsY nOt TiPpInG pEoPlE
    Home Page:
     
  9. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,940
    Likes Received:
    2,919
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Did this guy just steal my code? :D
     
    • Thanks Thanks x 1
  10. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,940
    Likes Received:
    2,919
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Hey all :D

    Here's a quick update.
    I have been playing a lot with the prediction system.
    A lot of reading about different strategies to make predictions.
    I have gained some new insights but the results are not better than what we already have so far.
    Also fixed several bugs (mostly UI wise).

    There is one interesting feature I added to the predictions tool v2:
    [​IMG]

    [​IMG]

    I was tired of switching between feature types, so I added the option "all" which will display all features on a single graph.
    This allows us to quickly see what the "general prediction trend" looks like, because some types will show a decrease while others an increase.
    You can see what I mean on the graph, there are two predictions that deviate away from the price (black line), while the other four ones are pretty much aligned with the price.

    note: when the "extend price" checkbox is enabled or there is no cache available yet, then it may take ±30 seconds before you will see the graphs.

    In the coming days I will shift my focus more on analytics than predictions.
    I plan to add more sources (blogs, forums, ...) -- and a bunch of other cool features.

    Stay tuned! :)
     
  11. Snoop Dawg

    Snoop Dawg BANNED BANNED

    Joined:
    Jan 31, 2018
    Messages:
    57
    Likes Received:
    36
    This is one of the most interesting threads I've ever read.
     
    • Thanks Thanks x 1
  12. BhutJolokia

    BhutJolokia Junior Member

    Joined:
    Feb 23, 2015
    Messages:
    116
    Likes Received:
    33
    How accurate is your prediction software so far? Really interesting thread!!! Keep up the good work! :)
     
  13. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,940
    Likes Received:
    2,919
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Hey :) I have not fully tested the accuracy of the predictions.
    This is something I plan to do in the coming days. But you can look at its accuracy manually by inspecting how well the predictions were at some historic interval:

    On the screenshot below I chose an interval from 5 hours ago.
    The black line is the actual price (which is extended until 'right now').
    The predictions are the red-ish lines, and as you can see there are 3 pretty accurate predictions (very similar slope).
    The other 3 predictions also went up but with a lower slope.
    [​IMG]

    [​IMG]

    ================
    === Feb 2, 2018 ===
    ================

    After having tweaked the predictions system (see my previous post),
    I started working on other financial graphs such as the Simple Moving Average (SMA):

    [​IMG]

    The SMA graphs (one with size 20, and the other of size 10) are both available for the "general chart" on our website.
    In the near future I will allow users to change the size of SMA's and add/remove additional ones.
    I will also add other moving averages such as Exponential Moving Average (EMA) and Double EMA as suggested by one of our readers.

    -- Edit --
    Using the SMA's graphs only makes sense if you have several days of data (with hourly intervals), otherwise they are too short to make much sense.
    You may also notice that on the screenshot I have set the trendline shape to 6, this only affects the price in my example, the SMA's are not categorized as trendlines per se.

    Stay tuned all :)
    - Ilya
     
    • Thanks Thanks x 2
    Last edited: Feb 3, 2018
  14. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,940
    Likes Received:
    2,919
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    I would like to thank @Christopher87 for mentioning the "Elliott Wave theory" in one of his replies.
    I did some quick research into this subject and it struck me with another thing I noticed on the 3D chart.

    I am not an expert on the Elliot Wave theory but it comes down to the fact that a stable/natural market has wave cycles.
    If anyone is curious, here is a good read: http://stockcharts.com/school/doku.php?id=chart_school:market_analysis:elliott_wave_theory

    On one of our 3D charts there is a specific pattern that I noticed early on, and I haven't written about this yet.
    More concrete it's the "time price volume" 3D plot.
    If you look closely (with some bit of imagination) you will notice that the data points look like a spiral traveling through time.
    It is a very irregular spiral whose parameters are not the same (radius, thickness, number of turns, ...).

    This is what a regular (mathematically correct) spiral looks like in 3D:
    [​IMG]

    On the gif below you can see a spiral-like trend formed by the data points.
    [​IMG]

    Below is a static screenshot of the same graph, but captured from the other side.
    [​IMG]

    You can use my settings of the chart to run the same simulation in your browser here: http://cryptanal.com/timepricevolume3d.php
    My observation is that from Jan. 20, until Jan. 27 there is a nice spiral. But then the spiral becomes very irregular (it flattens in the Y-dimension) because of the crazy changes in trading volume.

    If we link these observations to what Elliott's wave theory states, we can see a wave-like structure in the 3D structure (2D as well).
    I have this crazy idea right now that goes like this:
    1. Making predictions from 2D data is hard because there is exists no formula that defines that market (unless the market is rigged).
      In other words, a financial market cannot be defined by a formula like " y = 5x + 13 " , otherwise we'd all be rich.

    2. However, if my observations make any sense, then there do exist formulas for drawing (ir)regular spirals.
      In other words, we could use our 3D data to find parameters for a best-fit spiral and extrapolate it for a few time intervals (e.g. 5 hours into the future). Once we have done that we can look back at the prediction and compare it against the actual data.
    [​IMG]

    This got me so exited, now it'll be a big struggle falling asleep :D
    Good night all!
    - Ilya
     
    • Thanks Thanks x 3
  15. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,940
    Likes Received:
    2,919
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Thanks for the chat! I appreciate your time :)


    =================
    === Feb 04, 2018 ===
    =================

    Today I have started working on the backtesting part.
    I've noticed a few things that would cause me trouble, such as the delay in generating predictions.
    Sometimes it would take up to 30 seconds before I got the prediction results, so I refactored my code and incorporated a caching system for predictions.
    I have also removed the "extend actual price" slider -- this means if you look at historical intervals, the actual price will automatically be extended.
    You will also see that there is a red "Predicted price avg." line, which is the average of all predictions. And "all" is now the default feature type.

    Once this was done, I had to think of how to do backtesting.
    The thing is that almost all available libraries do not work by using future predictions, they only use historical data to backtest a certain trading strategy.
    On the other hand, at each interval, our tool provides us with a different prediction(s) because it learns and adapts itself.
    So I have not yet implemented a fully backtesting function yet, instead I started with adding a basic signals:

    [​IMG]
    As displayed on the screenshot above, it now indicates when to buy and sell based on the average of all predictions.
    These signals are super basic, they are just the minimum and maximum within the predicted range. The idea is to guarantee a profit margin for the trader.

    Then if we are at some historical interval (as in our example), we have the extended actual price (black line), and we can calculate ourselves how much we would've made/lost by following the signals. In this case we would have made a profit because we would buy low and sell high several hours later. However, if you are a trader using this basic system, you would be following the market hourly and sell at 14:00, 15:00 or 21:00 instead of at 00:00 as indicated on the chart (to have a higher margin than predicted).

    Here's another chart at a more recent historical interval:
    [​IMG]
    As you can see above, there are three signals: two buys and one sell.
    It shows three signals when we can have a sell in the middle and the first predicted price ( = buy-price) is lower than the future sell-price.
    However, be wary of the 3rd "buy" signal: the predictions do not guarantee there will be an opportunity to sell it later on.

    Here's a screenshot of the chart generated right now:
    [​IMG]
    We can see all six prediction graphs predict that the price is going to go up, as a result we see a buy indicator right now.

    Stay tuned for more soon.
    Have a good day all :)
    - Ilya
     
    • Thanks Thanks x 1
    Last edited by a moderator: Feb 5, 2018
  16. ttmschine

    ttmschine Power Member

    Joined:
    Mar 27, 2013
    Messages:
    629
    Likes Received:
    355
    And yet the price has continued to go down.

    Looking at your graph there where you predicted it would goup and it was a buy signal - lookslike it was about 8500 - it's now sub 8000 - if you continued to follow those predictions (which no doubt some people reading this thread will have) then before long you'd be broke.

    Why don't you get it to try predict tomorrows lottery numbers based on previous draws - it will be equally as accurate.

    i.e. not at all.
     
  17. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,940
    Likes Received:
    2,919
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    https://www.blackhatworld.com/seo/c...g-ai-and-big-data.998766/page-4#post-10729165

    Cheers,
     
  18. PinguSpy

    PinguSpy Jr. VIP Jr. VIP

    Joined:
    Dec 7, 2007
    Messages:
    1,869
    Likes Received:
    1,639
    Occupation:
    unemployee
    Location:
    2001:4860:4860::8888
    How’s your A.I doings OP?

    Based on current market dip. It’s accurate?
     
  19. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,940
    Likes Received:
    2,919
    Gender:
    Male
    Location:
    Somewhere in Europe
    Home Page:
    Hey :)

    These recent price drops are pretty crazy, I can imagine a lot of people are in panic.
    Maybe this post will bring some hope in the building.

    The predictions are not keeping up with the dramatic drops. It seems they are better at predicting prices in more regular situations.
    However here are some interesting findings.

    The screenshot below shows the predictions taken as of nine hours ago:
    [​IMG]
    You can clearly see that the price went down but the predictions vary a lot, some go down/up in waves while some show a steady increase. Not a single one indicates a drop in price as in reality occurred.

    As I went further back in time (20hrs back) I noticed a nice anomaly:
    [​IMG]
    As you see there is one prediction that that did go down for about 15 intervals before going up again. The next step would be to understand why this specific feature-combination predicted a drop while all other ones predicted an increase.

    A few minutes ago I took screenshots of several new forecasts:
    [​IMG]
    [​IMG]
    After this long decrease in BTC's price, the neural network starts to predict more "decreases" than "increases". Both of the above screenshots have two of these "anomaly" predictions that indicate a decrease -- and if a decrease does occur in the coming 20 hours, then that will be a nice find.

    Then I had a look at the 3D plot of Time Price and Volume to see what pattern was emerging:
    [​IMG]
    I noticed that every so often the price has a turnaround point (as indicated in red). That's the point when the price starts going up again.
    So right now (between 5 and 6 Feb) we are approaching a turnaround point and in my opinion it's a good moment to buy, so you can sell once it reaches some higher price 12-24 hours down the line.

    ====

    Apart from the above I have spent quite some time filtering out bugs and optimizing various components of the platform.
    And also added a new 3D chart that shows: Time, Volume and Hype. The idea behind this one is that Volume is semi-related to the price, and we already have a similar graph where we show the price instead of the volume -- I wanted to see what this one would look like.
    [​IMG]

    Stay tuned for more :)
    Cheers!
    Ilya
     
    • Thanks Thanks x 4
  20. TheGreyMatter

    TheGreyMatter Jr. VIP Jr. VIP

    Joined:
    Mar 23, 2011
    Messages:
    2,080
    Likes Received:
    2,299
    Home Page:
    Nice to see the latest additions made to the website.
    Like you mentioned in your last post, algorithms are never be guaranteed but can always reduce the guess work and gambling part, excited to see how the results turn out to be in the next few days.
     
    • Thanks Thanks x 1