Cryptocurrency analysis and predictions using AI and big data

Discussion in 'CryptoCurrency' started by healzer, Jan 3, 2018.

  1. ttmschine

    ttmschine Power Member

    Joined:
    Mar 27, 2013
    Messages:
    631
    Likes Received:
    359
    So to put it in a nutshell if I may.

    Your "predictions" always show the price going up.

    In reality the price is always going down.

    But your "predictions" still show the price going up.

    But the price is still going down.

    Perhaps (if your "predictions" model isn't totally useless then) there's something wrong with your calculations - like there's a "+" where a "-" should be?

    I only ask because if you're creating some software to predict market movements in any field, and it's 100% wrong all of the time - well, it suggests that either your model is flawed, your software is flawed, your brain is flawed, or maybe your software is right and reality is flawed.

    Perhaps watch the 2 series on netflix of Dirk Gently and try to understand the fundamental inter-connectedness of all things - or give up.

    The latter may be the easiest option versus the former?
     
    Last edited: Feb 5, 2018
  2. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,951
    Likes Received:
    2,936
    Gender:
    Male
    Location:
    Somewhere in Europe
    Unfortunately it's not that simple, if you've been following my progress you should know.
    The predictions are generated by a trained recurrent neural network, so it's not just sums "+" and/or subs "-".
    If you don't like this project you are not obliged to follow my progress and/or even reply with negativity.

    PS: not all predictions are going "always up".

    Cheers!
    Ilya
     
    • Thanks Thanks x 3
  3. mnunes532

    mnunes532 Elite Member

    Joined:
    Jan 21, 2014
    Messages:
    1,758
    Likes Received:
    599
    Gender:
    Male
    Location:
    Portugal
    Yup, its "idiotic" right right now but because it is always learning, it will get smarter day by day ;)

    @healzer good work on the 3d charts, maybe those are more useful than the 2d.
     
    • Thanks Thanks x 2
  4. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,951
    Likes Received:
    2,936
    Gender:
    Male
    Location:
    Somewhere in Europe
    That's exactly what is happening right now in my opinion.
    Several sources are artificially bringing down the price of crypto so they can buy low -- then they do the reverse to sell high again. But this cycle will take several days/weeks of tremendous effort.

    Hi all :)

    I hope everyone is doing well!
    After having received feedback from several of our readers/followers, I decided to rebrand our project to "CryptoPredicted".
    The old URL is being 301-redirected to the new site (here).

    Yesterday I have spent several hours tweaking the predictions system and in the coming few days I'll be releasing a new version of it.
    The current version will remain unaltered for about a week after the new release, but more info will be made available later.

    Regarding the big crash of BTC's price, I have been following its progress on the 3D plot.
    In particular expecting a turnaround point.

    This screenshot was made yesterday night:
    [​IMG]
    The red circle indicates a potential turnaround point (where the price would go up again).

    However the turnaround point did not happen, it was just a temporary increase in price.
    Here's the second screenshot taken approx. 13 hours later:
    [​IMG]
    We see there is yet another new and potential turnaround point where the price shortly went up.

    An hour later I took this one (with 5min intervals):
    [​IMG]
    The current situation has surpassed the two previous potential turnaround points. Maybe it's happening right now.

    The predictions also indicate a slight increase in price. But towards the end they indicate a decrease (again).
    [​IMG]

    Here's another version with a smaller parameter "sequence size":
    [​IMG]

    Let's see how it goes :)
    Stay tuned for the next update soon.

    Cheers!
    - Ilya
     
    • Thanks Thanks x 1
    Last edited by a moderator: Feb 7, 2018
  5. manolis111

    manolis111 Newbie

    Joined:
    Nov 9, 2017
    Messages:
    29
    Likes Received:
    5
    Gender:
    Male
    Following your project!!
     
    • Thanks Thanks x 1
  6. djelica

    djelica Regular Member

    Joined:
    Jun 26, 2009
    Messages:
    208
    Likes Received:
    88
    Awesome project. Also I'm very respectful for the reason that you are publishing your research and work here for us to see which takes good chunk of time. I enjoyed reading it in one go from the beginning.

    While I'm here, ill try to quickly write my opinion on this topic.

    First, you wont be able to trade based on signals you are getting from that data analysis. That is mostly because the cryptosphere is very turbulent and very influenced by some factors (govs, news, pumps and dumps and so on) especially right now, which are unpredictable. As some member mentioned when big bad Volkswagen news happened insiders already knew and already sold shares which resulted in drop before public had chance to know, same happened in cryptosphere now.

    Second, bitcoin is everywhere, take other cryptocurrency that has less news, articles, tweets etc.

    And third, in which I strongly believe is that you should focus on predicting new trends for some new altcoin(s). This industry depends on influence and almost every article that mentions altcoin, mentions bitcoin too (exclude it) but the coin that gets the most attention is much more likely to get nice price bump over time (NEO insta-pops on my mind, look at that uptrend, and many people talked about it in the past).

    I will be following your progress with great interest, keep it going.
     
    • Thanks Thanks x 1
  7. Sp3ctrum

    Sp3ctrum BANNED BANNED

    Joined:
    Mar 30, 2016
    Messages:
    140
    Likes Received:
    26
    Gender:
    Male
    a great read, I am following you for future update
     
    • Thanks Thanks x 1
  8. seohug

    seohug Jr. VIP Jr. VIP

    Joined:
    Feb 14, 2011
    Messages:
    374
    Likes Received:
    71
    Gender:
    Male
    Occupation:
    Automator
    Location:
    UK
    Home Page:
    Nice work.
    What's the predicted price Zeus, Jack, Davinci...etc are about? are you making multiple predications on separate factors?
     
  9. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,951
    Likes Received:
    2,936
    Gender:
    Male
    Location:
    Somewhere in Europe
    Yes that's correct :)
    The feature types represent different combinations of data to make predictions by.
    I have written it in detail on my blog: https://medium.com/@cryptopredicted/improving-our-autopilot-crypto-predictions-system-41360a82484b
    Here's a snippet:

    Hope this helps :)
     
  10. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,951
    Likes Received:
    2,936
    Gender:
    Male
    Location:
    Somewhere in Europe
    Bitcoin's price is recovering.
    We have reached a turn around point.
    It may take a day or two before the price has reached a new maximum value (before yet another turnaround occurs).

    Have a look at this beautiful pattern (time x price x volume):
    [​IMG]

    Static screenshot:
    [​IMG]

    Good night all ! :)
     
    • Thanks Thanks x 2
  11. megabytes

    megabytes BANNED BANNED

    Joined:
    Feb 6, 2018
    Messages:
    27
    Likes Received:
    0
    sad that bitcoin is dropping again!
     
  12. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,951
    Likes Received:
    2,936
    Gender:
    Male
    Location:
    Somewhere in Europe
    Hey all :) ,

    It looks like the turnaround point from my previous reply was spot-on.
    People who bought Bitcoin at its lowest point (± $6100 per BTC) have experienced a growth of ±35%.
    The current value is about ±$8200 per BTC.

    Here's a view from the back on Bitcoin's recovery (including the turnaround point)
    [​IMG]

    Most of today I have been working (and continue to work) on backtesting based on the predictions and basic signals.
    Stay tuned for more soon :)
    Cheers!
    Ilya
     
    • Thanks Thanks x 4
  13. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,951
    Likes Received:
    2,936
    Gender:
    Male
    Location:
    Somewhere in Europe
    Quick update regarding backtesting

    For the past few hours I've been immersed in optimizing, refactoring and cleaning up the predictions algo code. :)
    Simultaneously been working on basic backtesting algos/strategies.

    First I re-built the basic signals which are shown on the predictions tool into my custom backtesting script. I then generated predictions (length: 12), for 24 hourly intervals. The backtesting tool makes all of the trades as shown by the signals. Because every signal is in the future, we first fill up a queue of "future trades", and then step through time to obtain the actual price's value (simulating a real trade scenario).

    Example
    [​IMG]
    In our example "Maggy" is going to be our Bitcoin investor (trader).
    The green circle indicates our current interval (where Maggy is currently at).
    The red line indicates the predictions (generated by our neural network).
    The buy/sell arrows are the respective minimum/maximum values on the red line.
    The black line is the actual price, but the trader only knows the prices prior to the start of the red line (he/she cannot look into the future, duh!).
    Note: the red line does not fit the black line, however the most important part is that they look alike as much as possible!

    Every interval (i.e. green circle), Maggy will promise herself to buy/sell Bitcoins where and when the signal says "buy"/"sell".
    Maggy borrows $10,000 from her husband "Leonard" to make the trades.
    In this case, she always puts the whole amount into every "buy", and when she sells, she sells everything.
    Basically we make Maggy stay awake for 36 hours straight -- however she will not necessarily make trades every hour.
    * We also ensure that Maggy's last trade is always a "sell" (never a "buy") -- this allows us to calculate the ROI at the end of her trading journey by cashing-out.

    Maggy starts trading on "Feb 3, 2018 00:00 GMT", until "Feb 4, 2018 12:00 GMT" (exactly 24+12=36 hourly intervals)
    These are the predictions/signals she blindly obeys:
    [​IMG]
    IMHO, this animated gif beautifully illustrates how the red line follows a similar path as the actual price (= black line) -- they have slightly different Y-values, but that's not a problem at all.

    Below is a partial screenshot of the backtesting's output:
    [​IMG]
    (It's in JSON format).
    FYI, the fields in this JSON output are not important, but if you care to know then I'll explain it next:
    • Basically, we see (almost) every single interval. The first 24 intervals are included by default (00:00 until 23:00). Afterwards you may notice that some intervals are missing because no buys/sells were triggered for those prediction intervals.
    • When we expand some intervals, we will see a "_buy" object, "_sell" object or nothing. These are the signals assigned to the specified intervals, as you previously saw on the animated GIF. They indicate what the value of the prediction is ($ USD) and the buyprice/sellprice which is the actual/real price ($ USD) at that interval.
    • All other intervals also show "ap" = actual price ; "crypto" = how many Bitcoins Maggy holds ; "cash" = how much cash ($ USD) Maggy has got. So you can also see when she buys Bitcoins, her cash goes to zero, and vice verse for selling crypto.
    • At the end I added the key entry "_" which I explain up next.
    =========== Summary ==
    Maggy started with $10k and after 36 hours (34*) she ended up having $10,567.10 .
    This is an ROI of about 5.67% (i.e. $567 gross profit -- we have not taken any transaction fees into account).

    I noticed that the ROI was pretty low (between -1.5% and 1.5%) when I used the average of six different prediction graphs to generate signals by (as shown in my previous posts).
    But when I used only one feature type (e.g. "hopkins" in this example) then the ROI was almost twice as high and positive on every test-run.

    === additional info.
    Maggy could be too afraid of investing the whole $10k in a single "buy".
    So we can let her make buys with only 25% of the cash she's got.
    In this case the ROI is a bit lower but nonetheless positive: 4.73% ($473.- gross profit)
    [​IMG]
    Note: in this scenario Maggy never invests more than 25% of her $ cash into Bitcoins -- but when she "sells", she sells everything (100%).

    === next steps.
    1. A couple of weeks ago I mentioned that using more parameters in the predictions system yields more poor results than when using only 2 or 3 parameters. Some of you may remember me saying this. Unfortunately this remains true in our case. Hopkins is only based on two features (price and trading volume).
    2. My next priority is to understand "why" it is like that. The second thing would be to tweak the parameters to increase the ROI.
    3. Thirdly, this example uses very basic (almost stupid-like) signals which are nothing more than the min/max of a generated predictions graphs. I am quite surprised that we have a nice profit margin/ROI by obeying such a lame system. But I'm sure Maggy (nor her husband) can complain at all, but we definitely need to decrease our risk by optimizing the trading strategy and improving the signals system/algo.
    Time to hit the bed, it's currently 4am, so I hope I didn't write too many typos :D
    Good night everyone!
    - Ilya
     
    • Thanks Thanks x 4
    Last edited: Feb 8, 2018
  14. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,951
    Likes Received:
    2,936
    Gender:
    Male
    Location:
    Somewhere in Europe
    Hi guys :)

    Allow me to share a small insight regarding Bitcoin's price.
    Most of us remember Bitcoin's lowest point as of a day ago (indicated by the red rectangle):
    [​IMG]

    The blue rectangle is our current situation.
    And it looks like it's going to approach a turnaround point soon, i.e. the price will start going down again soon (hopefully not, but it does seem like it).

    Here are two screenshots of different predictions which indicate the same outcome:
    [​IMG]

    [​IMG]

    These drops/rises are natural occurrences in the crypto market. Some are more dramatic than others of course.
    But the most important part is that right now is a good time to sell crypto while it's high, and buy more once it reaches the next low point

    Hope this helps someone :)
    Cheers!
    Ilya
     
    • Thanks Thanks x 2
  15. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,951
    Likes Received:
    2,936
    Gender:
    Male
    Location:
    Somewhere in Europe
    Hey guys,

    Have spent all day tweaking the backtesting code.
    I realized that the backtests "only" yield positive margins/ROI in predictable regions.
    To give you a practical example -- from February 4 until February 6 there was a long decrease of BTC's price -- this appears to be non-trivial region that our predictions are unable to forecast, thus backtesting over this region yields negative margins (losses). I am still figuring out how to generate better prediction results for February 4, so this part remains work in progress.

    Yesterday I've posted a screenshot of several predictions, here's a followup of those predictions where you can see how it performed:
    [​IMG]
    There was indeed a small decrease, but much lower than indicated by the predictions.
    The buy/sell signals were pretty "okay" compared to the actual price as well.


    Have a nice weekend everyone! :D
    - Ilya
     
    • Thanks Thanks x 3
    Last edited by a moderator: Feb 10, 2018
  16. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,951
    Likes Received:
    2,936
    Gender:
    Male
    Location:
    Somewhere in Europe
    Today was yet another productive day :)

    Most of my time I've invested into optimizing predictions and running backtest simulations.
    This gave me a few new insights -- allow me to show you some interesting charts and patterns.
    I've also made several changes to the platform that I'll be discussing below.

    In the past two days my backtesting results weren't superb and in most of the cases had a negative ROI.
    This was primarily because I ran them on a very specific region where the BTC's price started crashing (between Feb 2 and Feb 4).
    In my previous post I noted that more stable (thus more predictable) regions already yield positive ROI on average.

    In the next example I extended the backtesting range to 10 days: from Feb 1 until Feb 10.
    Below is a list of the ROI (%) outcomes per feature and per date range:
    In the above results, you see that each iteration is one day longer than the previous one.
    The reasoning for this is that between Feb 1 and Feb 6 there was a big decrease in BTC's price. From Feb 6 it started to increase steadily again.
    I believe this decrease was influenced by factors that are not well measurable, I believe this huge drop was artificial and not a natural occurrence -- because of this the predictions are pretty terrible. So if we extend the backtesting beyond this uncertain period, we see that the ROI starts going up, because our trader (Maggy) is able to get better returns. You can see this in the last three date ranges where the ROI is positive and higher than in the first half.

    We also see that in almost every single case it was the "hopkins" feature type that gave the best positive ROI.
    I believe this is partly because there is much more price and volume data available -- these parameters did not suffer from my system crashes in the past few weeks.
    I wanted to investigate my theory more in-depth without involving many statistical formulas -- but just purely by logical thinking and visual analysis...

    Some time ago one of our readers suggested me to look into DEMA (Double Exponential Moving Average) index. So I thought of using a more basic index such as SMA (simple moving average) and use that formula on my training data. Thus instead of training the neural network using absolute values I would first transform it into moving average values.
    Before I did that I updated the "general chart". Basically I removed the "trendlines" concept and replaced it with SMA. I quickly realized that my trendline concept is actually a "broke" version of SMA itself -- thus SMA is better and more precise.
    * I haven't yet updated the FAQs on my website, but will do that soon.

    On this small GIF you can see what happens to the SMA(Price) graph when I change the "SMA size" from value 1 to 20:
    [​IMG]
    It's quite similar to the old trendline, except that it does not abruptly breaks -- instead the lines are smooth over the entire range.
    Notice that the first #'size' data points are missing, this is not a bug but just how SMA calculation works.

    Before going further into making predictions let me show a few of cool graphs.
    * (Pay attention to the selected SMA size)
    ** if you see a gap in the graphs below, it indicates there was an outage.

    [​IMG]
    On the above we see how news mentions and social mentions look quite alike, especially most of their peaks align nicely. On first sight they don't appear to define any movements in the price per se. But we do see that there is more hype in times of panic (when the price drops low).

    [​IMG]
    The above shows the SMA of sentiment analysis from news channels. Most values on the above graph are negative, meaning an overall negative sentiment. However the absolute values matter less than whether the value increases/decreases --> we see that there is a more positivity when the price goes up and negativity when the price is low. These sentiments do appear to have an impact on the price (and/or vice versa). It also validates that my basic sentiment analysis algorithm works pretty well to some degree :) .

    [​IMG]
    The above shows SMA of sentiment analysis from social channels (Facebook, Twitter and Reddit). It looks quite similar to the SMA of the price, and also to the previous SMA of news mentions but with less dramatic changes/peaks. So this appears to be yet another good indicator for making predictions.

    [​IMG]
    The above shows both SMA's for sentiments from news and social channels -- it's just a combination of what I discussed in the previous two cases. We clearly see how they look alike. Simply beautiful.

    [​IMG]
    Let us now take the SMA of news sentiments and SMA of news mentions -- It appears that the mentions are the inverse of the sentiments, which kinda makes sense. When the price drops the media is more active about the topic at hand.

    [​IMG]
    The above is the same scenario but now for social channels -- yet again a similar inverse pattern.

    [​IMG]
    In the above you see the "delta" traded volume (24h). If you've been trading for a while then it's no surprise when I explain that the volume goes up when the price drops --> people who are afraid start selling, but the smart ones start buying, and later sell for profit once it reaches a new high.

    [​IMG]
    You may have seen that there are two types of volume(24h) labels on the general chart. The first one is the "delta" which is derived from the regular "volume24h". The "delta volume24h" is calculated by subtracting the volume24h at interval (t-1) from (t). What this chart reveals is that the "delta" appears to be very in-sync with the price, while the regular one lags behind the price. I think this is quite unfortunate because most trading platforms (and their APIs) only show the regular volume24h data, so people end up with trading volume that's not really useful for "in the moment decisions".

    [​IMG]
    The above shows how some peaks of the SMA delta traded volume24h look similar to those of the news' & social mentions'.

    [​IMG]
    The above is a similar chart, but in this case I show the volume with the SMA sentiments of news and social channels.

    ====================================
    ====== making predictions (continued) ======

    These new insights, thanks to SMA, allowed me to make more accurate predictions.
    After many hours of tweaking parameters, testing and fixing bugs I had some nice results.

    The annoying part is that each time I want to predict a certain range (e.g. past 24 hours) it takes several minutes (up to 10 minutes) before the calculations are done.
    And at times there is a bug which you don't detect until a couple of hours later -- so a lot of time is occasionally wasted without realizing it.

    In the results below I have decreased the number of generated predictions to 4 intervals (previously I used 12 intervals, and on the live version it shows 20 intervals).
    I have decreased this range in the hope of obtaining better results from my super simple backtesting strategy.
    Initially decreasing the number of predictions did nothing, it still yielded negative ROIs. But this started to change when I began using SMA values instead of absolute values.

    Below are my new outputs that I've generated in the past 3-4 hours.
    I also noticed that sometimes the same data, parameters and same date range yield different ROIs (negative and positive) -- the first reason is self explanatory, each time the neural network is re-trained in a different manner so the results differ slightly. Secondly, the backtesting strategy is pretty dumb, the system can buy/sell even if it would mean a loss, while in reality a reasonable person would hodl until he/she can sell for a profit. That's why I run the same calculations over and over again to calculate an average value.

    Below you'll see that the feature types are now called "meany..." -- because they use transformed data using the SMA algorithm.
    As I did more tests I also added newer "meany" types:
    The average ROI of all these outputs is: 2.5465 %
    While the avg ROI of our initial output (with traditional feature types) was: -8.2110 %

    I am very impressed with these results!!
    Because with the traditional feature types we only had negative ROI values for the same date range.
    We also notice that meany3 and meany8 don't yield many positive ROIs, so these two are the worst feature types.

    Since it's already very late here, I won't have the time to test longer date ranges with the "meany" feature types.
    So in the coming few days I'll be running more tests on different date ranges with variable lengths.
    I am very excited because we have a lot of positive ROI outcomes from a "dumb" backtesting strategy; so I can't wait to improve the algorithm and allow it to make decisions whether to sell/buy as a real human would.

    Have a good night everyone :)
    - Ilya N.
     
    • Thanks Thanks x 4
  17. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,951
    Likes Received:
    2,936
    Gender:
    Male
    Location:
    Somewhere in Europe
    A couple of hours ago there was a possible turnaround point, where the price would go up again (and reach a maximum in the next 12-36 hours).
    You can see it happen on the 3D plot:
    [​IMG]
    The black rectangle indicates the turnaround point I'm referring to.
    But there is some risk/speculation in this prediction, because at midnight (between Feb 10 and 11, GMT) there was a quite similar turnaround point indicated by the red rectangle.
    It was a very short turnaround point that quickly changed into a decrease.

    I had pulled two different predictions from our system to see what the neural network can tell us:
    [​IMG]
    [​IMG]
    The first predictions set indicates a decrease, so this "black rectangle" is yet another very short turnaround point.
    From the second screenshot it looks like this is also a short turnaround point but that it would eventually evolve into an increase.

    Feb 11, 9pm GMT would've been the ideal time to buy BTC at a price of ± $7.9k -- by selling it at the current temporary turnaround point for about ±$8.3k would yield ±5% ROI.
    But if the second prediction is right, the price may rise even higher. If it goes beyond $8.7k in the coming hours this would yield 10% ROI (that's already double).

    Let's wait and see what will happen :)
    Cheers!
     
    • Thanks Thanks x 5
  18. waterworks

    waterworks Senior Member

    Joined:
    Sep 28, 2013
    Messages:
    1,019
    Likes Received:
    359
    On the bottom two plots, is the black line with black dots what has occurred already are those graphs of the same timeline as the 3d plot is showing?
     
    • Thanks Thanks x 1
  19. healzer

    healzer Jr. VIP Jr. VIP

    Joined:
    Jun 26, 2011
    Messages:
    2,951
    Likes Received:
    2,936
    Gender:
    Male
    Location:
    Somewhere in Europe
    The black line on the two bottom plots is historical data (the $price of BTC).
    The 3D plot's time axis is shown in decreasing order, while the bottom two plots are in increasing order.
    So if I horizontally flip one of the bottom ones, you can see the relationship between the "black rectangle area" and the black line.

    [​IMG]

    Sorry for the confusing explanation,
    let me know if it's clear now and/or if you have any other questions :)
    Cheers!
     
    • Thanks Thanks x 2
  20. LukaB

    LukaB Jr. Executive VIP Jr. VIP

    Joined:
    Apr 3, 2010
    Messages:
    4,526
    Likes Received:
    5,037
    Gender:
    Male
    Location:
    Planet Earth
    @healzer this is amazing - kudos to you.

    Haven't read everything but will this upcoming week - love the consistent updates.

    Luka
     
    • Thanks Thanks x 2