Cryptocurrency analysis and predictions using AI and big data

Advertise on BHW
====================================
=== February 28, 2018 and March 1, 2018 ===

In my previous post (Feb. 27) I've mentioned that we could use the notifications from our Anomaly Detection (AD) system as trading indicators.
So just out of curiosity I did a small and limited experiment to verify whether or not AD would be useful.

V4Rw5cd


My current backtesting script was designed to backtest predictions data, which has a different format/layout.
So after some time refactoring my code to include the AD's data, I could finally start testing.

I have done two types of experiments.
Each experiment is done over a range of 7 days ( == 168 hours), starting on February 20, 00:00 (UTC), until February 26, 23:00 (UTC).
We step over this range at hourly intervals. But our AD system checks for an anomaly at 2-minute intervals, and if there is one then we use it as a buy/sell indicator.

The biggest challenge is programming the buy/sell indicator itself.
How do we tell the system when it's time to buy/sell?
This is an important question because selling when we receive a negative AD, means we are selling at a loss and thus won't make a profit.
While buying when we receive a positive AD, means we might be selling at a pretty high price -- however we expect the price to keep going up.
So using the AD signals themselves is not a good strategy to enter/exit trades!!

A better way, and this is what I've implemented... we know for a fact that anomalies are pretty rare, so they don't occur every minute, but usually just a few times within several hours.
So in between anomalies there are stable periods, and pseudo-anomalies -- regions where the price goes up/down but not at a drastic rate (so we don't report these).
What I noticed is that after some anomaly (or a sequence of anomalies) it usually ends up at a stable period, a region where the price has reached some new level/plateau and no drastic changes occur. So my implementation is to wait for an anomaly, and when one occurs, we let the system wait until a stable period has been reached. Once it has reached some stability we tell it to buy/sell.

Here's an example of this strategy:
  1. Anomaly Detected for a huge drop in price (this event is remembered by the system).
  2. The system waits until a stable (or pseudo-stable) region has been reached.
  3. The system buys BTC (for a lower price), and the event is cleared/removed.
  4. ... time passes by...
  5. We detect an anomaly for an increase in price (this event is again recorded by the system).
  6. (same as step 2)
  7. The system sells BTC (for a higher price), and finally the event is cleared/removed.
This system does not guarantee that it will sell for higher than it bought.
It does not check whether the buy-price is higher then the sell-price.
And it's VERY important that we do not let the system do this -- otherwise we are unable to evaluate the effectiveness of this system (i.e. measuring the effectiveness of AD signals on as trading indicators).

All of this being said... time for the results.

Experiment 1
This experiment bought/sold only during very stable periods.
It made 7 full trades (buy+sell).
ROI: % -4.85753

Experiment 2
This experiment bought/sold only during pseudo-stable periods.
It made 6 full trades (buy+sell).
ROI: % -3.3229

Some additional info about these results:
The system made only 6 to 7 trades, that's about one trade per day.
It made this few trades because we only check for an anomaly every hour (not every 2 minutes), so there's a low chance of hitting an anomaly when checking this infrequently.
e.g.: we compare the price of February 22 at 17:00 against the prices 30 minutes prior [16:30 --> 16:58], and if there is an unusual change in price relative to that range then it counts as an anomaly.

Both of the ROIs are negative, the reason speaks for itself -- it entered/exited trades at unfortunate intervals.
Let us adjust our strategy and reduce the size of our steps from 1hour to just 10minutes.
This way we increase our chances of detecting anomalies by six.

Experiment 1 (10min steps)
This experiment bought/sold only during very stable periods.
It made 48 trades (buy/sell) in total.
ROI: % 2.6755

Experiment 2 (10min steps)
This experiment bought/sold only during pseudo-stable periods.
It made 39 trades (buy/sell) in total.
ROI: % -4.0240

Now we have slightly better results, the first experiment yielded +2% ROI over 7 days, even though it's positive but still not good enough.
Let us adjust our strategy again and reduce the size of our steps from 10min to just 2minutes.
This way we further increase our chances of detecting anomalies by five.

Experiment 1 (2min steps)
This experiment bought/sold only during very stable periods.
It made 83 trades (buy/sell) in total.
ROI: % 17.0293

Experiment 2 (2min steps)
This experiment bought/sold only during pseudo-stable periods.
It made 85 trades (buy/sell) in total.
ROI: % 4.6111

I am very surprised to see an ROI of %17.- for such a simple/dumb system (that's about %2.4 ROI/day )
This means that the ROI of a better system could be 2, 3 or even 5 times higher.
Reason: this system is not trading to make a profit, it trades by obeying the AD notifications.
So we can tweak & improve a live version of this system, by making it sell only when it makes a profit (or if the price is about to crash).

From our 6 experiments, it looks like that buying/selling at very stable periods is more lucrative than during pseudo-stable periods.
But more importantly, we see that the smaller our steps for detecting anomalies, and using those anomalies as trading signals, the better our outcome is.
We'll need to repeat these experiments a few dozen times (over different dates of variable lengths) to verify this statement so it remains work in progress. :)

==============
=== The app ===

About an hour ago I've released version 1.0.0.8 of the app.
Changes:
  • Enabled support for Android +4.1 ( @gulftrotter ).
  • New sidebar and toggle button.
    The forecast item is the default homepage showing the predictions.
    The price chart is new (and discussed next).
    Nc2ReUd


  • New price chart.
    As mentioned in one of my previous update posts, there was a need for a new chart that solely display the price.
    And since our AD system now works using 1 minute intervals, this chart has the ability to display the price at 1min (by default), 10min and 1hr intervals.
    OtXxkRV


  • Anomaly Detection displays on the new price chart.
    Instead of displaying AD messages on the predictions chart (on which prediction notifications are already displayed), these are now only shown on the price chart.

  • Separate notifications for predictions and anomalies:
    XKhqXqy
Thanks for reading, I hope you enjoyed!
- Ilya
 
Okay, this...is an idea. But.
There are too many factors affecting the price of a currency, let alone a cryptocurrency. Probably hundreds if not thousands of different factors.
So, is it really feasible that social rumors, hype, hate, speculations, suggestions etc are determining factor for a price movement?
I guess for crypto it's more viable than for other financial markets, but still...

We must know/remember that all we hear/see via tv/internet is the last spread of any potential original info. We, you, me, all are the last piece in the puzzle that gets to get a hand on any viable info. If it's already out there, when/to whom/how many do you think the original info was released to first and how viable in regards to making profit it is in the moment we get to know about it?

There is a theory in statistics known as "the wisdom of the crowd", this attempt seems to try to capitalize on an aspect of that theory. But it has its problems as well as its pros, and in the end of the day the ones that get the info first are with the advantage, all the rest are at a disadvantage.

Who is more likely to gain of an info about a financial market - the journalists who attended the press conference where a crucial announcement was made, the announcer, the ones that feed the info to the announcer or the general public who watched the press conference on tv/read about it on twitter? It's a matter of seconds. Always. it's not a matter of hours/days. After that it's too late and the 1/99 ratio of winning vs losing players on forex/stocks does prove that for years and years.
 
Important update.
The current version of the app has a small bug that prevents notifications from being received properly.
I have released version 1.0.0.9 and you should be able to update the app within a few minutes through the play store.

Cheers! :)
Ilya
 
healzer said:
Important update.
The current version of the app has a small bug that prevents notifications from being received properly.
I have released version 1.0.0.9 and you should be able to update the app within a few minutes through the play store.

Cheers! :)
Ilya
Click to expand...

How often do notifications happen? Is it every 10 minutes?
 
redjebo said:
Okay, this...is an idea. But.
There are too many factors affecting the price of a currency, let alone a cryptocurrency. Probably hundreds if not thousands of different factors.
So, is it really feasible that social rumors, hype, hate, speculations, suggestions etc are determining factor for a price movement?
I guess for crypto it's more viable than for other financial markets, but still...

We must know/remember that all we hear/see via tv/internet is the last spread of any potential original info. We, you, me, all are the last piece in the puzzle that gets to get a hand on any viable info. If it's already out there, when/to whom/how many do you think the original info was released to first and how viable in regards to making profit it is in the moment we get to know about it?

There is a theory in statistics known as "the wisdom of the crowd", this attempt seems to try to capitalize on an aspect of that theory. But it has its problems as well as its pros, and in the end of the day the ones that get the info first are with the advantage, all the rest are at a disadvantage.

Who is more likely to gain of an info about a financial market - the journalists who attended the press conference where a crucial announcement was made, the announcer, the ones that feed the info to the announcer or the general public who watched the press conference on tv/read about it on twitter? It's a matter of seconds. Always. it's not a matter of hours/days. After that it's too late and the 1/99 ratio of winning vs losing players on forex/stocks does prove that for years and years.
Click to expand...
This is an recent example of how some simple (!) info can change the price of a currency
https://www.reddit.com/r/CryptoCurrency/comments/80y51m/walton_got_busted_fake_winners_on_twitter/
Just one (also deleted fast) tweet caused a big drop on WTC price.
Great job, healzer!
 
lewy2k said:
How often do notifications happen? Is it every 10 minutes?
Click to expand...
Hey :)
There are two types of push notifications:
  1. Prediction notifications -- when a new prediction has been generated it will notify you. There are two modes: every 10min and every 1hr.
  2. Anomaly Detection notifications -- when an anomaly in the price has been detected (when price is going up/down rapidly) -- these are pretty rare but do occur once in a while.
On the settings page you can disable/enable any of these.
If anything is unclear or you have more questions let me know.
Cheers!
 
This is just wow. Thank you for putting your all into this. We appreciate.
Let us know when you release apk version of your app.
Thanks
 
I wish to start with 100$ and see how it goes. What is your advice?
I am not familiar with the app though.
I have downloaded it.
 
Hey, really interesting thread. Love your thinking and the app looks amazing. Do the new updates on the app include the tools you used to achieve:

Experiment 1 (2min steps)
This experiment bought/sold only during very stable periods.
It made 83 trades (buy/sell) in total.
ROI: % 17.0293

-----

You said the AD system checks every 1 minute. Does that mean the push notifcations are every 1-2 mins? I want to try and replicate the experiment you did.

Keep up the great work bro!
 
fredfredricks said:
I wish to start with 100$ and see how it goes. What is your advice?
I am not familiar with the app though.
I have downloaded it.
Click to expand...
Using $100 is a really good way to start, it's not a big amount -- so you can trade without worrying too much.
You can easily make %1 ROI per day from manually trading. First sign up on an exchange (e.g. Binance/Bittrex/...).
And then you can trade like I do at the moment: use 5 minute candlesticks to wait until there is a low price (or a valley) and buy in at 25 %or 50% of your cash.
Then wait until there is a peak or until you have between 1% and 3% ROI if you would sell.

You can use the app in two ways right now:
  1. Use the prediction notifications to estimate whether the price will go up/down in the next few intervals.
  2. The the anomaly detection notifications to wait until it tells you when the price is sharply going up/down and then closely monitor the price minute by minute to enter/exit a trade.

facebookdude said:
Hey, really interesting thread. Love your thinking and the app looks amazing. Do the new updates on the app include the tools you used to achieve: Experiment 1 (2min steps) ...
You said the AD system checks every 1 minute. Does that mean the push notifcations are every 1-2 mins? I want to try and replicate the experiment you did.
Keep up the great work bro!
Click to expand...

Thanks! :D
Yes the app includes this update.
Once in a while you'll be receiving anomaly notifications telling you when the price is quickly going up/down.
These same notifications are the ones I used in my backtesting experiments.
The push notifications are only received when the app is running (background or foreground), so keep it alive but not necessarily active.

I think it may be interesting to show and explain a bit deeper how they work:
8w3IkI5

On the screenshot above, on the left you have my trading view (5min candlesticks).
On the right is the log/output of the Anomaly Detection system. It stores in a log file when an anomaly is detected.
The entries containing "all good" are the very stable regions. The " ... 1 std" entries are pseudo-stable regions.
And the entries containing "+2 std" are anomalies.

So I have drawn two rectangles to show the correlation between the chart and AD system.
The green rectangle shows the (pseudo-)stable region -- in this region no notifications are generated.
And the red rectangle are the anomalies we need to know, so as soon as it detects one it notifies your device. But to prevent many push notifications every single minute, your device only shows the first one in the sequence.

And now you can basically reproduce my experiment for yourself. As soon as you receive an anomaly notification, login to your exchange and follow the price until it reaches some valley where the price is pretty low compared to the past few hours. As on the screenshot you want to aim to buy at the minimum "$10775/btc". Because from then on forward, the price appears to be going up. And then you either keep following the price and sell when the price gives you 1-3% ROI, or wait until you receive a 2nd anomaly of the price going up rapidly -- if so then wait until it reaches a peak (some new maximum).

Remember that my backtesting strategy did not have the intention of making a profit, but you should. You can and should verify whether you are selling higher than you bought.

I hope this makes more sense :)

PS: if anyone is having trouble receiving notifications, go to settings and toggle off then back on the "live anomaly detection" switch, and the "refresh predictions ..." switch.

Cheers! :D
Ilya

Edit: I may have drawn the green rectangle incorrectly on the chart above.
In the the green area (on the chart) I may have included an anomaly at its peak.
This is a more accurate version:
gnlbMN8
 
Last edited:
Looks like you don't receive the notifications on Bluestacks, damn. I'll have to boot up an old phone of mine.
 
@healzer dude, do a jv with clif high. I wonder what kind of results you would get, man. (You @healzer on twitter?)

You should contact CryptoBlood on twitter and tell him to connect you with clif high.

Dude, you guys would do an insane team!
 
royserpa said:
@healzer dude, do a jv with clif high. I wonder what kind of results you would get, man. (You @healzer on twitter?)

You should contact CryptoBlood on twitter and tell him to connect you with clif high.

Dude, you guys would do an insane team!
Click to expand...
Thanks man, I've reached out to him :)

============

Today was already a good day, and hopefully it'll just get better.
The price of BTC went up to ±$11480k and this allowed some of us make nice ROIs.
I personally exited at 2.5% profit for two trades from a week ago, which I purchased at a too high price ( a mistake I could've prevented ).

I'm also happy with the anomaly detection system, as it notified me about the rapid price changes:
oXkeaGg


and now it's like this:
tatHsSc



But the predictions are also very useful in a different way:

1WineGi


The predictions are not always on-point, but what I noticed is that they act more like predictive warning indicators.
What I mean is that when the price has reached a local maximum ( e.g. when it's at a peak ) then the predictions indicate that the price is most likely to go down.
The reverse is also true: when the price is pretty low (e.g. in a valley) and/or still declining, then they indicate that the price is going to go up in the next interval.
However, in reality this is not immediately true, but eventually true. The neural network has learned that after a decline comes an increase and vice versa.
So it seems it's pretty useful when we use these predictions as warning signals. They can't tell us exactly "when" the price is going to go down, but that's it's coming soon (it may take 10 minutes, it may take 100 minutes). But the chance of the price going down when it says it is, then it's just a matter of time before it does.

Have a great weekend all :D
- Ilya
 
Do you have a bot to automatically buy/sell on exchange markets whenever this app prompts you about market movements?
 
fredfredricks said:
Do you have a bot to automatically buy/sell on exchange markets whenever this app prompts you about market movements?
Click to expand...
Not yet, but it's on our bucket list :)
It has also been suggested by many other members.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top