healzer
Elite Member
- Jun 26, 2011
- 2,974
- 3,063
Nope, at this stage I only analyze the influx of mentions per minute.
Cryptocurrency analysis and predictions using AI and big data
- Thread starter healzer
- Start date
healzer
Elite Member
- Jun 26, 2011
- 2,974
- 3,063
========
======== 19 Jan. 2018
========
Today I have been fine-tuning my system and getting bug-free.
The next task is to add more data sources, more features per data and more cryptocurrencies.
The recent crypto crash got a lot of people worried, so I have made a few predictions as requested by some of you.
These predictions were generated on data at 12-hour intervals since 29 Dec. 2017, including what was available today.
Each prediction was generated using different parameters, so I could analyze its impact on the output:
* Red line: prediction of hype
* Dark black line: prediction of BTC's value (in USD)
As you can see, it predicts that BTC's USD-value will go up in the coming 1-2 weeks.
However, let me warn you that these are just predictions generated by a machine.
Stay tuned for the next update.
Have a great day all
-- Edit --
Here's a graph generated on 3-hour intervals:
* On the graph above we have 115 x 3hours of known-data = 345 hours; this equals 14.375 days. But it should be more than that. The reason it's only 14 days is that some days our system was down and no data was collected, these points are removed from the set.
======== 19 Jan. 2018
========
Today I have been fine-tuning my system and getting bug-free.
The next task is to add more data sources, more features per data and more cryptocurrencies.
The recent crypto crash got a lot of people worried, so I have made a few predictions as requested by some of you.
These predictions were generated on data at 12-hour intervals since 29 Dec. 2017, including what was available today.
Each prediction was generated using different parameters, so I could analyze its impact on the output:
* Red line: prediction of hype
* Dark black line: prediction of BTC's value (in USD)
As you can see, it predicts that BTC's USD-value will go up in the coming 1-2 weeks.
However, let me warn you that these are just predictions generated by a machine.
Stay tuned for the next update.
Have a great day all
-- Edit --
Here's a graph generated on 3-hour intervals:
* On the graph above we have 115 x 3hours of known-data = 345 hours; this equals 14.375 days. But it should be more than that. The reason it's only 14 days is that some days our system was down and no data was collected, these points are removed from the set.
Last edited:
healzer
Elite Member
- Jun 26, 2011
- 2,974
- 3,063
Update (continued)
In preparation for a public version I have implemented a caching system.
Caching allows me to store the results of a database query, so that on the next same request it won't have to re-calculate everything.
This decreases response time and increases browsing speed.
As I was thinking about the sentiment analysis part, I noticed that my list of words was not tuned for social media mentions.
So it did not include emojis❤ and slang words -- so I found a new list and extended my own. This should give better and more sentiment results.
Have a good night all
In preparation for a public version I have implemented a caching system.
Caching allows me to store the results of a database query, so that on the next same request it won't have to re-calculate everything.
This decreases response time and increases browsing speed.
As I was thinking about the sentiment analysis part, I noticed that my list of words was not tuned for social media mentions.
So it did not include emojis❤ and slang words -- so I found a new list and extended my own. This should give better and more sentiment results.
Have a good night all
D
Deleted member 969102
Guest
I was also reading about this too.
Things like "?!?!" can turn a positive sentence into a negative one.
Sarcasm is going to be much harder to detect, though, and I haven't come up with a solution, as of right now.
healzer
Elite Member
- Jun 26, 2011
- 2,974
- 3,063
That is true. Sentiment analysis is a complex thing, however, a few sarcastic mentions won't make a huge difference when analyzing over 20k mentions per hour.
The outcome of sentiment will lean towards negative or positive by a bunch of other strings such as, but not limited to: *sadface*
, sh!t, f*ck, ... and *happyface* , awesome!, great!, ...We don't need a 100% accurate outcome of every single mention but a general trend.
I also have another strategy for sentiment analysis (based on machine learning) instead of dictionary matching -- but that's something I'll test in the near future.
Cheers!
Christopher87
Regular Member
- Aug 27, 2015
- 308
- 70
I'm an hooked! Just found this this morning and just finished reading it all now. Amazing stuff!
Being able to predict small market trends is a definite win and for those playing around smaller alt coins would be able to make relatively good trades.. I think you said 3 hours accuracy which is amazing.
Anyways - please let me know if you do release this even as a beta, as I would love to test holding multiple Alt coins myself and alot of the hype/decisions comes from a personally view, multiple Reddit posts/threads and some basic research into the whitepaper as well as any underlying business partners that face the coin credibility.
I wish you all the success to follow and will be monitoring this closely for your next update.
Being able to predict small market trends is a definite win and for those playing around smaller alt coins would be able to make relatively good trades.. I think you said 3 hours accuracy which is amazing.
Anyways - please let me know if you do release this even as a beta, as I would love to test holding multiple Alt coins myself and alot of the hype/decisions comes from a personally view, multiple Reddit posts/threads and some basic research into the whitepaper as well as any underlying business partners that face the coin credibility.
I wish you all the success to follow and will be monitoring this closely for your next update.
- Jun 10, 2010
- 2,402
- 1,677
Awesome stuff. Interesting to see that hype goes down over time but the predicted btc is going up. Is the decrease in hype meaning more negativity?
Hopefully your bot isn’t programmed to feel pain. I’d hate to see it kill itself should the price of btc get depressingly low.
Hopefully your bot isn’t programmed to feel pain. I’d hate to see it kill itself should the price of btc get depressingly low.
mesbahuddin
Regular Member
- Nov 12, 2013
- 264
- 19
More and more people are getting familiar with Bitcoin and crypto world day by day. I don't think it's a bubble which will be banished any moment. If the traffic is increasing then I don't believe it's heading to be "fades away". Yes, it's changing the technology already. ICO is introduced to the world. Our mobile phones gonna be different pretty soon. I think it's gonna make a huge difference in Crypto currency.
healzer
Elite Member
- Jun 26, 2011
- 2,974
- 3,063
Hi guys, thanks for the great comments!
However, by looking at historic data I do notice that hype starts to go down when the price is stable (or increasing), then people start to calm down.
But when there's a huge drop in price, the media goes crazy.
Have a great weekend all!
At this stage it's hard to say "why" it's like that.
However, by looking at historic data I do notice that hype starts to go down when the price is stable (or increasing), then people start to calm down.
But when there's a huge drop in price, the media goes crazy.
I have never programmed Python before this project. So in just a couple of days it became my language of preference, I couldn't believe I was messing around with php/java/C# before -- Python allows me to be 2x more productive. I would encourage you to "just do it". I learn from experience not books. If you don't have any specific project where you can apply Python/R to, then just make something up. Someone previously complained that he/she doesn't have a good backlink checker, well that's already a demand you can fill. By helping that guy you will gain experiences (and maybe some money).This is amazing! I'm really interested in Machine learning, data visualization and data analysis, but I'm a complete newbie other than knowing Python and R
Have a great weekend all!
D
Deleted member 969102
Guest
Very interesting Reddit account I just stumbled on:
https://www.reddit.com/user/cryptochecker
https://www.reddit.com/user/cryptochecker/comments/7q3jum/faq_about_ucryptochecker/
https://www.reddit.com/user/cryptochecker
https://www.reddit.com/user/cryptochecker/comments/7q3jum/faq_about_ucryptochecker/
healzer
Elite Member
- Jun 26, 2011
- 2,974
- 3,063
======
====== 20 Jan. 2018
======
Today I've researched new charting libraries for javascript and found some much better ones than ChartJS.
ChartJS is free and easy to create simple/basic charts as I have shown until now.
But for financial and more complex data (especially some 3D stuff) I will changing to PlotlyJS soon
A few moments ago I've added 8 new currencies to track (as seen on screenshot below).
I am testing the stability and load on various APIs and scrapers. So if all goes well, I'll be adding another batch in a day or two.
--edit--
Forgot to mention that I also added a server status viewer.
It shows a summary of the system, so I know when something fails.
Have a nice weekend all
====== 20 Jan. 2018
======
Today I've researched new charting libraries for javascript and found some much better ones than ChartJS.
ChartJS is free and easy to create simple/basic charts as I have shown until now.
But for financial and more complex data (especially some 3D stuff) I will changing to PlotlyJS soon
A few moments ago I've added 8 new currencies to track (as seen on screenshot below).
I am testing the stability and load on various APIs and scrapers. So if all goes well, I'll be adding another batch in a day or two.
--edit--
Forgot to mention that I also added a server status viewer.
It shows a summary of the system, so I know when something fails.
Have a nice weekend all
healzer
Elite Member
- Jun 26, 2011
- 2,974
- 3,063
======
====== 21 Jan. 2018
======
Today I've spent most of my time changing the charting library (from ChartJS to PlotlyJS).
I must admit that Plotly is much richer, but also a bit harder to customize and lacks a few elements.
The screenshot below is the new chart. It looks almost identical to the previous ones.
However, it's not very convenient to set interval ticks on X-axis for the date/time (e.g. hourly, 1minute, 5 minutes, ...). In ChartJS this was automatically generated, here I have to mess around with code to get it right. I do like the fact that PlotlyJS comes with a toolbox which allows me to zoom in, select, drag, scroll, ... This was not possible in ChartJS.
Once I was done with the above, I immediately tackled a new challenge.
For quite some time I wanted to cluster data visually and this is one of the reasons I changed to PlotlyJS -- it has 3D charts.
The idea is to use clustering to categorize coins by their "stage", as suggested by @Ensili (on page 2). I haven't gotten to this part yet, so that's still work in progress.
My first attempt was to visualize 3 variables: time, hype and price. But this didn't appear to be very meaningful (a 2D chart was better for this).
My second attempt was: hype, price, trading volume (past 24hrs) -- yet again not much to conclude for this plot.
But the following did have something to say:
I have plotted (for the past 24hours, with 2-min intervals): time, BTC/USD price and trade volume (24hrs).
Here are a few things we can conclude:
!
--edit--
here's a slightly different angle of the above 3D plot:
====== 21 Jan. 2018
======
Today I've spent most of my time changing the charting library (from ChartJS to PlotlyJS).
I must admit that Plotly is much richer, but also a bit harder to customize and lacks a few elements.
The screenshot below is the new chart. It looks almost identical to the previous ones.
However, it's not very convenient to set interval ticks on X-axis for the date/time (e.g. hourly, 1minute, 5 minutes, ...). In ChartJS this was automatically generated, here I have to mess around with code to get it right. I do like the fact that PlotlyJS comes with a toolbox which allows me to zoom in, select, drag, scroll, ... This was not possible in ChartJS.
Once I was done with the above, I immediately tackled a new challenge.
For quite some time I wanted to cluster data visually and this is one of the reasons I changed to PlotlyJS -- it has 3D charts.
The idea is to use clustering to categorize coins by their "stage", as suggested by @Ensili (on page 2). I haven't gotten to this part yet, so that's still work in progress.
My first attempt was to visualize 3 variables: time, hype and price. But this didn't appear to be very meaningful (a 2D chart was better for this).
My second attempt was: hype, price, trading volume (past 24hrs) -- yet again not much to conclude for this plot.
But the following did have something to say:
I have plotted (for the past 24hours, with 2-min intervals): time, BTC/USD price and trade volume (24hrs).
Here are a few things we can conclude:
- The cluster where most trades were done ==> the price was lowest (so at midnight UTC Jan. 22): buy low.
- There is a specific cluster between volumes 115k and 120k, where the price was highest: sell high.
- All volumes in between the highest price and lowest price are people who were too late to the selling/buying party -- or don't know what the heck is going on.
!--edit--
here's a slightly different angle of the above 3D plot:
healzer
Elite Member
- Jun 26, 2011
- 2,974
- 3,063
Quick update regarding the Time-Price-Volume plot:
I've been playing a bit more with the 3D plot today (this time with 8 hours more data than yesterday).
First I have visualized the Volume on a regular 2D chart:
The yellow area is the price (at hourly intervals).
The dark red line is the traded volume (for the past 24 hours) at each hourly interval.
However, we are more interested in the purple line, which is the difference (delta) of the currently traded volume and the traded volume of the previous interval.
This allows us to see if at some time 't' there were more or less trades compared to the previous interval.
Notice that it can be negative if there were less trades, and positive if there were more ( zero if the volumes were equal ).
From this chart we can conclude that most trades were done when the price dropped. Each volume-peak corresponds to a price-valley.
But as soon as the price starts going up, the trading volume drops (people don't buy/sell).
Here's another way to look at this situation.
In the past ±30 hours the price has dropped from about $12.5k to just about $10.5k :
If we rotate the plot a bit, we see that the volume went up as people started to sell/buy.
They sold either out of fear or bought out of opportunity.
Here's a rotating gif animation I've made:
Cheers!
I've been playing a bit more with the 3D plot today (this time with 8 hours more data than yesterday).
First I have visualized the Volume on a regular 2D chart:
The yellow area is the price (at hourly intervals).
The dark red line is the traded volume (for the past 24 hours) at each hourly interval.
However, we are more interested in the purple line, which is the difference (delta) of the currently traded volume and the traded volume of the previous interval.
This allows us to see if at some time 't' there were more or less trades compared to the previous interval.
Notice that it can be negative if there were less trades, and positive if there were more ( zero if the volumes were equal ).
From this chart we can conclude that most trades were done when the price dropped. Each volume-peak corresponds to a price-valley.
But as soon as the price starts going up, the trading volume drops (people don't buy/sell).
Here's another way to look at this situation.
In the past ±30 hours the price has dropped from about $12.5k to just about $10.5k :
If we rotate the plot a bit, we see that the volume went up as people started to sell/buy.
They sold either out of fear or bought out of opportunity.
Here's a rotating gif animation I've made:
Cheers!
healzer
Elite Member
- Jun 26, 2011
- 2,974
- 3,063
I had that thought at first as well.
But as I kept looking more and more into the 3D shapes, it started making more and more sense.
Thanks to the 3D volume-price-time plot I could see the pattern of supply & demand much better than on a 2D plot.
So I'm looking for patterns in 3D shapes that could help me visualize things in 2D better and/or maybe even help me make better prediction algorithms.
For instance, I would personally use the 3D to determine when to buy or sell.
When I look at the plot for the past ±36 hours, I see when most trades were done.
And so if we could have such a plot across many days/weeks -- we could identify whether we are on the far left or the far right of the plot (relative to the volume). So for instance, right now is a good time to buy because a lot of people are selling.
We see a small cluster evolving "between 12:00 and 18:00 GMT" where people are buying/selling like crazy.
But that's more of a philosophical thing right now
You are right, the 2D data is much more important at this stage.
Thanks!
Last edited:
healzer
Elite Member
- Jun 26, 2011
- 2,974
- 3,063
Quick update (can't edit my previous post):
A couple of days ago I've added a whole range of new words and emoji's to the sentiment analysis algorithm.
Now I have plotted the price, volume and sentiments at 10min intervals for the past 24 hours:
It's quite stunning to me how much the sentiments (green) and volume (purple) are related.
I could go over the entire plot and discuss every hour, but I'll limit myself to the blue rectangle.
At some point the price dropped drastically (from 11.8k to 11.4k) and simultaneously the sentiments dropped (more negativity) on social media.
But then at its lowest the trade volume peaked very high and sentiment started going up (people started buying -- buy at a lower price).
So right now the sentiment is steadily increasing, so I wonder whether that will make the price go up or down in the next few hours.
PS: Once I have more volume-data I will add it as a new parameter to the machine learning algorithm (& make some predictions).
A couple of days ago I've added a whole range of new words and emoji's to the sentiment analysis algorithm.
Now I have plotted the price, volume and sentiments at 10min intervals for the past 24 hours:
It's quite stunning to me how much the sentiments (green) and volume (purple) are related.
I could go over the entire plot and discuss every hour, but I'll limit myself to the blue rectangle.
At some point the price dropped drastically (from 11.8k to 11.4k) and simultaneously the sentiments dropped (more negativity) on social media.
But then at its lowest the trade volume peaked very high and sentiment started going up (people started buying -- buy at a lower price).
So right now the sentiment is steadily increasing, so I wonder whether that will make the price go up or down in the next few hours.
PS: Once I have more volume-data I will add it as a new parameter to the machine learning algorithm (& make some predictions).
