Cryptocurrency analysis and predictions using AI and big data

[healzer]

========
======== Jan 16, 2018
========

I just realized I was writing "2017" in my posts instead of 2018 *facepalm*.
Today and Yesterday I kept analyzing Spark, trying to find the bug.
I got in touch with an expert who pointed out a few things that I'm testing out now.

I have also spent time "normalizing" different social sources and comparing facebook vs twitter vs reddit.
Until now all social hype was just a sum of all mentions, but facebook =/= twitter =/= reddit.

Cheers!

 

[allphase]

Keep up the good work healzer, you'll get it done. Interesting thread, I'm in when you need beta testers. Go crypto!

 

[NulledCode]

With all of the coins dropping it would be interesting to see what kind of data you produced.

 

[allphase]

Riding the bitcoin wave down... Be prepared to buy

 

[seoz87]

dude you should have created a backlink checker tool for us instead of this

 

[IamNRE]

So... did you predict the crash.... whats going to happen next? #SoapOpera

 

[healzer]

So... did you predict the crash.... whats going to happen next? #SoapOpera

To be honest, I have been too busy trying to predicting a crash in my technology.
Trying to keep the system up and running 24/7, but it still keeps crashing after several hours, but every time I'm getting closer to the core.

Regarding the crypto crash, here's a graph of BTC:

And the same one for ETH:

* 30 min intervals, and 8-point trendlines

Both prices are going down (duh!)
The hype of both cryptos has a similar trend.
But what's interesting is that most of BTC's sentiment is negative, while ETH has only positive sentiments.
Beware that my current sentiment algorithm is flawed, but this is still a huge difference.

--Edit--
It's crazy how similar the price trends are of ETH and BTC, isn't it?

 

[waterworks]

very interesting stuff. Thanks for sharing.

Does the prediction model carry to the end of the month?

 

[healzer]

very interesting stuff. Thanks for sharing.

Does the prediction model carry to the end of the month?

Right now the prediction model is very early stage, but works for predicting the next 1-5 time intervals (e.g. hours).
The further we try to predict, the less and less accurate it gets.
Once we have a better model and more stable system, we'll add long-term forecasting (e.g. one month ahead).

 

[healzer]

Super good news guys!
After almost 2 weeks of debugging our system, I have finally been able to resolve the crashes !!
If anyone wants to learn more about the problem, then you can read my post here: https://stackoverflow.com/questions...gs-forever-during-real-time-analysis/48309452

I also noticed that BTC's price is going up. I think it will continue to rise as more people are making use of its drop in price.
Since now I can finally have consistent 24/7 data, so I can finally shift my focus on developing the prediction framework and expanding to my coins.

Cheers!

 

[waterworks]

Well done man! Exciting to see this come together. If you need beta testers, let me know.

 

[mindmaster]

Good to hear that.

Well done.

 

[Deleted member 969102]

I was tinkering around with this idea while travelling into the City today. So far I have plotted the price of a bunch of coins & their release dates onto new exchanges. How are you measuring social media posts? All tweets with Bitcoin, BTC etc. in or is there an easier way to do it?

 

[Deleted member 969102]

Here are some interesting posts I have been reading:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4988639/ - Nice Supporting Information

https://dashee87.github.io/deep learning/python/predicting-cryptocurrency-prices-with-deep-learning/

https://blog.patricktriest.com/analyzing-cryptocurrencies-python/

Starting to piece it all together now.

 

[BigManTing]

Super good news guys!
After almost 2 weeks of debugging our system, I have finally been able to resolve the crashes !!
If anyone wants to learn more about the problem, then you can read my post here: https://stackoverflow.com/questions...gs-forever-during-real-time-analysis/48309452

I also noticed that BTC's price is going up. I think it will continue to rise as more people are making use of its drop in price.
Since now I can finally have consistent 24/7 data, so I can finally shift my focus on developing the prediction framework and expanding to my coins.

Cheers!

Beautiful work! this is what dreams are made off! *drools*

 

[healzer]

I was tinkering around with this idea while travelling into the City today. So far I have plotted the price of a bunch of coins & their release dates onto new exchanges. How are you measuring social media posts? All tweets with Bitcoin, BTC etc. in or is there an easier way to do it?

I don't think there's a way around this.
I did encounter a few service providers who "track keyword/brand mentions as a service", but in the end it all comes down to the same system.

====
==== 18 Jan. 2018
====
Yet another all-nighter, yet another productive day.
I have spent some time playing with the prediction algorithm, trying to understand it better.
There are exactly 4 parameters that I can play with, so for each parameter in a range ( i --> j ) , I let it generate a prediction model. So the end-result is just a hell lot of images:

However, I decided to do something more meaningful: scaling the system to more coins and more data.
I have successfully installed a distributed system (currently 2 servers). There is one broker that receives scraped data and sends it to workers (currently only one worker available).
The hardest part was making Spark work with Hadoop HDFS, as I couldn't find any well-documented tutorials. But it all worked out just fine ^phew.
I've also redesigned the API (which took many hours) due to complex array transformations.

The next step is to refine the system and prepare it for more crypto currencies.
Once I have at least one week's data then we can do some prediction stunts.

Have a great day !!

 

[mindmaster]

Are you taking into account other data than mentions in the media, like number of offers (sell/buy) and their prices?

 

[healzer]

Are you taking into account other data than mentions in the media, like number of offers (sell/buy) and their prices?

Not yet. Will add that soon There's so much I want to do but so little time.

 

[Deleted member 969102]

Not yet. Will add that soon There's so much I want to do but so little time.

Are you weighting the mentions?

Like say an account with 100,000 followers is more influential than an account with 10.

 

[PaulLAllen]

I personally hate crypto. it's always too much risky.