Need Historical Crypto Data

MaazAr said:

I would be needed data on a tick by tick basis. This is daily and monthly. Thank you for your reply

Click to expand...

MediaCircus56 said:

I am just curious. Why do you need it ?

Click to expand...

MaazAr said:

Hello, I want historical crypto trade/order data for as many exchanges as possible from 2019 or 2022 to current.
Any help would be appreciated. Thanks

We need data on a tick by tick basis. Is there anyone who will be able to get me this data.

Click to expand...

SirSmash said:

He's backtesting EA/Bots, working on ML, HFT (likely Triangular Arbitrage as he wants multiple exchanges), or possibility trying to be a crypto market maker. My money would be on Triangular Arbitrage.

I can go back to 2012 from Dukascopy, Yahoo, & Coinbase. 2017 for Binance. 2019 for Bittrex. The issue being the dataset size as just the crypto pairs [I keep data on >400 pairs at the moment] should be around the 300TB mark with CSV files (MT4/MT5 conversions are worse, lol). I only keep the last year on spinning disks, anything pre mid-2021 is on LTO-8 in autoloaders.

Those can be operated remotely but the upload speed here is only ~50mbps so you'll be waiting quite some time, looks like 500+ days, or I'd have to physically mail you boxes of drives or preferably LTO-8 tape as those travel better than spinning disks and it's simple to duplicate, but I'd imagine you don't have LTO-8 or LTO-9 (backward reading to 8) drives, or I'd have to like loan you a 2U autoloader. Then I'm not sure you have SAS hookups, so I'd probably have to mail a server too. At that point SATA drives might be simpler.

If you'd have needed this a few months ago it would have been easier as I used to do all this through Hetzner dedicated servers but they don't really have a good cold storage option and keeping it hot on storage servers was ~$1/TB/month. If you include the tick data for stocks/FX/commodities I'm nearing a PB, so I moved everything locally.

Click to expand...

MaazAr said:

Yes, you're right, I am working on triangular arbitrage.
As for the data that you have can you send me the data on LTP-8 Autoloaders. Ill work on arranging the infra to read them. Just the loaders are enough..

Click to expand...

SirSmash said:

I can certainly get you the data, but not sure if you meant just the tapes or the tapes + autoloaders as those are two seperate things and sending "just the loaders" wouldn't do you any good.

But I got to ask, do you have any experience with this? My gut feeling is you came across Hummingbot. Are you intending to actually make this a legitimate business (explained below)?

There's a pretty major flaw with backtesting triangular arbitrage. Even with tick data, you don't have a way to accurately account for lag between brokers (and yourself) which is constantly changing; this variable isn't recorded as part of tick data. You can't depend on the HH:MM:SS:XXX timecode as it's an OTC marketplace where the servers aren't synced like they would be on something like NYSE and a tick is recorded for every price change regardless of how much time was in-between.

Triangular abitrage works, or fails, based on lag between brokers and yourself. Running a back test for this would only prove that traingular arbitrage is possible, but we already know it's possible. You just have the software figure out the lag between brokers and do the transaction. You'd get far more accurate results doing a foward test for 24 hours than you would doing years of back tests.

What you need to be focusing on is where your servers are physically located and how you can reduce lag. If you have a broker in London and another in NY and another in Singapore it's possible if you know you can do the transaction quicker. But back testing will never give you this information.

And if you think you're doing this from a home desktop, forget it. Here's the hard reality of the HFT world that you don't want to hear but it'll probably save you a buttload of time and effort... People like myself have overclocked, watercooled, servers (these are specifically designed for finance) in datacenters like Equnix NY4 with 40gigabit fiber lines directly to the brokers that are just feet away (there's instances where we add extra fiber and keep spools on the floor to add lag). Unless you have the same infrastructure, there's no way you're beating us to the punch. You might get a few scraps here and there but things like Hummingbot are really just scams to add liquidity in the form of retail traders losing their money. Why do you think retail trading even exists? We ain't running the server infrastruture to make YOU money, we are running this stuff to make US money.

If you're going to make this a proper business, there's opportunity. If you're doing this to get rich from home, save yourself the time and effort.

Click to expand...