And... how not to make crypto currency exchange.
Recently I wrote a trading robot for Kraken crypto currency exchange. It's my job, to make custom software for trading on stock exchanges.
Very soon I realised that Kraken gives the data through API with huge delays. From one to three seconds. And if you request data more frequently, you will be cut off from the exchange by IP.
And I had an amazing insight: "The man who wrote the core of this exchange, does not realize how this core shall be arranged"
And I started studying other cryptocurrency exchanges. And I found out that a half of them is very bad and is made by people far from trading and understanding how to build the core architecture without cutting algotraders from bidding if they want to see the order book more than once every five seconds (WHAT?!).
In this post we'll talk about several ways of organizing ARCHITECTURE of exchange. They will lead us to a completely different result:
1) the Architecture of light, truth and distributed calculations – which can be painlessly expanded and where the load can be distributed. This is a hard but righteous path. Which, however, probably won't get you more money than the other.
2) the Architecture of evil, lack of competence, jack-off and theft – what most of the creators of criptocurrency exchanges have chosen. By creating this one you'll never be able to digest the normal number of traders. But don't despair. Such an exchange can be created in a month or a month and a half and then you can start to attract investors. To get the ICO, to blow a lot of cash. And when in a couple of years it turns out that your core is not working – you'll just get out of the project.
Table of content:
1) The architectural arrangement of the real exchange.
2) The architectural arrangement of an ordinary exchange
Let's start from the picture.
Fig. 1. A simplified diagram of the exchange core and its environment
They are all roughly similar.
There is a certain program written in C / C++, which is called the exchange core. The core has four main functions:
1) to check applications for coverage at the entrance
2) somehow to try to match applications at the auction
3) to generate order log of records from the auction
4) to serve order log and other data taken directly from it at the exit. This is a thread of anonymous transactions. The thread of images of order books. Etc.
All four of these functions can be threaded virtually (multithreaded execution) or physically (on different servers within the same local network).
The core is surrounded by different gateways for information and placing orders. In the case of ASTS and Spectra these are CGate, ASTS Bridge, Twime, Fix, Fast, etc.
Near the core of the exchange, physically. There are other servers that host premium users. These are intermediate servers of brokers’ robots that need quick access to the core.
All the rest
All the rest are forced to trade either via web interfaces or via brokers' terminals. Which allow you to trade through treir passthru servers in the collocation area. Through the Internet. The connection is slow and sad. But no less stable and acceptable.
If the core and Api were written by normal guys, everything should be perfectly extensible, and work really fucking fast at almost any loads.
It is possible to adapt this to crypto currency exchange.
Such a core will take from six months to a year, depending on the "trippy-ness" of your team. Consisting of three to five persons. During this time, you can make a good multi-threaded core and tie up one protocol. Fix Fast, as universal is OK. Next, using staging server Fix Fast, we connect user accounts on our site (which is a face of exchange). And we trade without any restrictions. For HFT and other super-high-speed robotics we'll allocate our own personal gateways and a place in the collocation for sub fee. A real exchange! Cheers!
Do you really think someone from crypto-exchanges has done this? Hell no!
Let us also start from the picture.
Fig. 2. A simplified diagram of the exchange core and its environment
Visually, at first glance, there are not many changes. The collocation area and the ability to get exchange data for anyone disappear completely.
We all have to get data through a unified HTTP interface.
Disclaimer: I'm not saying we meet it everywhere. But from what I saw, I think that this is the vast majority of cases.
Despite the fact that all functionality is preserved, a software component is completely changed.
1) Instead of C/C++ here we have PHP and a free database like MySQL.
2) All the operations inside the core are executed by one thread. Literally. From "Сheck of application coverage" + "Auction" + "Generation of order log" + "Update of all data threads" – all made by a single thread.
3) How does the "work" of the "core" look:
a. The order came through HTTP Api.
b. First, we expect that all work in the core stops.
c. For this user we request a table from the limits database. If the limits are ok we go on.
d. We requested a table from MySQL for a sell (let's assume we have a purchase order)
e. If the current sell orders in the table can match our purchase order, we begin cascading changes in the database.
f. We generate new records where orders were filled up in the table of orders
g. We generate new unallocated transactions
h. We generate change in a order book
i. Etc. We do this single thread in MySQL tables and make new records in it.
4) The request for the data is made directly from the database.
This "core" of "EXCHANGE" can be written by 80% of the sophomores of any technical collage of our planet. Writing this hurts, but looking at the list of criptocurrency exchanges I think that this is what they really did...
What else can I say. I don't want to delve into its essence. The normal protocol super suits small and medium projects. BUT. If you think that the main problem of cryptocurrency exchanges is that the core is written by children – it is not true.
When using the HTTP protocol, a single server address is used for all requests. This means that each API has a limit number of requests after which the server stops responding, as at DDOS attacks. In other words it falls.
And at the early stages, when pupils only started it a couple of months ago and took their first billion from investors suckers. You won't see this problem. At first glance everything seems to be decent.
Problems of this API will occur only if the exchange will get a little bit of success.
And if algorithmic traders come to this exchange, and start asking order books more often than once in a couple of seconds, it will be the end. The exchange will die.
Remark! In order to make this happen as rarely as possible, exchanges impose restrictions on the number of requests to API.
For example in my favorite Kraken this is from 1 to 4 seconds between requests. If you do it more often – you'll be banned by IP. https://support.kraken.com/hc/en-us/articles/206548367-What-is-the-API-call-rate-limit-
And they say that it's because they are afraid of DDOS ATTACKS!!! Get off! Idiots! On the other hand I can't just write that they are bunglers and lazyboness. Many users may get sick of such a truth.
To a programmer
I know that this topic's been extremely relevant for a very long time.
Probably this article will be a pain in the neck for all those startups which want to create their crypto exchanges. And they want to do it in a "bad" version...
Guys, I'm not against it. Do it. Just don't say in the interviews that you made the exchange in three weeks and are proud of this(https://cryptocurrency.tech/kak-za-3-nedeli-sozdat-svoyu-birzhu-kriptovalyut-s-nulya-istoriya-birzhi-bitflip/). Just do it quiet, you know... Through offshores. Use fictitious names. Eeveryone including you understand that what you've done will never be an exchange.
You've thrown up a bunch of bad code in someone else's subject area. That's why locals got a bit shocked and look at each other, not knowing what to say.
On the other hand. I can understand you too. This is actually a good spheare for your enrichment. If I had no conscience I would have fouled an exchange in a couple of weeks.
Blah. I even have a few investors who would have sponsored it. But how would I look them in the eyes later? How do you do it?
So. You are a serious guy who's been offered to invest in crypto currency exchange?
You think that you just need some time and you will be able to touch the advanced technologies of the future?
Invest safely! (no)
But first ask your programmers a few simple questions.
1) What language will they write their exchange core? If their answer is not C++, and even C# and not Java. If they start talking about PHP and database. – Run!
2) Will there be API programming for algotraders? If the answer is HTTP, it's bad too. This exchange will never get serious volumes. It will not be stable and will be slow.
But, in general. Of course its your choice. This is a hype topic and you can earn money both ways. The Kraken is alive. Bastard. It drops several times a day. It stops API. Users are in shock. But Kraken earns)
To constant reader
Friends. I meant no harm. Trust me. It just causes me pain.
I wanted to write to Kraken that they were wrong. But then I thought that they already know it. They just can't stop. And they know it at the stage of design of the exchange. So I decided to write to the descendants of the eighties.
Do not write shit exchanges. Big brother sees everything.
Have good algorithms!
Yes. Yes... There is also a medium variant of architecture when heavy data are sent using WebSocket. This is a very popular solution for many cryptocurrency "exchanges". It seems to be more or less acceptable from the point of view of algorithmic trading. At least the order books can be obtained not only every five seconds but at the moment of their change. But I honestly don't know how to apply this architecture. This is definitely not a stock exchange, but you can trade :). Something like that.