Blog

How to import the Bitcoin blockchain

In this article we are looking into why and how to import the Bitcoin blockchain, and also share tips and tricks. There are bits and pieces of information out there, therefore this paper is an attempt to bring them together and share our own experience.


In this article we are looking into why and how to import the Bitcoin blockchain, and also share tips and tricks. There are bits and pieces of information out there, therefore this paper is an attempt to bring them together and share our own experience.

Before diving into technical details, let’s start with the question: why import the blockchain. Hier is a list of possible reasons:

– perform analytics: after import, analytics will hopefully be more efficient, than accessing the blockchain directly, because you can query your database faster. For example, given an address what transactions were used in combination with that address.

– find malicious, fraudulent transactions: this speaks for itself, since there have been reported several cases of stolen crypto assets.

– extract data which has transaction fees.

– have multiple wallets, that connect to the blockchain directly.

– be independent from an external service, that might not have all the data you need.

– instantly compute the balance of one address.

How to read the blockchain

There are two possibilities to read Bitcoin blockchain: read directly in binary format or use bitcoin-core (free and open-source software that serves as a Bitcoin node (the set of which form the bitcoin network) and provides a Bitcoin wallet which fully verifies payments).

Directly in binary format

First option is to read the Bitcoin blockchain in binary format directly.

The big advantage is that this method is quick.

However, the big disadvantage is that this method requires a lot of work:

  • your program has to be synchronized all the time with the binary format,
  • It has to read the data in the proper way and,
  • parse the data correctly and ensure no mistakes.

Use bitcoin-core

Second option is to read the Bitcoin blockchain by using bitcoin-core.

Basically, your program makes an http connection to the bitcoin-core program, called bitcoin-d (“d” stands for demon). Bitcoin-d downloads blockchain and handels RPC-requests (“RPC” means Remote Procedure Calls).

Then, you need a client to connect to the bitcoin-d. You could use bitcoin-cli(“cli” stands for command line interface), which is a command line interface for bitcoin-d, connects to bitcoin-d, request stuff, create wallets, etc. You can query the data at bitcoin-cli and get it in Jason format. Or write your own client, which we did, because bitcoin-cli has limited functionality and we needed more in-depth blockchain analysis.

For example to compute a transaction fee, you need to query data that is randomly located over the blockchain. As a database we use MySQL.

As you can see, an advantage of this method is that you get a Jason string back which is easy to parse.

However, a disadvantage is the dependency on the bitcoin-core which is not very fast when reading large amount of data.

Algorithm for Bitcoin blockchain import

Let’s list steps for Bitcoin blockchain import using bitcoin-d.

Here, watch out for two problems:

  • Firstly, you will need input from other transactions that are connected to this processed transaction. We have measured that there is 60% chance that this transaction will be used again in the future. The best way to implement it is by using cache (meaning to store transactions that have been processed in the cache). We have seen speedup of 10x, from 1.5 hours to 10 min per 1000 blocks.
  • The second problem is that you don’t know what transactions will be needed in advance and at some point the cache will be full. In this case, a solution would be to randomly remove 50% of transactions.

Tips and tricks

In this section we list the tips and tricks that we found useful ourseves by importing the Bitcoin blockchain.

Tip 1: Make sure that bitcoin-d is setup correctly to allow RPC connections.

Tip 2: Make sure you have the option “txindex=1” (can retrieve the data ).

Tip 3: Fast drive (SSD) is important for cache and to read the data quickly.

Tip 4: Mapping transaction to address generates a lot of data, for which it is not worth to build the database. Therefore, send these data asynchronously, while reading new blocks.

Tip 5: How to manage the cache? We use FIFO: first in first out, no longer random. Also, a technical implementation detail: it is better to use list of blocks instead of a vector (or a linked list), because removing the first element of a vector is slow for very large vectors and requires memory copy.

Tip 6: Be careful with the timeouts, sometimes a query can take a long time. Af first it might look like bitcoin-d is hanging, but it is just in a process of setting up. It will abort, if the program is taking too long.

Tip 7: For some strange reason, import of block 256961 was taking longer than 30 sec, every time we have tried. Then we found out that 30 sec was the timeout of my program. After increasing timeout to 1 min, it worked.

Summary

Here we shared our experience by Bitcoin blockchain import. Basically, what worked for us was connecting to the bitcoin-core and writing our own client, since we needed advanced analytics on the Bitcoin blockchain.

Ones you have imported Bitcoin, the import of other cryptocurrencies such as Litecoin, Bitcoin Cash should be a piece of cake. Let us know in the comments, what was your experience with the blockchain import.

Crypto tax 2018 in the Netherlands

The year 2018 was tough for crypto holders: it started at the price close to 20,000 USD and end up below 4,000 USD, a true price rollercoaster. I wish that would be it, but the tax season is coming and we have to pay the consequences. The goal of this article to gather all information about the crypto tax in the Netherlands.

There are only a few things that are important for the Dutch crypto tax:
– the price of a cryptocurrency on the 1st of January 2018
– is crypto your source of income or not
– your situation: private individual, entrepreneur or a company

Let’s have a look at each aspect in the detail.

What was the price?

According to the Dutch Ministry of Finance, for every cryptocurrency, you need to take a value on the 1st of January 2018 at 0:00. If this is not possible, then the closest point in time to this date.

This should be easy to calculate, right? Sure, if you used only one exchange, then just check the price of the cryptocurrencies at this exchange.

What if you have used several exchanges? The price varies across different exchanges. What to do in this case, the Dutch Ministry of Finance does not say. I could guess that an average over the exchanges that you have used might make sense, but this is only my personal opinion.

Source of income

The income tax in the Netherlands, in a nutshell, is split across 3 ‘boxes’:

  • box 1= taxable income from home and work
  • box 2= taxable income from substantial interest
  • box 3= taxable income from savings and investments (capital)

Mining and trading of Bitcoin and other cryptocurrencies is taxed in the Netherlands in box 1 of the income tax act if it qualifies as a source of income, such as profit or result from other activity. To qualify as a source of income, certain conditions have to be met:
(1) aiming for profit and
(2) the reasonable expectation of profit.
Case law has clarified that there is no source of income when it comes to speculative transactions and in case the end result cannot be influenced by performing work. (MFFA)

If your crypto holdings do not qualify as a source of income, then they are taxed in box 3.

Who are you?

The crypto tax in the Netherlands is different for

  • private individual
  • entrepreneurs
  • BV (private limited liability company)

If you are a private individual in the Netherlands, then there are possible following situations:

  • you own cryptocurrencies : it is taxed as capital (box 3)
  • you mine : if you have profit from mining, then you have to declare it either in box 3 or box 1 (see the previous section)
  • you trade : trading is comparable with the investments and therefore most of the time is taxed in box 3 (capital) unless you spent extra time and effort to get “structurally positive results … which goes beyond speculation”, then it is taxed in box 1
  • you get paid in crypto : in this case, your employer has to convert it to EUR at the moment of payment and it is taxed in box 1

If you are an entrepreneur, then you can be

  • entrepreneur for income tax (“ZZP’er” or “eenmanzaak” in Dutch = “sole trader” or “freelancer” in English)
    In this case, the Dutch Ministry of Finance distinguishes whether you hold crypto assets privately or whether the crypto assets belong to the business. In the first case, the crypto is taxed in box 3. In the second case, the crypto forms part of your normal business operation and remains untaxed, but included in the company’s turnover.
  • entrepreneur for turnover tax (if you own a company : “eenmanzaak”, partnership, private liability company, etc). This is a simple one: in this case, you convert crypto to EUR and register at the turnover tax.

If you own a Dutch BV (private limited liability company), then all purchased or mined crypto assets belong to the company’s capital. As a result, realized gains are taken into account as profits and losses are deductible. The same holds if you are paid in crypto for goods or services.

How much taxes do I have to pay for my crypto?

In the previous section, we have seen different situations for crypto taxation. Unfortunately is not possible to pre-calculate how much taxes do you have to pay in all these cases, because of the influence of your personal situation. However, we can consider one case when your crypto assets are regarded as capital and taxed in box 3.

For a starter, you should add all your savings and capital, including the value of your crypto assets in EUR on 1 of January 2018 at 0:00. Let’s call it X.

Then subtract 30,000 EUR of tax-free capital.

If you have a fiscal partner, then X is your collective capital and the tax-free capital is 60,000 EUR : X – 60,000 EUR.

On this amount, you have to pay taxes.

The taxable amount is split into so-called discs, for each disk, there are different tax percentages, see Table 1.

Table 1: Dutch taxes for capital (box 3) in 2018 for a single person.

Example

Let’s have a look at an example. Let’s say you are a single private individual, a HODLer, and have 10 Bitcoins and nothing else. The price of Bitcoin on the 1st of January of 2018 is 11.325 EUR, which makes the total capital of 113.250 EUR.

Now, we know that on the first 30K EUR, you don’t have to pay taxes, which results in 83.250 EUR of taxable amount.

The from the Table 1 we see, that the first disk is taxed with 0,60 % and the second with 1,30% , which gives 70.800 * 0,006 + (83.250–70.800)*0,0013 = 586,65 EUR of taxes that you have to pay.

I know it is painful to see that at the moment the price is below 4K, but you have to pay for the higher price that was a year ago.

Tips for 2019

The advantage of the Dutch taxes for private individuals is that most of the time you already know how much taxes you have to pay in 2019 because the price is set right at the beginning of the year. Moreover, you can ask for voorlopige aanslag” (provisional tax calculation) at the Dutch Tax office. Knowing this you can set this amount aside or make for it during trading, or maybe arbitrage.

If the tax amount is large, you can request to split the payment, but you need to make an arrangement with the Dutch Tax office.

There are few ways you can reduce your capital like a gift to a goodwill or culture, extra mortgage repayment or study. They might help to reduce the taxes for 2018. In this case, you have to consult with the Dutch Tax office.

Summary

All in all, it sucks that crypto coins made such a fall in 2018, but now you have an idea what to expect at the tax declaration for 2018:

  • the 1st of January 2018 at 0:00 is the critical time to compute the value of your crypto assets
  • your status is important: private person, entrepreneur or a company
  • in what taxable ‘box’ belong your crypto assets.

Let’s be smarter in 2019 and be prepared for tax season next year.

How to make money on arbitrage with cryptocurrencies

Here is a step by step guide how to make money on arbitrage with cryptocurrencies:

Step 1: Find opportunities for arbitrage with cryptocurrencies (between exchanges or within exchange)
Step 2: Take a decision whether to buy or not to buy:
1) Estimate amount of fees: transaction, transfer, network, wallet fees
2) Be aware of risks: transaction and transfer time, market volatility
3) Estimate the amount of taxes

In this article we consider each step in great detail. The first step is of course essential, but please do not underestimate the following steps as well.

If you are experienced crypto trader, then you might skip the next section and jump to the finding opportunities. Otherwise we remind you on the terminology we will use in this article.

Terms to know

Arbitrage — “is the simultaneous purchase and sale of an asset to profit from an imbalance in the price. It is a trade that profits by exploiting the price differences of identical or similar financial instruments on different markets or in different forms.” Investopedia

Fiat or fiat money — “is a currency without intrinsic value that has been established as money, often by government regulation. Fiat money does not have use value, and has value only because a government maintains its value, or because parties engaging in exchange agree on its value”, Wikipedia. For example, dollars or Euros are fiat money.

Crypto asset — is a digital asset which utilises cryptography, peer-to-peer networking, and a public ledger to regulate the creation of new units, verify transactions, and secure the transactions without the intervention of any middleman. There are 4 types of crypto assets:

  1. Cryptocurrency (e.g. BTC)
  2. Platform tokens/cryptocommodities (e.g. ETH)
  3. Utility tokens (e.g. OMG)
  4. Transactional tokens (e.g. Stellar)

Volatility — “In finance, volatility (symbol σ) is the degree of variation of a trading price series over time as measured by the standard deviation of logarithmic returns”, Wikipedia

Deposit — amount of fiat money you have decided to put into crypto exchange to buy cryptocurrencies

Withdrawal — amount of cryptocurrencies you have sold for fiat or transferred to another exchange

Order book — a ledger containing all outstanding orders-instructions from traders to buy or sell bitcoin.

Find arbitrage opportunities

Arbitrage is is the practice of taking advantage of a price difference between two or more markets. For example, an arbitrage opportunity is present when there is the opportunity to instantaneously buy something for a low price and sell it for a higher price.

To find an opportunity for arbitrage with cryptocurrencies is an essential step. There are two major kinds of the arbitrage with cryptocurrencies:

  • Arbitrage between exchanges and
  • Arbitrage within an exchange.

Let us have a look at each of them and consider pro’s and con’s.

Arbitrage between exchanges

Arbitrage between exchanges is the most obvious type of arbitrage, because it is very similar to the fiat currency arbitrage (e.g. forex arbitrage) or to the sports arbitrage. The idea is simple: benefit from the differences in prices for the same coin but on different exchanges. For example, see the different prices for Bitcoin in US dollars for different exchanges on the Figure 1, where the price for 1 Bitcoin ranges between 6600 and 8730 US dollars. In this example you would buy for 6600 USD on and sell for 8730 USD, theoretically making 2130 USD or 32% in profits for one single transaction. Sounds good, right? Please do not rush to follow this particular example and read further.

Figure 1. Snapshot from https://www.coinhills.com/dashboard/ ,19.07.2018

Let us split the process in steps:

  1. Register on both exchanges of your choice
  2. Deposit fiat on one exchange and buy a Bitcoin or any other cryptocurrency
  3. Transfer cryptocurrency to the other exchange
  4. Sell your crypto asset for fiat
  5. Withdraw the profit

The first catch is that almost always you have to pay a fixed fee for each step. The subject of fees is quiet complex, you can read all about in the section below. But just to give you an idea, you might pay as little as 3% of your crypto asset or as much as 15%, depending on the exchanges.

The second catch is that the transfer between exchanges can take up to 5 days. Since the volatility of cryptocurrencies is high, the theoretical profit might diminish during this time.

It is possible to reduce the amount of fees and also waiting time. Here is how you could do it step by step:

  1. Register on the exchange 1 and 2
  2. Deposit fiat on exchange 1
  3. Deposit fiat on exchange 2
  4. For a part of deposit buy a cryptocurrency on exchange 1, now you have fiat and crypto on this exchange
  5. For a part of deposit buy a cryptocurrency on exchange 2 , now you have fiat and crypto on this exchange
  6. When arbitrage opportunity presents, buy a cryptocurrency on the exchange 1 and sell the same amount of the cryptocurrency on the exchange 2 at the same time, or vice versa.

Here there is no transfer of the cryptocurrencies between exchanges, that means neither waiting time, nor fee for this step. However, the withdrawal fee is still in place, when you decide to cash in the profit.

Obviously, arbitrage between exchanges is connected to several risks, see section on arbitrage risk below.

Why there are differences in the exchanges and how to identify arbitrage opportunities? Here are few ideas:

  1. Liquidity (see example above): difference in the trading volumes at different exchanges, meaning difference in supply and demand, affects the prices. On the established exchanges prices fluctuate less than on the smaller or new ones.
  2. Geography (while something can happen in the morning in Europe, that influences prices, the most of people in US are still sleeping, hence the price difference due to the geography)
  3. Listings (price difference when a crypto coin gets listed in one of the major exchanges).

Arbitrage within an exchange

Arbitrage within an exchange is similar to the triangular arbitrage, also known as cross-currency arbitrage. The step-by-step process is then as follows:

  1. Start with deposit of some amount of fiat on an exchange
  2. Buy cryptocurrency 1
  3. Sell cryptocurrency 1 and buy cryptocurrency 2
  4. Repeat steps 2 and 3
  5. Sell cryptocurrency 2 for fiat
  6. Withdraw the profit

You could substitute fiat with yet another cryptocurrency, or repeat step 2 many times with different cryptocurrencies. In the last case, it will be not a triangular arbitrage, but polygonal arbitrage.

By staying within an exchange and applying the same process over and over again to different cryptocurrencies, the major fee (withdrawal of cryptocurrency) is eliminated.

The catch in this case though is that the opportunity is less obvious than in case of arbitrage between exchanges.

Here is an example of triangular arbitrage. Let’s say that you have deposited some funds to an exchange and bought USDT, which you might consider a crypto equivalent of USD. You want to buy 1 Bitcoin (BTC). Of course you could buy 1 BTC for 6527.06 USDT (on 20th of August 2018, 04:26 CET).

Or you could use the triangular arbitrage strategy:

  1. Buy Ethereum or ETH for USDT, 1 ETH = 302.15 USDT
  2. Buy BTC for ETH, 1 ETH = 0.04643 BTC, or 1 BTC = 21.5378 ETH

Then your BTC would cost 21.5378*302.15 = 6507.64 USDT.

That means, to buy BTC via ETH you saved 19,41 USDT, which is about 0.3% of the Bitcoin price. If you sell immediately 1 BTC for 6527.06 USDT, you will make 0.3% profit of these 2 transactions.

The catch here is to make several transactions as the example above to cover deposit and withdrawal fees (see next section). The best practice is to run a bot that identifies the opportunity and if it is higher than a certain threshold (that includes fees and taxes), buy and sell while you are sleeping.

It can be even more complex, you could have bought first number of Litecoins (LTC), then ETH for LTC, BTC for ETH and finally cash out BTC for USDT. In this case you would make 0.306% profit (on 20th of August 2018, 04:26 CET).

Given the fact that the number of cryptocurrencies is approaching 2000, the combinations are endless, see example on Figure 1.

Figure 1: Comparison of the arbitrage gains with the fee on Binance. The orange line shows the gain from arbitrage opportunity of buying BTC via ETH. The purple line shows the gain from arbitrage opportunity of buying BTC via LTC and ETH. The blue line represents the fee on Binance. For arbitrage to be profitable, the gain lines (orange and purple) should be always above the fee line (blue).

Fees

By now, hopefully, you have identified an opportunity for arbitrage with cryptocurrencies and know how much percent you will gain if the orders will be executed in the ideal/theoretical case. Now let’s have a look how much fees you have to pay and what risks are associated with every trage.

Fees at exchanges:

There are three major sources of fees at the exchanges:

Fee #1: Fiat deposit/withdrawal fee is taken by an exchange for deposit/withdrawal of money from your bank account or by a credit card payment.

  1. Credit card payment is instant, you immediately can use your funds. However, the fee ranges between 3% to 8%. WEX exchange charges 15% for the fiat withdrawal.
  2. The wire transfer to the exchange deposit account has usually a small transaction fee (see below), but it can take several days.
  3. Direct deposit (might be even slower than the wire transfer, moreover direct deposits outside of country incur the conversion fee, e.g. in US it is 2.5%)

Fee #2: Transaction fee

Depending on the exchange, the transactions are charged with

  1. Fixed fee
  2. Maker fee
  3. Taker fee

The fixed fee is obvious: independent of the coin, volume and order books, the fee stays always the same. It ranges between 0.02% and 0.1%.

The maker and taker fee have been introduced by the Kraken exchange and some other exchanges followed.

“A trade gets the taker fee if the trade order is matched immediately against an order already on the order book, which is removing liquidity. A trade gets the maker fee if the trade order is not matched immediately against an order already on the order book, which is adding liquidity.”

Basically, if you want to sell/buy and you put the price such that your order will be executed immediately, then you pay lower fee (taker fee). Otherwise your order has to stay for some time and for the exchange it is less beneficial, in which case you pay the maker fee. Usually the maker fee is 2–3 times more than the taker fee.

These fees might change dependent on the amount of your order : the more you buy/sell the smaller the fee. Some exchanges only charge the taker fee, in this case the maker fee is 0%. See an overview of the fees per exchange here.

Fee #3: Cryptocurrency deposit/withdrawal

Usually, deposit of a cryptocoin is free, but if an exchange needs to create a new address for your chosen coin, then they will charge blockchain (or network fee), see below. Withdrawals fee are depending on the crypto coin, for example Kraken charges for Bitcoin withdrawal 0.0005 Bitcoin and Binance charges twice as much 0.001 Bitcoin. Also some exchanges don’t charge withdrawal fee, for example GDAX.

Let’s have a real life example with fees of an arbitrage on Coinbase, where you can buy Bitcoin directly. Let’s say that you want to buy Bitcoin for 10,000 USD at Coinbase via credit card (because it is the instant way of buying crypto). Coinbase charges 4% for this credit card transaction, that means you have 9600 USD left to buy Bitcoin. The prices are following (on 31st August of 2018):

  • 1 BTC = 7044,32 USD
  • 1 ETH = 283,49 USD
  • 1 ETH = 0.04032 BTC

The arbitrage opportunity is to buy BTC via ETH: 9600 USD -> 33,86 ETH -> 1,3653815 BTC, whereas if you would buy BTC directly you would get: 9600 USD -> 1,3628001 BTC. The gain from this arbitrage opportunity is 0.2581% (equal to 18,18 USD=0.00258139 BTC). If your orders are executed immediately, then 0% fees apply. However, if your order gets stuck in the order book, then the fee per 1 transaction is 0.3% (taker fee). Let’s assume that you have got lucky and do not have to pay transaction fees. In this case, you would need 22 transactions similar to these to cover the credit card fee for the deposit only.

Finally you need to pay the withdrawal fee. We suggest to use a wire transfer which has a little fixed fee (0,15€, 1£, 25USD).

You see, fees might be a profit killer, so you have to be very careful with the choice of the exchange.

Few ideas to minimize the fees:

  • Ideally match your transaction such that the order will be executed immediately
  • Use exchanges with no/little crypto withdrawal fee
  • Check carefully the fiat withdrawal fee on the exchange
  • Use wire transfer instead of the credit card or the direct deposit
  • Use exchanges with the deposit account

Fees on blockchain or network fees

Every crypto coin is connected to a blockchain. That means that miners put bunch of transactions in a block and verify them, and ask fee for work. This fee is called blockchain fee or network fee.

Some exchanges don’t take fee for the deposit or the withdrawal of a cryptocurrency. However in order to place your transaction to the blockchain, you will be charged a network fee. For example, you would place your freshly bought Bitcoin from Coinbase to your wallet or offline storage. At the moment of writing this article, the Bitcoin network fee was less than 1 USD.

Fees on wallets

Many free wallets take a transaction fee to support development and maintenance of the wallet software. You could check the fee in your wallet settings. Moreover, if the wallet creates a new address to store your cryptocurrency, it has to be added to the cryptocurrency blockchain. In this case, the network fee occurs (see above).

Risks

There are several risks associated with the crypto arbitrage. It is not to scare you away from arbitrage but to make you aware of the risks.

Risk #1: Hacking risk

There have been well known attacks resulting in millions of stolen Bitcoins (see top five hacks here).

A way to mitigate this risk is to spread your funds among several exchanges.

Another way is to keep the amount you are ready to lose on exchanges and the rest in the cold storage.

Risk #2: Execution risk due to fast moving market or market volatility: you need to perform at least 2 transactions for an arbitrage, which ideally should be executed immediately. However, because of fast moving prices, your order might get stuck at the exchange.

That means you also have to pay a taker fee.

A way to mitigate this risk is to use a bot that is doing trading for you.

Risk #3: Movement risks between exchanges/blockchains: most of the exchanges need a few days time to validate deposit and withdrawal. Here you can read a list of issues the author encountered .

Also, this risk category includes

  1. Wallet maintenance
  2. Network is down

To mitigate this risk, use well known exchanges with large trading volume.

Risk #4: Price decline risk: the trading funds will decline with higher % than profit from arbitrage. 
Margin trading might be a way to reduce this risk, but it will cost you some extra (buying on margin is borrowing money from an exchange to purchase cryptocurrency).

Risk #5: “Know your customer” (KYC) is posing a risk if you are just starting to trade on an exchange. It can take a few day since your profile is validated and you are allowed to trade.

Risk #6: Withdrawal limits might be a risk if you want to withdraw more funds than allowed at the exchange

Taxes

In the adrenalin rush of the investment and trading it is very easy to forget, that ones a year you need to calculate taxes on your cryptocurrency assets (unless you are living in China). The subject of taxation of the cryptocurrencies is very complex. The tax laws for natural person and legal entity are different. The tax laws are also different per country.

Taxes might actually reduce your profits and it is not easy to keep them in mind by posting a transaction order. By ignoring taxes, a crypto trader or crypto investor fails to get a very important piece of information to make a trade.

The taxes might be as simple as in the Netherlands, where cryptocurrencies are considered as a capital (overige bezittingen). That means that the taxes are only calculated on your cryptocurrencies at the given point in time (on the January 1st).

Or the taxes might be as complicated as in US, where cryptocurrencies are considered as assets, which means that you have to pay tax on every transaction.

https://cryptoresearch.report/crypto-research/taxation-cryptocurrencies-europe/

Conclusions

Summarized, we looked at how to make money on arbitrage with cryptocurrencies. Basically, we have identified 2 important steps. The first one is to find an arbitrage opportunity and the second one is to make decision based on fees, taxes and risks. The important factors to consider are

  • Exchanges: some have higher fees, some take longer transaction times, some are more trustworthy
  • Kind of arbitrage you want to execute, because for the arbitrage across exchanges you need relatively more investments than within one exchange.
  • Market volatility: some coins fluctuate more than others
  • Country of your residence, since as a private person or a company you might have to pay taxes
  • Your funds/investment in fiat and/or in cryptocurrencies on one or across exchanges. Remember, that to trade across exchanges you need to have a fiat and the crypto available on these in order to maximize your profits on arbitrage.

In my opinion it is also important to understand that you need several arbitrage transactions to cover your deposit, withdrawal fees and evenual taxes.

By taking into the account all these ingredients: fees, risks, taxes, you can increase your chances of success in the crypto arbitrage.

Bitcoin: how big is the bubble?

Is Bitcoin a bubble? Many posts suggest that Bitcoin is a bubble and compare it with the Dutch tulips bubble and dotcom bubble from the past. Some posts argue that Bitcoin is not a bubble and list all advantages of cryptocurrencies and features of this new technology. Few posts try to capture the intrinsic (or fundamental) value of Bitcoin. As mathematicians, we believe in hard numbers instead of subjective arguments. In this post we suggest a way to quantify the intrinsic value by estimating how much money has been invested in Bitcoin.

Why Bitcoin?

Why are we looking at Bitcoin, while there are more than a thousand cryptocurrencies out there?
A few reasons are that

  • Bitcoin has the largest market capitalisation which is nearly 50% of the total cryptocurrencies market share
  • most of the time you need Bitcoin in order to buy other cryptocurrencies

Apart from the intrinsic value, Bitcoin has a bunch of subjective added values that are impossible to quantify:

  • decentralisation – no more intermediate between you and the money, no centralised manipulation.
  • security – you can store Bitcoins on an offline computer/harddrive/USB (as a protection agains hackers), and still will be able to perform transactions online with just a signature.
  • transparency – all transactions are publicly published in the blockchain.
  • evolution – open source and public contribution make Bitcoin evolve with the current needs.
  • availability – cross-borders transactions 24/7.

The added value specifically for us as mathematicians is that the markets for crypto currencies are  not regulated. This is a big opportunity for machine learning algorithms: the raw data is “clean” (compared to a regular stock: no merges, no acquisitions, no dividend payments and other steps you need to calculate before you could actually see what is going on with the stock price).

 Market cap is not the right value estimator

How much value is in Bitcoin? This is a hot topic for every one who is trading cryptocurrencies. This question is impossible to answer, since the value of Bitcoin is determined by the trust and belief in the new technology. How do you estimate value of trust? Most of the articles I could find on the value of  Bitcoin are building up on the market capitalisation of Bitcoin.

Market capitalisation (or shortly: market cap) of Bitcoin is the total amount of circulating Bitcoins multiplied by the price of Bitcoin

 
At the time of writing this article the market cap for Bitcoin is just above 66 billion USD, which is larger than the CocaCola brand (55.6 billion USD).  The amount on Bitcoins in the circulation almost reached 80% of the 21 millions possible mined Bitcoins.

There are some calculations that “in 10 years Bitcoin’s market capitalization would be 10 times the average daily volume, giving a figure of $1.75 trillion for the market cap. In 10 years, the analyst thinks that there will be 17 million Bitcoin in circulation, up from the current 16.3 million figure. If the potential 17 million of Bitcoins in supply is divided by the $1.75 trillion market cap estimate, then each Bitcoin would be worth just over $100,000“.

There is a problem with the value of Bitcoin that is based only on market capitalisation. It only reflects the value at a given moment in time, with no reference to the price developments in the past. Moreover, it is nearly impossible to predict market cap in the future.

Comparison with gold and world money

Bitcoin is a currency with the fixed limit of total supply, which is 21,000,000 Bitcoins. Assuming the mining power stays constant, the last Bitcoin will be mined in 2140. Because of this property, Bitcoin is often compared to the gold.

The gold market is estimated at 8.7 trillions USD. If we assume that gold will be replaced by Bitcoins, then one Bitcoin should be worth ~414 thousands USD, which is 100 times more than the current price of Bitcoin.

The same calculations can be done with the world’s money that is estimated at 5.2 trillions of USD. If we assume that eventually Bitcoin will replace all the fiat money, then 1 Bitcoin will be worth ~247 thousands USD, which is 60 times more than the current price of Bitcoin.

If you do believe that Bitcoin will replace money or gold, then there is definitely room for growth. However, this is based on a belief. How can we quantify the value of a Bitcoin?

Calculating invested money in Bitcoin

We thought that the only value we could possibly estimate is the amount of invested money. At least this will give us a lower boundary for the intrinsic or fundamental value of Bitcoin.

“The intrinsic value is the actual value of a company or an asset based on an underlying perception of its true value including all aspects of the business, in terms of both tangible and intangible factors. This value may or may not be the same as the current market value. Additionally, intrinsic value is primarily used in options pricing to indicate the amount an option is in the money.” Investopedia

 
Unfortunately, also in this case we can not just read the invested amounts from the blockchain. So here is a way how to estimate them.

We know that several transactions are validated and combined in a block during Bitcoin mining. For this the miner gets a reward: 12,5 Bitcoins plus transaction fees. The amount of Bitcoins per block is halving every 4 years. Currently it is 12,5 Bitcoins, in 2020 it will be 6,25. Since transaction fees are negligible compared to the 12,5 bitcoins, we skip them in our calculations. Basically, every Bitcoin block “costs” 12,5 Bitcoins. If we assume, that the miner sells mined Bitcoins shortly after they are mined, then we can estimate the invested money for 1 block.

Investment per mined block = (the “cost” of the block) * (the Bitcoin price at the time the block was mined)


Now we take cumulative sum over all mined blocks and we get an amount of injected/invested money in the Bitcoin blockchain.

The rate of block creation is predetermined: about 6 blocks per hour. It is adjusted every two weeks, so that for our exercise we can assume it is constant over time. That means 144 block within a day. Note, that crypto markets are functioning 24/7 continuously. If we multiply the number of blocks per day by the investment per the block then we arrive at the investment per day.

Investment per day = (investment per mined block) * (number of blocks per day)


Summing up over all days of Bitcoin life, we finally obtain the cumulative injected money in the Bitcoin chain.

Of course, not all of the mined Bitcoins are sold shortly after they were mined. Because the blockchain does not contain information about it, we make an assumption to get an estimate.

To get the daily Bitcoin price, we use cryptocompare API and Python. This post describes nicely how to do it. The cryptocompare API provides the prices from 2010-07-17. To get earlier cumulative prices we consult blockchain.

The estimated injected dollars in the Bitcoin blockchain is about 3 billion USD (the final date is 2017-09-20). The market cap, however, at 2017-09-20 is just above 66 billion USD. For the prices this means that on average the Bitcoin community has paid about 200 USD for one Bitcoin, whereas the current price is just above 4100 USD.

How big is the bubble?

Is Bitcoin a bubble?

As you can see, the market cap is much larger than invested dollars, so the answer might be: yes, it seems like a bubble. If we take the market cap as 100% then invested dollars comprise roughly 5%.

How big is the bubble?

The difference between invested dollars and the market cap is factor of 20 (at 2017-09-20).

The figure below shows the historical market cap in USD as a green curve and cumulative injected dollars (USD) as a blue curve, respectively.  The horizontal axis represent the time starting at 2010-07-17 as the data is available at cryptocompare. The vertical axis represents the amount of USD on a logarithmic scale.

If we have a look at the development over time, then we see that before 2011 the market cap and the injected dollars were overlapping. However, starting from 2011 till now, the market cap curve is above the injected dollars curve with few exceptions. This means that Bitcoin might be currently overvalued compared to the investment. Of course as we mentioned earlier this is only one part of the intrinsic value, there are number of subjective added values that we can’t quantify.

Cumulative injected dollars versus market cap (in USD) on logarithmic scale

Conclusions

The technology in the crypto currencies is very promising and can be applied to many aspects, not only in finance. Whether Bitcoin is going to make it to the future is still a question. Right now Bitcoin is the market leader for the crypto currencies.

Whether Bitcoin is a bubble or not, the fact is that its market capitalisation is much larger than amount of invested dollars. If Bitcoin owners decide to sell more than 5% of their Bitcoins at the same time, the crypto currency exchanges will not be able to pay out.

What do you think is the real value of Bitcoin? Is Bitcoin a bubble? Let us know in the comments.

 


Liked this article? Get EZNumeric’s future articles in your inbox:

 

VCRS matrix compression for GPUs

Have you considered using compression to solve huge linear systems of equations? How can a lossy compression speedup numerical solvers without affecting accuracy? Do you need compression that will be suitable for GPU(s)?

Well, the good news is that we have developed a matrix compression scheme that you can use on GPUs. We call it VCRS – Very Compressed Row Storage. In this post, firstly, we focus on “why do we need matrix compression”. Secondly, we describe “what is VCRS” and “how to use it for compression”. Finally, we give few examples from a real-world applications.

Why matrix compression?

If you are doing modelling or simulation of a physical process, most of the time, you end up with differential equations describing this process. Very often we can’t solve these differential equations analytically in the continuous space.  Therefore, we need to discretise these and solve numerically. Discretisation can be viewed as representation of your differential equation as a system of linear equations in a matrix form

A x =b

where A is the matrix, x is the solution vector, b is the right-hand side vector.

Most of the time, the matrix A is huge (millions of rows) and sparse (few non-zero elements in a row). We can’t just invert this matrix to get the solution, since the matrix inversion is very costly in terms of memory and computational costs. To solve this system of linear equations, we use iterative methods. Very often we can speedup iterative methods using a preconditioner M^{-1} such that

M^{-1}A x = M^{-1}b

Basically, the preconditioner M^{-1} is a matrix which is close to the inverse of the matrix A, in other words M^{-1}\approx A^{-1}.

If M^{-1}A  is the identity matrix, then we automatically obtain solution of our system of linear equations.

Sometimes, the original matrix A can be implemented matrix-free, meaning that the elements are calculated on the fly and not stored. However, most of the time we have to store the preconditioner matrix M in the memory. The less storage both matrices take, the larger problems we can compute. It is especially important if we use accelerators, for example a GPU with limited memory.

An important aspect of a preconditioner is that it does not have to be exact. It is acceptable to have an preconditioner that is an approximation of an approximation.

Therefore lossy compression of the preconditioner enables bigger problems to be computed on accelerators.

There are many ways to compress a matrix. In this blog we suggest to use a compressed matrix M using VCRS compression. This way we take two pigeons with one bean:

  • speedup iterative solver with a preconditioner
  • suitable preconditioner for GPU(s)

What is VCRS?

VCRS stands for Very Compressed Row Storage format. This method was developed during Hans‘s PhD at TU Delft. VCRS format was inspired by the well-known CSR (Compressed Sparse Row) format.

CSR format versus VCRS format

To illustrate compression, let’s consider a small matrix A from a one-dimensional Poisson equation with Dirichlet boundary conditions, see picture above.

CSR format consists of two integer and one floating point arrays:

  1. The non-zero elements of the matrix A are consecutively, row by row, stored in the floating point array \text{data}.
  2. The column index of each element is stored in an integer array \text{cidx}.
  3. The second integer array \text{first} contains the location of the beginning of each row.

To take advantage of the redundancy in the column indices of a matrix constructed by a discretization with finite differences or finite elements on structured meshes, we introduce a new sparse storage format: VCRS.

VCRS format consists of five integer and one floating point arrays:

  1. The first array \text{col offset} contains the column indices of the first non-zero elements of each row.
  2. The second array \text{num row} consists of the number of non-zero elements per row.
  3. The third array is \text{col data} which represents a unique set of indices per row, calculated as the column indices of the non-zero elements in the row \text{cidx} minus \text{col offset}. Here, the row numbers 0 and 4 in the matrix A have the same set of indices, \text{col data} = {0 1}, and the row numbers 1 and 2 have the same set of indices as well, \text{col data} = {1 2 1}.
  4.  To reduce redundancy in this array, we introduce a fourth array \text{col pointer} that contains an index per row, pointing at the starting positions in the \text{col data}
  5. The approach from previous array 4 is also applied to the array \text{data} containing values of the non-zero elements per row, i. e., the set of values is listed uniquely.
  6. Therefore, also here we need an additional array of pointers per row \text{data pointer} pointing at the positions of the first non-zero value in a row in \text{data}

Why VCRS is better than CSR?

At a first glance, it seems that the VCRS format is based on more arrays than the CSR format, six versus three, respectively. However, the large arrays in the CSR format are cidx and data, and they contain redundant information of repeated indices and values of the matrix. For small matrices, the overhead can be significant, however, for large matrices it can be beneficial to use the VCRS, especially on GPUs with a limited amount of memory.

Summarizing, the following factors contribute to the usage of the VCRS format:

  1. The CSR format of a large matrix contains a large amount of redundancy, especially if the matrix arises from finite-difference discretisations;
  2. The amount of redundancy of a matrix A can vary depending on the accuracy and storage requirements, giving the opportunity to use a lossy compression;
  3. The exact representation of matrices is not required for the preconditioner, an approximation might be sufficient for the convergence of the solver.
The lossy compression of preconditioners can be very beneficial for a GPU-implementation, as it allows to store the data on hardware with a limited amount of memory, but at the same time takes advantage of its speed compared to CPU hardware.

VCRS with lossy compression

Of course you can already use VCRS format as described above. If you want to get even more advantages of the VCRS format, here we list two mechanisms to adjust the data redundancy:

  • Quantization
  • Row classification

Quantization

Quantization is a lossy compression technique that compresses a range of values to a single value. It has well-known applications in image processing and digital signal processing.

By lossy compression, as opposed to lossless compression, some information will be lost.

However, we need to make sure that the effect of the data loss in lossy compression does not affect the accuracy of the solution. The simplest example of quantization is rounding a real number to the nearest integer value.

The quantization technique can be used to make the matrix elements in different rows similar to each other for better compression. The quantization mechanism is based on the maximum and minimum values of a matrix and on a number of so-called bins, or sample intervals.

Figure above illustrates the quantization process of a matrix with values on the interval [0, 1]. In this example the number of bins is set to 5, meaning there are 5 intervals [ 0.2(i-1),\ 0.2i\ ),\ i = 1, 2,3,4, 5. The matrix entries are normally distributed between 0 and 1, as shown by the black dots connected with the solid line. By applying quantization, the matrix values that fall in a bin, are assigned to be a new value equal to the bin center. Therefore, instead of the whole range of matrix entries, we only get 5 values. Obviously, the larger number of bins, the more accurate is the representation of matrix entries.

Row classification

Next, we introduce row classification as a mechanism to define similarity of two different matrix rows.
Given a sorted array of rows and a tolerance, we can easily search for two rows that are similar within a certain tolerance. The main assumption for row comparison is that the rows have the same number of non-zero elements.

Let R_i = \{ a_{i1} \; a_{i2} \; \dots \; a_{in}\} be the i-th row of matrix A of length n and R_j = \{ a_{j1} \; a_{j2} \; \dots \; a_{jn}\} be
the j-th row of A.

The comparison of two rows is summarized in Algorithm 3 below. If R_i is not smaller than R_j and R_j is not smaller than R_i, then the rows R_i and R_j are “equal within the given tolerance \lambda“.
Algorithm 4 then describes the comparison of two complex values and Algorithm 5 compares two floating-point numbers.

Figure bellow illustrates the classification of a complex matrix entree a_{ij}. Within a distance \lambda the numbers are assumed to be equal to a_{ij}.  Then, a_{ij} is smaller than the numbers in the dark gray area, and larger than the numbers in the light gray area.

Impact of quantization and row classification

The number of bins and tolerance have influence on

  1. the compression: the less bins and/or the larger lambda, the better is the matrix compressed
  2. the accuracy: the more bins and the smaller lambda, the more accurate are matrix operations (such as matrix-vector multiplication)
  3. the computational time: the less bins and/or the larger lambda, the faster are computations
  4. the memory usage: the less bins and/or the larger lambda, the less memory is used
  5. the speedup on modern hardware (which is calculated as a ratio of the computational time of the algorithm using the original matrix and of the computational time using the compressed matrix)
There is a trade-off between performance and accuracy: the more accurate the compressed matrix, the slower matrix-vector multiplication.

Example: Helmholtz equation

From our previous posts, you might know that one of the problems we had to solve is the Helmholtz equation – the wave equation in the frequency domain.

 

Real part of the preconditioner

Imaginary part of the preconditioner

Using the VCRS format to store this matrix results in three to four times smaller memory requirements and three to four times faster matrix-vector multiplication, depending on the compression parameters. In general, we observed the compression factor between 3 and 20 depending on the matrix.

Based on our experiments, the most reasonable parameter choice would be a tolerance λ = 0.1, and number of bins equals to 100 000.

Summarising our experiments for the whole solver (Bi-CGSTAB preconditioned with shifted Laplace matrix-dependent multigrid method), we can conclude that the VCRS format can be used

  • to reduce the memory for the preconditioner as well as
  • to increase the performance of the solver on different hardware platforms (CPUs, GPUs),
  • with minimal effect on the accuracy of the solver.

Example: Reservoir simulation

The VCRS compression can also be used in other applications where the solvers are based on a preconditioned system of linear equations. For example, an iterative solver for linear equations is also an important part of a reservoir simulator.

It appears within a Newton step to solve discretized non-linear partial differential equations describing the fluid flow in porous media. The basic partial differential equations include a mass-conservation equation, Darcy’s law, and an equation of state relating the fluid pressure to its density. In its original form the values of the matrix are scattered. Although the matrix looks full due to the scattered entries, the most common number of non-zero elements per row is equal to 7, however the maximum number of elements per row is 210.

The distribution of the matrix values is shown in the figure below. Note, that the matrix has real-valued entries only. It can be seen that there is a large variety of matrix values, that makes the quantization and row classification effective.

Using the VCRS format to store this matrix results in two to three times smaller memory requirements and two to three times faster matrix-vector multiplication, depending on the compression parameters. 

Summary

In this post

  • we introduced a VCRS (Very Compressed Row Storage) format,
  • we have shown that the VCRS format not only reduces the size of the stored matrix by a certain factor but also increases the efficiency of the matrix-vector computations,
  • we introduced two parameters for lossy compression: number of bins and lambda,
  • we concluded that with proper choice of these compression parameters, the effect of the lossy compression on the whole solver is minimal,
  • we used VCRS compression on CPUs and GPUs,
  • we applied VCRS on real-world applications.

What matrix compression are you using? Let us know in the comment box below.


Liked this article? Get EZNumeric’s future articles in your inbox: