Tezos suffers minor glitch, block production stops for nearly 90 minutes
Tezos suffered a minor glitch on Friday and the block production stopped for nearly 90 minutes. According to a Reddit post, the waiting time was between at level 200703 which was pushed at 2018-11-23T 16.01.03z and level 200704 which was finally baked at 2018-11-23T 17:14:33Z.
The post goes on to add that the normal average time between blocks is between 60 and 104 seconds, there were other occurrences with similar implications since the Tezos protocol’s betanet’s launch.
The issue was related to one of the double baking instances that happened recently. The baker under the public key tz1KksC8RvjUWAbXYJuNrUbontHGor26Cztk committed a safety fault at the level 194934, which was later denounced, to the loss of security deposits, rewards, and fees for the baker. These security deposits constitute of the bakings and endorsements slots until that height. So, even if the baker committed only one conflicting signature, the entire amount of all the slots are slashed.
Reason for the temporary halt
Someone on Reddit also shared that the bakers stopped all active banking instances to stop double baking. Then they restarted another instance to avoid missing bakings and endorsements for the rest of the year. However, this opened the opportunity for another denunciation to be included. Well, this time it was the slashing of the security deposits of the remaining blocks and endorsements from the rest of the cycle. As the baker noticed the second denunciation, they decided to shut down even the initial baker for the rest of the cycle and this was the main reason that led to the temporary break in operations.
Solution to the problem
The baker mentioned above stopped all baking nodes. By protocol, all bakers must submit all of the nonces at the beginning of the next cycle or anytime during the cycle. Nonetheless, as the baker shut down all the validating nodes, the nonces file was not shared yet. At the beginning of cycle 48, the baker failed to remove the nonces, leading the stalling of the network at level 200703. The state machine then checked and found out that there was a missing nonce from that of the baker and it executed the function to slash the rewards.
However, the attempt failed as there were no more rewards to be slashed. According to the post, this is a bug to be fixed in future Tezos protocol releases. The fix to this problem was to inject the nonce manually. This could have been done by the baker as long as the baker provided the value. The nonce was injected and the network returned back to normalcy.
Image via Shutterstock