Security & Redundancy
Read the Docs
We highly recommend reading our docs thoroughly to understand how the Chainflip network operates.
Applying changes to your Node or Engine
The Chainflip network relies on validators running two binaries, the chainflip-node and the chainflip-engine. The chainflip-node is a blockchain node, responsible for communicating with and authoring blocks on the Network; the chainflip-engine runs alongside the node, and is responsible for witnessing external block chain events, and maintaining the validator's share of a the sharded threshold signature key underlying the security of Chainflip's vaults.
You can think of the chainflip-node as a node of any other proof-of-stake blockchain. It has to be up all the time and connected to the network.
The chainflip-engine is a bit different. It acts as a multi-signature crypto wallet and it performs its cryptographic operations using its share of the key generated in the current authority set. The chainflip-engine has also to be up at all times.
You might want to update your engine config to use a different RPC endpoint or to update the public IP address of your validator node. Or you might want to start your node with additional flags or update any other node config. That is generally fine, but you should be aware of the following:
DO NOT restart the chainflip-node and the chainflip-engine at the same time, unless absolutely necessary. This means you should decouple the node and engine deployments if you are using container orchestration tools like Kubernetes or Docker Compose.
- In case you want to change the location of your chaindata, you can do so by following these steps:
- Stop your
chainflip-node(This will make the engine enter a crash loop. That is expected due to the state chain node not being available.) - Move the chaindata to the new location
- Update the location of the chaindata using the
base_pathargument in/etc/systemd/system/chainflip-node.d/override.conf - Start the
chainflip-node
- If you want to change the location of your engine keyshares (located at
/etc/chainflip/data.dbby default), you can do so by following these steps:
- Stop your
chainflip-engine - Move the engine keyshares to the new location
- Update the location of the engine keyshares in the
signing.db_fileitem in/etc/chainflip/config/Settings.toml - Start the
chainflip-engine
Backups
It is crucial to have a backup and recovery strategy in place due to the possibility of data loss or corruption caused by hardware failures, unexpected interruptions from cloud providers, loss of access to the machine, or accidental deletion. It is also important to understand the separate, yet complimentary, concepts of providing security and redundancy to the Chainflip Network and also to the Validator's $FLIP bond. There are absolute minimum backup requirements and some best practices to increase your chances of a successfully recovery.
Absolute minimum backup requirements
Validator's $FLIP Bond
Backing up and securing the Validator's $FLIP bond is akin to backing up any cryptocurrency wallet. The seed-phrase that was created by generating validator keys must be securely and remotely stored; ideally in a physically secure location offline. This seed-phrase will be required to recover your Chainflip account keys and your Ethereum keys. This is the absolute minimum requirement to backing up your Validator's $FLIP bond.
Chaindata and Session Keys
Chaindata and session keys are stored under /etc/chainflip/chaindata/chains/Chainflip-Berghain/. All of this data is required to author new blocks and authenticate to the p2p network. If any of this data is lost or corrupted, the chainflip-node will not function correctly and your account will not be able to author blocks and as a result will not be rewarded for the rest of the epoch. This can also lead to getting slashed until the next epoch where you will be rotated out.
Session keys are used by the chainflip-node for authoring blocks and are stored in the substrate keystore. By default the keystore is a directory is under the chaindata directory.
It is critical to the Chainflip network that Validators back up their session keys and chaindata after each rotation. If you lose these irrecoverably, your node will miss out on up to two Epochs' worth of rewards; it won't be able to author blocks for the remainder of the current epoch; and it will be banned from the next keygen ceremony.
If you do lose your session keys irrecoverably, run sudo chainflip-cli --config-root /etc/chainflip rotate to generate and register a fresh key (assuming you still have access to the VPS and the data is not corrupted). However note that key cannot author blocks until the following epoch.
To ensure faster recovery of the chainflip-node's ability to author blocks for the network, it is important to back up your session keys located at: /etc/chainflip/chaindata/chains/Chainflip-Berghain/keystore/.
Backing up the chaindata can speed up recovery in case of data loss, however this is not essential: the data can be recovered from the network. The fastest way to recover by syncing to the network is to use the --sync=warp argument when syncing from scratch.
Validator Keys
By default, these keys are stored under /etc/chainflip/keys. These are the keys used by the chainflip-node and chainflip-engine to communicate with and authenticate over the network.
These keys are extremely important and should be treated with the utmost care. Losing them can result in a loss of funds and compromise the security of your validator. It is crucial to keep them safe and secure.
If you do lose access to these keys, you can recover your validator account and restore it to a state where it can participate in the network again.
However, there are important considerations to keep in mind:
- These keys are separate from the session keys and engine key shares. The latter are not recoverable from a seed.
- If you only backed up your validator keys without backing up your session keys or engine key shares, your validator will be penalized for missing authorship slots even if you rejoin the network and submit heartbeats. This is because session keys are rotated at the beginning of every epoch, so your rejoining will not be effective until after the next rotation.
- The same applies to engine key shares. Losing these keys will prevent you from participating in keygen ceremonies or performing any multi-signature operations. Failing a keygen will result in penalties and a potential ban, which could lead to your validator node being removed from the authority set. In such cases, you would have to wait for the next auction to rejoin.
- If you think the keys might be compromised, we recommend that you immediately secure your FLIP funds by binding a redeem address, stopping your bid and then, when your funds are unbonded, redeeming them to your wallet before funding a new node with new keys.
Engine Keyshares
The keyshares are used by the chainflip-engine are stored in a database. By default this is located at /etc/chainflip/data.db.
Note it is not possible to recover these key shares from a seed if they are lost, so it it crucial to back up this database, ideally at the beginning of each epoch.
In order to secure the Chainflip Protocol, it is imperative for Validator operators to continuously back up their engine keyshare database as the network generates fresh keys for every epoch.
The key shares collectively secure all the funds on the Chainflip network so should be treated with the utmost attention to their safety and security.
Best Practices
To provide another layer of redundancy for a quick and easy recovery, we suggest configuring hard disk snapshots. Additionally, it is recommended to mount three volumes and use them to store the chaindata, validator keys and engine keyshares. This adds an extra layer of resilience in case of computing layer failures and simplifies the recovery process.