Common Problems
We'll continue updating this page so you can easily troubleshoot your node.
How does this page work?
This page is designed to help you fix all of your problems on your own!
First of all you need dump recent logs from your terminal.
journalctl -f -u "chainflip-*"
In order to quickly understand your issue, please first check if your log output resembles any of these.
Not Funded
You will see this log before the Chainflip network witnesses that you added funds to your validator. Once you have successfully added funds, it can take a while for the network to register it. Your node might appear Offline
until then.
Logs
{
"tag": "",
"msg": "Your Chainflip account cFNugEinCy5Zxv44B4o8uU4PSigKmmzdUYabbZbfNjAAZx18a is not funded. WAITING for account to be funded at block: 143222",
"level": "warn",
"ts": "2022-11-18T09:45:37.474214013+00:00",
"component": "StateChainConnector"
}
Solution
Please make sure you followed all the steps outlined in Funding.
If after 30 minutes your engine is still not online, try restart your engine
sudo systemctl restart chainflip-engine
.
Incorrectly configured the ETH RPC
This can have a number of possible issues.
Logs
{"tag":"","msg":"Connecting new HTTP web3 client to https://eth****","level":"debug","ts":"2022-11-18T09:25:21.576142234+00:00","component":"Eth-DualRpcClient"}
Error: Failed to create EthDualRpcClient
Caused by:
Inconsistent chain configuration. Terminating.Expected ETH chain id 5, received 1 through WebSocket., Expected ETH chain id 5, received 1 through HTTP.
Solution
You have not selected the correct Ethereum network. In this case, you have selected mainnet
. The Chainflip network uses Sepolia. Please reread #pointing-to-the-ethereum-client.
Logs
{"tag":"","msg":"Connecting new WebSocket web3 client to wss://eth****","level":"debug","ts":"2022-11-18T10:09:02.643481895+00:00","component":"Eth-DualRpcClient"}
Error: Failed to create EthDualRpcClient
Caused by:
0: Failed to create EthWsRpcClient
1: Error: 'web3::transports::WebSocket::new(ð_settings.ws_endpoint).await' with type 'core::result::Result<web3::transports::ws::WebSocket, web3::error::Error>' failed at https://github.com/chainflip-io/chainflip-backend/tree/5e37992a14bb14d8d0609687ebe8f72e9402fbc0/engine/src/eth/rpc.rs#L266#C5
2: code 401
Solution
You have entered your RPC endpoint in incorrectly. Please make sure you copy it over without any mistakes.
Logs
{"tag":"","msg":"Connecting new WebSocket web3 client to wss://SOM****.sepolia.ws.rivet.cloud","level":"debug","ts":"2022-11-18T10:14:58.473000904+00:00","component":"Eth-DualRpcClient"}
Error: Failed to create EthDualRpcClient
Caused by:
0: Failed to create EthWsRpcClient
1: Error: 'web3::transports::WebSocket::new(ð_settings.ws_endpoint).await' with type 'core::result::Result<web3::transports::ws::WebSocket, web3::error::Error>' failed at https://github.com/chainflip-io/chainflip-backend/tree/5e37992a14bb14d8d0609687ebe8f72e9402fbc0/engine/src/eth/rpc.rs#L266#C5
2: Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 337047686, library: "SSL routines", function: "tls_process_server_certificate", reason: "certificate verify failed", file: "../ssl/statem/statem_clnt.c", line: 1913 }]))) }, X509VerifyResult { code: 62, error: "Hostname mismatch" })
Solution
You have copied the chainflip-engine
config without changing any of the values! Please re-read the steps in Engine Settings . You need to make that you use the URL from your Ethereum Client. #pointing-to-the-ethereum-client.
Node killed Out Of Memory
Logs
If your node logs contain the following error:
chainflip-node.service: Main process exited, code=killed, status=9/KILL
It means that your node is running OOM and it is getting killed by the OS.
Solution
You need to create a swapfile (8GB), you can follow this guide in order to do so: https://linuxize.com/post/create-a-linux-swap-file/ (opens in a new tab)
Auctions App
Node Offline — Reasons
![validator offline](/perseverance/validator_offline.png)
Likely you haven't completed all the steps. Please start from Generating Keys section and work your way very slowly, making sure you have read everything.
It can be, that you did not successfully submit the chainflip-cli
commands. Please run all the commands in Funding and Bidding section.
Turning Authority Online
![validator online](/perseverance/validator_authority.png)
You need to wait until the next Rotation. Make sure your node is Online with the labels Backup and Bidding and Qualified.
![validator backup](/perseverance/validator_backup.png)
Authority Offline — Reasons
![](/perseverance/validator_authority_offline.png)
Your engine has stopped submitting Hearbeats. This means either your node, your engine or both have crashed or are misconfigured.
Verify this with:
sudo systemctl status chainflip-node
sudo systemctl status chainflip-engine
Now check the logs:
journalctl -f -u "chainflip-*"
Backup and Bidding but not Qualified — Reasons
![validator backup](/perseverance/validator_backup.png)
You did not successfully submit the chainflip-cli
commands. Please run all the commands in Funding and Bidding section.
The commands that already ran successfully will fail with an error resembling the following:
![validator cli commands failing(command already executed before)](/perseverance/validaotr_cli_commands_failing.png)
This is expected, continue with the next command until you complete them all, after completing this your node should appear Backup and Bidding and Qualified.
Ethereum Keyfile Error
thread 'main' panicked at 'called Result::unwrap() on an Err value: Failed to decode Ethereum Private Key file at /root/chainflip/ethereum_key_file
This is due to your key likely either having 0x
on the front, or having a newline at the end. You can use nano
to edit the file. Make sure to include the -L
argument, otherwise nano will save the file with a new line at the end! Make sure that you replace /path/to/ethereum_key_file
with the correct file path on your system.
nano -L /path/to/ethereum_key_file
FailedToBroadcastTransaction
If your node fails to broadcast a transaction there can be multiple reasons for that:
- Your validator is out of
sETH
- Your connection or the connection to the endpoints is unstable
This can happen from time to time (due to temporary connection problems) and it results only in a loss of reputation, but if this keeps happening then be sure to top up you sETH and check the health of your endpoints.
Your validator wallet is out of sETH
This is a very common problem if you have joined the network early.
Make sure you have at least 0.1 sETH at all time in your validator's wallet. Otherwise you'll fail to submit transactions to the Ethereum network.
Why Slash? Why low reputation?
Slashoor has paid you a visit, hasn't it?
Overview
If you find yourself being penalized and your reputation points dropping, it is possible that the chainflip-engine is not functioning optimally. This discussion will focus specifically on ETH witnessing issues.
One potential cause is using a free-tier managed service for your RPC endpoint provider (such as Alchemy, Rivet, or Infura). This can lead to rate-limiting problems based on network activity and the stability of your engine. Rate-limiting models differ among services. Some have a fixed daily request limit, while others offer unlimited daily requests but impose a cap on "compute units" per second. These varying models may result in different behaviors under certain conditions.
If you encounter rate-limiting, your engine will be unable to witness ETH events, which can lead to immediate penalties or reductions in reputation points, potentially culminating in slashing. Additionally, failure to properly witness ETH events can cause rotations to take longer and reduce your chances of becoming an Authority member.
Solution
You have multiple options here:
Using Two Different ETH RPC Providers
This is a good method for setting up your RPC configuration. Sign up for two accounts, such as Infura (opens in a new tab), Rivet (opens in a new tab), and Alchemy (opens in a new tab) (choose two) and update your engine configuration to use Infura for the rpc
and Rivet for backup_rpc
, or vice versa, for each chain. This approach ensures protection against service outages from either provider.
Subscribe for a paid tier at your RPC Endpoint provider
Subscribe to a paid tier at your RPC endpoint provider: Though this option incurs a monthly fee, it requires minimal effort and is recommended for those who want a hassle-free start without setting up their own ETH nodes or worry about rate-limiting.
Run your own ETH node and expose it as an RPC endpoint
This option, which involves connecting directly to the ETH network, is the most reliable solution. However, it demands strong technical skills for setting up and operating ETH nodes, which is beyond the scope of this guide. For more information, visit: https://geth.ethereum.org/docs/getting-started (opens in a new tab)
Updating your config file
After setting up your RPC endpoints as described above, you need to update the config file and restart your engine. You can do this by running:
sudo nano /etc/chainflip/config/Settings.toml
Your config file will look something like this:
![](/perseverance/image (9).png)
Make sure that both endpoints (HTTP and WebSockets) are accessible. Otherwise your engine will fail to start.
Save your changes and exit then restart the engine by running:
sudo systemctl restart chainflip-engine.service
Missing Authorship Slots
A common problem with the chainflip-node
is missing an authorship slot which basically means the node has missed authoring a block and as a result missed the rewards of doing so. This penalty is not as severe as a slash but it will cost you some reputation points as well as missing on the block authoring rewards.
This shows as an Offence Penalty event in the block explorer with the reason being MissedAuthorshipSlot
.
Potential Causes
-
Node errors: Always start by checking for any errors in
chainflip-node
logs by running:journalctl -f -u chainflip-node
-
Node is out of disk space: Check your disk space by running
df -h
and make sure you have enough space for the node to operate. -
Session Keys are missing or corrupted: You can check that by running the following:
ls -la /etc/chainflip/chaindata/chains/Chainflip-Perseverance/keystore/
You should see something like this:
total 24
drwxr-xr-x 2 root root 4096 Jan 3 12:09 .
drwxr-xr-x 5 root root 4096 Dec 7 18:34 ..
-rw------- 1 root root 80 Jan 3 12:09 6175726140fc6d50c7b5a8fefaff58d763f3f46fb753ae457530bc6def3ca4a8477be50d
-rw------- 1 root root 77 Dec 7 18:35 61757261be7e07fdd5314a89b9c353b8730593ce98c8d728eacff1db065320854a49cc49
-rw------- 1 root root 77 Dec 7 18:35 6772616e669a751d3d129605145a9876e06034469e680db4c0da4b1d442c249bcebe5094
-rw------- 1 root root 77 Jan 3 12:09 6772616e9d6d1af0f2a0ce1c0ac1faaa07c835e42d0be799bbf98dfcb7e8edb7192093df
If you see any files with size 0 or any files that are not named like the ones above, then you'll need to rotate your keys by running:
sudo chainflip-cli --config-root /etc/chainflip rotate
This will take effect in the next epoch and you'll continue to miss authorship slots until then.
- System Clock is not synced: If your system clock isn't synced with the network, then you'll miss authorship slots. The reason is that the block you are trying to author will be rejected by the network because it's timestamp is not within the allowed range. You can check your system clock by running
date
and you can sync it by running:
sudo apt-get update
sudo apt-get install ntp
sudo systemctl restart ntp
sudo systemctl status ntp
ntpq -p
Make sure your firewall allows NTP outbound traffic (Port 123 UDP).
It still doesn't work?
Follow instructions under Submitting an issue.
Please do this step before raising an issue on Discord. It makes our lives much much easier.
ParticipateKeygenFailed
Problem
Your validator is consistently failing to participate in keygen and as a result gets penalized because of the following offence: ParticipateKeygenFailed
.
It is likely that your engine isn't communicating properly with other engines in the network. This can be due to a number of reasons, such as a misconfigured firewall, or a misconfigured engine config.
Solution
Start by making sure the port 8078
is open and accessible from the internet. You can do this by using online tools like PortChecker (opens in a new tab), or by running the following command from another machine:
telnet <your-validator-public-ip> 8078
You should see something like this:
Trying <your-validator-public-ip>...
Connected to <your-validator-public-ip>.
Escape character is '^]'.
Connection closed by foreign host.
If you don't see the above, then you'll need to open the port 8078
on your firewall.
Is the port open but you are still getting slashed? In that case you'll need to check your engine config under /etc/chainflip/config/Settings.toml
and make sure that the ip_address
and port
under the [node_p2p]
section are set correctly.
FailedToWitness
Problem
Validators are expected to witness calls from external chains accurately and in time. When enough validators have witnessed an external event the appropriate action is then dispatched.
This means that all Validators are expected to witness external states on time. Given a validator could theoretically choose to sit out of voting if they know that enough validators have already witnessed the call, this could lead to a situation where validators are incentivized to wait until the last moment to witness a call.
Thus, we have a FailedToWitnessInTime
penalty. This penalty is dispatched after the following:
-
A call that requires witness is triggered on the external chain.
-
Some nodes has witnessed the call and makes the
witness_at_epoch
extrinsic call. -
After certain amount of time as passed, a certain threshold has been reached (currently 2/3), the call is dispatched. At this moment, a deadline is set for all remaining validators - the call must be witnessed by the remaining nodes before this deadline is up. This deadline (
current_block
+LATE_WITNESS_GRACE_PERIOD
) is currently set at 10 blocks. -
When the deadline is reached, all nodes that has not witnessed this specific call will be reported with the offense of
FailedToWitnessInTime
, and their reputation will be deducted accordingly.
Solution
Usually failing to witness a call indicates networking problems. Chain Events are emitted to help the operators to identify the potential causes for the failure. specifically:
Then to find the specific call:
-
From your authority's page on scan.chainflip.io and under the "Penalties" tab you will see the block number associated with
FailedToWitnessInTime
. Click on that to see all the events in the block. -
Search the page for
ReportedWitnessingFailures
. There may be multiple. Expand these and find if your account id is listed under any of them. Note this block number and call hash. -
Go to the block number specified using the search bar of the explorer.
-
Search the page for that hash. Fastest would be too inspect the page and search the HTML for the hash - Note that if you're not witnessing chain-tracking for a particular chain, this process won't help. So if you don't have a match, it's probably chain tracking which is your issue. Which could mean one of your RPC node is slow.
NB: Witnessing will be changed quite substantially in the future, which should help make detection of these issues much easier in the future.