Block Distribution Network
Here, we provide a more detailed overview of components of the Block Distribution Network. In particular, we cover Gateways, Relays, the Control Plane, and Remote Blockchain Nodes.
The bloXroute Gateway is an application that connects your blockchain full node to the bloXroute BDN. Using the Gateway, the BDN is designed to propagate blocks and transactions to your full node faster than the traditional peer-to-peer blockchain network. It is recommended that you install the Gateway on the same server as your full node, or on a server with low latency to your full node.
The Gateway software is written in a combination of Python and C++. It may be installed by downloading a Docker Container with the gateway already installed, or via installation using pip.
While the Gateway software communicates using the blockchain protocols, it does not maintain the blockchain state. Accordingly, the Gateway may be connected to a remote blockchain node, explained later in the text, to proxy messages when requested by the user’s blockchain node.
How the gateway decides on best relay to connect to
Gateways are responsible for determining which relay to connect to from a list provided by the Control Plane. We explain in more detail both the relay and the Control Plane components of the BDN later in the text.
When starting a gateway, a user may specify the IP address of the Gateway as a command line parameter. If the user does not specify an IP address at startup , the Gateway will attempt to determine its own IP address by reaching out to a remote IP detection service. If the Gateway cannot determine its IP address, it terminates itself with an error instructing the user to provide the IP address at the command line.
After determining it’s IP address, the Gateway connects to the Control Plane and provides its IP address. The Control Plane then looks up the geographic region of the IP address using an IP geolocation service. Based on the determined geolocation of the IP, the Control Plane returns a list of potential relays for the gateway to connect to. The list of potential relays contains the top relays closest to the gateway, sorted by distance. If no location is returned by the IP geolocation service, the Control Plane will send the Gateway a list of potential relays including two relays from North America, and three relays from other regions.
Using the list of potential relays, the Gateway checks the latency of the provided relays and connects to the block and transaction relays with the lowest latency. Latency is measured using ping latency between the gateway and each relay. In addition to connecting to the relay with the lowest ping, the gateway additionally stores at the second lowest latency pair of transaction and block relays as a backup in case the initial relay connections are lost.
Relays form the backbone of the bloXroute BDN. Gateways connect to Relays to receive notifications of new transactions and blocks faster than they would otherwise be received from the peer-to-peer network. bloXroute Relays are located around the globe at points optimized to provide faster propagation of transactions and blocks between Gateways.
In order to provide improved block propagation, bloXroute uses a split relay design. In the split relay design, the BDN includes two types of relays, transaction relays and block relays. As the names suggest, transaction relays propagate transactions and block relays propagate blocks. The split relay design ensures that transactions do not interfere with the propagation of blocks, and vice versa.
Currently, transaction relays and block relays are paired on a single machine and IP. When registering with the Control Plane, the Gateway receives a single IP and port for a pair of relays. The Gateway then attempts to connect to the block relay on the given port and connect to the transaction relay on the next sequential port. (For example, if the block relay is on port 1609, then the transaction relay will be on port 1610.)
When attempting to connect to the bloXroute BDN, Gateways register with the BDN Control Plane. The Control Plane then checks for a geographic region of the Gateway’s IP address. The Control Plane uses the IP address and region to determine suggested relays for the Gateway to connect to. The Gateway may then select from the provided relays to determine a relay to connect to. As described above, the default Gateway behavior is to ping each of the provided relays and to select the relay with the lowest ping.
Gateways connect only to a single relay of each type (block and transaction) in the bloXroute BDN. If the Gateway loses a connection with a relay, the Gateway may connect to a different relay to maintain its connection with the bloXroute BDN. The Control Plane also provides the Gateway with IP addresses of other Gateways it is aware of, so that the Gateways can form a P2P network independent of the BDN Control Plane. Gateway are designed to maintain the P2P network even if the Control Plane fails.
Remote Blockchain Nodes
bloXroute Gateways communicate with blockchain nodes using their native peer-to-peer protocols but are not themselves full blockchain nodes. Thus, while a Gateway can, for example, send and receive a Bitcoin blocks from a Bitcoin full node, it cannot validate that the block is valid. However, blockchain nodes frequently make requests to a connected Gateway that the Gateway cannot fulfill because it does not fully validate the chain. In order to permit the Gateway to respond, bloXroute Gateways may connect to remote blockchain nodes, run by bloXroute, to fulfill certain requests by its blockchain node.
bloXroute runs multiple versions of the bloXroute BDN. For example, bloXroute maintains a testnet BDN and a mainnet BDN. The mainnet is bloXroute's production network meant for use by miners, wallets, and any user that runs a full node. The bloXroute testnet is the test environment for future versions of bloXroute Gateways, relays, and associated infrastructure. bloXroute maintains separate Control Planes for each network. See the
sdn-url command line parameter in the Command Line Arguments section for details on connecting to a different network.
Denial of Service Protections
In order to protect against denial of service attacks, relays include several protective measures. One important anti-DOS measure employed by the relays is rate-limiting.
bloXroute implements rate-limiting using a token bucket algorithm. The token bucket algorithm is based around the analogy of a bucket of tokens that permit the passage of a fixed amount of data. Tokens are added to the bucket at a fixed rate, and tokens are removed upon receiving messages from a client.
Each relay connection has an associated token bucket with a bucket size and a bucket fill rate. Connection priority is determined by the token count available in a connection’s bucket as recorded in a connection priority list. Every time the relay parses a message from the connection, tokens are removed from the bucket for that connection. A relay will periodically refresh its connection priority list and check for suspicious behavior.
If the bucket for a connection is fully depleted, the relay will suppress incoming messages for cooldown period and remove the connection from the priority list. The removed connection will be re-added to the connection priority list after the cooldown period ends. The cooldown period is the time needed to fill the bucket back completely.
Additionally, the relay maintains a global bucket for all connections. Every time the relay parses a block or a transaction message from any connection, tokens are removed from the global bucket as well. If the global bucket is emptied, the relay removes connections from the bottom of the connection priority list. The cooldown period for each connection will be subtracted from the global bucket cooldown period and the relay will continue depleting connections until the total period reach 0 or no more connections are available in the priority list .
Each connection connected to a relay is assigned to one of three groups that determine its bucket size and bucket fill rate. Connections in a trusted group are known connections from valid block miners or valid transaction providers. Connections in the trusted group are given above normal bandwidth. The default group for connections is the untrusted group. The untrusted group receives normal bandwidth. For connections that are suspected of being part of a denial of service attack there is a malicious group. The malicious group is assigned very limited bandwidth via a low bucket size and bucket fill rate. Connections may move between the default untrusted group, the trusted group, and the malicious group based on their behavior. Additionally, the parameters for each group vary based on the blockchain of the connection to reflect the differing needs and uses for each blockchain.