Skip to main content

Load Balancing With Endpoint Pools

Endpoint pools automatically load balance traffic between their endpoints. Load balancing is applied in two places:

  • When pooled public and kubernetes endpoints load balance traffic received from external clients.
  • When pooled internal endpoints receive traffic that is forwarded to them via the forward-internal Traffic Policy action.

Granularity​

ngrok intelligently chooses the load balancing layer automatically for you.

Traffic to HTTP and HTTPS endpoints balanced on a per-request basis (layer 7). If multiple endpoints in a pool have different on_tcp_connect phases in their Traffic Policy, load balancing is instead done on a per-connection basis (layer 4).

Default load balancing strategy​

ngrok's default balancing strategy attempts to strike a balance between performance and faireness by prioritizing the following:

  1. Identify all endpoints in the pool which are in the point of presence closest to where a connection is received.
  2. Balance the connections in a random distribution among those endpoints.

For the purposes of the first step, cloud endpoints are considered to be online in all points of presence.

Example​

If an endpoint pool consist of the following endpoints:

  • Endpoint A: Agent Endpoint connected to eu
  • Endpoint B: Agent Endpoint connected eu
  • Endpoint C: Agent Endpoint connected us
  • Endpoint D: Cloud Endpoint

Then:

  • A connection received in eu will be balanced among endpoints A, B, D
  • A connection received in us will be balanced among endpoints C, D
  • A connection received in jp will be balanced among endpoints D

Connection retries with load balancing​

Currently, there's no support for connection failover with endpoint pools. Failed connections to an endpoint in a pool are neither retried not sent to other endpoints in the pool.

Custom Strategies​

Coming Soon

Custom load balancing strategies are not yet generally available. Request access to the developer preview.

When balancing traffic to internal endpoints, you may define your own balancing strategy by setting the endpoint_selectors and endpoint_weights fields on the forward-internal Traffic Policy action configuration.

Using Traffic Policy with pools​

Endpoints in a pool may have different Traffic Policies. This lets you run old and new Traffic Policies side-by-side when transitioning endpoints into and out of a pool, so you can control and monitor traffic as you make changes.

It also enables you to roll out new Traffic Policy changes without risking an all-or-nothing deployment, because you can apply changes to endpoints individually rather than being forced to apply them to all endpoints in a pool at once

Example scenario​

You might want to transition from using one data center to a different data center for request handling, and the new data center might require a different Traffic Policy configuration. Because traffic policy is applied on a per-endpoint basis in endpoint pools, you could add your new endpoints associated with your new data center to the pool of old endpoints on by one, while gradually removing the old endpoints.

Such a transition would be easier to monitor and control than if you had to change the traffic policy for all endpoints in the pool at once. You could also use this strategy to roll out a new Traffic Policy to a subset of your endpoints, and then gradually roll it out to the rest of your endpoints.

How it works​

When traffic is balanced among endpoints with different Traffic Policies, ngrok first chooses an endpoint to balance to, then executes the Traffic Policy of the chosen endpoint.