resilience reference¶

Package gitlab.com/phpboyscout/go/transit/resilience — the transport-neutral primitives the HTTP and gRPC layers wrap. It imports no HTTP or gRPC types, so both transports share one tested implementation of each concern instead of maintaining two that drift.

Most callers never import this package directly: they configure http.CircuitBreakerConfig or grpc.CircuitBreakerConfig and let the middleware construct the primitive. Read this page when you need to know exactly what those settings do, or when you want a breaker around something that is not an HTTP or gRPC call — a database handle, a message publisher, a shell-out.

Exported defaults¶

const (
    DefaultFailureThreshold = 5
    DefaultCooldown         = 30 * time.Second
    DefaultHalfOpenMax      = 1
    DefaultMaxTrackedKeys   = 8192
)

Both transports' Default*Config constructors reference these rather than re-typing the literals, so the HTTP and gRPC defaults cannot drift apart.

Breaker — the closed/open/half-open state machine¶

func New(cfg Config) *Breaker
func (b *Breaker) Allow() (done func(failure bool), allowed bool)
func (b *Breaker) State() State
func (b *Breaker) Record(failure bool)

New returns a breaker in the closed state. It is safe for concurrent use; every method takes an internal mutex.

The breaker is a stability primitive, not a cache. It never stores a response and never serves a previously-seen one — while open it simply refuses, and the caller decides what a refusal means.

Config fields¶

Normalisation happens inside New. The zero Config{} is therefore a perfectly good breaker with the default policy — resilience.New(resilience.Config{}) trips at 5 and cools down for 30 seconds.

Using Allow correctly¶

done, allowed := br.Allow()
if !allowed {
    return ErrDownstreamUnavailable // fail fast; done is nil
}

err := doTheCall()
done(err != nil) // exactly once, with the verdict

Two rules, and breaking either one is the usual cause of a breaker that behaves oddly:

• When allowed is false, done is nil. Calling it panics.
• When allowed is true, done must be called exactly once. A call that never reports holds a half-open trial slot. Use defer with a recover if the work between them can panic — that is what both transports' middleware does.

The breaker has a time-based escape for the case where you get this wrong anyway: if the half-open trial budget is exhausted and no verdict has arrived within a further Cooldown of entering half-open, the outstanding trial is treated as expired and the breaker re-opens. An abandoned done can wedge the breaker for one cooldown, not forever.

A late done from an expired or superseded admission is ignored. The breaker stamps a generation on every entry into half-open and each admission captures it, so a stale verdict cannot release a newer trial's slot or adjudicate its outcome.

State values¶

type State int

const (
    StateClosed State = iota // admit everything, count consecutive failures
    StateOpen                // admit nothing until the cooldown elapses
    StateHalfOpen            // admit up to HalfOpenMaxRequests trial calls
)

String() returns "closed", "open", "half-open", or "unknown" for anything else.

How each transition happens¶

Failures must be consecutive: with the default threshold of 5, a downstream failing four calls in five will never trip the breaker, however long it does so. A breaker is a detector of total failure, not of elevated error rate. If you need the latter, wrap a sliding-window error-rate check around it and feed the verdict in through IsFailure.

Record — reporting an outcome with no matching Allow¶

func (b *Breaker) Record(failure bool)

Applies an out-of-band outcome — one observed without a paired admission. The gRPC stream interceptor uses it for per-message failures after the establishment verdict has already been delivered.

Record counts only while the breaker is closed. In open it is ignored (nothing was admitted), and in half-open it is ignored because that state's verdict belongs exclusively to its admitted done callbacks. This is deliberate, and it means a burst of out-of-band failures cannot re-trip a breaker that is mid-recovery.

Store — bounded per-key token buckets¶

func NewStore(limit rate.Limit, burst, maxKeys int) *Store
func (s *Store) LimiterFor(key string) *rate.Limiter
func (s *Store) Len() int

A mutex-guarded LRU map of *rate.Limiter values from golang.org/x/time/rate, used by both server-side rate limiters when a KeyFunc is set. Each key gets its own bucket at the same rate and burst.

The cap is what makes it safe to key on client-controlled data. Without it, an attacker rotating source addresses allocates a limiter per fabricated key until the process runs out of memory. When the map exceeds maxKeys, the least-recently-used key is evicted.

Only maxKeys is clamped here, because the transport wrappers normalise limit and burst before constructing the store. Calling NewStore yourself means validating them yourself.

What eviction costs you¶

An evicted key that reappears gets a fresh, full bucket — its consumed tokens are forgotten. A client that can force eviction can therefore reset its own limiter.

That is an acceptable trade because eviction only happens under key churn: it requires more than maxKeys distinct keys active at once, which is itself the attack the cap exists to survive. If the churn is legitimate — you are keying on something with a naturally large cardinality, such as a user ID on a large service — raise MaxTrackedKeys rather than accepting silent resets. Len() reports the number of tracked keys, which is what to watch to find out whether you are evicting at all.

What this package does not do¶

• No shared state between processes. Every breaker and every bucket is in-memory and per-process. Ten replicas mean ten independent breakers, each needing its own FailureThreshold failures, and a rate limit that is effectively ten times the configured one across the fleet. There is no Redis backend and no gossip.
• No error-rate or latency triggers. The breaker counts consecutive failures only. It does not open on a percentage of failures, on p99 latency, or on a concurrency limit.
• No metrics. OnStateChange is the only observation hook, and there is no equivalent for the store.
• No adaptive behaviour. The cooldown is fixed; it does not back off further when a trial fails again, and there is no jitter on it.
• No persistence. State is lost on restart, so a process that restarts into a downed dependency starts closed and has to trip again.