Skip to content

Configure restart policy

By default a service is not restarted: if its WithStart returns an error, the controller records it and moves on. To make a service self-healing, attach a RestartPolicy with WithRestartPolicy.

Enable restarts

c.Register("worker",
    controls.WithStart(runWorker),
    controls.WithRestartPolicy(controls.RestartPolicy{
        MaxRestarts:    5,
        InitialBackoff: time.Second,
        MaxBackoff:     30 * time.Second,
    }),
)

Now, if runWorker returns a genuine error while the controller context is still live, the supervisor waits a backoff interval and starts it again, up to MaxRestarts consecutive failures.

The policy fields

type RestartPolicy struct {
    MaxRestarts            int           // cap on consecutive failures (0 = unlimited)
    InitialBackoff         time.Duration // first wait before restart (default 1s)
    MaxBackoff             time.Duration // backoff ceiling (default 30s)
    HealthFailureThreshold int           // consecutive Status failures before restart
    HealthCheckInterval    time.Duration // how often Status is polled (default 10s)
    RestartResetInterval   time.Duration // healthy run length that resets the counter (default 30s)
}
  • InitialBackoff / MaxBackoff. The wait before each restart doubles from InitialBackoff, capped at MaxBackoff. A zero field selects its default (1s / 30s).
  • MaxRestarts. Caps consecutive failures (see below), not lifetime restarts. It counts restarts, so MaxRestarts: 5 lets the service run six times in all. When exceeded, the supervisor gives up on the service and forwards a max restarts exceeded error that errors.Is matches against ErrRestartsExhausted, and against the last StartFunc error when there was one. 0 means unlimited.
  • The controller keeps running either way. Giving up on a service does not stop the process or the other services; the error is recorded on ServiceInfo.Error, logged, and delivered as a terminal ServiceEvent to a WithOnEvent callback and on the error channel. If a dead service should take the process down, react to EventFailed or EventUnableToStart in the callback and call Stop() yourself.
  • But readiness goes false if the service never started at all. Giving up on a service that never once started cleanly means it never will, so the controller moves to UnableToStart and reports unready. An orchestrator then routes no traffic to it and restarts the pod, which is usually what you want and needs no code from you. A service that started and failed later does not trip this, and neither does one that fails once and recovers.

Health-driven restarts

A service whose WithStart returns nil immediately (it serves in a background goroutine) never "exits", so there is no error to restart on. To supervise such a service, combine a WithStatus probe with a HealthFailureThreshold:

c.Register("streamer",
    controls.WithStart(startStreamerInBackground), // returns nil once serving
    controls.WithStatus(func() error {
        return streamer.Ping() // non-nil == unhealthy
    }),
    controls.WithRestartPolicy(controls.RestartPolicy{
        HealthFailureThreshold: 3,               // 3 consecutive bad pings...
        HealthCheckInterval:    5 * time.Second, // ...polled every 5s
        MaxRestarts:            10,
    }),
)

The supervisor polls Status every HealthCheckInterval. Each failure increments a counter; a single success resets it. When the counter reaches HealthFailureThreshold, the service is stopped and restarted (subject to MaxRestarts and backoff).

HealthFailureThreshold requires a Status probe. Health monitoring only runs when both HealthFailureThreshold > 0 and a WithStatus func are present. Without them, a clean-start service simply runs until shutdown.

Which restart path calls WithStop. A health-threshold restart stops the service first: WithStop runs before each restart, and again at shutdown, so it must be idempotent. An error-triggered restart does not call WithStop at all. The StartFunc has already returned, and the supervisor goes straight to the backoff wait. Release resources on the way out of your StartFunc if the error path needs cleaning up.

The probe must report the current run. A restart calls WithStart again but reuses your WithStatus closure, so a status cell it captured and never cleared keeps reporting the previous run's failure, which trips the threshold again, which restarts again. That is a service churning to exhaustion with no log line naming the cause. See What a restart shares.

Consecutive failures and the reset window

MaxRestarts counts consecutive failures, not the lifetime total. A service that fails, restarts, and then runs healthily for RestartResetInterval has its counter reset to zero, so a service that flakes once an hour is never permanently killed by an old failure.

The reset window defaults to 30s (DefaultRestartResetInterval). Tune it with the policy field, or with WithRestartResetInterval:

c.Register("flaky",
    controls.WithStart(run),
    controls.WithRestartPolicy(controls.RestartPolicy{MaxRestarts: 3}),
    controls.WithRestartResetInterval(2*time.Minute), // must run 2m clean to reset
)

WithRestartResetInterval implies a policy. If the service has no RestartPolicy yet, this option creates a default one so the interval takes effect, and a default policy has unlimited MaxRestarts. Order does not matter; apply it before or after WithRestartPolicy.

Exempt expected terminal errors

Some errors are a normal end of run, not a failure. The canonical example is http.ErrServerClosed, returned by ListenAndServe after a graceful Shutdown. Register a controller-wide predicate with WithValidError so the supervisor treats a matching error as a clean stop: it is neither counted toward the restart total nor forwarded on the error channel.

c := controls.NewController(ctx,
    controls.WithValidError(func(err error) bool {
        return errors.Is(err, http.ErrServerClosed) ||
            errors.Is(err, context.Canceled)
    }),
)

WithValidError is a controller option (passed to NewController), and the predicate applies to every service. Contrast it with the per-service restart options above, which are ServiceOptions passed to Register.

A supervised Child declares its own, as a ValidError field rather than an option, so children of different kinds can exempt different errors:

sup.Attach(controls.Child{
    Name:  "api",
    Start: srv.Run,
    ValidError: func(err error) bool {
        return errors.Is(err, http.ErrServerClosed)
    },
})

Observe restart counts

info, _ := c.GetServiceInfo("worker")
fmt.Printf("restarts=%d lastErr=%v\n", info.RestartCount, info.Error)