Resolving errors
================

Ensemble internally tracks the state of units and their relations.
It moves them through a simple state machine to ensure the correct
sequencing of hooks and associated unit agent behavior. Typically
this means that all hooks are executed as part of a workflow transition.

If a hook fails, then ensemble notes this failure, and transitions
either the unit or the unit relation (depending on the hook) to a
failure state.

If a hook for a unit relation fails, only that unit relation is
considered to be in an error state, the unit and other relations of
the unit continue to operate normally.

If a hook for the unit fails (install, start, stop, etc), the unit
is considered not running, and its unit relations will stop responding
to changes.

As a means to recover from hook errors, ensemble offers the
``ensemble resolved`` command.

This command will operate on either a unit or unit relation and
schedules a transition from the error state back to its original
destination state. For example given a unit mysql/0 whose start
hook had failed, and the following command line::

  $ ensemble resolved mysql/0

After being resolved the unit would be in the started state. It
important to note that by default ``ensemble resolved`` does
not fire hooks and the ``start`` hook would not be invoked again
as a result of the above.

If a unit's relation-change/joined/departed hook had failed, then
ensemble resolved can also be utilized to resolve the error on
the unit relation::

  $ ensemble resolved mysql/0 db

This would re-enable change watching, and hook execution for the
``db`` relation after a hook failure.

It's expected that an admin will typically have a look at the system to
determine or correct the issue before using this command line.
``ensemble resolved`` is meant primarily as a mechanism to remove the
error block after correction of the original issue.

However ``ensemble resolved`` can optionally re-invoke the failed hook.
This feature is particular beneficial during formula development, when
iterating over an in development hook. Assuming a mysql unit with
with a start hook error upon executing the following command::

 $ ensemble resolved --retry mysql/0

Ensemble will examine the mysql/0 unit, and will re-execute its start
hook before marking it as running. If the start hook fails again,
then the unit will remain in the same state.