An important facility in any distributed system is the ability to introspect the running system, and to debug it. Within ensemble the actions performed by the system are executing formula defined hooks. The debug-log cli provides for inspecting the total state of the system via capturing the logs of all agents and output of all hooks run by the system.
To facilitate better debugging of hooks, the debug-hooks cli provides for interactive shell usage as a substitute for running a hook. This allows a formula author or system adminstrator the ability to interact with the system in a live environment and either develop or debug a hook.
When the ensemble user utilizes the hook debug command like so:
ensemble debug-hooks unit_name [hook_name]
Ensemble is instructed to replace the execution of the hook from the formula of the respective service unit, and instead to execute it in a screen shell. If no hook name is given, then all hooks will be debugged in this fashion. Multiple hook names can also be specified on the command line. Shell regular expressions can also be utilized to specify hook names.
The debug-hooks command line invocation will immediately connect to the remote machine of the remote unit and start a named screen shell there.
The native capabilities of screen can be exploited to construct a full debug/development environment on the remote machine.
When a debugged hook is executed a new named window will pop up in the screen shell. The new window’s title will match the hook name.
This shell environment has all the ensemble environment variables in place, and all of the hook cli API may be utilized (relation-get, relation-set, relation-list, etc.).
It’s important to note that Ensemble serializes hook execution, so while the shell is active, no other hooks will be executed on the unit.
The user can stop the hook execution when done, by closing the hook’s window. At which point the system is free to execute additional hooks.
Its important to note that any state changes performed while in the hook window via relation-set are buffered till the hook is done executing.
The debug-hooks can be used to debug the same hook being invoked multiple times as long as the user has not closed the debug screen session.
The user can exit debug mode, by exiting the screen shell, and the unit will resume its normal processing.
Internally the debug-hooks cli begins by verifying its arguments, namely the unit exists, and the named hook is valid for the formula. After that it modifies the zookeeper state of the unit node, setting a flag noting the hook to debug. It then establishes an ssh connection to the machine and executes the screen command.
The unit-agent will establish a watch on its own debug settings, on changes introspecting the debug flag, and pass any named hook values down to the hook executor, which will construct debug hook scripts on the fly for matching hooks. These debug hook scripts are responsible for connecting to screen and monitoring the execution of the hook therein.
Special care will be taken to ensure the viability of the screen session and that debug mode is active before creating the interactive hook window in screen.