seamless-remote-debugging
Seamless Remote Debugging Guide
The Cardinal Rule
When a remote operation fails, always consider the possibility that the bug is on the server side, not the client side.
Seamless remote clients talk to separate server processes (hashserver, database, jobserver, daskserver) that may run on different machines, in different repos, with their own dependencies. A ClientConnectionError in buffer_client.py might mean the hashserver crashed due to a Starlette API change — a completely different codebase. If you only look at the client code, you might "fix" the problem at the wrong level of the stack (e.g., adding error handling in the client when the server needs patching).
When you see a remote failure related to connection or resource retrieval, dig up the server-side logs before attempting any fix. The logs will tell you whether the error originates on the server, which changes the diagnosis entirely.
SSH Guard and Helper Commands
remote-http-launcher installs a set of helper commands (rhl-*) on the remote server. When the SSH guard is configured, only these helpers and the specific command patterns sent by the launcher itself are permitted — naked shell commands like pkill, rm, or kill are rejected.