Today I wanted to test a service directly on my developer system. That is, I wantedto start it with systemd to prove that it was working (it does not work right on a full install, though.)

I bumped in a few problems, though.

Aug 23 09:32:38 halk systemd[1]: snapwatchdog.service: Main process exited, code=exited, status=200/CHDIR

First, I got this cryptic error. I though that "CHDIR" would mean that when systemd tries to change directory, it wasn't happy. Sure enough, that was it. I have an entry which tells systemd what my Current Working Directory should be:

WorkingDirectory=/var/lib/snapwebsites

And that is owned by me on my system so that way I can tweak things in there quickly (without the need to sudo here and sudo there all the time.)

So, I had to also change the user and group names, because without doing so, the default would not work. That is, systemd checks that it is compatible for the user you say your daemon runs as. If you run your daemon as root, this should not be a problem, though.

So I changed this:

User=snapwebsites
Group=snapwebsites

Into that:

User=alexis
Group=alexis

And that error was replaced with the next one.

Aug 23 09:38:46 halk systemd[1]: snapwatchdog.service: Main process exited, code=exited, status=203/EXEC

It seemed sensible to just start my process from my BUILD folder. After all, that way each time I recompile I can just restart and the newer version gets executed.

However, it looks like systemd understands /home as a user directory and disallow daemons from being started from there if you don't use the user daemon mechanism. Only, I did not want to test with the user daemon thinking that it could very well not work the same as the main daemon system. So to prove that the feature I was testing works as expected, I decided to run snapwatchdogserver as a full system daemon.

So... in my case, the path had to be changed to something else. All I had to do is copy my file under /usr/sbin. That worked as is, even though the binary includes RPATH back to my home folder (i.e. so if I changed libraries, it picks up my newer version of the library.) Very practical in my case, although that may not be quite secure for a daemon system like systemd... Maybe that will be changed at some point.

Aug 23 09:48:41 halk systemd[1]: snapwatchdog.service: Main process exited, code=exited, status=127/n/a

You would think that would finally make it all work... well, nope. Not in my case. This onewill very much depend on your security awareness and thus how many options you've used in your service file.

In my case my Unit section includes the following:

[Unit]
...
ProtectHome=true
...

I'm not too sure how that works, but somehow, it did not like it. I commented it out and tried again and this time the service started as expected.

Since the Home directory is the same on my full installs (i.e. /var/lib/snapwebsites), the fact that the libraries are under /home/alexis should have no consequences. So I'm not too sure what they are testing here. My daemon does not chdir() anywhere...

And it worked?!

"Unfortunately" the final result was that it worked as expected on my developer system. You see, I have a script that generates an error if a certain file exists on the system. It prints the error in stderr using the following syntax:

echo "The file exists, check it out [blah blah blah]" >&2

The >&2 means write to file descriptor number 2, which is stderr.

I was afraid that if run through systemd, the stderr would be captured by systemd and put in the journal instead of going back to the calling process (my daemon). Not the case, it goes to the daemon as expected when run from my programmer's system.

So the problem on the full install of the test system is different.

Final Answer

The final answer: the fact is that the snapwatchdog daemon uses a starter script which runs the user's (other packages) scripts to test this or that.

The old version would do just this at some point:

$SHELL $WATCH_SCRIPT

This means start the $WATCH_SCRIPT using the shell as defined in $SHELL.

The problem with this concept is that by default the snapebsites user has a shell setup to /bin/false. Obviously that wouldn't work to start a new shell script.

What I've done to the final version of that realm is forcibly define the SHELL variable in the default parameters (under /etc/defaults/...).

SHELL=/bin/bash

Now the scripts are started as expected.