Problem

In our snapwebsites.org environment, we install many daemons.

Most of these daemons are expected to be started as soon as they get installed even if they can't really run properly. For example, the snapserver needs a valid connection the snapdbproxy which requires Cassandra to be ready for the purpose. It still works since snapserver waits on a CASSANDRAREADY signal (sent by snapdbproxy through snapcommunicator.) While waiting, daemons are sleeping so they use their normal share of memory, but no CPU at all.

Now, a few of our daemons need to not be started by default. We have a few backend daemons that should only run on a backend computer, but you end up installing that backend package on all front-end computers too because you need to run the snaplistjournal service.

Somehow, all daemons seem to auto-start when they are installed on a Debian-like system (we're developing for Ubuntu.) So I looked closer to see whether there would be a way to avoid that auto-enable feature and I have to say that it took me forever to understand what I was doing wrong.

Set Up Service to Auto-Start

The fact is that in most cases you're going to see an example of .service file that looks something like this:

# Documentation available at:
# https://www.freedesktop.org/software/systemd/man/systemd.service.html

[Unit]
Description=Snap! Websites images backend
After=snapcommunicator.service snapfirewall.service snaplock.service snapdbproxy.service

[Service]
Type=simple
WorkingDirectory=~
ProtectHome=true
ExecStart=/usr/bin/snapbackend --cron-action images::images
ExecStop=/usr/bin/snapstop --service $MAINPID
Restart=on-failure
RestartSec=1h
User=snapwebsites
Group=snapwebsites
Nice=10
LimitNPROC=1000
# For developers and administrators to get console output
#StandardOutput=tty
#StandardError=tty
#TTYPath=/dev/console
# Enter a size to get a core dump in case of a crash
#LimitCORE=10G

[Install]
WantedBy=multi-user.target

# vim: syntax=dosini

Everything looks fine at first glance.

The option that tells the system to auto-enable this service is actually the WantedBy=... line. Frankly, It wasn't that obvious to me.

The concept is very simple, but the implementation is quite complex as it involves many things that test what is where and the current status of things, etc. etc. etc.

In our case, we were saying that the snapimages service was wanted by the multi-user.target. That means once the multi-user.target gets enabled, anything it wants, including our snapimages, gets activated. If you are in your X11 desktop and install snapimages, you are in the multi-user.target and since it wants snapimages to be running, the installer auto-enable and activate (start) that service for you.

Solution

The solution is very simple. Remove that target. You have to remove the dependency and voilà, now your target will remain disabled on installation.

#WantedBy=multi-user.target

For example, you can comment it out as shown here. You can also delete it although it could be a good idea to properly document why it is not there or comment out. (Because later you may think it's missing and re-add it and not notice that it gets started when it shouldn't be.)

A very strong side effect of this change is that you can't just enable that service. Without the WantedBy, it is viewed as a static service. This means you can start it and stop it manually, but running enable against it has no effects. Instead of the normal enable, you have to specifically specify who wants to run it as follow:

sudo systemctl add-wants multi-user.target snapimages

That command creates the symlink that the enable command would have created, had you kept the WantedBy=... definition.

In Snap! C++, this special enable is done in our snapmanager.cgi tool so you don't have to remember that syntax. But it's definitely a gotcha. I think there should be a way to define that in your file so you can choose how to get the service started for the first time (i.e. by the install command or explicitly) without the need for a special command like this. That being said, I'm still glad that I can make these work as expected.

deb-systemd-helper was-enabled ...

The deb-systemd-helper perl script is used to check various flags, one of which is the was-enabled. This is not really a systemd flag, but it is determining using the systemd installation files. It creates a file with a list of filenames that systemd installs. When such files exist, the system decides that the service is enabled.

In my case, the deb-systemd-helper was-enabled snapimages command checks a file named /var/lib/systemd/deb-systemd-helper-enabled/snapimages.service.dsh-also and that file contains this link generated by systemd because of the WantedBy=multi-user.target definition we had (as shown above.)

/etc/systemd/system/multi-user.target.wants/snapimages.service

So by removing that WantedBy=... definition we get a snapimages.service.dsh-also file which is empty and that results in a disabled service.

Third Party Services

Those services you want to install in the disabled state from packages that you do not control... well, from what I understand, it's just not possible. Not without editing the package and tweaking the postinst script to your liking.

There is a way to install the files of a package and not run the configuration scripts. This is done by using the --unpack command line option. After that, you could tweak the postscript and run your version.

This can be really complicated to do it right and I really don't recommend doing such. Also using dpkg directly means you installed the dependencies which means the autoremove feature of apt-get won't work quite right for you. I'm hoping that someone will understand that there is a need for this one. One good example is installing the Apache server and auto-starting it. Then you need to make many modifications to the settings and most likely restart Apache (especially if you add SSL certificates and such.) It also means that while you first install Apache, it may not be secure as expected by your application.

One solution in this circumstance is to install a firewall that blocks port 80 (and probably 443), install Apache, make changes to your settings, restart Apache, then unblock ports 80 and 443. At that point things are safe.

Another example is a service that requires settings to communicate with other services on your local network. You probably need to set up one or more IP addresses with which this computer can communicate. We have such a problem with our snapcommunicator service.

So there is probably always a solution to make things work as expected and safely, but it's just additional work. Now, don't get me wrong, most people in most cases like to have services automatically installed and activated. It works great for many services such as NTP. It just doesn't always work well when the service requires changes to its settings.

Masking Failure

I found a web page that said that you could mask the service before you install it so that way it is prevented from being started. The truth is that the postrm script of your package uses the mask feature of systemd.

In my case I can see these few lines:

# On "remove"
deb-systemd-helper mask snapimages.service
# On "purge"
deb-systemd-helper purge snapimages.service
deb-systemd-helper unmask snapimages.service

That can cause a problem:

The script may fail between the time the mask is put in place and the unmask gets called. In other words, the system may never correctly remove the mask.

Also, if you remove and later reinstall, you need to unmask, don't you?

This is done in the postinst script as follow:

deb-systemd-helper unmask snapimages.service >/dev/null || true

And as we can see in our postinst script, that line is always executed and therefore trying to mask a service to prevent it getting enabled on installation will have absolutely no effect. Masking is definitely not a solution.

Vendor Preset Disabled Failure

The preset feature is used to describe what status a daemon should be given on installation. The preset statuses are defined in preset files with the following syntax:

disable snapimages.service

To add a preset file to my Debian package, I create a file with the proper name, in my case:

90-snapimages.preset

and then added one line to my snapbackend.install file:

snapbackend/conf/90-snapimages.preset  lib/systemd/system-preset/

(note that we use the system preset folder for packages, these files can be overridden by an administrator in /etc/systemd/system-preset.)

The expected result is a service that gets installed in the disabled state, so let's check just after installation:

systemctl status snapimages
● snapimages.service - Snap! Websites images backend
   Loaded: loaded (/lib/systemd/system/snapimages.service; enabled; vendor preset: disabled)

So we can see that the vendor preset status changes accordingly, but it definitely doesn't prevent the service from being started because of the dependency (WantedBy=...). In other words, systemd preset on Ubuntu (and most certainly Debian and other distributions based on Debian) doesn't work.

Having the preset is probably not a bad idea to still convey the concept but to really not have the target enabled, you need to use my solution above.

The preset can be used to make the systemctl preset command work as one would expect. You can use the following:

systemctl preset snapimages

and the status will change to disable as per the default vendor status.