Systemctl does not stop and start contrast-server as expected

Issue

After running systemctl stop contrast-server you find that systemctl start contrast-server does not result in the server starting up (as evidenced by the UI not being available, or by the fact that the last entry in the <contrast home>/logs/server.log file still shows completion of the previous startup with no additional entries related to the current attempt).

Cause

The command systemctl stop contrast-server will time out on some Linux distributions (verified to be an issue on RHEL7.x and Ubuntu 18.04), leaving systemd in a confused state.

Resolution

The following steps will resolve the confusion:

1. First reset the systemd status to allow systemd to recognize the desired status as stopped, by running systemctl stop contrast-server and wait for return of the command line prompt.
2. Determine the process ID(s) of the contrast-server process(es) by running ps aux | grep contrast.
3. You will find the java process that runs the application server and possibly also the mysqld process that runs the database. Terminate the java process using kill -9 <PID> (to force the termination) and the mysqld process using kill -15 <PID> (to avoid any chance of data corruption if a transaction is in progress).
4. Start the server using systemctl start contrast-server.
5. Verify that the server is starting up by watching the server.log file using tail -f <contrast home>/logs/server.log - you should see newly logged entries indicating the server is starting, with a final entry indicating "Contrast TeamServer Ready".
6. Log into the UI to verify the server is up and running.