Take Control of Your Cron Jobs With Rundeck


Take Control of your Jobs with Rundeck

In this tutorial, we’ll show you how to setup and get started with Rundeck, a free (as in beer) job control solution.

For some simple needs, crontab suffices.  If you have a job you want to run at 1am, you stick it in crontab (or use atd), it runs, and you’re happy.

But what if you have more complex needs?  For example, “backup these databases, and if that succeeds, tarball this source code, then git pull, then restart these systems, then run these maintenance jobs…”  You could do that all as different cron jobs, but you lose the ability to set dependencies.  You could write a mother script to call many scripts, but then you lose modularity and create a brittle sprawl.  And what if you want to manage jobs across many servers?   What if the results of jobs on some drive what happens on others?  And what if you want to make jobs easy to run one-off – perhaps even allowing end-users to do so without needing to give them all the keys to the kingdom?

This type of complex job management was for years the domain of stratospherically expensive job enterprise control solutions.  Rundeck is a free product that gives you tremendous power in managing systems.

Personally, I have a home Rundeck server that controls backups on all my home systems and VPSes, housekeeping jobs, seedbox maintenance and conversion jobs, rotating pictures on our digital photo frame, etc.

In this tutorial, I’ll create a Rundeck server (master.lowend.party) and a single node (slave.lowend.party).  Rundeck can scale to control hundreds (perhaps thousands) of nodes, but the principles are the same.

In this case, I’m using a Debian 9 VPS with 4GB of RAM for my master, and a Debian 10 VPS with 1GB of RAM for the salve.  The reason for Debian 9 on the master is that Rundeck requires OpenJDK 8, which is easily installable on Debian 9.  Support for OpenJDK 11 is coming but at the moment, JDK 8 is required.

After imaging the master, I installed some required packages:

apt-get install openjdk-8-jre-headless mariadb-server gnupg2 apt-transport-https

After installing MariaDB, I secured it.

root@master:~# mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? (Y/n) y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!

By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.

Remove anonymous users? (Y/n) y
... Success!

Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? (Y/n) y
... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? (Y/n) y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? (Y/n) y
... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

Now we’ll create a database for Rundeck.

root@master:/etc/rundeck# mysql
Welcome to the MariaDB monitor. Commands end with ; or g.
Your MariaDB connection id is 55
Server version: 10.3.22-MariaDB-0+deb10u1 Debian 10

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

MariaDB ((none))> create database rundeck;
Query OK, 1 row affected (0.000 sec)

MariaDB ((none))> grant ALL on rundeck.* to 'rundeckuser'@'localhost' identified by 'rundeckpassword';
Query OK, 0 rows affected (0.000 sec)

MariaDB ((none))> flush privileges;
Query OK, 0 rows affected (0.000 sec)

MariaDB ((none))> Bye

Now let’s install Rundeck itself.

echo "deb https://rundeck.bintray.com/rundeck-deb /" > /etc/apt/sources.list.d/rundeck.list
curl 'https://bintray.com/user/downloadSubjectPublicKey?username=bintray' | apt-key add -
apt-get update
apt-get install rundeck

Before we startup Rundeck, we want to make some configuration changes.  All of Rundeck’s configuration is in /etc/rundeck.

Modify framework.properties:

framework.server.name = master.lowend.party
framework.server.hostname = master.lowend.party
framework.server.port = 4440
framework.server.url = http://master.lowend.party:4440

Modify rundeck-config.properties to change from using h2 (a slow internal database only suitable for testing) to the MariaDB database we set up:

grails.serverURL=http://master.lowend.party:4440
#dataSource.url = jdbc:h2:file:/var/lib/rundeck/data/rundeckdb;MVCC=true
dataSource.url = jdbc:mysql://localhost/rundeck?autoReconnect=true&useSSL=false
dataSource.username=rundeckuser
dataSource.password=rundeckpassword
dataSource.driverClassName=com.mysql.jdbc.Driver

Modify realm.properties to set a password fo the ‘admin’ account.  The default password is ‘admin’.  Change it to something better:

admin:complex-password,user,admin,architect,deploy,build

Now start Rundeck:

systemctl start rundeckd

Now you can see Rundeck running (via ps -ef):

rundeck 6810 1 99 12:44 ? 00:00:21 java -Drundeck.jaaslogin=true -Djava.security.auth.login.config

It may take Rundeck a few moments to startup.  If you look at /var/log/rundeck/service.log you’ll see this message towards the end that Rundeck is ready:

(2020-05-19 12:47:46.891) INFO BootStrap --- ( main) Rundeck startup finished in 2202ms

Rundeck runs on port 4440, so I surfed to master.lowend.party:4440.

Login as admin with the password you set in realm.properties above.

Now click New Project.

Click Create.  Now in the left menu bar, expand Project Settings and click Edit Nodes.

We now need to setup our list of nodes.  We need a read from a “source”.  This could be something exotic like the output from a giant enterprise database system, Ansible, a URL, etc.  But for this tutorial, we’ll read from a file.

Click File.

We’ll use YAML.  We’re going to let Rundeck generate a new file for us.  Fill out the boxes as shown.

Now click save.  On master, in /var/lib/rundeck/data, this file now exists:

root@master:/var/lib/rundeck/data# cat nodes.yaml

master.lowend.party:
  nodename: master.lowend.party
  hostname: master.lowend.party
  osVersion: 4.9.0-12-amd64
  osFamily: unix
  osArch: amd64
  description: Rundeck server node
  osName: Linux
  username: rundeck
  tags: ''

I added a new entry for slave:

master.lowend.party:
  nodename: master.lowend.party
  hostname: master.lowend.party
  osVersion: 4.9.0-12-amd64
  osFamily: unix
  osArch: amd64
  description: Rundeck server node
  osName: Linux
  username: rundeck
  tags: ''
slave.lowend.party:
  nodename: slave.lowend.party
  hostname: slave.lowend.party
  osVersion: 4.9.0-12-amd64
  osFamily: unix
  osArch: amd64
  description: Rundeck slave node
  osName: Linux
  username: rundeck
  tags: ''

Now let’s setup SSH keys.  Once again select Edit Configuration on Project Settings.

Click Default Node Executor.

You’ll see that SSH is the default way that Rundeck will connect to systems and run jobs.  Let’s use the name already in use and create a passwordless SSH key:

root@master:/var/lib/rundeck# ssh-keygen -f /var/lib/rundeck/.ssh/id_rsa
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/lib/rundeck/.ssh/id_rsa.
Your public key has been saved in /var/lib/rundeck/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:jKEPwDsvSDZ7WB1Sc9tRiBFgC+JI3c6IO1xhriTL1NE root@master.lowend.party
The key's randomart image is:
+---(RSA 2048)----+
| o..o=.++.o. |
|+..=+E+.o.. |
|..B.*o.. . |
|.+ Bo+.+ |
|*=*.o.. S |
|+B=o o |
|.oo.. . |
| .. |
| |
+----(SHA256)-----+
root@master:/var/lib/rundeck# chown rundeck:rundeck /var/lib/rundeck/.ssh/id*

On the slave server, I created rundeck account and placed the id_rsa.pub ssh key in rundeck’s authorized_keys.  I also allowed the rundeck user to execute jobs as root by placing a file in /etc/sudoers.d/rundeck with these contents:

rundeck ALL=(ALL:ALL) NOPASSWD: ALL

Now I’ve created a job to run:

root@slave:/etc/sudoers.d# cat /usr/local/bin/maintenance.sh
#!/bin/bash

apt-get update
apt-get -y upgrade

Back in the Rundeck interface, click Jobs.

Click “Job Actions”, and then click “New Job”.  Fill out the form with a suitable name and description.

Now click Workflow.

Click Add Step.

Select Command.

Now click Save, then click Nodes.

Select Dispatch to Nodes.  Under Node Filter, I’ve entered the node name.  If you had nodes in a specific format, you could use wildcards (e.g., prod*, etc.), list nodes, or use * for all nodes.

You can also configure instructions on how to handle failed nodes:

Now save the job definition.  On the main job page, select the job.

And then Run Job Now:

In the subsequent window, you’ll see the job’s result:

Let’s set the job to run without need for manual execution.

On the project’s jobs page, select Actions and then Edit This Job.

Click Schedule.  The default “simple scheduler” will appear:

This is sufficient for most scheduling, but if you need more advanced options, you can switch to Crontab scheduling by clicking “Crontab”:

We can also setup notifications.  For various states, such as job success or job failure, Rundeck can send an email or POST to a web hook (see the documentation for details).  Here I will configure the job to email me if it fails.  Note that I’m using one of the available Rundeck variables (${job_name}) so I don’t have to type the job name.  These variables are covered in the documentation.

Let’s make our job a bit more complicated by linking several steps together.

On the slave, I’ve created a script to do MariaDB backups.  I also want to clean out old backups as well.  For that, I will use an Inline Script as opposed to a Command.  Here’s how my workflow looks now:

Now when I execute the job, I see all steps firing sequentially:

We’ve barely scratched the surface of Rundeck’s capabilities.  To learn more, visit Rundeck.com and peruse the documentation.

raindog308

I’m Andrew, techno polymath and long-time LowEndTalk community Moderator. My technical interests include all things Unix, perl, python, shell scripting, and relational database systems. I enjoy writing technical articles here on LowEndBox to help people get more out of their VPSes.