Automating Tasks With Unix Crontabs
do system updates, database backups, periodically and automaticallyWeb developers often use Unix like operating systems like Ubuntu, centos and others to deal with server installation, configuration, and management. Linux provides many effective tools to make some things easier to maintain, one of these tools is called the Crontab.
- Cron means “time” in Greek language
Intro
Web developers often use Unix like operating systems like Ubuntu, centos and others to deal with server installation, configuration, and management. Linux provides a lot of very effective tools to make some things easier to maintain, one of these tools is called the Crontab.
How can you benefit from Cron jobs, here’s some case scenarios I had to deal with:
- Task automation, checking email, downloading files over the internet.
- Maintain system services, rotating logs (done by the OS automatically).
- Making database backups.
- Generating and sending reports.
- Rotating custom logs.
- Updating search indexes.
- Checking external APIs.
- Basically. whatever you need on a recurring basis.
Crontab is basically a service running (in daemon mode) on the operating system, this service does the following:
- Read the Crontab files, which are just text files containing Cron Tasks (more on this later).
- Parse all Cron Jobs in that file, and execute the ones that are scheduled for this time as a super user.
- After one minute, repeat the whole thing.
You can trust the Cron Job service, it’s very reliable. It doesn’t occupy the memory when it’s idle. And if the service exits and fails it will restart at a certain time automatically.
Info: Cron jobs on Unix systems and event scheduler in Microsoft Windows operating system have the same main idea.
Let’s get down to business:
Now there are 2 ways of making Cron Jobs:
Method 1: Using the Crontab command
This method alters the files at /var/spool/cron/crontab
directory, each user will have a file named by the username in that directory. we should not alter these files manually however. The Crontab command is responsible for handling these files.
To open the current user Crontab use:
Crontab –e
(edit)
this will open the Crontab file like so:
# Edit this file to introduce tasks to be run by cron.
# # Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# # To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
# # Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# # Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# # For more information see the manual pages of crontab(5) and cron(8)
# # m h dom mon dow command * * * * * echo ‘Run this command every minute’ >> /home/vagrant/cron_output.txt
At the bottom of this file, you can start writing your Cron Jobs.
Opening the file for the first time will prompt you for your favorite text editor, at the moment I advise you to use the Nano text editor, option 2 from the list. Alter the file, click F2 to exit then Y to save the changes.
That’s it, nothing else needs to be done, the Cron service will pick up the changes.
Method 2: writing to a pre-configured scheduled Cron directories
These files are located at:
/etc/cron.daily
/etc/cron.hourly
/etc/cron.monthly
/etc/cron.weekly
You just write a file and save it at one of the above directories, say /etc/cron.daily/myCron
(leave the file extension), and it will be executed in the background once every day. Very simple huh!
These files are system wide and always run by the root user. Adding Cron Jobs in these files would require you to specify the user that the Cron Job will run as. take a look at /etc/Crontab
:
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly 25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ) 52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
You see how this file is responsible for running all Crontabs in the directories in method 2.
Info: it’s rarely that you would need to add Crontabs in:
/etc/Crontab
/etc/cron.d/*
The order of execution of multiple Crontab files in a directory like /etc/cron.daily/
. is according the ASCII code. So a file with a name starting with 0 will run first
After writing your Cron Jobs, To list the current user’s Crontabs
Crontab –l
(for listing)
Crontab –u someuser –l
(view Cron Jobs for the current user)
Now, how to write a Cron Job
Let’s say we have a command to update a search index for a website, don’t care much about the command itself:
The command is python manage.py update_index
, and we need to make the system run it once every day.
So, obviously, instead of doing this manually. We use Cron Jobs.
The syntax for a user’s Cron Job is:
Minute hour dayofmonth month dayofweek command
And for system wide Cron Jobs (which we rarely use), the syntax is:
Minute hour dayofmonth month dayofweek
user
command
For each minute, hour, day of month, month and day of week, there are many options:
*
for any (ex:*
in minutes means at any minute, all 60 of them)X
, where x is a number stating the actual time (ex:12 * * * *
means anytime every 12thminute of an hour, like6:12, 9:12
…etc.)X-y
, a time period (ex:* 6-8 * * *
, means any minute between 6 am and 8am of any day of the year)X,y,z,…,
different options for the same placeholder (ex:* * 10,15,25 * *
, means any minute of only the 10th, 15thand 25thdays of any month in the year)x/4
, a division, (ex:* */8 * * *
, means every 3 hours. Same as* 0,3,6,9,12,15,18,21 * * *
)
there are short hands to user which may help as more readable, like @daily
. You can use it instead of using the first part of the Cron Job (* * * * *). For example:
@daily python manage.py update_index
Is the same as:
0 0 * * * python manage.py update_index
options for short hands are:
@hourly
, @daily
, @weekly
, @monthly
, @midnight
@annually
and @yearly
which are the same thing.
There’s a special one, @reboot
. Crons with @reboot
schedule run only once at system start.
With the 5 placeholders (* * * * *
), you can pretty much pick any time interval in a year you can imagine, here’s an exercise for you:
Write a Cron Job to echo your name every Sunday in July only.
It’s a good idea to save whatever return from the executed command by appending to some text file, this is a good practice for monitoring and checking the command behavior, I use this to see errors and warnings from my command that may help for problem troubleshooting.
Another good practice is to just run a bash script at a separate file from the command in the Cron Job, this makes it easier to run multiple commands in multiple lines of script instead of using &
or &&
between commands in one line
Example usage:
0 0 * * * source /some/path/myscript.sh >> /some/path/log.txt
you can send the results of the Cron Job command to your email by specifying:
[email protected]
(in the same Crontab file).
info: For this to work you need to have your system or server setup and configured to send emails.
There are many options like ssmtp, postfix, mailgun, sendmail and others. For me I recommend using mailgun, because they give you 10000 emails per month on their free plan (at the time of writing this).
I would like to mention https://Crontab-generator.org which is an online helper tool to generate Cron Jobs. It uses a graphical user interface to pick the right time schedule, which may be easier for newbies to start with
Who has access to Cron Jobs
As a system administrator, you may need to limit users who are allowed to make Cron Jobs on the system. This can be done with 2 files located at /etc/cron.deny
and /etc/cron.allow
, these files simply contain lines of usernames, empty lines are not allowed. The allow and deny files are read each time a user tries to add, modify or delete a Cron Job, so there’ no need to restart the Cron daemon for changes to take effect.
for example if you want to only allow only userA and userB to make Cron Jobs, then in the cron.allow enter:
# /etc/cron.allow userA userB
and if you want all users to make Cron Jobs the you either enter “ALL
” in the cron.allow file, or just delete the file (which doesn’t exist initially).