Page 1 of 4  

Tue, 08 May 2012

Keeping Tabs on Cron

Cron

Email is a great way to keep track of automated tasks on servers you maintain, but it's not necessary to have scripts explicitly send email - you can keep a watch on cron jobs with cron's built-in email facility. This works with the standard Vixie cron used on GNU/Linux and the various BSD's. The way it works is that the standard output and standard error of any commands run will be emailed to the recipient(s) specified in the MAILTO environment variable. This variable is defined in the crontab file itself, as a single email address, or a comma-separated list of email addresses. If it is not defined, email is sent to the owner of the crontab. If it is set to empty (""), or no output is produced by a command, no email is sent. Here are examples showing the various ways this can work.

Capture All Output

Emails both STDOUT and STDERR of the command /usr/local/bin/backup.sh to doug@example.com.

MAILTO=doug@example.com 0 3 * * * /usr/local/bin/backup.sh

Capture No Output, Selectively

Discard both STDOUT and STDERR, effectively sending no email, even though MAILTO is defined. This is useful if you have several programs to run in the same crontab - you can selectively choose which will have their output captured and emailed.

MAILTO=doug@example.com 0 3 * * * /usr/local/bin/backup.sh > /dev/null 2>&1

Capture No Output, Never Send Email

Don't send an email at all, for any commands.

MAILTO="" 0 3 * * * /usr/local/bin/backup.sh

Capture All Output and Send Email to Crontab Owner

This just sends an email to the owner of the crontab, as no MAILTO is defined. I prefer to define MAILTO in all cases, since an undefined MAILTO is a common reason why system mailboxes fill with email - root's mailbox, for example. If such system mailboxes are not aliased, they will typically not be checked often (or at all).

0 3 * * * /usr/local/bin/backup.sh

Capture Error Output

Capture only STDERR by redirecting STDOUT to /dev/null. Since this is per-command, you can mix and match and have some commands that discard all output, and some that capture only STDOUT.

MAILTO=doug@example.com 0 3 * * * /usr/local/bin/backup.sh > /dev/null

Capture All Non-Error Output

Capture only STDOUT by redirecting STDERR to /dev/null.

MAILTO=doug@example.com 0 3 * * * /usr/local/bin/backup.sh 2> /dev/null

Sample Email

Here is what the email from cron will typically look like, notice the subject line contains the hostname, the program that was executed, and the user that executed it, so it can be filtered quite easily. A line like Starting backup... would have come from an echo statement inside the shell script.

Date: Sun, 6 May 2012 03:00:01 -0400 (EDT) From root@nix.example.com Sun May 6 09:00:05 2012 From: Cron Daemon To: doug@example.com Subject: Cron /usr/local/bin/backup.sh Starting backup... First line of output Next line of output Backup complete...

More Reading

posted at: 10:51 | path: / | read more | bsd, cron, email, linux, sysadmin, tips, unix

Wed, 21 Mar 2012

Using Install to Copy Files and Change Ownership/Permissions at the Same Time

The little-known but still useful Install command is most commonly seen in makefiles when implementing the 'install' target (i.e., as the last part of the ubiquitous configure; make; make install sequence). You can use the install command as a convenient replacement for separate cp, chmod and chown commands in shell scripts, or just from the command line when you need to copy a file and change its ownership or permissions at the same time. It's a great tool to add to your shell-scripting arsenal.

Using Install

Usage is very simple, here is an example with some common options:

install -C -m 770 -u root -g staff backup.sh /usr/local/bin/

This will copy the file backup.sh from the current directory into /usr/local/bin, changing the permissions to 770, the owner to root, and the group to staff. The -C option compares the source file and supplied attributes (in this case permissions, owner and group) to the destination file, if it exists, and performs the copy only if they differ in some way. Note that -u implies that you are running the install command as root.

More Reading

posted at: 09:21 | path: / | read more | chmod, chown, coreutils, cp, install, scripting, shell, tips

Thu, 15 Mar 2012

Four Tips to Moving Around the Filesystem Quickly With the Bash Shell

I used to teach Linux/Unix fundamentals and Linux sysadmin courses, which were both by nature centered around use of the shell. What I observed was that many of the frustrations of novice shell users revolved around mistakes they made while typing, and time spent trying to remember directory paths or filenames. Part of getting comfortable with the command line environment was learning how to save time and either avoid or work around these issues. Luckily, there are several tools we can use to make our life in-shell easier.

Tab Completion

The first trick is in my opinion the most useful, even in its most basic form. That is, at any point while entering a command or command argument, you can just hit the TAB key to complete it. If there are multiple choices, hitting TAB twice will show you a list. It's that simple to use. Here is how this works under the hood, from the completion section of the bash manual page:

complete (TAB) Attempt to perform completion on the text before point. Bash attempts completion treating the text as a variable (if the text begins with $), username (if the text begins with ~), hostname (if the text begins with @), or command (including aliases and functions) in turn. If none of these produces a match, filename completion is attempted.

Completion can also be programmed, for example you can specify that the shopt bash builtin command will complete from the list of available shell options. Debian-based GNU/Linux systems have a very useful bash-completion package with a lot of the most useful programmable completions enabled, including the aforementioned shopt option completion.

Change Directories With Variable Names by Setting cdable_vars

There are some directories you will find yourself cd'ing into all the time. These are good candidates to save for future use, by putting each directory name in a variable, at the same time setting the shell option cdable_vars. Here is an example of what would go in your ./bashrc, just change the directory path to suit.

shopt -s cdable_vars export doc="/home/dmaxwell/Documents"

Here is an example of how to use cdable_vars.

dmaxwell@kaylee:~$ source .bashrc dmaxwell@kaylee:~$ shopt cdable_vars cdable_vars on dmaxwell@kaylee:~$ echo $doc /home/dmaxwell/Documents dmaxwell@kaylee:~$ cd doc /home/dmaxwell/Documents dmaxwell@kaylee:~/Documents$

One note - tilde expansion, which normally allows you to use ~ as a shortcut for your home directory, will not work with cdable_vars - so if you had this in your .bashrc for example:

export doc="~/Documents" # Warning - will not work

Then typing cd doc would result in a bash: cd: doc: No such file or directory error.

Correct Spelling Mistakes On-the-fly with cdspell

It happens to all of us - transposed letters or minor case errors mean recalling the last command from history and editing it to fix the mistakes. Setting the cdspell bash shell option will fix many such minor errors for you on-the-fly. Just enable it in your .bashrc as follows:

shopt -s cdspell

Now let's see how it works, with a case error and some transposed letters:

dmaxwell@kaylee:~$ ls -ld Documents drwx------ 37 dmaxwell dmaxwell 4096 Mar 15 08:54 Documents dmaxwell@kaylee:~$ cd documents bash: cd: documents: No such file or directory dmaxwell@kaylee:~$ source .bashrc dmaxwell@kaylee:~$ shopt cdspell cdspell on dmaxwell@kaylee:~$ cd documents Documents dmaxwell@kaylee:~/Documents$ cd dmaxwell@kaylee:~$ dmaxwell@kaylee:~$ cd tpm tmp dmaxwell@kaylee:~/tmp$

The above also illustrates another nice shell shortcut - a simple cd alone on a line will always bring you back to your home directory.

Store Commonly Used Directories with CDPATH

It's not uncommon to forget where you are in a directory tree, or to forget where that latest projects subdirectory is. CDPATH can help in these situations. Start by putting something like the following in your .bashrc:

export CDPATH=":~/Documents/projects"

CDPATH is an environment variable that should contain a colon-separated list of directories. In this case, we have two - the current directory (given by the first empty string), and the directory ~/Documents/projects. When you use the cd command, if the target path is not found, the shell looks under those directories you specify in CDPATH. Let's see how it works.

dmaxwell@kaylee:~$ ls -ld Documents/projects/foo drwxr-xr-x 2 dmaxwell dmaxwell 4096 Mar 15 09:06 Documents/projects/foo dmaxwell@kaylee:~$ cd foo bash: cd: foo: No such file or directory dmaxwell@kaylee:~$ source .bashrc dmaxwell@kaylee:~$ echo $CDPATH :/home/dmaxwell/Documents/projects dmaxwell@kaylee:~$ cd foo /home/dmaxwell/Documents/projects/foo dmaxwell@kaylee:~/Documents/projects/foo$

We can see that ~/Documents/projects/foo would not normally be accessible directly from the home directory, being two levels deeper. Once we add ~/Documents/projects to our CDPATH, the shell can now find the directory foo.

More Reading

posted at: 11:58 | path: / | read more | bash, bash-completion, basics, cd, cdable_vars, cdpath, cdspell, completion, shell, shopt, tips

Mon, 12 Mar 2012

Monitoring and Alerting on Patterns in Linux Logfiles

As a sysadmin, I've found it's always useful to monitor system logs on your Linux or Unix servers for specific patterns of activity, things that can indicate security or system issues. Even nicer to get alerts when activity occurs. I know there are tools that will do this, but they tend to be all-inclusive and complex. I wrote this perl script that monitors any logfile for a specific pattern and generates email or syslog alerts because I had the need to alert on one specific pattern, and wanted something self-contained that was easy to configure.

Installing Logmon

It needs a few non-core Perl modules to run, namely Mail::Mailer, Proc::Daemon, Unix::Syslog and File::Tail, but these can be installed pretty easily as packaged modules or via CPAN. On Debian/Ubuntu systems, all the needed modules are pre-packaged for you:

apt-get install libmailtools-perl libunix-syslog-perl libfile-tail-perl libproc-daemon-perl

On Red Hat/Fedora servers, you can use yum:

yum install perl-MailTools perl-Unix-Syslog perl-File-Tail perl-Proc-Daemon

To pull in all the modules from CPAN, you can use this one-liner:

for m in Mail::Mailer Proc::Daemon Unix::Syslog File::Tail; do perl -MCPAN -e "install $m"; done

Once the modules are installed, download the script and double check that it will run without error, then copy it to your path and make it executable. You should see no errors about missing modules when you run the script under 'perl -cwT'.

perl -cwT ./logmon.pl install -m 755 logmon.pl /usr/local/bin/

Running Logmon

It's meant to be both simple and secure, here is the usage summary:

logmon.pl synopsis: Daemon that periodically checks logfile for a pattern and send alerts Pattern is always required. If no other options are given, defaults to syslog alerts and monitors /var/log/messages for given pattern. Usage: logmon.pl -p pattern [-m alerts@example.com] [-f logfile] [-u run as user] [-g run as group] [-i max interval] [-v] [-d] [-h] -m: Email destination for alerts -f: logfile to monitor -p: Pattern to match against lines in logfile (Perl regexp, match is case-insensitive) -u: Run with permissions of user -g: Run with permissions of group -i: Max time to sleep between checks -d: Debug output to STDERR, do not daemonize -v: Verbose logging (use with caution or you may have endless alerts) -h: This help text

Running the script is pretty straightforward, you specify a pattern to match against with -p (this is the only required parameter), and optionally an email recipient (-m) and logfile to watch (-f). Here is an example. Let's say you want to get alerts whenever MySQL detects a crashed table. The syslog event for this looks like this on my Ubuntu box:

Jul 17 08:02:49 kaylee mysqld[1532]: 110717 8:02:49 [ERROR] /usr/sbin/mysqld: Table './mysql/user' is marked as crashed and should be repaired

And here is the command line usage:

logmon.pl -p 'mysqld.+?table.+?crashed' -m you@example.com -u nobody -g adm -f /var/log/syslog -i 30

Note that the pattern match is case-insensitive. When run this way, Logmon will detach itself from your terminal and run as a daemon, checking /var/log/syslog every 30 seconds for the supplied pattern. I recommend you use the -u and -g options to force Logmon to drop it's privileges, just make sure you specify a user or group that have read-permissions on the specified logfile. On Debian and Ubuntu servers, all the system logs are readable by the group 'adm'.

Other Options and Tips for Testing Logmon

Logmon will dump alerts to your default system log if you leave off the -m option. If you also use the -v option for verbose logging, these syslog entries and alerts will have pattern and match data. If you end up monitoring the same file you are dumping alerts into you'll get an endless series of alerts continuously being added to the system log. For this reason when alerts are sent to syslog, by default they are very generic (email alerts are always verbose).

To test Logmon, use the -v and -d options together. Run this way, Logmon will not daemonize itself, and will just print alerts and activity to your console (STDERR).

Errors

Any errors that cause the script to die while it's running as a daemon can be found in your system log.

Startup and Shutdown

I haven't yet written any startup scripts for Logmon, although I plan to. For now, just start it from one of your system's startup scripts, perhaps /etc/rc.local, and if you have to stop it you can just use pkill logmon. Please send any bugs or suggestions for improvement to the email address in the script header, or leave a comment here.

posted at: 16:12 | path: / | read more | linux, logs, monitoring, perl, regexp, scripts, sysadmin, tips

Fri, 09 Mar 2012

Learning Network Security with Linux Iptables Firewall Scripts

The essence of security is simplicity, and when it comes to workstation or small-office Linux firewalls I have always been a fan of direct iptables use over some of the more popular alternatives ([g]ufw, fwbuilder). While they may be easier to use, they also hide a lot of the details. Especially when you are starting to learn about firewalls and network security, I believe you are better served using customizable firewall scripts like the two I detail below. When you get comfortable with iptables and networking concepts, then you can look to some of the other solutions. At that point, you'll fully understand what they are doing under the hood.

Iptables Scripts

The first aptly-named shell script, 'firewall.sh', is meant to protect a SOHO or home office network behind a dual-homed (two interface) firewall. It doesn't support DMZ hosts, but does support the most common scenario of SOHO or home firewalls doing double-duty as SSH or web servers. It features forwarding, NAT (network address translation), syn-flood protection and rate-limiting for log entries.

The next script, 'bastion-host.sh', is much simpler, and is meant to be used on any host directly connected to the Internet, like a home workstation or laptop. It drops all inbound connections by default. Both scripts are well-commented, with any variables and each section explained. You can download the scripts here:

Startup Options

The way I like to use these scripts on a Debian or Ubuntu system (see below for an alternative if you use network manager) is very simple and is as follows:

First, put your chosen script in /usr/local/sbin, and make it owned by root with permissions 0700.

sudo cp ./firewall.sh /usr/local/sbin; chown root.root /usr/local/sbin/firewall.sh; chmod 0700 /usr/local/sbin/firewall.sh

Edit /etc/network/interfaces, and add the following line to the interface stanza of your external interface (usually eth0):

pre-up /usr/local/sbin/firewall.sh

So the stanza for your external interface will probably look something like this when you are done:

interface eth0 inet dhcp pre-up /usr/local/sbin/bastion-host.sh or interface eth0 inet static address 10.1.1.254 netmask 255.255.255.0 gateway 10.1.1.1 pre-up /usr/local/sbin/firewall.sh

On desktop systems where you are using the network manager application, or on Red Hat, CentOS or Fedora systems, you can put scripts like this in /etc/rc.local (On Red Hat systems the comments and touch command are there already by default):

#!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff. touch /var/lock/subsys/local /usr/local/sbin/firewall.sh

Just make sure that if you use this method on Red Hat based systems, you stop the default iptables firewall:

/etc/init.d/iptables stop chkconfig --level 2345 iptables off

If you would rather integrate your firewall into the Red Hat startup scripts, run the firewall script of your choice directly. Then save the rules so they will be read by the iptables init script:

/usr/local/sbin/firewall.sh iptables-save > /etc/sysconfig/iptables service iptables restart

You would then need to do this every time you made a firewall script change.

Testing

When you make changes to the script, just run it again directly and check the firewall status (see below). There are times when an erroneous change will lock you or a network client out of your server. If you have direct access to the host, you can correct any errors that occur immediately from the console. If you are making firewall changes over an SSH session, rename the firewall script first, and make changes on the copy, so that you can just reboot the box as a last resort to get a known-good configuration. Some hosting providers also provide a remote console that is ideal for fixing mistakes.

Monitoring Firewall Status

You can view the currently loaded ruleset as follows:

iptables -L -nvx iptables -t nat -L -nvx

The -nvx options give you the most detail possible - showing IP addresses instead of hostnames and full packet counts. This comes in handy if you want to see how often a rule is being hit, or if some rules never get hit. The option -t nat shows just the rules in the NAT table.

One final note, there is an ip6tables command that is the analogue of iptables for IPv6, and can be used independently of and alongside it.

References:

posted at: 12:08 | path: / | read more | firewall, ip6tables, iptables, linux, networking, scripts, security

Page 1 of 4  
[FSF Associate Member #286]