Configuration files best practices

Overview

Modifying configuration files is a normal part of system administration and in this document I describe using version control software for managing configuration file revisions. I consider this to be a best practice in system administration.

You’re going to need to modify various configuration files to do something, either to install some new software, or modify some default system behavior. Rather than make manual backup copies of the “original” configuration files, and having lots of copies such as “vfstab-orig”, “vfstab-“, “vfstab–“, “vfstab-old”, “vfstab-Jul17”, etc, you should use some version control software to manage the multiple revisions of such files. This provides a clean system, you’ll only have *one* “vfstab” and with the revision control in place, you can retrieve any previous version. When used properly, you can document each change you make so next week or next year you’ll know why you added/deleted/modified some lines in the configuration file.

I pretty much do this for all files I need to edit, including hosts, inetd.conf, /kernel/drv/sgen.conf, vfstab, dfstab, nsswitch.conf, services, sshd_config, httpd.conf etc. I use RCS and coupled with emacs it’s easy to update files as emacs has built in commands to perform the version control “check in” and “check out”. Emacs will also prompt you for the change comments which will be checked into the system along with your changes. You’ll always have the latest version, and the files are made “read only” (so you should know that you shouldn’t just up and edit the file without first properly checking it out). And if some changes you made don’t work, you can easily retrieve the last (hopefully working) revision and recover your previous configuration.

Some files aren’t edited often, so it’s important to enter the change comments to document what changes were made because it could be weeks, months or years before you need to look at the file again and then you’ll easily know what was done when. If you have multiple people that have system administrator responsibilities, it can be very helpful for the next administrator to track down changes made to a system. But it’s important that everyone with such responsibilities are aware that version control is in place. I’ve come across some administrators that just chmod the file and edit away (the version control software always places read-only copies of the current version on the system), not realizing that they’re not properly editing the file. Coming across a file which is read-only should be a clue that maybe version control might be in place for that particular file.

In RCS, I always add two keywords in the file (in RCS these are known as keyword substitutions): the $Id$ and $Log$ keywords. The $Id$ keyword expands a one line summary of the file name, latest version and date, and the last person modifying the file. The $Log$ keyword inserts the revision history (which includes the change comments) of the file being edited. RCS will automatically expand the log lines with the prefix string used in the $Log$ keyword (such as comments that begin with #, etc). SCCS has similar ID keywords.

Here’s a sample of a vfstab that has these tags.

# $Id: vfstab,v 1.19 2006/12/10 19:02:23 root Exp $
#
# $Log: vfstab,v $
# Revision 1.19  2006/12/10 19:02:23  root
# moved vpm/domains from c1t14d0s6 to c1t14d1s0 (about twice as large)
#
# Revision 1.18  2006/07/20 15:41:51  jkwan
# added old system disk
#
# Revision 1.17  2006/07/19 22:00:15  jkwan
# updated for c0t0 new system disk, updated Solaris 8
#
# Revision 1.16  2005/10/27 14:46:30  jkwan
# replaced c0t0 disk - was IBM 18g; now Seagate 36g;
# trimmed comments from 1.1;
# removed c0t0 fs from new disk;
# added swap back on on c0t0d0s1

For an initial version control file check in, I use something like:

# $Id$
#
# $Log$
# comment about this file

Basic RCS commands

Here’s some basic RCS commands you can used. You can use other version control software such as SCCS (I’ve used that myself in the past) but check the appropriate man pages for command syntax.

• co -l file — checks out file for editing (lock)
• ci file — check in file. This is also used to check in the initial file. You’ll be prompted to enter change comments. Important After you’re done you’ll need to do a “co file” to check out the latest version. Be sure to do this otherwise you won’t have the configuration file in place which could cause problems.

Using with emacs

• ^Xvi — checks in the file you’re editing as a new file (initial file)
• ^X^Q — to check in/check out the file you’re editing (toggles)

Software Installations

Sometimes when you’re installing some third party software, during the installation process the installation scripts will make backup copies of files it edits. You should review the standard directories (/etc, /kernel etc) for any files which might have been modified and manually integrate the changes into the version controlled files.

For example, after a Veritas Netbackup software installation, these files were found in /etc:

inetd.conf.NBU_060107.11:05:31
services.NBU_060107.13:16:02

These are copies of the original versions of these files prior to the software installation (which should be the latest version checked in under the version control system). Do a diff and incorporate the changes into the version controlled configuration files.

bash-3.00$ diff -c inetd.conf.NBU_060107.11\:05\:31 inetd.conf
*** inetd.conf.NBU_060107.11:05:31      Fri Jun  1 13:16:02 2007
--- inetd.conf  Wed Jul 18 22:26:27 2007
***************
*** 24,26 ****
--- 24,30 ----
  100235/1 tli rpc/ticotsord wait root /usr/lib/fs/cachefs/cachefsd cachefsd"
  # TFTPD - tftp server (primarily used for booting)
  #tftp dgram   udp6    wait    root    /usr/sbin/in.tftpd      in.tftpd -s /tftpboot
+ bpcd  stream  tcp     nowait  root    /usr/openv/netbackup/bin/bpcd bpcd
+ vnetd stream  tcp     nowait  root    /usr/openv/bin/vnetd vnetd
+ vopied        stream  tcp     nowait  root    /usr/openv/bin/vopied vopied
+ bpjava-msvc   stream  tcp     nowait  root    /usr/openv/netbackup/bin/bpjava-msvc bpjava-msvc -transient

rm inetd.conf.NBU_060107.11:05:31 — configuration prior to software installation

the file should be the same as the last checked in verion

mv inetd.conf inetd.conf-netbackup — configuration file after software installation
co -l inetd.conf
edit file incorporating changes
rm inetd.conf-netbackup — clean up

Summary

Using version control software on configuration files is an excellent way to manage systems. It keeps the system tidy, properly documents changes over time of configuration files, and makes it easier to keep track of changes. It also provides a means to roll back a configuration file to a previous state. Additionally, when doing a system upgrade or installation, you can review the configuration files that were previously configured by looking at the RCS directories or looking for the “,v” files that RCS uses (this is for RCS). It’s important that responsible individuals are aware of version control configuration usage and how to use it. Using version control software only requires a few basic commands and only requires an additional step or two in editing a configuration file.

Authorized ssh access

aka remote ssh access without passwords

Secure shell (ssh) and it’s related utilities (scp, slogin) should be used whenever possible to provide encrypted data communications. ssh also provides forwarding of X11 data by automatically setting up the appropriate encrypted channels. Therefore you never need to mess with the DISPLAY shell environment variable when using ssh. To authorize ssh remote access, so a password is not required, the following steps should be done.

NOTE: this document specifically references OpenSSH’s version of ssh2.

• generate your personal ssh keys

Use the “ssh-keygen” program to generate your ssh keys. Typically the following will work:

ssh-keygen -t dsa -N ""

This will generate a DSA key (for ssh2, stronger encryption). The key will be stored in your home directory under .ssh/id_dsa and .ssh/id_dsa.pub. The .pub file is your public key, which is used on remote systems. The other key is your private key and should never be revealed. Your ~/.ssh directory can be read/write (plus execute) for owner only with no permission for group and other (chmod 700 ~/.ssh).

• Copy your public key to the remote system

Of course, you should use scp to do this.

local host% scp -p id_dsa.pub remote:~/.ssh/authorized_keys2

This assumes, of course you have an account on the other system and you’ve created the .ssh directory on that system. You will be prompted for your password on the remote system. The above command will copy your local host key to the remote host and it will copy it to the file called authorized_keys2 (if you already had one, then it will replace it!). The authorized_keys2 file specifies which host(s) are authorized access. It may contain more than one key (in such case, you’d want to get your public key over to the remote host and append it to the authorized_keys2 file.

After you’ve completed these step, you should be able to use ssh on the local host to the remote machine without being prompted for a password.

---

Example

Generate the personal key
fireworks 14) ssh-keygen -t dsa -N "" 
Generating DSA parameter and key.
Enter file in which to save the key (/home/jjob/.ssh/id_dsa):
Your identification has been saved in /home/jjob/.ssh/id_dsa.
Your public key has been saved in /home/jjob/.ssh/id_dsa.pub.
The key fingerprint is:
79:17:23:df:d3:f4:54:8d:c0:48:a6:f3:c2:df:ca:18 jjob@fireworks

fireworks 17) cd .ssh
fireworks 18) lla
total 206
drwx------ 2 jjob sysadmin   512 Nov 14 15:30 .
drwxr-sr-x 9 jjob sysadmin  3584 Nov 14 15:08 ..
-rw------- 1 jjob sysadmin   668 Nov 14 15:30 id_dsa
-rw-r--r-- 1 jjob sysadmin   604 Nov 14 15:30 id_dsa.pub
-rw------- 1 jjob sysadmin 38150 Nov 14 08:53 known_hosts
-rw-r--r-- 1 jjob sysadmin  5296 Nov 10 14:59 known_hosts2

• Now copy the key to the remote system

fireworks 20) scp -p id_dsa.pub eclipse:~/.ssh/authorized_keys2
jjob@eclipse's password: 
id_dsa.pub 100% |*****************************| 604 00:00

• Now test it

fireworks 21) ssh eclipse who
jjob pts/1 Jun  4 10:55 (fireworks)
jjob pts/2 May 29 08:51 (fireworks)
jjob pts/3 Jun  6 09:57 (fireworks)
...

Note: on the remote system, the destination home directory must not be group writeable. If it is, sshd won’t allow the DSA authentication.

For more information, see the man pages for ssh, ssh-keygen and sshd.

Using rdist/rsync with sudo for remote updating

Overview

Direct root login via ssh should be avoided. Instead administrative (aka root) work should be done by su’ing to root or using sudo. Apple’s Mac OS X doesn’t even have the root account enabled – everything must be done by using sudo. My systems are managed by having one master host upon which I make all my updates and rely on various scripts to update all the other systems running the same OS and configuration. That required root ssh permission on the master machine to the client machines.

The following technique allows remote updates via rdist and rsync using ssh from a master machine by using sudo on the client machines from a non-privileged account. That specific account is only authorized to execute two sudo commands. No other privileged authorization is permitted. The master machine has an ssh key generated for root, and this key is stored on each of the remote’s non-privileged account for authorized ssh access (ssh access without a password). This allows the master root machine to have ssh access to each remote through the non-privileged account. sudo is then configured to allow the non-privileged account to execute the rdist/rsync commands as root. This is required since the master machine will send file updates requiring permission, file creation/deletion updates which must be done as root. Only the master machine’s root account has access to the remote account (non-privileged) and only the one or two sudo authorizations are allowed.

Setup

Set up the non-privileged account. In my configuration I use remupd.
Note: I configure the account to use bash as the shell. There’s a small shell function that needs to be set up for rdist. If a different shell is used, that will need to be modified.

1. /etc/passwd entry

remupd:x:4761:60001:Remote Update Admin:/var/sys/remupd:/usr/bin/bash

2. /etc/shadow entry

remupd:NP:::::::

Note: this shouldn’t be “*LK*” which specifies a locked account as ssh might not allow access if an account is locked. I saw the following log message in syslog:

Nov  8 12:45:07 underscore sshd[1491]: [ID 800047 auth.info] User remupd not allowed because account is locked

3. sudoers
I define the following to authorize remupd to execute a couple of commands as root.

# Cmnd alias specification
Cmnd_Alias   RSYNCDIST=/usr/bin/rsync, /usr/sbin/rdistd
...
remupd	ALL=NOPASSWD:RSYNCDIST

The NOPASSWD flag allows remupd to execute the sudo without authentication.

4. .bashrc in home directory (/var/sys/remupd in my configuration)

rdistd() (
 /usr/local/bin/sudo /usr/local/bin/rdistd -S
)

This just sets up a shell function for rdistd for inbound rdist. If rdist isn’t being used this can be removed.

5. .ssh/authorized_keys2
On the server root account, generate an ssh key and copy the public portion to each remote in the authorized_keys2 file (see authorized ssh access for more details).

ssh-keygen -d -N ""

then copy the .ssh/id_dsa.pub to the remotes ~remupd/.ssh/authorized_keys2 file.

Testing

as root

ssh -l remupd remote host who

if set up correctly, this command should run without prompting for a password. This is the authorized ssh access test.

rsync

The following command will run rsync manually and not update any files. It’ll print out a list of files that need to be updated between the master and remote.

rsync -avHDn –delete –rsh=”ssh -l remupd -x -c blowfish” \

–rsync-path=”/usr/local/bin/sudo /usr/local/bin/rsync” \

/usr/ dixieland:/usr

Adjust the paths for sudo and rsync as necesary. This allows the ssh connection to be non-root which means it doesn’t require the “PermitRoot” ssh setting on sshd to be enabled. The server root’s ssh key is configured for authorized access to the non-root remupd account. The sudo rsync on theremupd account is is done to set up the root rsync on the remote so the link between the rsyncs run as root for proper operation.

rdist

To test rdist, create a small script, and in the update portion, specify remupd@remote which specifies to use the user name remupd on the remote for running the rdistd (client portion of rdist).

rdist -v -P rdsh -f r-s10-root

script fragment
${UPDATE_LIST} -> ( remupd@${HOSTS} )
       install -oremove,chknfs ;
       except ${EXCEPTS} ;

This works with the non-root account remupd on the remote calling sudo rdistd (specified in that .bashrc function above), to start the rdistd as root so the files it receives from the rdist server can be updated appropriately. So the communication between the server and remote doesn’t require root ssh to the remote but the end point link between them are executed as root via the sudo.

The standard rdist wrapper script I use is as follows.

rdsh wrapper script

#!/bin/sh
# rdist shell wrapper
# basically calls ssh with additional ssh options
#
if [ -x /usr/local/bin/ssh ]; then
       exec /usr/local/bin/ssh -x -q -o "BatchMode yes" -c blowfish $* 2>/dev/null
else
       echo "can't execute /usr/local/bin/ssh"
       exit 1
fi

Logging

In syslog (usually auth facility), the following example log messages are recorded when these commands are executed:

Nov 14 00:03:07 sabrina sshd[6595]: Accepted publickey for remupd from 169.232.144.42 port 59746 ssh2
Nov 14 00:03:07 sabrina /usr/local/bin/sudo: remupd : TTY=unknown ; PWD=/var/sys/remupd ;
     USER=root ; COMMAND=/usr/local/bin/rdistd -S

For rsync, the log messages look like:

Nov 14 00:05:45 sabrina sshd[6599]: Accepted publickey for remupd from 169.232.144.42 port 3407 ssh2
Nov 14 00:05:45 sabrina /usr/local/bin/sudo: remupd : TTY=unknown ; PWD=/var/sys/remupd;
     USER=root ; COMMAND=/usr/local/bin/rsync --server -vlHogDtpr --delete . /usr

Summary

For more secure configurations, the PermitRoot sshd configuration should be disabled to prevent root logins via ssh. Using rsync/rdist over ssh from a master machine to update files (such as /usr, /opt) requires a communication link between the end points that must be privileged. Using authorized ssh access coupled with sudo on the remote end accomplishes this.