Beeper Duty Overview

Contents

• Basics
• AFS
  • AFS database service
  • AFS Fileservers
• Backups
  • Amanda (workstation backups)
  • STAGE (AFS backups)
• Mail & News
  • Cyrus Mail System
    • The Cyrus IMAP Aggregator (Murder)
  • Sendmail mail routing
  • Webmail
• Printing
  • Some cluster printer is down.
• Unix Servers
  • Runaway process/Heavy processes from users who have logged out
  • Fork Bombs
• LDAP
  • Everything you might ask about LDAP, including:
  • Where are the servers running?
  • What do the LDAP servers serve?
  • Starting and stopping the servers.
  • Search for an entry
  • Change an entry
• WebISO (pubcookie)
  • Everything you might ask about WebISO
• Oracle Information
  • General Information:
  • Locations of Oracle Instances:
  • What to do if Oracle misbehaves:
  • Where to find additional information:
• Monitoring
  • Event monitoring
  • Historical data
• Miscellaneous
  • Getting Root
  • Downloading Machines
  • Service Account
  • Copying SPARC Disks
  • Solaris
  • License Servers Condensed
  • Calendar Servers Condensed
  • The Web Publishing System
  • MyAndrew

AFS database service

Amanda workstation backup system

You can find instructions on how to do a restore from Amanda via http://asg.web.cmu.edu/isam/howto/amanda.html. Man pages exist for all the amanda helper commands.

Generally you will not need to worry about amanda.

Cyrus Mail System

The Cyrus IMAP Aggregator is a system that allows a single IMAP namespace to exist (and be accessed by any IMAP client) across multiple machines. To do this, there are 3 types of machines involved in mail storage: frontends (mail-fe*), backends (mail*), and a MUPDATE server (mupdate1.andrew.cmu.edu).

The backends are normal IMAP servers with a bit of added intelligence whenever a mailbox operation is performed. In other words, clients can connect directly to a backend and have a typical IMAP session, except they will not be able to see all of the mailboxes that exist on the murder. In fact, this is how we ran the beta test: mail1 continued to be the primary point of access for most users, but was actually a full member of the murder. We currently have 5 backends, (mail1, mail2, mail3, mail4, and mail5), with mail5 being used primaraly for testing purposes.

The frontend servers are fancy IMAP proxys. They are fancy because they can switch servers mid-session and perform some operations (such as LIST) locally. Otherwise, they proxy requests to the appropriate backend server, or (client-willing) refer the client to the backend directly. Frontends are, for all intents and purposes, identical, and exist as a loadbalanced pool. If one goes down and is removed from the pool, the only clients who lose are the ones who were connected to it at the time. Otherwise, no one should notice or care what frontend they get. We currently have 6 frontends (mail-fe[1-6]), with mail-fe1 being used primaraly for testing purposes.

The Aggregator requires an authoritative server to manage the namespace. The MUPDATE server performs this task. There is also a slave MUPDATE server running on each of the frontends, to allow the frontends a local copy of the entire mailbox namespace. When updates happen on the master they are pushed to the frontends in a relatively short period of time.

Mail delivery happens to the aggregator via a process that runs on the mx and smtp servers called lmtpproxyd. This process recieves a message, queries the mupdate server for the location of its destination mailbox, and forwards the message along to that server. Note that we currently run sieve on the backend machines only, so all of a user's mailboxes must live there. (user.rjs3 and user.rjs3.spam couldn't live on different backend servers for example)

When things go wrong

The murder is designed so that restarting master on a machine (excpet for the mupdate server, which is, for the most part, considered authoritative) should bring it in sync with the state of the world as the murder sees it. That is, when a frontend restarts, it gets a fresh copy of the entire mailbox list. When a backend restarts, it performs a series of checks between its local mailbox list and what the MUPDATE server thinks it has, and either updates the MUPDATE server (if it has a mailbox that is not in MUPDATE, or the MUPDATE data is stale), or deletes the local mailbox (if MUPDATE claims the mailbox is hosted on another server). Note that if mailbox transfers fail a certain way, it may be necessary to restart both the source and target servers.

A common problem we are currently experienceing is that mon will report a "timeout" for the mupdate server on a frontend. If this is the case, typically this means that fdsync() calls have started to take a very long time (due to a Solaris bug). Generally, running /etc/remount-root will fix this problem. There is a cron job that runs this on a somewhat daily basis. Presumably this will be fixed if we upgrade to Solaris 10 or manage to wrangle a patch for the kernel out of Sun (unlikely) [rjs3 5/19/03]

Locating a mailbox

Mailboxes have two parts to their location, just like AFS: a server and a partition on that server. To get this information for a given mailbox, use the info command in cyradm. If you need to find the physical files on a given server, look in /imap/<partitionname> on the correct backend (the command mbpath on the backend will help).

Sendmail is the world-infamous MTA (mail transfer agent) that we use for moving electronic mail through a maze of twisty-little passages. It runs on every machine, but most machines run very stupid configurations of it.

When things go very wrong, complain to Larry.

• Client machine problems: remedy1, web3, web4, bugzilla, etc.
  • Look for problems with Sendmail in /var/log/mail.log.
  • This is usually due to skew between the Sendmail cf file (/etc/mail/sendmail.cf) and the Sendmail binary (/usr/sbin/sendmail). To detect this, run /usr/sbin/sendmail -bs:

    # /usr/sbin/sendmail -bs
    220 web3.andrew.cmu.edu ESMTP Sendmail 8.12.0.Beta7/8.12.0.Beta3; Wed, 22 Aug 2001 15:00:28 -0400 (EDT)
    

    The binary version is 8.12.0.Beta7 (the first number) and the configuration file is the second number (8.12.0.Beta3). It's very bad when the config file is more recent than the binary. Regardless, make sure the machine is running sane versions of the sendmail collection (in local) and the smailcf collection (in host). They should both either be the current beta release or the current gamma release.
  • Sendmail can also be very picky about file permissions. Look for "unsafe file" in /var/log/mail.log to diagnose this, and judicious use of chmod (and longer-term, package) can fix this.
  • Generally, client machines should be forwarding all of their mail to other places.
  • Client machines that accept mail (like remedy) can also fall prey to denial of service attacks; see below for possible solutions.
• SMTP server problems: smtp5, smtp6, smtp7 (may change).
  
  Similiar problems occur on the Andrew MXs (mx1, mx2, mx3)
  
  or the cmu.edu MXs (cmu4, cmu5).
  • When paged for SMTP server problems, it's usually because one of the machines has stopped responding (either due to high load or Sendmail dying). Find out which one by telneting to smtpN.andrew.cmu.edu port 25, and whichever says "connection refused" is the culprit.

    Log in and try to determine what's wrong. Generally, rebooting these systems are safe. If it keeps happening, it might be due to a denial-of-service attack (intential or otherwise) from some client machine. See below.

  • It's also possible to be paged due to high queue size. This is generally a result of other failures preventing mail from going through. Running mailq as root will probably show the problem machines.

    mail1 (the IMAP server) is generally the biggest culprit of this, and start looking at it to diagnose problems if there are a large number of messages queued for it. However, the smtp machines will queue mail for people who are over quota for up to 5 days, so generally mail queued because of over quota reasons isn't very interesting.

  • You can try to force a queue run manually by running Sendmail by hand. However, since Sendmail wants to authenticate to other SMTP servers on the network, it needs to run with special tickets. Generally you can do this via:

    # /usr/local/bin/kauth -n smtp.smtp6 -f /etc/srvtab -- /usr/sbin/sendmail -q -v
    

    where smtp6 is the first part of the hostname. You can use -qRcmu.edu to run only *cmu.edu* mail. There is also a script to run andrew.cmu.edu mail (with proper authentication) in /afs/andrew/acs/asg/coverage/forceq. This is useful if delivery has stalled for a reason that has been resolved and you want to restart mail delivery immediately.

  • If you need to kill Sendmail on an smtp server, you probably also want to kill kmemsrv so the load balancer doesn't send clients towards that machine.
  • Using graphs.andrew.cmu.edu to look at historical queue usage or other statistics can be enlightening.
• Dealing with denial-of-service attacks
  • Lots of mail from a given machine, all of it junk: That machine can be put in /etc/mail/access to refuse connections. To do it temporarily (until the next update):

    # cat >> /etc/mail/access
    badhost.example.com REJECT
    # /usr/local/sbin/makemap hash /etc/mail/access < /etc/mail/access
    

    More permanently, edit /afs/andrew/data/db/sendmail/etc/mail/access. This will be integrated into the Sendmail servers every 5 minutes.

  • Using qtool.pl to cleanup. If everything is being sent with a single sender, something like this might work:

    # mkdir /var/spool/mqueue/hold-mar-05-2002
    # cd /var/spool/mqueue
    # /afs/andrew/acs/asg/coverage/qtool.pl -e '$msg{sender} =~ /zx22/' hold-mar-05-2002 q1
    

    where hold-mar-05-2002 is an arbitrary hold directory and /var/spool/mqueue/q1 is the mail queue with all the junk in it. If there are multiple queues with junk in them, you can specify multiple from directory at the end of the qtool.pl command line.

    You can now use mailq to make sure you got everything or mailq -oQ/var/spool/mqueue/hold-mar-05-2002 to see that everything in the hold directory is junk. If it is junk, feel free to remove the directory. [leg 03/05/02]

• Setting up new servers
  • Look at /afs/andrew/wsadmin/cyrus2/README

Webmail service is provided on the loadbalanced pool of webmail machines, using the host sqmail collection, which contains our local version of Squirrelmail, an open-source PHP based IMAP client. Each of the webmail servers runs an SSL-capable apache, which uses webiso to authenticate user.

Because webmail must maintain session information, as well as preferences information that is shared among all the servers, the /afs/andrew/data/db/squirrelmail tree exists. While the sess/ tree should basicly maintain itself, the prefs/ tree (which contains a large number of volumes, which are used based on hashes of the username), may occasionally have a volume fill. In this case, some users will not be able to write out preferences files, address books, or upload attachments. In this case, the most likely cause is attachments that haven't been properly garbage collected. The attachment temporary files have fairly obvious names (they look like hashes), as opposed to the preferences and addressbook files (which are username.pref and username.abook). Deleting any attachments that are older than a half hour or so should be safe. If this doesn't clear enough space, just increase the quota on the volume.

There is also a webmail bugzilla project which may be enlightening for known issues. If something goes wrong that you can't fix, complain to Rob or Larry.

To bring up a new webmail machine, just copy the package.proto, pubcookie key, and SSL keys and certs from an existing machine. Do NOT run keyclient, since it will invalidate the keys on the other machines. (alternatively, you can run keyclient -d to just download the current key)

Printing

• No, there's no FAQ. If there was, it would say:
  • printer-N down, reboot spooler-N.
  • If that doesn't fix it, contact Shawn, and we'll fix it and add the fix to the FAQ.
  Up until now, rebooting has always worked. :) [ryan 12/12/96] Well, now it didn't. [shadow 04/10/99]

  Other things to try include nmapping the printer to ensure that the jetdirect port is working, e.g.:

  /afs/andrew/system/gamma/@sys/ng/bin/nmap cyh-8000.cyert.andrew.cmu.edu
  
  Starting nmap V. 2.54BETA36 ( www.insecure.org/nmap/ )
  Interesting ports on CYH-8000.CYERT.ANDREW.CMU.EDU (128.2.111.219):
  (The 1551 ports scanned but not shown below are in state: closed)
  Port       State       Service
  21/tcp     open        ftp                     
  23/tcp     open        telnet                  
  80/tcp     open        http                    
  280/tcp    open        http-mgmt               
  515/tcp    open        printer                 
  631/tcp    open        ipp                     
  9100/tcp   open        jetdirect               
  

  Note that you may need to do this from the spoolers themselves, since printers can be configured to restrict access by IP. Some machines may also have nmap accessable as /usr/ng/bin/nmap

  If port 9100 isn't open, this can indicate a problem with the printer. Additionally, printers grab some of their configuration from sentry.net, so it helps to see of the tftp server is running:

  [11:06 12 powertool:~]$ tftp 128.2.4.8
  tftp> get ClusterJetDirect.cnf
  Received 3609 bytes in 0.0 seconds
  

  [rjs3 7/23/02]

  Mike Kelleher adds the following advise about printers needing to be powercycled (a distinct posibility if 9100 isn't open, as above):

  Between 8am-12am weekdays and 10am-10pm weekends (12am for Sunday) you should be able to reach a CCon at x8-8776 who will be happy to do that.

  All CCons on duty are also instructed to subscribe to the "ccon" zephyr class. If you want to private z the CCon on duty run /afs/andrew/acs/ac/clusters/bin/whoson and talk to the Wean CCon.

  After hours, well we won't mind if you do it :), otherwise send email to clusters+ and we'll get it in the morning.

  [rjs3 11/14/02]

  There's a bug in the printing software that is triggered when something like this happens. Basically, once the spooler has failed to contact a printer, it stays in the broken state indefinitely. you need to 'klpc restart <printername>' as some appopriately privilged user (root on the affected spooler, or someone listed in /etc/lpc.permits). There are some scripts which attempt to do this automatically, but they don't always work

  [cg2v 11/21/02]

• Runaway process/Heavy process from user who logged out

  If operations gets called about processes on unix servers, the basic policy is that the coverage staff should *not* kill the processes if possible. But they can feel free to lower the priority (nice +20) on the processes.

  The exception to this rule is if the jobs appear to be runaway or if the processes are "seriously" hurting performance for everyone else. ie. use your best judgement or you can try to contact John Lerchey if you don't want to make the decision.

  At any time that you have to do something, please save the ps output and send that, along with the unix server and the time to John Lerchey. [wcw 15-feb-1996]

• Fork Bombs

  Every so often someone will accidentially or intentionally set off a fork bomb on a Unix server. This is when a process gets into a loop where all it does is call fork repeatedly, causing lots of processess to come up, each of which is also calling fork until it can't fork anymore.

  When this happens, you can su root, su user, and kill -9 -- -1, which blows away all the user's processes (including the shell you're running). "-1" is a special argument for kill that says "kill all my processes".

  Anything else is ineffective, since the processes are spawning as quickly as they can be killed.

  Because we (Derrick) put in user limits after the last round of fork bombs, this probably won't be necessary, but it's here if it is.

  There's an Operating Systems project where students write shells, and as a result this seems to happen a few times at the beginning of the semester. [tjs 12-oct-1998]

LDAP

All questions are preceeded by a "Q: " string, and the question is written in upper case letters. You may search for keywords in the questions by doing upper case only searches.

Q: WHAT THE HECK IS LDAP?

Good place to start. LDAP (Lightweight Directory Access Protocol) is a database that holds records for all people and accounts on campus. The database has a tree structure similar to a filesystem so that entries can be placed in a heirarchical system. Each entry is identified by a "distinguished name" or "DN", which reads like the path to a file in a filesystem, except the divider between components is a comma instead of a slash. The order of the components in the DN is from specific to general. Example:

    uid=adamson,ou=Account,dc=andrew,dc=cmu,dc=edu

    ou=Person,dc=cmu,dc=edu    Humans past, present, and future on campus
    ou=Account,dc=cmu,dc=edu             cmu.edu accounts
    ou=Account,dc=andrew,dc=cmu,dc=edu   Andrew accounts
    ou=Account,dc=cs,dc=cmu,dc=edu       Computer Science accounts
    ou=AuthEntity,dc=cmu,dc=edu          Special authentication identities 

The LDAP system is used by the Email system to route Email in the Andrew and cmu.edu domains. It is used by "finger" to find and fetch all information about people. It is used by web pages on www.cmu.edu to find people. Mail clients can use LDAP to find and resolve names on the "To:" line of composed messages.

Q: WHERE IS LDAP RUNNING?

The main machine is metadir.andrew.cmu.edu. There are slave servers running on these machines:

• ldap1.andrew.cmu.edu
• ldap2.andrew.cmu.edu
• ldap3.andrew.cmu.edu
• ldap4.andrew.cmu.edu
• mail-ldap1.andrew.cmu.edu
• mail-ldap2.andrew.cmu.edu

The main machine, metadir, takes updates and is the only one that can actually write new data to the others. People can only query the others, not make changes to them. The other servers are intended for:

	mail-ldap*	Andrew and cmu.edu Email routing. VERY IMPORTANT
	ldap*		Andrew mail client To: line name resolution

Since we add and delete servers, you might want to look at CNAMEs to figure out what server is doing what. Currently defined CNAMEs [leg 8/22/01]:

• finger-ldap
• remedy-ldap (remedy help center stuff)
• mail-ldap (load balanced, Sendmail usage)
• search-ldap (web page)
• pine-ldap (the pine e-mail client)
• netreg-ldap (for netreg)
• imsp-ldap (IMSP proxy usage)
• update-ldap (for write access)

Solaris:   (currently metadir,ldap1,mail-ldap[12])

/etc/init.d/openldap start

/etc/init.d/openldap stop

/etc/init.d/openldap restart

Linux:   (currently ldap[234])

/etc/rc.d/init.d/openldap start

/etc/rc.d/init.d/openldap stop

/etc/rc.d/init.d/openldap restart

Use restart when possible. There are other some other services that may need to be started and stopped alongside the LDAP server. If the server stopped unexpectedly and you want to start it up again, the restart command will prevent the situation where you get multiple copies of the other services running.

There is a wrapper program called slapd.wrapper that is run by the init script's start and stop commands. The wrapper will attempt to restart the slapd daemon if it were to stop unexpectedly (aka "crash"). The wrapper will wait about 10 seconds after the child dies and then restart it. If it crashes fast several times, it will send a zephyr to the people listed in /usr/openldap/etc/admin and slow down the restart process. So if you try to kill slapd by sending the slapd process a signal, it will be restarted. Use the init scripts. The stop command will signal the wrapper to exit and it will signal the slapd daemon to exit too. The restart command will only signal the slapd daemon to exit, and then the wrapper will restart it.

Q: HOW DO I TELL IF LDAP IS RUNNING ON A MACHINE?

A good way is to see if some program is listening to the LDAP port:

	% netstat -an  | grep 389

	*.389      *    *   0 LISTEN

	% ps -e | grep slapd

Q: WHERE ARE ALL THE FILES THAT LDAP USES?

Look in /usr/openldap/ on any LDAP server:

	bin/			client programs
	libexec/		daemon binaries
	db/			BDB database files
	etc/			update scripts
	etc/opendldap/		config files for slapd
	etc/openldap/schema/	LDAP schema files
	share/			odd config files, e.g. nicknames
	feeds/			on the master server, the update feed files
	logs/			output from slapd and slurpd
	var/			LDIF files, core files, PID files
	var/openldap/replica/	bookkeeping on replication
	lib/			LDAP libraries (not installed by default)
	lib/perl/		modules for performing updates on master 
	lib/perl/Feed/		modules for each feed type

Q: WHERE DOES THE SOFTWARE COME FROM?

The base LDAP service is in the source collection "local/openldap", or,

	/afs/andrew/system/src/local/openldap/

	/afs/andrew/system/src/host/cmuldap

	%define doesopenldap
	%define doescmuldap

	/afs/andrew/wsadmin/services/lib/openldap.generic
	/afs/andrew/wsadmin/services/lib/cmuldap.generic

	/afs/andrew/wsadmin/services/etc/openldap.*

Q: HOW DOES THE MASTER MACHINE UPDATE THE SLAVE MACHINES?

A process called slurpd runs on the master machine (metadir). When updates occur in the LDAP database, slapd writes to a log file called /usr/openldap/var/rep.ldif. There may be several processes that want to read that file, so a program called "repd" will take rep.ldif and make multiple copies of it to other filenames, according to its config file /usr/openldap/etc/openldap/repd.conf. One of the copies it makes is called /usr/openldap/var/rep.slurpd.ldif. The slurpd process runs in the background, polling that file for updates. When updates appear, it uses normal LDAP calls to tell all of the slave servers.

Q: HOW DOES SLURPD START AND STOP?

/etc/init.d/openldap.rep start

/etc/init.d/openldap.rep stop

/etc/init.d/openldap.rep restart

Just like slapd.

Q: HOW DOES THE LDAP DATABASE GET BACKED UP?

The LDAP server uses Sleepycat BDB database files to keep data. These files are always open, so writing them to tape is no good. To properly write them to tape or get a flat file copy of them, the LDAP server must be stopped, which is an interruption in service.

For this reason, there is a second LDAP server on the master server, listening on port 3890. The main server sends updates to this second server through the normal slurpd replication model. The backup server is not intended to service any queries, so it does no indexing or optimizations, and its ACL's are all shut off.

Each evening, CRON runs /usr/openldap/etc/ldapbackup.sh at 7:00pm. This time was chosen because it is about an hour before the nightly backup system, Amanda, runs. The ldapbackup.sh script shuts down the backup LDAP server and then runs /usr/openldap/sbin/slapcat to make a flat file copy of the LDAP database held by the backup server, then restarts the server. There is no interruption in service, since the backup server does not serve users.

When Amanda runs, it will make copies of the BDB files, but these are of little use. However, it will make a copy of the flat file database, which could be used in a disaster recovery situation.

This backup server gets started and stopped like the master LDAP server, except its script is

	/etc/init.d/openldap.bak  {start|stop|restart}

Q: HOW IS IT THERE ARE TWO LDAP SERVERS ON THE MAIN MACHINES?

The main server uses files in the default directory

	/usr/openldap

Q: HOW DO I RESTORE THE LDAP DATABASE?

First you have to determine how widespread is the problem that makes you want to restore the database. There's one of 3 situations, listed in increasing order of unfriendliness:

1. The master server crashed and its database is corrupt
2. Bad data got into LDAP today and sent to the slave servers too
3. Bad data has been in the system for a while, and we have to go back a ways

Solution to scenario 1:

	# on metadir:
	su $LOGNAME.root
	echo ${LOGNAME}@ANDREW.CMU.EDU >> /.klogin
	cd /usr/openldap
	/etc/init.d/openldap stop
	mv db db.corrupt           (optional)

	# on mail-ldap2:
	su $LOGNAME.root
	cd /usr/openldap
	/etc/init.d/openldap stop
	tar -cf db.tar db
	klog $LOGNAME
	scp db.tar root@metadir:/usr/openldap/.
	/etc/init.d/openldap start

	# when that is done, on metadir:
	tar -xf db.tar
	/etc/init.d/openldap start

Solution to scenario 2:

	# on metadir
	su $LOGNAME.root
	echo ${LOGNAME}@ANDREW.CMU.EDU >> /.klogin
	cd /usr/openldap
	/etc/init.d/openldap stop
	/etc/init.d/openldap.rep stop
	echo > var/rep.slurpd.ldif
	mv db db.corrupt
	mkdir db
	sbin/slapadd -l  backup/db/backup.ldif

	tar -cf db.tar db

	/etc/init.d/openldap.back stop
	/bin/cp db/nextid.dbb    backup/db
	/bin/cp db/dn2id.dbb     backup/db
	/bin/cp db/id2entry.dbb  backup/db
	/etc/init.d/openldap.bak start

	/etc/init.d/openldap start
	klog $LOGNAME

	# on the slave:
	su $LOGNAME.root
	echo ${LOGNAME}@ANDREW.CMU.EDU >> /.klogin
	cd /usr/openldap
	/etc/init.d/openldap stop
	mv db db.corrupt           (optional)

	# on metadir:
	scp db.tar root@<SLAVE SERVER NAME>:/usr/openldap/.
	/etc/init.d/openldap start

	# when that is done, on the slave server:
	tar -xf db.tar
	/etc/init.d/openldap start

When all slaves are updated, restart the replication server:

	/etc/init.d/openldap.rep start

Solution to scenario 3:

	/usr/openldap/backup/db/backup.ldif

You can try getting the LDAP database more up to date if you think the feed files are uncorrupted. You'll need to look in the directory

	/usr/openldap/feeds/

	# on metadir
	cd /usr/openldap
	etc/ldapfeeds.pl -old 010305 -new 010320

	/usr/openldap/feeds/ldapfeeds.output.010320

(Also see the next question to MODIFY an entry)

The "ldapsearch" command should be all you need. It's in /usr/local/bin/ on most machines, and can be run with --help for a list of all command line switches. The command tends to look like this:

      % ldapsearch -h ldap2 -b dc=cmu,dc=edu  cmuAndrewID=adamson 

If you get an error about command line options, make sure you are using the OpenLDAP ldapsearch and not one from the OS vendor or Netscape or other vendors, as the command line switches can vary. Specify /usr/local/bin as the path to the program.

There are more general searches you can conduct, changing the filter at the end of the command line to something like

      'cmuDepartment=Services Development Group (Comp Services)'

      (&(attr1=value1)(attr2=value2)(!(attr3=value3)))

If there are only 1 or 2 attributes you are interested in instead of the entire entry, you can append them to the commandline. For the ldapsearch command if anything appears after the filter, it is taken as a list of attributes you want to see.

      % ldapsearch -h ldap2 -b dc=cmu,dc=edu  cmuAndrewID=adamson homePhone cn

(Also see the next question to ADD an entry)

You need to write an LDIF file and feed it to ldapmodify. A quick crash course on LDIF... It is flat ASCII, with lines separated by carriage returns, and entries separated by blank lines. Example:

dn: uid=blef,ou=Account,dc=andrew,dc=cmu,dc=edu
mail: blef@andrew.cmu.edu

dn: uid=fnork,ou=Account,dc=andrew,dc=cmu,dc=edu
cn: John Fnork
cn: John Q Fnork

The ldapmodify program lives in either /usr/openldap/bin or /usr/local/bin on most machines. You have to give it a few commandline args, and you will need a Kerberos ticket. A typical line looks like this:

    % ldapmodify -h metadir -f ./ldif

IMPORTANT NOTE: If your LDIF is all new values to APPEND to entries, you are fine. If you want the old values that are in the entry removed and REPLACED with the values you have in your LDIF file, add a "-r" switch to the command. Again, you don't have to list the entire entry in your LDIF -- if you add -r to the command, only attributes explicitly included in your LDIF file will be overwritten. If one entry lists "cn" and a second lists "mail", and you use the -r switch, the "mail" attr of the first entry WILL NOT BE CHANGED, nor will the "cn" of the second entry. "Inclusion" of attributes is on an entry by entry basis, not the entire file.

In order to REMOVE a SINGLE value from an attribute, list all of the other values in the LDIF file and use the -r switch. To remove ALL of the values, say all of the "labeledURI" attribute values, add "labeledURI: " to the LDIF. BE SURE to have the colon and the space.

What could go wrong?

If there is a schema violation, you are trying to add/change attributes in an LDAP entry that can't have those attributes. For example, if you try to append a "cmuSIScat" attribute to an Account entry, it will fail -- you can only add that attribute to a Person entry (with the guid=<hex> in the DN).

If an error occurs with one entry, the ldapmodify command will stop. You'll have to take the tail of the LDIF file and run it through ldapmodify again. You can add the "-c" switch to tell ldapmodify to (c)ontinue after errors.

Q: HOW DO I ADD AN ENTRY TO THE LDAP DATABASE?

(See also the previous question for MODIFYING an entry, and the next question for creating a GUID)

You will need a file containing a complete LDIF copy of the entry you want to add to the database. See the previous question for a crash course in LDIF. The file will contain the COMPLETE entry, with all values filled in. One way to start is to fetch a similar entry from LDAP and plagarize.

With the LDAP file ready, feed it into ldapadd:

    % ldapadd -h metadir -f ./ldif

You shouldn't have much need to create entries by hand.

Q: HOW DO I CREATE A NEW GUID?

This will come up if you're manually adding a new entry for a Person for some reason. (This situation could arise if you find a Person entry that is the combination of two humans into one entry, and you need to splice them apart into two Person entries.) All Person entries have a DCE GUID in their DN to uniquify them across the campus forever.

There is a program on the main LDAP server

	/usr/openldap/etc/guid

The program produces slightly better GUIDs if run as root, since on SUN machines only root can get to the MAC address of the ethernet card, which is used as an input into GUID production.

Q: WHO IS AUTHORIZED TO MAKE CHANGES IN THE DATABASE?

You can read the slapd config file to get the list of LDAP DN's that can make writes to the database. The config file is

	/usr/openldap/etc/openldap/slapd.conf

  access to attrs=telephoneNumber,homePhone
    by group="cn=Administrators,ou=Group,dc=cmu,dc=edu" write
    by * read

In general, you will find that to make changes you must be in the cn=Administrators group, or bind as the cn=Manager identity listed in the "rootdn" directive in the slapd.conf file.

The cn=Administrators group lists several people's admin instance. These people, with their Admin ticket, can change anything and add new entries.

Q: WE'RE ALL LOCKED OUT! HOW DO WE CHANGE THE DATABASE TO ADD SOMEONE BACK IN?

Some idiot deleted everyone from the administrators list, eh? These things happen. You will need to log into the master server (currently metadir.andrew) and change the slapd config file.

	/usr/openldap/etc/openldap/slapd.conf

By default, the rootpw line has a bogus value, because we don't want people using the rootdn directly, but this is an emergency, right? You will need to DES encrypt the new password. You did think of a new password, right? Something that people aren't going to guess, right? You can DES encrypt it using the crypt() function, available from perl quickly with

    % perl -e '$a=<STDIN>; chop $a; print crypt($a, "SALT") . "\n";'

    rootpw {CRYPT}SAYoATkkQuOb7

Now add someone to the cn=Administrators group. Write an LDIF like:

dn: cn=Administrators,ou=Group,dc=cmu,dc=edu
member: uid=adamson.admin,ou=Account,dc=andrew,dc=cmu,dc=edu
member: uid=shadow.admin,ou=Account,dc=andrew,dc=cmu,dc=edu

   % ldapmodify -h metadir -D cn=Manager,ou=AuthEntity,dc=cmu,dc=edu \
      -x -W -f ./ldif
   Password:

The "-x" switch says to use simple authentication, and "-W" says to prompt for the password. DO NOT USE "-w <passwd>" since that leaves the password sitting in your shell, and possibly in your ~/.history file. If the ldapmodify command doesn't like "-W", then you are using the Netscape version -- go find the OpenLDAP version (/usr/local/bin, /usr/openldap/bin).

WHEN YOU ARE DONE reset the "rootpw" directive in slapd.conf to remove the password, like

    rootpw  {CRYPT}NO_PASSWORD

Q: HOW DO I REINDEX THE DATABASE?

The index files in LDAP speed up lookups for indexed items. As values are changed over time the hash tables in the index files become less and less efficient. This can be cleared up by removing the index files and running the OpenLDAP utility 'slapindex'. Of course, this can't be done while the server is running, and reindexing takes about an hour, so the indexing should be done on a separate offline copy of the database. Therefore, we use the space in the backup database server on the main LDAP server, in

     /usr/openldap/backup/

	dn2id.dbb
	id2entry.dbb
	nextid.dbb

Another problem to be faced is that a snapshot of the database is going to be used to create the indecies. While the indecies are being created, which takes about an hour (by June,2001 standards), the live database can undergo updates. If the newly indexed database is swapped in as the live database, those changes will be lost. For this reason, some games must be played with the replication system to recover any changes. Nevertheless, this process should be done during non-peak hours because there is a window of about 60 seconds during which any changes made have to be re-entered by hand. Just make sure not to attempt this process while the nightly feed update is going on!

1. Log into the main LDAP server, which also has the nightly backup server.

   	telnet metadir
   	  login
   	su root
   	cd /usr/openldap/
   

2. Modify the slapd.conf file for the backup server to include all indecies. Normally, the backup server doesn't do any indexing.

   	cp backup/etc/openldap/slapd.conf backup/etc/openldap/slapd.conf.orig
   	grep ^index etc/openldap/slapd.conf >> backup/etc/openldap/slapd.conf
   

3. Stop the backup server

   	/etc/init.d/openldap.backup stop
   

4. Reindex in the backup area. Takes an hour or so

   	sbin/slapindex -c -f backup/etc/openldap/slapd.conf
   

5. Restart the backup server (using the indexing config file)

   	/etc/init.d/openldap.backup start
   

6. Let slurpd update the backup server with any changes that occurred while it was reindexing. Because the backup server is using indexing, the new index files you created with slapindex will be updated. Watch the slurpd status file to see when the most recent update timestamp for the backup server matches the last entry in the slurpd replog

   	tail var/openldap-slurp/replica/slurpd.replog
   	grep metadir var/openldap-slurp/replica/slurpd.status
   

   You'll have to do some date conversions.
7. This next multistep should be done as swiftly as possible, during a period when updates to the database are not expected to happen. Make a tarball of the reindexed database then swap the new index into the place of the live database.

   	cd backup/db
   	/etc/init.d/openldap.rep stop
   	/etc/init.d/openldap.backup stop
   	tar -cf db.tar *
   	cd /usr/openldap
   	/etc/init.d/openldap stop
   	mv db db.old
   	mv backup/db .
   	mv var/rep.slurpd.ldif var/redo.ldif
   	/etc/init.d/openldap start
   

8. Between the time the backup server was stopped and the main server was stopped, any changes made to the database are now sitting in the LDIF file that was moved to var/redo.ldif. These changes must be fed back into the main database, or they will be lost. Since the time window was small, hopefully there won't be any changes.

   	edit the redo.ldif file
   		remove each block of "replica" lines
   		remove any lines that change system attributes (modify* creat*)
   		remove the "-" lines after them, too.
   	feed the redo.ldif file into ldapmodify
   

9. The main database is now happily running and taking updates. The slurpd server is NOT running, so any changes made after the backup server was stopped are queuing up in the slurpd LDIF file.

10. Recreate the environment for the backup server, and restart it

    	mv backup/etc/openldap/slapd.conf.orig backup/etc/openldap/slapd.conf
    	mkdir backup/db
    	cd backup/db
    	tar -xf ../../db/db.tar dn2id.dbb id2entry.dbb nextid.dbb
    	cd ../..
    	/etc/init.d/openldap.backup start
    

11. Now start copying the reindexed database to the slave servers. Note that the scp will take as much as a few minutes. The untar can take a minute or so, which is why it is done into a separate directory and then copied into place with the near-instant "mv" commands. Note that this requires 3x the disk space of the directory, (the running db, the tar file, and the new db) so watch your df.

    	foreach host ( mail-ldap1 ...... )
    	 scp db/db.tar root@${host}:/usr/openldap/db.tar
    	 ssh -l root $host '\
    	   cd /usr/openldap; \
    	   mkdir db.new; \
    	   cd db.new; \
    	   tar -xf ../db.tar; \
    	   cd ..; \
    	   /etc/init.d/openldap stop; \
    	   mv db db.old;\
    	   mv db.new db; \
    	   /etc/init.d/openldap start;  \
    	   rm -r db.tar db.old'
    	end
    	rm db/db.tar
    

12. Restart the replication server. It will have an LDIF file to work on that contains all the changes since the database was copied from the backup area into the live main database. You can clear out the logs that slurpd keeps, since you know all of the databases are the same now.

    	rm var/openldap-slurp/replica/*
    	/etc/init.d/openldap.rep start
    

The downside is this took a while, and the database is missing all changes that happened since the 3 main files were copied over. If you stop slurpd on the main server before you copy the 3 files, you won't miss any updates, but ALL updates will not be replicated while that is going on. You have the option of leaving slurpd running, but tell "repd" on the main server to make another copy of the LDIF file, and when the slave is done reindexing you can feed that new LDIF file into a "one shot mode" slurpd (use the -o and -r).

WebISO (pubcookie)

• What is Webiso?

  WebISO provides centralized authentication to web servers. The webiso login servers take care of authenticating users and issue cookies that can be used by other webservers to verify the user's identity.

• So, what about those login servers?

  The login servers run on the websio.andrew pool of machines (currently webiso1 and webiso2). The authentication is handled by a CGI program that lives in /usr/www/htdocs/login.cgi. The login CGI gets its configuration information from /usr/www/pubcookie/config (a normal ascii file, one option per line, formatted as option:value).

  The login servers log data to two different places: /usr/www/logs/error_log and /var/adm/messages. (Work to make this logging better is in progress.) At the default debug:1 and logging_level:1, the error_log logs when a user successfully authenticates and when a user gets redirected back to an application server. The system messages logs when users visit the login server (including what app server sent them there), information about the existing login cookie (if any), information when it issues login cookies, and failed authentications.

• (Somewhat) common problems with webiso

  There aren't many problems with the webiso login servers, but other problems come up on occasion.

  • A specific user can't authenticate to websio

    WebISO verifies usernames/passwords by trying to get a Kerberos 5 tgt, and then using that TGT to get a krb5 host/webiso.andrew.cmu.edu ticket. If the user doesn't have a krb5 passwd entry, this will fail. Have the user change their password and try again.

  • A specific application server keeps redirecting people back to the login server in a loop.

    This happens when the app server can't validate the cookies issued by the login server. The app server sends the user back to the login server to get new cookies, which issues more cookies that the app server can't validate repeatedly. Often, this is caused by a misconfigured app server (missing the pubcookie_granting.cert or encryption key file, or bad permissions on them), or mismatched encryption keys between the app server and login server. The login server keeps the encrpytion key for each app server in /usr/www/pubcookie/keys/, one per hostname. The hostname used is the name listed in the app server's SSL certificate, not the real hostname. Verify that the keys are the same on all login servers and the application server(s). If they're not, either create a new one from the app server by running keyclient, or just copy one of the key files around to make them all the same.

  • ...?

    If you come across another problem not documented here, get in touch with jeaton@andrew.cmu.edu and let him know so he can update this FAQ.

General Oracle Information.

If a DB acts up, the first place to look is in one of the alert_{SID}.log files. This file contains information about log switches, major DB alterations, and errors. All errors are preceeded by "ORA-XXXXX". Where XXXXX is a very useful error number. These numbers can be used to access further information, and possible solutions, for that error.

Locations of Oracle Instances

CONTAINS	MACHINE	Oracle Home	AOracle SID	LOCATION OF ALERT LOG
primary BlackBoard Server	courseinfo4.andrew.cmu.edu	/oracle/app/oracle/product/8.1.7	blkboard	/oracle/app/oracle/admin/blkboard/bdump/alert_blkboard.log
Primary Help Center and Generic Developer Server	ora1.andrew.cmu.edu	/oracle/m01/app/oracle/product/8.1.7	thebigdb	/oracle/m01/app/oracle/admin/thebigdb/bdump/alert_thebigdb.log
Primary DAMS Trigger Server	metadir.andrew.cmu.edu	/oracle/m01/app/oracle/product/8.1.6	dams	/oracle/m01/app/oracle/admin/dams/bdump/alert_dams.log

What to do when Oracle misbehaves.

% su root
Password: 
skydiver.andrew.cmu.edu# su oracle
skydiver.andrew.cmu.edu# source /oracle/oracle.env
skydiver.andrew.cmu.edu% stoplsnr
skydiver.andrew.cmu.edu% dbshut
skydiver.andrew.cmu.edu% startlsnr
skydiver.andrew.cmu.edu% dbstart

If the drive containing the binaries failes, it will have to be restored from amanda. Since the DB control files live on the RAID unit, after you restore the DB binaries and bring the DB back on line, it should pick up where it left off as in an instance failure. If the entire RAID goes bad, the nightly amanda dump of /oracle will need to be restored and again the DB should recover although only up to the state of the DB as of the nightly dump.

Where to find additional information

You can get pretty reasonable results just by typing the Ora-XXXXX error into any search engine. Finally, most of the Oracle documentation can be found at Oracle 8I Documentation

Monitoring Infrastructure

The monitoring systems previously documented here (nadine, tsvmon) have been superseded by "mon", a system that supports active notification in addition to polling by user consoles, and is easier to extend to test new services as well. Mon is currently used to test services provided by the Systems group (including AFS, kerberos, mail, ldap, and various web servers) and the network group (dns, radius) as well as network devices maintained by datacomm.

The primary interface to mon is a cgi script running on monitor.andrew.cmu.edu (aka opermon1.andrew.cmu.edu) This interface allows you to see what, if any, services are currently failing, and also allows you to adjust some of mon's settings

Mon is significantly different from earlier monitoring systems used at cmu in that it does not rely on the operators to contact the primary in the event of a problem.

In most cases, two consecutive failures of a service will result in the primary being sent a text page with a brief description of the problem (or at the very least, identification of the service and machine that have failed). The primary will be re-paged at varying intervals until either the service starts functioning again, or the failure is acknowleged.

The acknowlegement can either be done from the web console or directly from a 2-way pager. The messages mon sends to the pagers include information that allows the pager user to reply to the page. Once an acknowlegement has been processed, mon will not send further alerts about this failure, and the web console will display the text of the acknowlegement.

Alternatively, if the circumstances warrant it (the outage will be extended, the machine will need to be rebooted), monitoring of a host or service may be disabled instead. This can also be done from either the web console or a pager.

There are 3 kinds of disable actions that can be performed: disabling a host none of the services on the specifiv host will be monitored disabling a service prevents that specific service from being monitored on all the hosts in the hostgroup. disabling a group prevents all the services from being monitored on all the hosts in the group.

Because mon automatically notifies the primary of an outage, people who maintain services should make sure not to trigger it while doing maintenece. The relevant services or hosts should be disabled before any interruption in service, and not be re-enabled until there is a reasonable expectation that the service is stable.

[I'll presumably add more detail to this next section when my brain is feeling less frazzeled]

quick navigation hints for using the web console:

Acknowlegement messages are set by clicking on the service name in the second column, and using the text box on the following page

services, hosts, and groups can be disabled or enabled by clicking on the hostgroup name in the first column and using the radio buttons on the following page. Single services can also be disabled/enabled by clicking on the service name in the second column and using the "disable/enable service xxx in group yyy" link on the following page.

Historical data on machines is currently maintained on graphs.andrew.cmu.edu.

Historical data on the network is available at stats.net.cmu.edu. People trying to track down denial-of-service problems might want to look at the Top 10 usage link at the bottom. "cyh-a100.sw.cmu.edu" is the machine room switch.

Netdev takes care of stats.net; Larry is probably good to complain to about graphs.andrew. [leg 8/22/01]

Local Machine Accounts

If you need a root instance and don't have one, ask the System Manager.

Service Account

The content and driver CGI for MyAndrew are in a Web Publishing collection called myandrew, which mounts on web3's disk as /collections/myandrew. Related collections such as myandrew/software are in there as well: grep 'myandrew' in /usr/www/conf/cmualiases.conf for particulars.

MyAndrew access is forced to the secure port: see the httpsdirect.conf file. The front door (/myandrew/index.html) is open for access without authentication.

Authentication-requiring services pass through the CGI at the URL /myandrew/auth (/collections/myandrew/auth/myandrew.pl). This script accepts a few variables that tell it which service is being requested, such as the Web Publishing system, and redirects accordingly.