Web-based Email

part of the ArsDigita Community System by Jin Choi

User-accessible directory: /webmail/
Site adminstrator directory: /admin/webmail/
data model: /doc/sql/webmail.sql
procedures: /tcl/webmail-defs

The big picture

In the beginning, email would get sent to your big Unix box and delivered to a central mail queue, where you would pick it up from time to time with your mail reader and store it permanently in your home directory somewhere. You telnetted to the machine, used your favorite mail reader, all your mail was available from one place, it got backed up every night along with everything else on the machine, and life was good.

Then along came POP. Mail was sent to your big Unix box where it got delivered to a central mail queue, from where you would pick it up from time to time with your mail reader. Only now, it would get stored on whatever machine you happened to be at the time that you decided to read it. You could tell it to keep all the mail on the server, but then every time you read your email, you would see things you'd already seen before, even mail you had deleted elsewhere. If you didn't tell it to keep your mail on the server it would get deleted as soon as you picked it up, and the only copy would be on your desktop machine or your laptop, with no backups whatsoever.

IMAP was invented to address this problem, giving both the benefits of POP along with the centralized mail reading convenience of the old-school style of doing things. True, probably your favorite mail reader didn't support IMAP just yet, but you could get by with what was available, and more readers were adding support all the time. Still, you might be stuck somewhere without a readily available IMAP client, and in today's hostile net environment, telnetting to the server in the clear is a no-no. This is where web-based email shines. Constant access to your email store wherever you may be, as long as a web browser is available. Combined with an IMAP interface to the same store, you can have the benefits of both worlds, and be almost as happy as when you were telnetting to your mail server.

This architecture is available today. You can sign up for an account at criticalpath.net, and they will set you up with an IMAP and web-based email account. You can install Oracle email server from a CD and get the same thing. Why yet another system? Ease of modification and flexibility. If you find yourself missing a feature from Critical Path's system that you would dearly love, you can request it of them and they might deign to provide it in the indeterminate future. You may or may not be able to modify the Oracle email system to your particular needs, but I wouldn't know as I haven't been able to figure out how it works. This module is flexible, easy to extend, and easy to understand. Or if not, at least all the source is there for you to look at.

Architecture

As far as possible, I've tried to avoid getting into implementing systems that have been done well before. That means no reinventing of mail software; no implementing of IMAP or SMTP protocols; no worrying about file locking and mail delivery. qmail and the UW IMAP server were written years ago by people who dedicated lots of effort to make sure they were done correctly. This module doesn't even look at email until it has been delivered by qmail as it would be to any user.

Once the mail has been delivered, it is parsed and inserted into Oracle by some Java code running inside Oracle. Using a relational database as the message store has many benefits. It provides a standard data model for multiple interfaces to hook up to. It makes it easy to do the web interface. It is fast and scalable. It makes implementing powerful email search facilities easy. Using Java lets us leverage the JavaMail API to do the hard work of parsing the messages. It also has the benefit of tight integration to Oracle through the JServer JSQL interface. For example, messages are read into a CLOB column by being streamed from the file system in bite sized chunks. If we were using, say, a perl DBI program to do the insertion, we would have to read the entire message into memory before writing it to the CLOB. With multi-megabyte emails becoming more common in today's media-rich environment, this has obvious benefits.

On the front end, we provide a more or less traditional ACS web module for the web-based front end, and an Oracle "driver" (mail store hookup) for the UW IMAP server to provide the IMAP interface (not yet written).

Receiving Mail

The webmail module interfaces with qmail to receive mail. All of the heavy lifting involved in getting email delivered reliably and in a standards-conformant manner is done by qmail.

The webmail system defines domains and email accounts. A domain is defined by a host name and a short string to be able to refer to it easily. The full domain name must correspond to a virtual domain that qmail is set up to receive email for. In the following, let us assume that "webmail.arsdigita.com" is the full domain name and "wm" is the short name.

Set up DNS so that the full domain name resolves to the host on which you are setting up this system (or set up an appropriate MX record). Check the DNS entry. Usually the sysadmin first gives you only a C name, and you'll see that nslookup yourservername gives you Name: some other name, Address: your ip address, and Alias your server name. If this happens, put in another ticket to have this fixed, so you get an A record. You should get Name: your server name and Address: your ip address. (An MX record should theoretically work too, and is the preferred way to do it.)
Check the qmail installation.
- Make sure the hostname is spelled correctly in /var/qmail/control/me /var/qmail/control/locals /var/qmail/etc/smtp.txt /var/qmail/control/rcpthosts (optional; you'll add to this file)
- Make sure the users qmaild, qmaill, qmailp, and alias are in the group nofiles. In tcsh, you might do
```
foreach i (alias qmail{d,l,p,q,r,s})
  id $i
end
```
  and you should get something like
```
uid=7790(alias) gid=103(nofiles)
uid=7791(qmaild) gid=103(nofiles)
uid=7792(qmaill) gid=103(nofiles)
uid=7793(qmailp) gid=103(nofiles)
uid=7794(qmailq) gid=2107(qmail)
uid=7795(qmailr) gid=2107(qmail)
uid=7796(qmails) gid=2107(qmail)
```
  although the id numbers will likely be different. If you need to change them, a command like sudo usermod -g nofiles alias should do it.
- If you had to change things, stop and restart qmail:
  - kill the qmail-send process. (Hint: ps -e | grep qmail-se)
  - Wait for 'exiting' to show op in the log, which typically is in /var/log/qmail/current.
  - To start up qmail, sudo /var/qmail/boot/home &
    According to All About Qmail -- run csh -cf '/var/qmail/rc &', or /etc/rc2.d/S88qmail start. The latter is sometimes installed as S99qmail, though. Be sure the log goes to the file, not onto your terminal.
Add a line consisting of the full domain name to /var/qmail/control/rcpthosts so that qmail knows to receive email for this domain.
Select a name for an email account that will be used to handle all the email for this system. This can be arbitrary, so we will just use "webmail" (you probably want to use something else, and you must edit the Java code). This name should be distinct from any that might already be in /var/qmail/users/assign
Create a virtual user by adding the following line to /var/qmail/users/assign:
```
+webmail-:oracle:101:102:/home/nsadmin/qmail/alias:-::
```
Make sure this file is ended by a line consisting of a single dot (.). This says any email intended for "webmail-*@webmail.arsdigita.com" will be handled by the Unix user "oracle" (uid 101, gid 102) according to the alias files in the directory /home/nsadmin/qmail/alias. Make sure that directory exists, and can be written to by the nsadmin user:
```
(as root):
mkdirhier /home/nsadmin/qmail/alias
chown nsadmin /home/nsadmin/qmail/alias
```
The location of this alias directory can be changed by setting the AliasDirectory parameter in the webmail section of the ACS .ini file for this server. It is ok to share this directory with other virtual servers running on the same machine.
Run /var/qmail/bin/qmail-newu to have the previous change take effect.
Add the following to /var/qmail/control/virtualdomains:
```
webmail.arsdigita.com:webmail-wm
```
This tells qmail that any mail sent to "*@webmail.arsdigita.com" will be handled by the webmail virtual user, and have "webmail-wm-" prepended to the email address. So, mail sent to "foo@webmail.arsdigita.com" will be delivered to the webmail user as "webmail-wm-foo@webmail.arsdigita.com".
Choose a queue directory for deliveries to the webmail user. This directory must not be shared with any other virtual server running on the same machine; /home/nsadmin/servers/yourservername/qmail/queue should be a safe choice. Create the directory and make sure it is writable and readable by the oracle Unix account (or whatever user Oracle runs at):
```
mkdir /home/nsadmin/servers/yourservername/qmail
(as root:)
/var/qmail/bin/maildirmake /home/nsadmin/servers/yourservername/qmail/queue
chown -R oracle /home/nsadmin/servers/yourservername/qmail/queue
```
Set the QueueDirectory parameter in the webmail section of the ACS .ini file for this server to the location of the queue directory. Once every minute, Oracle polls the new directory of the maildir queue and picks up any mail that has been delivered. Using the JavaMail library running inside Oracle, it stores and parses the message, and saves off various pieces of information (parsed headers, attachments, etc.; see the data model).
To install the Java portion of this module, download the JavaMail library from http://java.sun.com/products/javamail/ and the JavaBeans Activation Framework from http://java.sun.com/beans/glasgow/jaf. Unpack the distributions and load activation.jar and mail.jar into Oracle:
```
loadjava -user dbuser/dbpasswd -resolve -verbose activation.jar
loadjava -user dbuser/dbpasswd -resolve -verbose mail.jar
```
(Note: loadjava takes about 5 minutes to install activation and 14 minutes for mail. You may get a verification warning attempting to resolve javax/activation/ActivationDataFlavor which you can ignore.)
If you are using Oracle 8.1.5, you will not be able to load the JAF and the JavaMail packages into a user's schema, even with JAVASYSPRIV. You must load them into the SYS schema:
loadjava -user internal/internalpasswd -synonym -resolve -verbose activation.jar loadjava -user internal/internalpasswd -synonym -resolve -verbose mail.jar
Also, the activation.jar and mail.jar files come with some of the files compressed. You must unpack and repack the files without compression:
jar xvf activation.jar; rm activation.jar jar cf0 activation.jar META-INF javax com rm -rf META-INF javax com jar xvf mail.jar; rm mail.jar jar cf0 mail.jar META-INF javax com rm -rf META-INF javax com
Then load the various Java files in the /webmail/java directory:
```
loadjava -user dbuser/dbpasswd -resolve -verbose BlobDataSource.java ClobDataSource.java MessageParser.sqlj MessageComposer.sqlj
```
Make sure you create the PL/SQL bindings in the data model file and create the job which will poll the mail queue (if you decide to do so -- see next item).
In 8.1.5, add a resolver spec to the above loadjava statement:
```
-resolver "((* dbuser) (* public) (* -))"
```

Webmail by default schedules wm_process_queue with Oracle to run every minute. For some webmail installations this results in messages without bodies, only headers. The problem is not understood, but the fix is to schedule with AOLserver instead of Oracle. If you want this, comment or delete the following lines from www/doc/sql/webmail.sql,

-- Parse the queue every minute. Queue directory is hardcoded.
-- Only works for some people (Karl Goldstein, Jin Choi) under Oracle 8.1.5
-- and 8.1.6.  Does not work for others.  Hence commented out.  Christian
  declare
    job number;
  begin
    dbms_job.submit(job,
                    'wm_process_queue(''/home/nsadmin/qmail/queue/new'');',
  		  interval => 'sysdate + 1/24/60');
  end;
  /

and put the following into tcl/webmail-defs.tcl:

proc_doc gt_cs_process_email {directory} "
Looks in the directory variable and processes all the emails there.
This is a fix/hack for the normal webmail processing where oracle does
it.  Oracle 8.1.5 doesn't process scheduled jobs correctly, and 8.1.6
loses the message bodies.

Originally by Richard Perng for Greentravel/Customer Service (hence gt_cs)
" {
    
    ns_db exec $db "call wm_process_queue('[DoubleApos $directory]')"
    db_release_unused_handles
}

ns_share -init {set gt_cs_process_email_run -1} gt_cs_process_email_run

if {$gt_cs_process_email_run < 0} {
    set gt_cs_process_email_run \
	[ ns_schedule_proc -thread 60 gt_cs_process_email \
	      [ad_parameter QueueDirectory webmail]new/]
}

The file must be sourced by the AOLserver to have effect.

Either way, make the argument of wm_process_queue indicate your queue directory.

Your tablespace might not have been set up with the assumed 250M space, but rather like 50M (see also Making a New Tablespace. In this case you should reduce the memory size from 250M to 50M (=52428800) in the following two statements in /doc/sql/webmail.sql.

    -- interMedia index on body of message
    create index wm_ctx_index on wm_messages (body)
    indextype is ctxsys.context parameters ('memory 250M');
    
    -- INSO filtered interMedia index for attachments.
    create index wm_att_ctx_index on wm_attachments (data)
    indextype is ctxsys.context parameters ('memory 250M filter ctxsys.inso_filter format column format');

Otherwise both will throw the same error,

    *
    ERROR at line 1:
    ORA-29855: error occurred in the execution of ODCIINDEXCREATE routine
    ORA-20000: interMedia Text error:
    DRG-10754: memory size must be between 1024 and 52428800
    ORA-06512: at "CTXSYS.DRUE", line 126
    ORA-06512: at "CTXSYS.TEXTINDEXMETHODS", line 54
    ORA-06512: at line 1

Load the datamodel. For example,

cd www/doc/sql
sqlplus dbuser/dbpassword
  @ webmail

Register this domain with the webmail system by using the administration pages, with (again, just for this example) "wm" as the short name and "webmail.arsdigita.com" as the domain name.

Once the domain has been set up, you can start adding email accounts. An email account is tied to an ACS user; an ACS user can receive email at any number of different accounts, on any domains that this host receives email for. Once received, they are treated identically, and are indistinguishable other than from the email headers. Email accounts can be assigned to users by using the administration pages. (An extension in the iluvCAMP project also lets you rename and delete email accounts. There's a Tcl API for account creation, renaming, and removal.)

When an email account is added, a file is created automatically in the alias directory of the form ".qmail-[short domain name]-username" (e.g., ".qmail-wm-jsc") that contains one line with the full path to the queue directory (/home/nsadmin/servers/yourservername/qmail/queue/). This file specifies that mail sent to "webmail-wm-jsc@webmail.arsdigita.com" be delivered to the maildir directory that we have set up. All email to be handled by the webmail system ends up in the same place (/home/nsadmin/servers/yourservername/qmail/queue/new). The system uses the RFC822 Delivered-To header to distinguish who it should be displayed to. The indirection through the .qmail alias files is done so that only email sent to valid accounts will be received. Email sent to an address that does not have a .qmail file set up for it will bounce normally.

Once every minute, Oracle polls the new directory of the maildir queue and picks up any mail that has been delivered. Using the JavaMail library running inside Oracle, it stores and parses the message, and saves off various pieces of information (parsed headers, attachments, etc.; see the data model).

To test that you are receiving mail properly, send a message to user@full_domain_name. A new file should immediately be created in /home/nsadmin/servers/yourservername/qmail/queue/new. If one does not appear, check out the qmail error logs (usually /var/log/qmail/current) to see what the problem might be. If you have started the polling job, the file should disappear in less than a minute, and the message should be waiting in the user's INBOX.

Reading Email

The web interface should be self-explanatory and documented with Help links. The IMAP interface isn't yet written, so there is no documentation for it.

Last modified: Wed May 31 22:25:30 EST 2000