MajorCool

Majordomo is a mailing list manager written in Perl by Brent Chapman. Chances are, anyone who has ever subscribed to an Internet mailing list has probably used Majordomo to do it. List configuration, subscription management, identifying the purpose of a list or who is on it, or even finding what lists are available are all functions easily provided by commands sent via e-mail to a Majordomo server address.

As popular and as easy as Majordomo is to use, there are still numerous opportunities for users to encounter difficulties. Being subscribed under multiple addresses, the word-wrapping behavior of some mailers, and even the "delayed gratification" nature of the mail-based system often confuse users. And if end-user functions such as "subscribe" and "unsubscribe" are confusing, imagine the novice list administrator staring at a 300+ line configuration file that was requested (and must be returned) via Majordomo e-mail commands!

By its very nature, Majordomo has always lacked a GUI tool to help simplify its usage by non-power users. However, with the WorldWide Web providing the perfect foundation, it may be an appropriate time for just such an application.

MajorCool, also written in Perl, is a Web interface for Majordomo mailing list configuration and subscriber management. Unlike Web-to-mailinglist interfaces currently available (for example, MailServ and LWGate), MajorCool is uniquely different.

"Off-line" Web interfaces, aware of the appropriate syntax, merely help to format and send a mail query to the mailing list service. There is no direct resource manipulation or immediate user feedback for the requested actions. Most require redundant setup and administration, since adding a new list to Majordomo requires adding the list to the configuration of the Web interface as well.

MajorCool, in contrast, is directly tied to the Majordomo service and is designed to be co-located with the Majordomo program suite. MajorCool offers users immediate notification of current status and the ability to interact directly for viewing lists and their various configurations. Lists added to Majordomo are automatically made available to MajorCool, so no additional maintenance is required for list synchronization.

Another significant difference is the fact that MajorCool also provides an administrative interface for list owners. Most previous attempts to gateway between the Web and list managers focused only on the end-user interface for subscription management and list browsing. Using MajorCool, however, list owners may access many of the more common list configuration options via a Web browser GUI as opposed to requesting, modifying, and re-submiting a Majordomo list configuration file via e-mail.

Like other Web interfaces, MajorCool assists the user in constructing a syntactically-correct Majordomo mail message to affect a change. However, unlike others, MajorCool does not send the commands via e-mail. Taking advantage of the fact that it is run on the same server as Majordomo, MajorCool simply sends the commands directly to the Majordomo application. The advantages of this are two-fold. Delays incurred by e-mail are eliminated, making for immediate interaction with Majordomo resources. At the same time, Majordomo ultimately controls what is done, so MajorCool is neither a re-invention of the wheel, nor will it ever do something that Majordomo permissions typically disallow.

MajorCool is currently in use by the Southern California campuses of NCR Corp, presently serving close to 2000 local users, countless remote subscribers, and over 100 active lists. It has been successfully implemented under NCSA httpd 1.4 and 1.5 servers in conjunction with Majordomo 1.93. MajorCool has proven to be a most valuable service, and much preferred by users over the standard Majordomo e-mail interface.

Usage

The opening screen of MajorCool contains two sections: one for end-users and the other for list administrators, each implemented as individual forms.

List Subscriptions

The LIST SUBSCRIPTIONS form queries for an e-mail address (or, in the case of the NCR-specific implementation, a name). This siteaddr value will be passed to the regexp-generation routine defined in majorcool.cf. The user also has the option at this time to select a search for all available lists, or only ones currently subscribed to.

When the user submits the form, the regexp addresses will be compared against all Majordomo lists, and a set of valid list-names will be generated. This regexp-matching method allows MajorCool to catch multiple addresses that a user might be subscribed under rather than a primary address alone.

Subscribed/Available Lists

The LISTS page displays the generated set of valid list-names as a table of checkboxes, list-names, and list descriptions. Checkboxes indicate which lists the user is subscribed to; if "subscribed" lists were selected, the user will only see lists that they are currently a member of. The list-names are hyperlinked to allow a drill-down to additional list details.

After modifying subscription selections via checkbox, the user applies the changes and the submission is compared against the last known subscription selections. Any changes will be converted into the appropriate "subscribe" or "unsubscribe" commands using the user's "primary address" (as defined by the siteaddr() results).

Depending on the Majordomo list settings of advertise and noadvertise and the MajorCool $member_advertised configuration, some lists may not be visible even if "all lists" was selected or the user is a member. Please note, as explained in the ToDo section below, that advertise/noadvertise is compared against the siteaddr() primary address only, rather than each possible address regexp.

List Details

The LIST DETAILS screen shows additional list information such as the list-owner address, additional description via the info file, and which address the user may be subscribed under. Depending on the subscribed address and the setting of private_who in the list configuration, the option to view all current list members may also be available.

Administer List

The ADMINISTER LIST section of the initial MajorCool page provides access to the list-manager functions for a selected list. The form prompts for a list name (via typed entry or pull-down menu, depending upon configuration) and the list password. When submitted, the password is validated and, if successful, the user is given access to the following list configuration functions.

Configuration Options

Access to some of the more "common" list settings is possible via the LIST CONFIGURATION screen. From here, list administrators (having been validated by password) can change such things as list description, subscription policy, approve password, and much more.

Depending upon the majorcool.cf setting of $hidden_passwd, the list admin password is configurable here or in a separate form below. If $log_suffix is defined, the time and sender of the last message to the list is shown. Additional links to modify the list's info file and the list itself are also provided.

Info File

The INFO FILE page allows the list admin to view and modify the $list.info as contained in an HTML textarea. If changes are submitted, a newinfo command is issued to Majordomo.

Subscriber Administration

The SUBSCRIBER ADMINISTRATION screen allows the list admin to view and modify the members of the list as contained in an HTML textarea. If changes are submitted, a newwho command is issued to Majordomo. As will be discussed later, this newwho command is non-standard and must be added to Majordomo.

The newwho command initiates a wholesale replacement of the specified list file. As such, the "standard" subscribe process is bypassed and the user does not receive confirmation of the addition to the list. For this reason, an additional MEMBERSHIP APPROVAL function is provided to invoke the standard "approve subscribe list address" process.

Change List Password

If $hidden_passwd is enabled in the MajorCool configuration file, then the changing of the list-admin password is moved to a more "safe" double-entry method in this form. This section is not present if the option is disabled.

Send Full Config

The final form available at the list configuration level is the SEND FULL CONFIG BY EMAIL which simply issues a config command to retrieve the current configuration file in the user's name. This is useful for accessing additional configuration options not currently supported the the Web version's configuration screens.

Installation Vagaries

Installation of MajorCool is relatively straightforward (exceptions below notwithstanding). Most of the work is done by a script called via the Majordomo wrapper program, so all permissions are honored accordingly. A CGI "shim" is used to link the world of the Web to the wrapper-based world of MajorCool. Since the wrapper procedure provides a limited environment (and does not pass parent env-vars), the sole purpose of the "shim" is to pass needed CGI variables to the MajorCool routine.

Before beginning the installation of MajorCool, there are some things to be made aware of. MajorCool was not designed on a plain-vanilla Majordomo system. Many local modifications have been made, some of which are critical requirements for the use of MajorCool.

Majordomo Keywords

Many config file keywords have been added to our local Majordomo implementation, such as "announcements" (notification of sub/unsub sent to list-owner?) and "welcome" ("welcome" message sent to new subscribers?). One of those new keywords, "owner", is critical to the implementation of MajorCool.

The "owner" keyword is designed to identify the e-mail address of the list owner (or owners). This data is used to automatically define the set of sendmail aliases for each list. In our configuration, a list is not active until an owner is defined. Using this mechanism, Majordomo administration is completely hands free. Given a new list, users define the ownership (thus activating the -owner, -approval, -request, and -outgoing aliases) and, if necessary, can later reassign the list responsibility without administrative assistance.

MajorCool uses the value of the list "owner" to determine the reply address of administrative mail messages that are simulated and passed on to Majordomo. By performing configuration actions as the list owner, unauthorized tampering will be made known by confirmation mail returned to the owner. If the definition of an "owner" keyword is unacceptable, MajorCool will require modification to redirect to the standard list-owner alias rather than the true resolved "owner" address.

If you are open to the prospect of a new field, the addition of an "owner" keyword is very easily done using the config file interface:

&main'new_keyword("owner", "", "grab_word", "address",
	"The owner of the list. The owner will receive all 
	list errors. This keyword is required in order for 
	the list to become active.");

The existence of the "owner" attribute also opens the door for automatic sendmail aliases generation as mentioned above. If this idea appeals to you, our utility can be be found in the contrib directory of the distribution.

Majordomo Commands

The administrative "edit" interface to the full subscriber list is implemented as an HTML text area field, where subscribers may be added and deleted freely. In order to submit that edited list back to the list server, a Majordomo action that does not normally exist needs to be created. newwho, analogous to the newinfo and newconfig commands, provides the ability to do wholesale replacement of the mailing list file.

Addition of newwho support to Majordomo is incredibly easy. Simply duplicate the do_newinfo function, call it do_newwho, modify as appropriate, and add a "newwho" invocation to the command processing loop.

Because the newwho operation is a complete file replacement of the subscriber list, new list additions do not follow the Majordomo process typically encountered by a first-time subscriber. That is to say, there is no "welcome to the list" message sent to the user. For this reason, an "Approve" form is also provided to allow Web-based list administrators the ability to (un)subscribe individual users per the usual process.

Key Cache

Another unique feature, marginally related to the "owner" field mentioned above, is the implementation of a "key cache" to speed access to all the available Majordomo list configurations. List-by-list access via get_config was found to be exceedingly slow when dealing with a large number of lists. While get_config was fine for Majordomo's non-interactive mail processing, it was death to the Web interface. A way was needed to obtain necessary config information without incurring the cost of numerous run-time list accesses via the config library.

Because of the presence of the non-standard "owner" field and its use to automatically generate the sendmail aliases file, a mechanism for knowing when list configs have changed was already in place. It was decided to extend that procedure to build a "key cache" at the same time the alias table was constructed.

The "cache" is a flat-file database of one-line keyword entries indexed by the list name. The keys cached are all the config attributes that are necessary to perform any initial data display and processing (such as "owner", "description", advertise/noadvertise, etc). Other keys (not cached) are assumed to be retrieved via the get_config function when specifically needed.

In order to make MajorCool work for you, you will need to run the key cache generator regularly. This function, mj_key_cache, will generate the .majordomo_keys file needed by the MajorCool application. In our implementation, the key cache is rebuilt every time the sendmail aliases are made (keying on the "owner" keyword).

User Look-Up

Another site-specific distinction is the use of a username "lookup" function. This function is not critical to the operation of MajorCool, but it is a nice feature and thought it appropriate to mention here.

The NCR mail infrastructure utilizes a Rolodex-style mail delivery tool to allow delivery based on name as well as traditional host addresses. Thus, my own address is both bhoule@www.sandiegoca.ncr.com and Bill.Houle@sandiegoca.ncr.com.

While Majordomo can be configured to know the equivalencies between bhoule@www.sandiegoca.ncr.com and bhoule@sandiegoca.ncr.com via the $mungedomain option, it cannot handle the address mapping when provided by an external function. It was decided that, given our addressing complexities, MajorCool would be infinitely more useful if it was aware of the mapping equivalencies provided by external functions such as the NCR Rolodex.

We opted to base our list subscription tests on a regular expression string containing all possible e-mail addresses for the given user. This address regexp is constructed via the siteaddr() function defined in the MajorCool configuration file. This siteaddr() is completely configurable and is assumed to return the user's full name, preferred address, and address pattern suitable for use as the target regexp.

For example, the resultant NCR siteaddr() on "Bill Houle" might return the regexp

	/^Bill.Houle$|^Bill.Houle@|^bhoule@.*sandiegoca.ncr.com$/

defining address matching patterns for (respectively): my unqualified Rolodex address, fully-qualified Rolodex address, and any host-based address in my domain that matched my login-id.

The sample file provided shows how the NCR Rolodex mapping is implemented (using an external program to map fullname to e-mail address). If no address mapping is needed, you may simply define the siteaddr() function to return the minimum 3-tuple required by MajorCool.

Installation

Unpack the archive somewhere on your Majordomo server. A typical location might be in a www subdir of the Majordomo $homedir. The remainder of the instructions assume that you will do likewise, but feel free to be creative.
Add owner and newwho modifications as described above.
Link (or copy) majorcool.cf, mj_key_cache, and mj_www_interface to Majordomo's $homedir.
Copy the MajorCool icons to the appropriate Web server icon directory.
Implement
```
	$homedir/wrapper mj_key_cache
```
via cron or similar mechanism.
Link (or copy) majorcool.cgi to your Web server's cgi-bin directory. The name majordomo is recommended so that there is no doubt as to the function.
Edit the majorcool.cf configuration file (below) to suit your local preferences, and you are ready to give it a try as "/cgi-bin/majordomo".

Configuration

Basic behavior of MajorCool is controlled by the majorcool.cf configuration file. The settings are defined as follows:

$site: The site or installation location name to be used in page titles.
$siteinfo: A local help file displayed when the MajorCool icon is clicked. A majorcool.info file is provided as an example of what can be placed there.
$header
$footer: Optional header/footer pages (to incorporate common site "look and feel" into MajorCool pages.
$hidden_lists: If disabled (0), the "List Administration" portion of the MajorCool opening screen will display a pick-list of all lists available for configuration. Sites wishing to keep all list names hidden may enable (1) this setting to force the user to explicitly enter the list name.
$hidden_passwd: If disabled (0), the admin_passwd setting will appear as a plain-text field on an equal level with all other fields on the list configuration screen. If enabled (1), the password field would be elevated to a "secret" double-entry process within a separate form. Sites wishing to protect their users from accidental changes or "glance-at-the-screen" compromises might prefer this option.
$log_suffix: Based on a tip from the Majordomo-workers list, we implemented logging to track list activity (saving the From line of each posting). We save this info in the list.last file, but if an alternate suffix is used at your site, please change as appropriate. If logging is not implemented, comment out this variable.
$member_advertised: Majordomo 1.93 can be configured to advertise lists to its members even if the list is not normally advertised. Unfortunately, to enable/disable this behavior, the appropriate source lines must be commented in/out. This setting achieves the same function in MajorCool programatically rather than via code changes.
sub siteaddr(): As was previously explained, MajorCool has the ability to match multiple address classes using regular-expressions rather than absolute addresses. The sitaddr() function is used to create this regexp list. Based on the input value of the siteaddr variable (obtained on the opening MajorCool screen), the function returns a 3-tuple "real name", regexp list, and "preferred address".
%siteaddr: The "List Subscriptions" portion of the MajorCool opening screen queries for the value to be passed to the above siteaddr() look-up function. The %siteaddr associative array is used to define the prompts related to this form field. The prompt value defines the actual field prompt, while the intro string can be used to further explain exactly what is being asked for in the query.
sub by_siteaddr(): Since the siteaddr() look-up function defines a set of addresses that MajorCool uses to support multiple address classes, it may also be valuable to sort list members according to an order appropriate to your local address definitions. The by_siteaddr() function is used to do just that -- provide a customizable address sorting mechanism.

Conclusion

Design Concerns

Blah blah blah...

Authentication -- Using the Web to augment a mail-based system is always problematic. Barring the use of htpasswd or other server-based authentication, there is no way to trust that the user is whom [s]he says [s]he is. For MajorCool, a decision was made to "bite the bullet" and trust that the end-user can correctly specify his/her e-mail address (or name). However, this does leave the door open for tampering with list subscriptions under a false name. In order to minimize this risk, the httpd REMOTE_HOST and REMOTE_ADDRESS CGI variables are sent as part of address comments in the simulated e-mail header. Thus, Majordomo maintains an accurate Log of who submitted Web requests.
Results Feedback -- The single biggest limitation of the chosen design is the fact that Majordomo still sends results via e-mail. Although MajorCool does a majority of the front-end work interactively, the interpretation of the final Majordomo results is not possible and the user must instead watch for e-mail confirmation.
Performance -- Because Majordomo is used to perform any actual work for list or configuration changes, the MajorCool app is at the mercy of how fast Majordomo can process the submitted commands. Given this and the above Feedback limitation, it may be more appropriate to simply hand off results to Majordomo as a background process and not have MajorCool wait for command completion as it currently does.
Web Server Locality (NFS) -- We do not actually know how the overall system would perform if both Web and Majordomo were on the same physical server. In our implementation, the Web server actually resides on a separate system from the Majordomo server, but through NFS, has full access to all Majordomo files. Whether through blind luck or the homogeneity of our environment, we have not run into file-locking or list contention problems (knock wood).

Attributions

MajorCool development was begun by Matt Bateman of NCR Corporation (before he left for greener pastures [literally]). With Matt's departure, the torch was passed to me (Bill Houle), whereupon I modularized & extended the application and prepared it for release back into the Majordomo community.

Aside from the obvious debt to Matt, I would like to thank the following people who shaped the current release of MajorCool:

Rick Hernandez
Chris Claborne

Humble apologies to Patrick Fitzgerald, creator of MailServ, from whom I blatantly stole some images and modified them for MajorCool. We aren't graphic artists, and I needed to slap a quick logo out. This attribution shall remain until someone makes some (original) snazzy artwork.

Future Directions

Archive list & file 'get' interface. Interface archives with HyperMail.

The use of 'mj_key_cache' will have to be updated if Majordomo moves to the discussed DBM format configuration. If the newer formats prove to be sufficiently responsive, the use of the cache could be eliminated altogether.

Bugs/Misfeatures/ToDo

MajorCool is by no means complete. Things to work on include:

Porting for Majordomo 1.94. At the time of MajorCool's inception, v1.94 was still in alpha-test.
Porting for Perl 5. I'd imaging that escaping the '@' will be all that is necessary, but I have not tried it yet....
Better MajorCool logo/images.
Support for string_array in configuration form and the config file submission routine. The advertise/noadvertise regexp arrays are the most-asked-for feature yet to be implemented -- the limitation is the config routines' assumption that every item is a single-line "var=value" pair.
is_advertised function should probably match against address patterns rather than primary address. But this would involve comparing RE's to RE's, and I was too lazy to investigate.
Cross-pollinate siteaddr(), sort by_siteaddr, and $member_advertised to Majordomo as well. I think they would be valuable additions. ($member_advertised would be easy and mostly cosmetic -- just augment the "comment this out" section currently in Majordomo)
List membership maintenance via Majordomo "newwho" addition may not be a good feature. 'newwho' was the quickest and easiest way to accomplish what was needed, but a list 'diff' would also be appropriate. Anyone want to write a 'diff' in Perl to implement member list changes?
Add an "html" command to Majordomo to support an HTML version of "list.info" as $list.html. The html file can then be displayed by MajorCool if present.