IMPORTANT NOTE: This is a “running blog” today, updated as I complete several steps. It will continue to change until I’ve nailed this process down — I just wanted it recorded, and my blog seems a good spot to put up technical documentation.
This is my personal how-to on how I am migrating users from Groupwise to Cyrus Mail. I know this may sound backwards to some people! If it sounds odd to you, guess you’re not a free/open-source software aficianado.
Background
We were given a directive by our CIO that was quite simple. “Replace Groupwise. We’re done running Novell stuff on our network. If you choose an alternative where we can own the source code, you may implement it without question. If you choose a proprietary alternative, you must provide written cost justification.”
Exhibiting the enthusiasm that all systems administrators show for options that require cost-justification, I, of course, chose the free software option 🙂 Spencer Tuttle did the initial Cyrus implementation and LDAP integration (a process well-documented elsewhere on the web), and eventually it fell to me to figure out how to migrate the Groupwise mailboxes to Cyrus. This, unfortunately, is much tougher than it sounds. Yeah, there’s an easy way, but given the choice between doing it the easy way and relying on users to get it right, and doing it the hard way but being in control of the process and knowing it’s done right, I’d rather be in control.
Products evaluated
- Mozilla Thunderbird. This is our chosen mail client for post-Groupwise life, so we figured we could just have all the users log into both their Groupwise and Cyrus mail accounts at the same time, and copy the folders over that they want to keep.
Upsides:
- It just works™. You can log into both mailstores and drag folders.
- Users gain complete control over what gets moved.
Downsides:
- It requires training on the part of every user.
- If the user screws it up (deleting the folder, for instance), I’m left holding the bag to restore a Groupwise mailbox from backup. That’s really something that’s painful and time-consuming! Particularly if they continue to receive messages between the incident and the mailbox restore.
All things considered, I’d rather avoid this option.
- imapcp, the IMAP Copy Tool.
Upsides:
- Written in Python. I know Python a little, and it’s pretty easy to read and debug.
- A single script, rather than a hard-to-compile collection of utilities.
- Straightforward usage. Typing “python mbcp.py” gives you a brief help, and the README provides sufficient instructions for synchronizing your mailbox.
- Copies folders as well as messages, so you can just point to, say, INBOX, and get all the subfolders.
The downsides:
- The single biggest downside to this, and I need to stress this for all the Googlers who are coming here searching for mbcp.py: THIS SCRIPT DOES NOT WORK!. I’m trying to hack it now, but at the moment it creates mailbox names with the first letter truncated, fails with many obscure errors for no apparent reason, and appears to build some kind of table in memory, leading to crashes due to out of memory errors on very large mailboxes.
- It doesn’t do differential comparisons between mailboxes, just a blind copy. Which means if you have the messages already in your box, you’ll get ’em again if you run the program again.
- It doesn’t seem to handle the “alternative namespace” we use in Cyrus Mail very well. It crashes, actually, trying to create more folders.
- It doesn’t read Groupwise mailboxes; it appears the Groupwise mailstore occasionally stores messages with an odd message ID (or something, haven’t fully debugged it). The program dumps out right after reading the Groupwise INBOX.
The problems with this script prevented me from using it, although I might revisit it if I can’t create a superior working option.
- The IMAP Migration Tool
Upsides:
- With just a little hacking, this script seems to mostly work.
- Follows a disconnected mode of operation: define your parameters, then set up cron (or whatever) to actually run the shell script and migrate the mailbox.
Downsides:
- Required hacking to make it work 🙂 But I think I’m resolved that this will be the case no matter what.
- Created a unique set of interdependent shell scripts and PHP for management of each mailbox. It seems to be geared towards a “one user at a time” approach rather than all 160+ users in our corporation.
- It seemed sloppily coded and documented. For instance, the fact that it deletes and expunges all the messages in the old mailbox without checking that the message was actually migrated is a big downside, and completely undocumented. I could hack around it, but would rather not if I can find a tool that doesn’t have this unfriendly behavior.
- No two-way diff mechanism to synchronize boxes rather than a blind copy.
Overall, it seems a really nifty piece of coding, and is pretty cool in that it invokes PHP from a shell script for processing (leveraging the strengths of both languages), but combining hard-coded nuking of old messages with blind copying and no error checking makes for a package I’m uncomfortable using.
- Mailsync. This little utility is designed to keep various mailboxes synchronized in varying formats, including IMAP and POP.
Upsides:
- Very mature, been around many years.
- Has non-live test modes to enumerate mailboxes.
- Can handle arbitrary namespaces.
- Reasonably well-documented and well-commented code; at least, it’s better than the alternatives.
- Is the only utility I’ve found that can be run multiple times without copying the same messages over and over. It’s designed to synchronize stuff like CVS does — and that’s pretty cool.
Downsides:
- Relies on a library called “c-client”, which is turning out to be a real bummer to compile and make work, particularly if you want SSL support. I eventually gave up trying to compile this library. Even though I’d built it, and pointed mailsync’s “./configure –with-c-client=/some/path/” appropriately, it refused to recognize it as a working c-client. Instead, I just installed the “imap” and “imap-devel” libraries from my Redhat 9 CD, and they worked fine.
- Little support to speak of. The maintainer of the project really, really doesn’t want you emailing him for free support (and I empathize!). The sole mailing list runs about 2 or 3 posts a month, and is housed with the Project Home on Sourceforge.
- Lots of unnecessary complexity for our task — we just want to sync up some IMAP mailboxes, not handle tons of other mail formats.
- Confusing configuration file syntax. I’m sure I’ll get a handle on it with usage, and like Fetchmail eventually come to regard it as “normal”, but right now it’s fairly cryptic.
Important tips:
- It appears that cyrus and the imap package (yeah, c-client is just the usual “imap” package on redhat) conflict. So you can’t run Mailsync on your Cyrus mail server.
- If you are running Redhat9 and have the imap and imap-devel packages installed, getting mailsync compiled relies on the standard five-step:
- tar xzvf mailsync_5.1.0.orig.tar.gz
- cd mailsync-5.1.0
- ./configure
- make
- make install
(And, if you wish)
Me, I chose not to install it, and instead ran it out of the src/ directory.
- You need to be triple-careful with your .mailsync file in your home directory. I recommend in the strongest possible way that you test this on a non-important mailbox first! Otherwise, you are nearly certain to regret it.
- If you’re using Cyrus with an alternate namespace, your delimeter on mailboxes is a “/” instead of a “.”. This is very important! Some scripts assume, if the IMAP server says it’s Cyrus, that it uses the Cyrus default namespace separator of “.” which just isn’t so in our case.
EDIT 5 September 2003: Unfortunately, after getting mailsync working fine, I discovered it only can import/export folders that are already set up with the exact same folder names on both sides. For instance, you can have a /sync folder in IMAP where you store messages you want to sync, and that will work. In this case, though, we need it to propagate folder names as well. I’m sure I can do manual c-client commands to create these folders and then mailsync them (and I may yet do that), but at this point just debugging the problems with imapcp seems easier.
