Migrating IMAP mailboxes using Linux and Mailsync

matthew's picture

IMPORTANT NOTE: This is a "running blog" today, updated as I complete several steps. It will continue to change until I've nailed this process down -- I just wanted it recorded, and my blog seems a good spot to put up technical documentation.

This is my personal how-to on how I am migrating users from Groupwise to Cyrus Mail. I know this may sound backwards to some people! If it sounds odd to you, guess you're not a free/open-source software aficianado.

Background

We were given a directive by our CIO that was quite simple. "Replace Groupwise. We're done running Novell stuff on our network. If you choose an alternative where we can own the source code, you may implement it without question. If you choose a proprietary alternative, you must provide written cost justification."

Exhibiting the enthusiasm that all systems administrators show for options that require cost-justification, I, of course, chose the free software option :) Spencer Tuttle did the initial Cyrus implementation and LDAP integration (a process well-documented elsewhere on the web), and eventually it fell to me to figure out how to migrate the Groupwise mailboxes to Cyrus. This, unfortunately, is much tougher than it sounds. Yeah, there's an easy way, but given the choice between doing it the easy way and relying on users to get it right, and doing it the hard way but being in control of the process and knowing it's done right, I'd rather be in control.

Products evaluated

  1. Mozilla Thunderbird. This is our chosen mail client for post-Groupwise life, so we figured we could just have all the users log into both their Groupwise and Cyrus mail accounts at the same time, and copy the folders over that they want to keep.

    Upsides:

    • It just works™. You can log into both mailstores and drag folders.
    • Users gain complete control over what gets moved.



    Downsides:

    • It requires training on the part of every user.
    • If the user screws it up (deleting the folder, for instance), I'm left holding the bag to restore a Groupwise mailbox from backup. That's really something that's painful and time-consuming! Particularly if they continue to receive messages between the incident and the mailbox restore.



    All things considered, I'd rather avoid this option.



  2. imapcp, the IMAP Copy Tool.

    Upsides:

    • Written in Python. I know Python a little, and it's pretty easy to read and debug.
    • A single script, rather than a hard-to-compile collection of utilities.
    • Straightforward usage. Typing "python mbcp.py" gives you a brief help, and the README provides sufficient instructions for synchronizing your mailbox.
    • Copies folders as well as messages, so you can just point to, say, INBOX, and get all the subfolders.



    The downsides:

    • The single biggest downside to this, and I need to stress this for all the Googlers who are coming here searching for mbcp.py: THIS SCRIPT DOES NOT WORK!. I'm trying to hack it now, but at the moment it creates mailbox names with the first letter truncated, fails with many obscure errors for no apparent reason, and appears to build some kind of table in memory, leading to crashes due to out of memory errors on very large mailboxes.
    • It doesn't do differential comparisons between mailboxes, just a blind copy. Which means if you have the messages already in your box, you'll get 'em again if you run the program again.
    • It doesn't seem to handle the "alternative namespace" we use in Cyrus Mail very well. It crashes, actually, trying to create more folders.
    • It doesn't read Groupwise mailboxes; it appears the Groupwise mailstore occasionally stores messages with an odd message ID (or something, haven't fully debugged it). The program dumps out right after reading the Groupwise INBOX.



    The problems with this script prevented me from using it, although I might revisit it if I can't create a superior working option.



  3. The IMAP Migration Tool

    Upsides:

    • With just a little hacking, this script seems to mostly work.
    • Follows a disconnected mode of operation: define your parameters, then set up cron (or whatever) to actually run the shell script and migrate the mailbox.



    Downsides:

    • Required hacking to make it work :) But I think I'm resolved that this will be the case no matter what.
    • Created a unique set of interdependent shell scripts and PHP for management of each mailbox. It seems to be geared towards a "one user at a time" approach rather than all 160+ users in our corporation.
    • It seemed sloppily coded and documented. For instance, the fact that it deletes and expunges all the messages in the old mailbox without checking that the message was actually migrated is a big downside, and completely undocumented. I could hack around it, but would rather not if I can find a tool that doesn't have this unfriendly behavior.
    • No two-way diff mechanism to synchronize boxes rather than a blind copy.

    Overall, it seems a really nifty piece of coding, and is pretty cool in that it invokes PHP from a shell script for processing (leveraging the strengths of both languages), but combining hard-coded nuking of old messages with blind copying and no error checking makes for a package I'm uncomfortable using.



  4. Mailsync. This little utility is designed to keep various mailboxes synchronized in varying formats, including IMAP and POP.

    Upsides:

    • Very mature, been around many years.
    • Has non-live test modes to enumerate mailboxes.
    • Can handle arbitrary namespaces.
    • Reasonably well-documented and well-commented code; at least, it's better than the alternatives.
    • Is the only utility I've found that can be run multiple times without copying the same messages over and over. It's designed to synchronize stuff like CVS does -- and that's pretty cool.



    Downsides:

    • Relies on a library called "c-client", which is turning out to be a real bummer to compile and make work, particularly if you want SSL support. I eventually gave up trying to compile this library. Even though I'd built it, and pointed mailsync's "./configure --with-c-client=/some/path/" appropriately, it refused to recognize it as a working c-client. Instead, I just installed the "imap" and "imap-devel" libraries from my Redhat 9 CD, and they worked fine.
    • Little support to speak of. The maintainer of the project really, really doesn't want you emailing him for free support (and I empathize!). The sole mailing list runs about 2 or 3 posts a month, and is housed with the Project Home on Sourceforge.
    • Lots of unnecessary complexity for our task -- we just want to sync up some IMAP mailboxes, not handle tons of other mail formats.
    • Confusing configuration file syntax. I'm sure I'll get a handle on it with usage, and like Fetchmail eventually come to regard it as "normal", but right now it's fairly cryptic.

Important tips:

  • It appears that cyrus and the imap package (yeah, c-client is just the usual "imap" package on redhat) conflict. So you can't run Mailsync on your Cyrus mail server.
  • If you are running Redhat9 and have the imap and imap-devel packages installed, getting mailsync compiled relies on the standard five-step:
    1. tar xzvf mailsync_5.1.0.orig.tar.gz
    2. cd mailsync-5.1.0
    3. ./configure
    4. make
    5. (And, if you wish)

    6. make install
    7. Me, I chose not to install it, and instead ran it out of the src/ directory.

  • You need to be triple-careful with your .mailsync file in your home directory. I recommend in the strongest possible way that you test this on a non-important mailbox first! Otherwise, you are nearly certain to regret it.
  • If you're using Cyrus with an alternate namespace, your delimeter on mailboxes is a "/" instead of a ".". This is very important! Some scripts assume, if the IMAP server says it's Cyrus, that it uses the Cyrus default namespace separator of "." which just isn't so in our case.

EDIT 5 September 2003: Unfortunately, after getting mailsync working fine, I discovered it only can import/export folders that are already set up with the exact same folder names on both sides. For instance, you can have a /sync folder in IMAP where you store messages you want to sync, and that will work. In this case, though, we need it to propagate folder names as well. I'm sure I can do manual c-client commands to create these folders and then mailsync them (and I may yet do that), but at this point just debugging the problems with imapcp seems easier.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
matthew's picture

The wrap-up..

Sorry to have left this blog entry unfinished. In the final analysis, what we decided to do with this corp was have people log into both their Groupwise IMAP mailbox and their Cyrus mailbox using Mozilla Thunderbird as the email client, and drag the folders they wished to move wherever they needed to go.

It's crude, but effective.

Some of these tools are cool, but don't work well with Groupwise's brain-dead IMAP implementation. In at least one case (mbcp.py), it just simply doesn't work at all, due to poor handling of strings truncating off the first letter of subfolder names; if I'd been more motivated, I might have debugged further, but I got sick of the crappy code and moved on. I was able to get Mailsync working really well, but unfortunately that whole "create the folders in both places" prerequisite stopped that effort cold. On the other hand, Mailsync is now a nice tool in my repertoire for use when I want to make sure there are certain messages that stay synched up in my home and work email boxes!

--
Matthew P. Barnson

My experience

I'm in the reverse position to you: my company is moving from mail on a Cyrus IMAP server to GroupWise. From your comments, Mozilla Thunderbird looked like the easiest option; unfortunately, at least the Windows build of 0.3, the latest version as I write this, doesn't allow dragging folders between servers. Which version were you using?

On the other hand, what I'm currently looking at is mailutil, which is part of the UW IMAP toolkit. On Debian (which I'm using), it's the uw-mailutil package. It uses the c-client library, but it's much simpler for basic copying than mailsync, as you don't need to set up a config file (though you do still need to use c-client syntax for server names).

The downside is that some messages aren't getting transferred; there's a problem when mailutil tries to APPEND them onto the GroupWise server. I'm continuing to investigate; hopefully it's solvable, as otherwise it's almost perfect. (One fairly simple command line per mailbox hierarchy to be transferred; unfortunately you can't give passwords on the command line, so it's not quite scriptable for large numbers of users...)

Followup

As it turns out, the error messages I was seeing had nothing to do with the mail that apparently wasn't transferring; they were two separate issues. The latter actually wasn't an issue at all - I'd forgotten to turn off my GroupWise rules while testing, so they were grabbing some of the incoming mail and moving it into folders other than the one I was using for testing. Oh, the embarrassment...

The error messages ("Permission denied") seem to come up when mailutil tries to append messages with flags GroupWise doesn't like; () and (\Deleted) cause problems, but (\Seen) seems to be fine. It's actually not a problem, as mailutil simply re-copies the messages using (\Seen) as the flags, and all is well.

I see you actually tried mailutil before, but had problems with it. Aside from the scripting issue, though, it's by far the easiest option I've found for the Cyrus-to-GroupWise transition, and unlike a number of the other options I've tried, it actually works.

matthew's picture

Moving mail...

Yeah, after writing this I discovered that Thunderbird requires you move individual or groups of messages -- you're still stuck re-creating folders between mailboxes. What a pain!

I'll have to check out mailutil. Seeing that there is such a dearth of utilities to do exactly this job, I find myself wanting to write up a quick script to make it happen the way I want things happening! Not that I'm going to do it any time soon, but it sure seems like a gap in available free software tools that begs for filling...

--
Matthew P. Barnson

--
Matthew P. Barnson

Another tool worth looking at.

I know this is a bit late, but there is also 'imapsync' (http://www.linux-france.org/prj/imapsync/dist/). Creates subfolders and all. Written in perl. Easy to install - I just needed to install a CPAN library I didn't have by default.

I have just copied across approx 48,000 messages, and there were only 173 errors - and at this point evidence suggests that there errors are due to inherent differences between the two imap servers, rather than a problem with imapsync.

Cheers,

Rasjid.

matthew's picture

imapsync...

Very cool! Although it is a bit late in the game at this point (I no longer work for that company), migrating mailboxes is a common enough chore that I'm sure to revisit this topic in the future. Thanks for your post!

--
Matthew P. Barnson

--
Matthew P. Barnson

matthew's picture

One more tool...

One more tool I've really enjoyed using now is offlineimap. I've enjoyed it enough that I've begun hacking it; it's written in Python, fairly easy to understand, and doesn't lose mail.

There are a few niggling errors when working with it over the long-term (mostly related to state differences between mailboxes that somehow cause a dump, then I need to blow away one of my offlineimap-synched sources and re-establish it from the other), but it's an excellent utility. It can synchronize IMAP to Maildir, IMAP to IMAP, and Maildir to IMAP. I touch on the topic in this post, where I mention using Mutt under Cygwin on a "managed mount". offlineimap can do that, and do it pretty well.

Should I ever need to do this abominable chore again, it will be nice to refer back to this blog page to remember the available tools :)

--
Matthew P. Barnson

--
Matthew P. Barnson

Sorry my english.

Sorry my english.

I have gw 5.5 and want to move this to cyrus imap.
But when i run imapsync dont "import" messages.

matthew's picture

FAQ?

Sounds like your problem is answered in the imapsync FAQ. I'd suggest hitting the documentation for details.

Unfortunately, I am not an expert in imapsync; in fact, I've yet to use the product...

--
Matthew P. Barnson

--
Matthew P. Barnson

Thanks for posting this

I was in the middle of much the same evaluation that you were doing. I finally decided to go with Mailsync and I'm pretty pleased with it. I'm glad I didn't write my own---I started. I also tried to fix imapcp and was frustrated that it wouldn't pick up my IMAP folders. THe problem here, though, seems to be my lack of understanding of python's IMAP client. How did you get imapcp to do sub mailboxes?

Are you going to post your fixed imapcp?

matthew's picture

Offlineimap

For me, offlineimap eventually solved all my mail sync problems. Under Cyrus, I just had to do the following:

  • Create a "master" account which I would only use for the migration, and give global admin privileges.
  • Write a script, which does the following as this master account:
    • List all users, dump to file
    • List all user folders, dump to file
    • Add itself with read/write privilege to every account and folder
    • Iterate all those folders into offlineimap's config file to synchronize between the old and new server
  • On the target server side, using Dovecot, I just moved the subdirectories out of my Maildir and into their respective user directories.
  • On the target server, I changed ownership of those Maildirs to the users who owned them

I've done a few dry runs, and it seems to work. Now I just need to do it for real, and send out a notice to my users :)

--
Matthew P. Barnson

--
Matthew P. Barnson

Try creating the destination

Try creating the destination folder, then highlighting the source folder, then select all of the messages in the source folder and ctrl-drag them to the new destination folder.

oops, already suggested as a

oops, already suggested as a pain in the.........

crude

nice avatar. I agree that's crude, but effective.
Some of these tools are cool, but don't work well with Groupwise's brain-dead IMAP implementation. - Sorry, but any examples?

matthew's picture

Very old, and versions

Well, it's been a very long time since I posted this. Perhaps Novell has addressed some of the grossest inadequacies in that time. Some of the examples I recall off the bat were:
* Re-usage of IMAP UIDs for messages that rotated. IIRC, when you'd expunge a message, new messages coming in could get assigned that ID. This is death for programs which attempt to keep IMAP mailboxes synched by UID. I could be mis-remembering the exact symptom.
* Doesn't respect the APPEND command
* Case insensitivity on folder names. Can't create "Sent" and "sent" for instance.
* General crashiness of the GWIA (1-3X per week if heavily used. We generally just restarted it every morning when we got in due to this issue.)

There were more, but I recall digging in deep enough to go "yuck" and giving up on using an IMAP synchronization tool for this particular migration.

As I said, though, you're looking at old stuff. Things that happened nearly three years ago at this point, on a version of Groupwise a couple of years out-of-date at the time. It could very well not apply to reasonably recent versions.

--
Matthew P. Barnson

--
Matthew P. Barnson

You may also want to look

You may also want to look into Yippiemove (http://www.yippiemove.com). It's a basically a fancy (and easy to use) web tool that does the same thing as imapsync (but without the hassle).