IMAP Copy

EQ Admin

EQ Forum Admin
Staff member
imapcopy.pl

imapcopy is a utility for copying a user's messages from one IMAP server to another.

imapcopy is called like this:

./imapcopy -S host1/user1/password1 -D host2/user2/password2

Optional arguments:
-d debug
-I show IMAP protocol exchanges
-L logfile
-m mailbox list (copy only certain mailboxes,see usage notes)
-r reset the \DELETE flag on copied messages
-p <root mailbox> put copied mailboxes under a root mbx
-M <file> mailbox mapping (eg, src:inbox -> dst:inbox_copied)
-i initialize mailbox (remove existing msgs first)
-U run in "update" mode
Run imapcopy.pl -h to see complete set of arguments.

If you have a question about using this script please reply.

If you would like to download the script please see How to get the IMAP Tools.

If you only need to copy a few email accounts you may be interested in the Copy My Mail service.
 

EQ Admin

EQ Forum Admin
Staff member
Out of Memory!

Any chance this script can be updated to handle large email accounts more gracefully?

I got around the problem with the combination of both more real physical memory and more swap space added to the server running the imapcopy.pl

We're taking about having run into an email acocunt that was over 12GB big.
 

rfs9999

IMAP Tools
>Any chance this script can be updated to handle large email accounts more gracefully?

This is the first report I've gotten of a large account causing an "out of memory" problem with imapcopy. I'll take a look into it.

-Rick
 

EQ Admin

EQ Forum Admin
Staff member
I suspect the out of memory came from a php script that was running in the background.

The largest IMAP structure I've moved (single account with lots of folders) was over 15GB.
 

EQ Admin

EQ Forum Admin
Staff member
It took me a little while to figure this one out on a special case:

-z Don't require messages to have a Message-ID in the header

Do you think it makes sense to add to the output, when the -z flag isn't used, that there were X number of skipped messages missing message id's?

I think that would be better than only reporting that there were X good messages (that was in my case) in the source folder.

I'm all set, but it might help someone else figure out the same problem faster.
 

rfs9999

IMAP Tools
>Do you think it makes sense to add to the output, when the -z flag isn't used, that there were X number of skipped messages missing message id's?

Or maybe "don't require message-id" should just be made the default?

-Rick
 

EQ Admin

EQ Forum Admin
Staff member
Yes, that would have worked perfect for me too, and then seeing the option -something to skip emails with no message id?

Why would someone have wanted to skip those?

When you were making the script was that a way of skipping "broken" emails by default?
 

rfs9999

IMAP Tools
Without message-IDs "update" mode won't work because imapcopy uses it to determine whether a message already exists on the destination.

-Rick
 

EQ Admin

EQ Forum Admin
Staff member
Is there a way to use imapcopy.pl and -m to copy only an Inbox but not any folders?
 

rfs9999

IMAP Tools
> The CAPS matters.

Actually when it comes to the Inbox the case does not matter. The following are all the same to IMAP:

Inbox, INBOX, InBoX, inboX, inbox

However, the Inbox is the only mailbox which is not case-senstive. All other mailboxes must be specified in the exact case.

-Rick
 

EQ Admin

EQ Forum Admin
Staff member
For me, with -m Inbox, the script went on to try and copy every folder.

Maybe it depends a bit on the imap server?
 

rfs9999

IMAP Tools
>For me, with -m Inbox, the script went on to try and copy every folder.

That should not happen with imapcopy and -m <inbox>. If so, I'd like to see the logfile with -I and -d.

By the way RFC3501 says...
The case-insensitive mailbox name INBOX is a special name reserved to mean "the primary mailbox for this user on this server".

Any IMAP server that insists on case sensitivity of the Inbox is non-compliant.

-Rick
 

EQ Admin

EQ Forum Admin
Staff member
I was trying this:

/usr/local/imap-tools/imapcopy.pl -z -m "INBOX" -S {src info} -D {dst info}

I get the same result for INBOX, Inbox, etc

Number of mailboxes to process: 1
unexpected response: 1 NO Mailbox does not exist, or must be subscribed to.
There are 0 messages in INBOX.INBOX on the source
INBOX.INBOX mailbox is empty
Copied 0 total messages

Here is the debugging output (slightly sanitized) when I enable those options:

Attempting a non-SSL connection
Connected to x.x.x.x on port 143
method
Authenticating to x.x.x.x as username@example.com
>> 1 LOGIN username@example.com "XXXX"
<< * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2004 Double Precision, Inc. See COPYING for distribution information.
<< 1 OK LOGIN Ok.
Logged in as username@example.com
>> 1 NAMESPACE
<< * NAMESPACE (("INBOX." ".")) NIL (("#shared." ".")("shared." "."))
<< 1 OK NAMESPACE completed.
prefix >INBOX.<
delim >.<
Connecting to y.y.y.y
Attempting a non-SSL connection
Connected to y.y.y.y on port 143
method
Authenticating to y.y.y.y as username@example.com
>> 1 LOGIN username@example.com "XXXX"
<< * OK Waiting for authentication process to respond..
<< * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE NAMESPACE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
<< 1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE NAMESPACE QUOTA] Logged in
Logged in as username@example.com
>> 1 NAMESPACE
<< * NAMESPACE (("INBOX." ".")) NIL NIL
<< 1 OK Namespace completed.
prefix >INBOX.<
delim >.<
>> 1 LIST "" "*"
<< * LIST (\HasChildren) "." INBOX
<< * LIST (\HasNoChildren) "." INBOX.Drafts
<< * LIST (\HasNoChildren) "." INBOX.Junk
<< * LIST (\HasNoChildren) "." INBOX.Sent
<< * LIST (\HasNoChildren) "." INBOX.Trash
<< 1 OK List completed.
Number of mailboxes to process: 1
src mbx INBOX.INBOX
src prefix INBOX.
src delim .
dst prefix INBOX.
dst delim .
>> 1 SUBSCRIBE "INBOX"
<< 1 OK Subscribe completed.
Mailbox INBOX has been subscribed
>> 1 SELECT "INBOX"
<< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
<< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
<< * 0 EXISTS
<< * 0 RECENT
<< * OK [UIDVALIDITY 1393428950] UIDs valid
<< * OK [UIDNEXT 1] Predicted next UID
<< * OK [NOMODSEQ] No permanent modsequences
<< 1 OK [READ-WRITE] Select completed (0.000 secs).
>> 1 EXAMINE "INBOX.INBOX"
<< 1 NO Mailbox does not exist, or must be subscribed to.
unexpected response: 1 NO Mailbox does not exist, or must be subscribed to.
There are 0 messages in INBOX.INBOX on the source
INBOX.INBOX mailbox is empty
Copied 0 total messages
>> 1 LOGOUT
<< * BYE Courier-IMAP server shutting down
<< 1 OK LOGOUT completed
>> 1 LOGOUT
<< * BYE Logging out
<< 1 OK Logout completed.

If there is anything interesting here, or something you want to test against, I can make a debugging account on both the src and dst for you.
 

rfs9999

IMAP Tools
OK, I see the problem. Your IMAP server uses a mailbox prefix of "INBOX." according to the NAMESPACE data:

* NAMESPACE (("INBOX." ".")) NIL NIL

When imapcopy processes the -m <inbox> argument it appends the prefix and comes up with INBOX.Inbox" which is not a valid mailbox (the mailbox prefix is only used for non Inbox mailboxes):

<< * LIST (\HasChildren) "." INBOX
<< * LIST (\HasNoChildren) "." INBOX.Drafts
<< * LIST (\HasNoChildren) "." INBOX.Junk
<< * LIST (\HasNoChildren) "." INBOX.Sent
<< * LIST (\HasNoChildren) "." INBOX.Trash

So this is a bug in imapcopy that is not correctly parsing the mailbox value from -m when it's the inbox and there is a prefix.

I will fix it.

-Rick
 

EQ Admin

EQ Forum Admin
Staff member
FYI, the format for the -M file is a simple src:dst per line (no ->), for example:

INBOX:Inbox
Deleted Items:Trash

Something about specifying the inbox mapping was giving me some grief and caused the actual inbox to not copy.

The fix for me was to run two commands for each user, the first like the above -M to copy all of the folders, and a second with a -m INBOX that migrated the actual inbox emails.
 
Top