IMAP Copy

Discussion in 'IMAP Tools' started by popowich, Oct 27, 2013.

  1. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    imapcopy.pl

    imapcopy is a utility for copying a user's messages from one IMAP server to another.

    imapcopy is called like this:

    ./imapcopy -S host1/user1/password1 -D host2/user2/password2

    Optional arguments:
    -d debug
    -I show IMAP protocol exchanges
    -L logfile
    -m mailbox list (copy only certain mailboxes,see usage notes)
    -r reset the \DELETE flag on copied messages
    -p <root mailbox> put copied mailboxes under a root mbx
    -M <file> mailbox mapping (eg, src:inbox -> dst:inbox_copied)
    -i initialize mailbox (remove existing msgs first)
    -U run in "update" mode
    Run imapcopy.pl -h to see complete set of arguments.

    If you have a question about using this script please reply.

    If you would like to download the script please see How to get the IMAP Tools.

    If you only need to copy a few email accounts you may be interested in the Copy My Mail service.
     


  2. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    Any chance this script can be updated to handle large email accounts more gracefully?

    I got around the problem with the combination of both more real physical memory and more swap space added to the server running the imapcopy.pl

    We're taking about having run into an email acocunt that was over 12GB big.
     

  3. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    >Any chance this script can be updated to handle large email accounts more gracefully?

    This is the first report I've gotten of a large account causing an "out of memory" problem with imapcopy. I'll take a look into it.

    -Rick
     
  4. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    I suspect the out of memory came from a php script that was running in the background.

    The largest IMAP structure I've moved (single account with lots of folders) was over 15GB.
     
  5. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    It took me a little while to figure this one out on a special case:

    -z Don't require messages to have a Message-ID in the header

    Do you think it makes sense to add to the output, when the -z flag isn't used, that there were X number of skipped messages missing message id's?

    I think that would be better than only reporting that there were X good messages (that was in my case) in the source folder.

    I'm all set, but it might help someone else figure out the same problem faster.
     
  6. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    >Do you think it makes sense to add to the output, when the -z flag isn't used, that there were X number of skipped messages missing message id's?

    Or maybe "don't require message-id" should just be made the default?

    -Rick
     
  7. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    Yes, that would have worked perfect for me too, and then seeing the option -something to skip emails with no message id?

    Why would someone have wanted to skip those?

    When you were making the script was that a way of skipping "broken" emails by default?
     
  8. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    Without message-IDs "update" mode won't work because imapcopy uses it to determine whether a message already exists on the destination.

    -Rick
     
  9. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    Is there a way to use imapcopy.pl and -m to copy only an Inbox but not any folders?
     
  10. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    Yes, -m INBOX will copy just the inbox.

    -Rick
     
  11. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    The CAPS matters. Inbox vs INBOX for anyone wanting to do this.
     
  12. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    > The CAPS matters.

    Actually when it comes to the Inbox the case does not matter. The following are all the same to IMAP:

    Inbox, INBOX, InBoX, inboX, inbox

    However, the Inbox is the only mailbox which is not case-senstive. All other mailboxes must be specified in the exact case.

    -Rick
     
  13. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    For me, with -m Inbox, the script went on to try and copy every folder.

    Maybe it depends a bit on the imap server?
     
  14. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    >For me, with -m Inbox, the script went on to try and copy every folder.

    That should not happen with imapcopy and -m <inbox>. If so, I'd like to see the logfile with -I and -d.

    By the way RFC3501 says...
    The case-insensitive mailbox name INBOX is a special name reserved to mean "the primary mailbox for this user on this server".

    Any IMAP server that insists on case sensitivity of the Inbox is non-compliant.

    -Rick
     
  15. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    I was trying this:

    /usr/local/imap-tools/imapcopy.pl -z -m "INBOX" -S {src info} -D {dst info}

    I get the same result for INBOX, Inbox, etc

    Number of mailboxes to process: 1
    unexpected response: 1 NO Mailbox does not exist, or must be subscribed to.
    There are 0 messages in INBOX.INBOX on the source
    INBOX.INBOX mailbox is empty
    Copied 0 total messages

    Here is the debugging output (slightly sanitized) when I enable those options:

    Attempting a non-SSL connection
    Connected to x.x.x.x on port 143
    method
    Authenticating to x.x.x.x as username@example.com
    >> 1 LOGIN username@example.com "XXXX"
    << * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2004 Double Precision, Inc. See COPYING for distribution information.
    << 1 OK LOGIN Ok.
    Logged in as username@example.com
    >> 1 NAMESPACE
    << * NAMESPACE (("INBOX." ".")) NIL (("#shared." ".")("shared." "."))
    << 1 OK NAMESPACE completed.
    prefix >INBOX.<
    delim >.<
    Connecting to y.y.y.y
    Attempting a non-SSL connection
    Connected to y.y.y.y on port 143
    method
    Authenticating to y.y.y.y as username@example.com
    >> 1 LOGIN username@example.com "XXXX"
    << * OK Waiting for authentication process to respond..
    << * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE NAMESPACE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
    << 1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE NAMESPACE QUOTA] Logged in
    Logged in as username@example.com
    >> 1 NAMESPACE
    << * NAMESPACE (("INBOX." ".")) NIL NIL
    << 1 OK Namespace completed.
    prefix >INBOX.<
    delim >.<
    >> 1 LIST "" "*"
    << * LIST (\HasChildren) "." INBOX
    << * LIST (\HasNoChildren) "." INBOX.Drafts
    << * LIST (\HasNoChildren) "." INBOX.Junk
    << * LIST (\HasNoChildren) "." INBOX.Sent
    << * LIST (\HasNoChildren) "." INBOX.Trash
    << 1 OK List completed.
    Number of mailboxes to process: 1
    src mbx INBOX.INBOX
    src prefix INBOX.
    src delim .
    dst prefix INBOX.
    dst delim .
    >> 1 SUBSCRIBE "INBOX"
    << 1 OK Subscribe completed.
    Mailbox INBOX has been subscribed
    >> 1 SELECT "INBOX"
    << * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
    << * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
    << * 0 EXISTS
    << * 0 RECENT
    << * OK [UIDVALIDITY 1393428950] UIDs valid
    << * OK [UIDNEXT 1] Predicted next UID
    << * OK [NOMODSEQ] No permanent modsequences
    << 1 OK [READ-WRITE] Select completed (0.000 secs).
    >> 1 EXAMINE "INBOX.INBOX"
    << 1 NO Mailbox does not exist, or must be subscribed to.
    unexpected response: 1 NO Mailbox does not exist, or must be subscribed to.
    There are 0 messages in INBOX.INBOX on the source
    INBOX.INBOX mailbox is empty
    Copied 0 total messages
    >> 1 LOGOUT
    << * BYE Courier-IMAP server shutting down
    << 1 OK LOGOUT completed
    >> 1 LOGOUT
    << * BYE Logging out
    << 1 OK Logout completed.

    If there is anything interesting here, or something you want to test against, I can make a debugging account on both the src and dst for you.
     
  16. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    OK, I see the problem. Your IMAP server uses a mailbox prefix of "INBOX." according to the NAMESPACE data:

    * NAMESPACE (("INBOX." ".")) NIL NIL

    When imapcopy processes the -m <inbox> argument it appends the prefix and comes up with INBOX.Inbox" which is not a valid mailbox (the mailbox prefix is only used for non Inbox mailboxes):

    << * LIST (\HasChildren) "." INBOX
    << * LIST (\HasNoChildren) "." INBOX.Drafts
    << * LIST (\HasNoChildren) "." INBOX.Junk
    << * LIST (\HasNoChildren) "." INBOX.Sent
    << * LIST (\HasNoChildren) "." INBOX.Trash

    So this is a bug in imapcopy that is not correctly parsing the mailbox value from -m when it's the inbox and there is a prefix.

    I will fix it.

    -Rick
     
  17. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    FYI, the format for the -M file is a simple src:dst per line (no ->), for example:

    INBOX:Inbox
    Deleted Items:Trash

    Something about specifying the inbox mapping was giving me some grief and caused the actual inbox to not copy.

    The fix for me was to run two commands for each user, the first like the above -M to copy all of the folders, and a second with a -m INBOX that migrated the actual inbox emails.
     

Want to reply or ask your own question?

It only takes a minute to sign up (and it's free!). Click the orange sign up button to choose a username and then you can ask your own questions on the forum.
Loading...