How to prevent duplicate emails after server migration for pop3 users with leave mail on server

Discussion in 'IMAP Tools' started by popowich, Oct 30, 2013.

  1. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120


  2. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    Re: IMAP Copy

    That would be a great feature to have. Unfortunately there's no way to do it.

    POP clients keep track of whether they have already downloaded a message using a unique number assigned to the message by the POP server (called the UID). You can see the list of UIDS when you are connected to the POP server by issuing the UIDL command.

    # telnet host 110
    user <user>
    pass <pass>
    UIDL
    1 1001
    2 1022
    3 1023
    4 1999
    etc

    The POP mail client records the UID when it downloads a message. Each time the mail client checks the POP server for messages it compares the numbers in the UID list with its record of previously downloaded messages and determines whether the message has already been downloaded.

    When you copy a message using IMAP to another server that server assigns a new UID to the message and that number bears no relation to the original UID on the other server. So when the POP client makes a connection to the new server it will download all the messages. It may or may not be the same as the IMAP UID.

    The only possibility is to get a list of the UIDs from the old server and somehow load them into whatever database the POP client users to store UIDs.

    Each type of mail client uses a different way of storing UIDs. In Outlook Express the file is named pop3uidl.dbx. I'm not certain about Outlook but the name may be pop3uidl.dat.

    As far as I have been able to determine there aren't any tools for inserting entries into the pop3uidl file. I found some for reading dbx files but nothing for manipulating them. I imagine that Microsoft does not want people changing the file and possibly corrupting it. The same is no doubt true for other clients like Thunderbird.

    -Rick
     

  3. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    I think there might be a special case where this is possible.

    When the source server supports both IMAP and POP3, and the destination server is qmail + Maildir,

    The qmail-pop3d uses the filenames in the UIDL list, for example:

    305 1379790535.M813973P12875V04F0697EI019E250D_0.mx005,S=2024
    306 1379803891.M669698P18044V04F0692EI019E250E_0.mx003,S=2978
    307 1379813910.M811138P10947V04EC69BFI019E250F_0.mx009,S=2549
    308 1379818103.M515188P5341V05286A63I019E2510_0.mx002,S=2983
    309 1379819381.M854401P3576V050C6A10I019E2511_0.mx010,S=2892
    310 1379821714.M7554P8934V05586B86I019E2512_0.mx012,S=2749

    If the IMAP to Maildir script had a flag that specified to also check pop3 at the source, and write the Maildir filenames based on the pop3 UIDL's of the source server (instead of in Maildir filename format) , would that work?

    I'll check how the messages are put in order.

    Hopefully based on the timestamps of the files so the order can be duplicated with a touch loop down all the names?

    Would making this change cause IMAP to get confused? (test)
     
  4. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    Creative thinking, Ray. But I don't think it will work.

    As you said, for each message imap_to_maildir would create a filename with the same value as the POP UID on the source server. UIDs on most POP servers are numeric values.

    imap_to_maildir could certainly create filenames like that but maildir filenames have to conform to certain rules. For example, here is the line of code which generates a maildir filename:

    $msgfn = $sec . '.M' . $msec . 'P' . $$ . '.' "$localhost,S=$size,W=$rfc822_size:2$seen";

    S=XXX gives the size, W=XXX the size of the RFC822 data, and "seen" is ",S" if the message has been seen. I've forgotten what the 'M' signifies but it's supposed to be there.

    For this to work, the filename must match exactly what the POP client has in its UID database from the old POP server and creating filenames based on the UID values would create problems for maildir.

    -Rick
     
  5. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    Apparently I'm a rule breaker. :cool:

    Once in a while I'll drop a special message directly into a Maildir

    Every mail account gets a filename such as messageFromRay.txt

    For uniqueness I'll usually add a time stamp like messageFromRay-2013103001.txt (like a DNS serial number)

    I understand when I'm doing that, that I'm not following Maildir naming conventions.

    But it does work, and they look normal to mail programs, as long as I include some basic header info at the top of the text file too.

    It's possible there have been some side effects, but I don't think it's ever broken pop or imap access to an account, or caused mail program confusion.
     
  6. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    It may be that qmail is less strict about filenames than the maildir-based implementations I have worked with. :)

    Next issue. Are the UIDs the same for POP as IMAP on the source server? When imap_to_maildir runs it fetches the IMAP UID which it could use for the basis of the maildir filename. If the IMAP UID is the same as the POP UID then the scheme is workable. But there's no guarantee that they are the same.

    Try this. Pick a POP account with just a few messages in the inbox and compare the POP UIDs with the IMAP UIDs.

    # telnet server 110
    user <user>
    pass <pass>
    UIDL
    1000
    1029
    1033
    quit

    # telnet server 143
    1 login <user> <pass>
    2 select inbox
    3 fetch 1:* uid
    * 1 FETCH (UID 8547)
    * 2 FETCH (UID 8548)
    * 3 FETCH (UID 8549)
    4 logout

    If they match then this may work. Otherwise more complicated logic would be needed to correlate POP UIDs with IMAP UIDs...maybe using message-ids to map between the two.

    -Rick
     
  7. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    They share some of the same info, but they do not match.

    In this case it might be doable if the pop3 UIDL was done first, and that info was used while creating rule breaking Maildir file names in my less strict qmail implementation?

    It's cool to know it might be possible for that particular environment, even if it's generally going to be near impossible for most others or my other email servers.

    POP3:

    1 1294690616.429
    2 1294690616.430
    3 1294690616.431
    4 1294690616.432
    5 1294690616.433
    6 1294690616.434
    7 1294690616.435
    8 1294690616.436
    9 1294690616.437
    10 1294690616.438
    11 1294690616.439
    12 1294690616.440
    13 1294690616.441
    14 1294690616.442
    15 1294690616.443
    16 1294690616.444
    17 1294690616.445
    18 1294690616.446


    IMAP:

    * 1 FETCH (UID 429)
    * 2 FETCH (UID 430)
    * 3 FETCH (UID 431)
    * 4 FETCH (UID 432)
    * 5 FETCH (UID 433)
    * 6 FETCH (UID 434)
    * 7 FETCH (UID 435)
    * 8 FETCH (UID 436)
    * 9 FETCH (UID 437)
    * 10 FETCH (UID 438)
    * 11 FETCH (UID 439)
    * 12 FETCH (UID 440)
    * 13 FETCH (UID 441)
    * 14 FETCH (UID 442)
    * 15 FETCH (UID 443)
    * 16 FETCH (UID 444)
    * 17 FETCH (UID 445)
    * 18 FETCH (UID 446)
     
  8. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    Ray,

    POP3:
    1 1294690616.429
    2 1294690616.430
    3 1294690616.431
    4 1294690616.432

    Is the prefix part of the POP UID (1294690616) the same for each user? Or is it specific to a user? If it is unique to a user but never varies then a table could be built mapping the username to the prefix. The table could then be used to link POP UIDs to IMAP UIDs.

    But if it is random then all bets are off.

    -Rick
     
  9. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    Hi Rick,

    I checked several accounts.

    The prefix part of the UIDL list is always the same for a user, and each user had a unique start to those names.

    Example from a different account:

    1 1330722277.1
    2 1330722277.2
    3 1330722277.3
    4 1330722277.4
    5 1330722277.5
    6 1330722277.6
    7 1330722277.7
     
  10. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    OK, then you could build a table of user POP UID prefixes that would allow an app to map IMAP UIDs to POP UIDs which could then be used as the filenames for messages in the user's Maildir directory.

    For each user
    Get user's POP UID prefix (133072277 for example)
    For each IMAP message
    Get IMAP UID ( eg "200")
    POP UID = 13307277.200
    maildir filename = POP UID
    write message to maildir file system as "13307277.200"

    This sounds like it should work but of course by using POP UIDs as maildir filenames you sacrifice the metadata that is supplied by the standard maildir filename like size, message read/unread status etc.

    If that's acceptable then this may work. Because POP users only access the INBOX I would suggest applying this file naming scheme to just INBOX messages. Use the regular naming conventions for all other folders brought over from the server.
     
  11. rfs9999

    rfs9999 IMAP Tools

    Joined:
    Oct 27, 2013
    Messages:
    114
    Likes Received:
    6
    Ray, just out of curiosity what server would you be migrating from (where the POP and IMAP UIDs are the same except for the prefix)?

    -Rick
     
  12. popowich

    popowich EQ Forum Admin Staff Member

    Joined:
    Aug 12, 2008
    Messages:
    8,983
    Likes Received:
    120
    I'm not 100% sure but I think it's one of these: Message Server

    It's a Mirapoint appliance, but it might be a older model than what is listed in their current web pages.
     

Want to reply or ask your own question?

It only takes a minute to sign up (and it's free!). Click the orange sign up button to choose a username and then you can ask your own questions on the forum.
Loading...