How to prevent duplicate emails after server migration for pop3 users with leave mail on server

popowich

EQ Forum Admin
Staff member

rfs9999

IMAP Tools
Re: IMAP Copy

That would be a great feature to have. Unfortunately there's no way to do it.

POP clients keep track of whether they have already downloaded a message using a unique number assigned to the message by the POP server (called the UID). You can see the list of UIDS when you are connected to the POP server by issuing the UIDL command.

# telnet host 110
user <user>
pass <pass>
UIDL
1 1001
2 1022
3 1023
4 1999
etc

The POP mail client records the UID when it downloads a message. Each time the mail client checks the POP server for messages it compares the numbers in the UID list with its record of previously downloaded messages and determines whether the message has already been downloaded.

When you copy a message using IMAP to another server that server assigns a new UID to the message and that number bears no relation to the original UID on the other server. So when the POP client makes a connection to the new server it will download all the messages. It may or may not be the same as the IMAP UID.

The only possibility is to get a list of the UIDs from the old server and somehow load them into whatever database the POP client users to store UIDs.

Each type of mail client uses a different way of storing UIDs. In Outlook Express the file is named pop3uidl.dbx. I'm not certain about Outlook but the name may be pop3uidl.dat.

As far as I have been able to determine there aren't any tools for inserting entries into the pop3uidl file. I found some for reading dbx files but nothing for manipulating them. I imagine that Microsoft does not want people changing the file and possibly corrupting it. The same is no doubt true for other clients like Thunderbird.

-Rick
 

popowich

EQ Forum Admin
Staff member
I think there might be a special case where this is possible.

When the source server supports both IMAP and POP3, and the destination server is qmail + Maildir,

The qmail-pop3d uses the filenames in the UIDL list, for example:

305 1379790535.M813973P12875V04F0697EI019E250D_0.mx005,S=2024
306 1379803891.M669698P18044V04F0692EI019E250E_0.mx003,S=2978
307 1379813910.M811138P10947V04EC69BFI019E250F_0.mx009,S=2549
308 1379818103.M515188P5341V05286A63I019E2510_0.mx002,S=2983
309 1379819381.M854401P3576V050C6A10I019E2511_0.mx010,S=2892
310 1379821714.M7554P8934V05586B86I019E2512_0.mx012,S=2749

If the IMAP to Maildir script had a flag that specified to also check pop3 at the source, and write the Maildir filenames based on the pop3 UIDL's of the source server (instead of in Maildir filename format) , would that work?

I'll check how the messages are put in order.

Hopefully based on the timestamps of the files so the order can be duplicated with a touch loop down all the names?

Would making this change cause IMAP to get confused? (test)
 

rfs9999

IMAP Tools
Creative thinking, Ray. But I don't think it will work.

As you said, for each message imap_to_maildir would create a filename with the same value as the POP UID on the source server. UIDs on most POP servers are numeric values.

imap_to_maildir could certainly create filenames like that but maildir filenames have to conform to certain rules. For example, here is the line of code which generates a maildir filename:

$msgfn = $sec . '.M' . $msec . 'P' . $$ . '.' "$localhost,S=$size,W=$rfc822_size:2$seen";

S=XXX gives the size, W=XXX the size of the RFC822 data, and "seen" is ",S" if the message has been seen. I've forgotten what the 'M' signifies but it's supposed to be there.

For this to work, the filename must match exactly what the POP client has in its UID database from the old POP server and creating filenames based on the UID values would create problems for maildir.

-Rick
 

popowich

EQ Forum Admin
Staff member
Apparently I'm a rule breaker. :cool:

Once in a while I'll drop a special message directly into a Maildir

Every mail account gets a filename such as messageFromRay.txt

For uniqueness I'll usually add a time stamp like messageFromRay-2013103001.txt (like a DNS serial number)

I understand when I'm doing that, that I'm not following Maildir naming conventions.

But it does work, and they look normal to mail programs, as long as I include some basic header info at the top of the text file too.

It's possible there have been some side effects, but I don't think it's ever broken pop or imap access to an account, or caused mail program confusion.
 

rfs9999

IMAP Tools
It may be that qmail is less strict about filenames than the maildir-based implementations I have worked with. :)

Next issue. Are the UIDs the same for POP as IMAP on the source server? When imap_to_maildir runs it fetches the IMAP UID which it could use for the basis of the maildir filename. If the IMAP UID is the same as the POP UID then the scheme is workable. But there's no guarantee that they are the same.

Try this. Pick a POP account with just a few messages in the inbox and compare the POP UIDs with the IMAP UIDs.

# telnet server 110
user <user>
pass <pass>
UIDL
1000
1029
1033
quit

# telnet server 143
1 login <user> <pass>
2 select inbox
3 fetch 1:* uid
* 1 FETCH (UID 8547)
* 2 FETCH (UID 8548)
* 3 FETCH (UID 8549)
4 logout

If they match then this may work. Otherwise more complicated logic would be needed to correlate POP UIDs with IMAP UIDs...maybe using message-ids to map between the two.

-Rick
 

popowich

EQ Forum Admin
Staff member
They share some of the same info, but they do not match.

In this case it might be doable if the pop3 UIDL was done first, and that info was used while creating rule breaking Maildir file names in my less strict qmail implementation?

It's cool to know it might be possible for that particular environment, even if it's generally going to be near impossible for most others or my other email servers.

POP3:

1 1294690616.429
2 1294690616.430
3 1294690616.431
4 1294690616.432
5 1294690616.433
6 1294690616.434
7 1294690616.435
8 1294690616.436
9 1294690616.437
10 1294690616.438
11 1294690616.439
12 1294690616.440
13 1294690616.441
14 1294690616.442
15 1294690616.443
16 1294690616.444
17 1294690616.445
18 1294690616.446


IMAP:

* 1 FETCH (UID 429)
* 2 FETCH (UID 430)
* 3 FETCH (UID 431)
* 4 FETCH (UID 432)
* 5 FETCH (UID 433)
* 6 FETCH (UID 434)
* 7 FETCH (UID 435)
* 8 FETCH (UID 436)
* 9 FETCH (UID 437)
* 10 FETCH (UID 438)
* 11 FETCH (UID 439)
* 12 FETCH (UID 440)
* 13 FETCH (UID 441)
* 14 FETCH (UID 442)
* 15 FETCH (UID 443)
* 16 FETCH (UID 444)
* 17 FETCH (UID 445)
* 18 FETCH (UID 446)
 

rfs9999

IMAP Tools
Ray,

POP3:
1 1294690616.429
2 1294690616.430
3 1294690616.431
4 1294690616.432

Is the prefix part of the POP UID (1294690616) the same for each user? Or is it specific to a user? If it is unique to a user but never varies then a table could be built mapping the username to the prefix. The table could then be used to link POP UIDs to IMAP UIDs.

But if it is random then all bets are off.

-Rick
 

popowich

EQ Forum Admin
Staff member
Hi Rick,

I checked several accounts.

The prefix part of the UIDL list is always the same for a user, and each user had a unique start to those names.

Example from a different account:

1 1330722277.1
2 1330722277.2
3 1330722277.3
4 1330722277.4
5 1330722277.5
6 1330722277.6
7 1330722277.7
 

rfs9999

IMAP Tools
OK, then you could build a table of user POP UID prefixes that would allow an app to map IMAP UIDs to POP UIDs which could then be used as the filenames for messages in the user's Maildir directory.

For each user
Get user's POP UID prefix (133072277 for example)
For each IMAP message
Get IMAP UID ( eg "200")
POP UID = 13307277.200
maildir filename = POP UID
write message to maildir file system as "13307277.200"

This sounds like it should work but of course by using POP UIDs as maildir filenames you sacrifice the metadata that is supplied by the standard maildir filename like size, message read/unread status etc.

If that's acceptable then this may work. Because POP users only access the INBOX I would suggest applying this file naming scheme to just INBOX messages. Use the regular naming conventions for all other folders brought over from the server.
 

rfs9999

IMAP Tools
Ray, just out of curiosity what server would you be migrating from (where the POP and IMAP UIDs are the same except for the prefix)?

-Rick
 

popowich

EQ Forum Admin
Staff member
I'm not 100% sure but I think it's one of these: Message Server

It's a Mirapoint appliance, but it might be a older model than what is listed in their current web pages.
 
Top