IMAP mailbox/folder support
Hi, This may be slightly too long. RFC 3501, dated March 2003, extends the folder naming rules of the previous versions of IMAP. In the new version there’s support for Unicode folder names, and it’s mandatory, and I believe that it’s also widely implemented. There are a couple of mandatory aspects of 3501 that software sometimes… but I digress. Unicode names aren’t one of those areas. So support for Unicode folder names has been mandatory for almost 20 years. So far so good. RFC 6855, the one that adds EAI support to IMAP, adds something new, namely UTF-8 support for folder names. Confused yet? Well, the older RFC had support for Unicode, but used a tortuous encoding called mUTF-7 instead of nice sensible UTF-8. mUTF-7 is bad. It’s a pain for people like myself, but for end-users mUTF-7 and UTF-8 are equivalent. You could have a folder called maßkrug before and you could have one afterwards, the only difference is that the SELECT command sent to the server is easier to read and debug for nerds to need to read and debug protocol traces. All praise nerds and let’s make their job simpler. It is however doubtful to me that IMAP.18 has any place in the Guide, given that the functionality it tests is mandatory in unextended IMAP. Now to Harsha’s mailboxes. Harsha uses Sendmail, Dovecot and Roundcube on Centos. Sendmail basically doesn’t use logins/passwords, it bases its decisions mostly on IP addresses and outsources the last question to either the system login database or to Dovecot’s database, which is often also the system login database. Roundcube makes no decisions, it does whatever Dovecot uses. Dovecot is very configurable. It can be configured to use the system’s logins/passwords or to have a database of its own. In addition, both Sendmail and Dovecot support aliases, so you can have an ASCII login, an email address that corresponds exactly to the login, and another email address that’s configured in a table. The Centos system mostly supports Unicode for logins and passwords. I believe most scripts are perfectly supported, but it ought to have problems with a few, since it doesn’t do the transformations described in RFC 4013. (RFC 4013 are generally not binding on Centos, but they exist for a reason, and the reason applies to Centos.) Sendmail supports EAI well, except that if the system/Dovecot database has restrictions, Sendmail will have some of the same restrictions. This means that if you use just Sendmail+CentOS, EAI will tend to work, except for the few scripts where RFC4013 makes a difference. Using Sendmail+CentOS+Mutt is the same: Should work perfectly, as far as I understand. Dovecot has only accidental support for EAI. If you configure it to use the system’s login database, it’ll partly work. It’s not exactly tested, but some things work anyway. Receiving and reading mail along the chain Sendmail Dovecot Mutt will generally work, replies… Harsha: I’m not at all surprised if you get a weird patchwork of features that work and don’t 😉 I did a three-line change to Roundcube to get it to work. That still hasn’t been merged, I have to explain and persuade. Without the patch it’ll be another of those patchworks, where some things work by accident. I have the impression that many things work by accident if you use Sendmail, Dovecot and Roundcube together. Folder names using Unicode ought to work, for example, but if a message arrives with From/To/Cc grå@xn--gr-zia.rg<mailto:grå@xn--gr-zia.rg> it’ll be displayed that way instead of a more sensible grå@grå.org<mailto:grå@grå.org>. -- Arnt Gulbrandsen UA Technology Sr. Manager, ICANN +32 492 374706
Dear Arnt, I implemented CentOS stream 9 as Operating System, Dovecot, and Roundcube as the Webmail client. To my surprise, CentOS works very well although it does not properly render Sinhala with ZWJ (It looks as if it does not strip off ZWJ, I still have to check and confirm). Roundcube has a bug when I type an email address with U-Label, it seems to drop the last character or a Vowel Modifier. If I type an email address with A-Label, it seems to work without a hitch. Dovecot has partial UTF 8 support it looks like as Arnt pointed out. It does not allow me to authenticate either IMAP or POP. I'm in the process of implementing the MariaDB database as the email storage. Thanks and regards Harsha Harsha On Tue, Feb 28, 2023 at 11:01 PM Arnt Gulbrandsen via UA-EAI < ua-eai@icann.org> wrote:
Hi,
This may be slightly too long.
RFC 3501, dated March 2003, extends the folder naming rules of the previous versions of IMAP. In the new version there’s support for Unicode folder names, and it’s mandatory, and I believe that it’s also widely implemented. There are a couple of mandatory aspects of 3501 that software sometimes… but I digress. Unicode names aren’t one of those areas.
So support for Unicode folder names has been mandatory for almost 20 years. So far so good.
RFC 6855, the one that adds EAI support to IMAP, adds something new, namely UTF-8 support for folder names. Confused yet? Well, the older RFC had support for Unicode, but used a tortuous encoding called mUTF-7 instead of nice sensible UTF-8. mUTF-7 is bad. It’s a pain for people like myself, but for end-users mUTF-7 and UTF-8 are equivalent. You could have a folder called maßkrug before and you could have one afterwards, the only difference is that the SELECT command sent to the server is easier to read and debug for nerds to need to read and debug protocol traces. All praise nerds and let’s make their job simpler. It is however doubtful to me that IMAP.18 has any place in the Guide, given that the functionality it tests is mandatory in unextended IMAP.
Now to Harsha’s mailboxes.
Harsha uses Sendmail, Dovecot and Roundcube on Centos. Sendmail basically doesn’t use logins/passwords, it bases its decisions mostly on IP addresses and outsources the last question to either the system login database or to Dovecot’s database, which is often also the system login database. Roundcube makes no decisions, it does whatever Dovecot uses. Dovecot is very configurable. It can be configured to use the system’s logins/passwords or to have a database of its own. In addition, both Sendmail and Dovecot support aliases, so you can have an ASCII login, an email address that corresponds exactly to the login, and another email address that’s configured in a table.
The Centos system mostly supports Unicode for logins and passwords. I believe most scripts are perfectly supported, but it ought to have problems with a few, since it doesn’t do the transformations described in RFC 4013. (RFC 4013 are generally not binding on Centos, but they exist for a reason, and the reason applies to Centos.)
Sendmail supports EAI well, except that if the system/Dovecot database has restrictions, Sendmail will have some of the same restrictions. This means that if you use just Sendmail+CentOS, EAI will tend to work, except for the few scripts where RFC4013 makes a difference. Using Sendmail+CentOS+Mutt is the same: Should work perfectly, as far as I understand.
Dovecot has only accidental support for EAI. If you configure it to use the system’s login database, it’ll partly work. It’s not exactly tested, but some things work anyway. Receiving and reading mail along the chain Sendmail Dovecot Mutt will generally work, replies… Harsha: I’m not at all surprised if you get a weird patchwork of features that work and don’t 😉
I did a three-line change to Roundcube to get it to work. That still hasn’t been merged, I have to explain and persuade. Without the patch it’ll be another of those patchworks, where some things work by accident. I have the impression that many things work by accident if you use Sendmail, Dovecot and Roundcube together. Folder names using Unicode ought to work, for example, but if a message arrives with From/To/Cc grå@xn--gr-zia.rg it’ll be displayed that way instead of a more sensible grå@grå.org.
--
Arnt Gulbrandsen
UA Technology Sr. Manager, ICANN
+32 492 374706 _______________________________________________ UA-EAI mailing list UA-EAI@icann.org https://mm.icann.org/mailman/listinfo/ua-eai _______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Dear All, My apologizes for some off topic request but probably anyone of you knows whether Dovecot is going to enhance their support of EAI? Or whether UASG has an plans to motivate them to do it? As I know, the Dovecot is widely used for making corporate mail systems and well-known as one of the best open source mail solutions with good functionality which can hardly be replaced by any other solution. In case Dovecot will start to support EAI correctly it seems it may have a valuable impact on EAI implementation level in general. Thank you in advance, Maria Kolesnikova Chair, UA CIS-EE Local Initiative From: UA-EAI <ua-eai-bounces@icann.org> On Behalf Of harsha wijayawardhana via UA-EAI Sent: Tuesday, February 28, 2023 9:15 PM To: Arnt Gulbrandsen <arnt.gulbrandsen@icann.org> Cc: ua-eai@icann.org Subject: Re: [UA-EAI] IMAP mailbox/folder support Dear Arnt, I implemented CentOS stream 9 as Operating System, Dovecot, and Roundcube as the Webmail client. To my surprise, CentOS works very well although it does not properly render Sinhala with ZWJ (It looks as if it does not strip off ZWJ, I still have to check and confirm). Roundcube has a bug when I type an email address with U-Label, it seems to drop the last character or a Vowel Modifier. If I type an email address with A-Label, it seems to work without a hitch. Dovecot has partial UTF 8 support it looks like as Arnt pointed out. It does not allow me to authenticate either IMAP or POP. I'm in the process of implementing the MariaDB database as the email storage. Thanks and regards Harsha Harsha On Tue, Feb 28, 2023 at 11:01 PM Arnt Gulbrandsen via UA-EAI <ua-eai@icann.org <mailto:ua-eai@icann.org> > wrote: Hi, This may be slightly too long. RFC 3501, dated March 2003, extends the folder naming rules of the previous versions of IMAP. In the new version there’s support for Unicode folder names, and it’s mandatory, and I believe that it’s also widely implemented. There are a couple of mandatory aspects of 3501 that software sometimes… but I digress. Unicode names aren’t one of those areas. So support for Unicode folder names has been mandatory for almost 20 years. So far so good. RFC 6855, the one that adds EAI support to IMAP, adds something new, namely UTF-8 support for folder names. Confused yet? Well, the older RFC had support for Unicode, but used a tortuous encoding called mUTF-7 instead of nice sensible UTF-8. mUTF-7 is bad. It’s a pain for people like myself, but for end-users mUTF-7 and UTF-8 are equivalent. You could have a folder called maßkrug before and you could have one afterwards, the only difference is that the SELECT command sent to the server is easier to read and debug for nerds to need to read and debug protocol traces. All praise nerds and let’s make their job simpler. It is however doubtful to me that IMAP.18 has any place in the Guide, given that the functionality it tests is mandatory in unextended IMAP. Now to Harsha’s mailboxes. Harsha uses Sendmail, Dovecot and Roundcube on Centos. Sendmail basically doesn’t use logins/passwords, it bases its decisions mostly on IP addresses and outsources the last question to either the system login database or to Dovecot’s database, which is often also the system login database. Roundcube makes no decisions, it does whatever Dovecot uses. Dovecot is very configurable. It can be configured to use the system’s logins/passwords or to have a database of its own. In addition, both Sendmail and Dovecot support aliases, so you can have an ASCII login, an email address that corresponds exactly to the login, and another email address that’s configured in a table. The Centos system mostly supports Unicode for logins and passwords. I believe most scripts are perfectly supported, but it ought to have problems with a few, since it doesn’t do the transformations described in RFC 4013. (RFC 4013 are generally not binding on Centos, but they exist for a reason, and the reason applies to Centos.) Sendmail supports EAI well, except that if the system/Dovecot database has restrictions, Sendmail will have some of the same restrictions. This means that if you use just Sendmail+CentOS, EAI will tend to work, except for the few scripts where RFC4013 makes a difference. Using Sendmail+CentOS+Mutt is the same: Should work perfectly, as far as I understand. Dovecot has only accidental support for EAI. If you configure it to use the system’s login database, it’ll partly work. It’s not exactly tested, but some things work anyway. Receiving and reading mail along the chain Sendmail Dovecot Mutt will generally work, replies… Harsha: I’m not at all surprised if you get a weird patchwork of features that work and don’t 😉 I did a three-line change to Roundcube to get it to work. That still hasn’t been merged, I have to explain and persuade. Without the patch it’ll be another of those patchworks, where some things work by accident. I have the impression that many things work by accident if you use Sendmail, Dovecot and Roundcube together. Folder names using Unicode ought to work, for example, but if a message arrives with From/To/Cc grå@xn--gr-zia.rg it’ll be displayed that way instead of a more sensible grå@grå.org. -- Arnt Gulbrandsen UA Technology Sr. Manager, ICANN +32 492 374706 _______________________________________________ UA-EAI mailing list UA-EAI@icann.org <mailto:UA-EAI@icann.org> https://mm.icann.org/mailman/listinfo/ua-eai _______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
It appears that Maria Kolesnikova via UA-EAI <masha@cctld.ru> said:
My apologizes for some off topic request but probably anyone of you knows whether Dovecot is going to enhance their support of EAI? Or whether UASG has an plans to motivate them to do it?
I talked to the Dovecot people a while ago. They would be willing to add support for the new EAI features in RFCs 6855 and 6866 if someone paid them, but they won't do it on their own. It would require a lot of work reorganizing the internal structure of Dovecot so it would be quite expensive, I estimate on the order of $100,000. But when I did EAI conformance tests on Dovecot, I found the same things that Arnt did. Even though it doesn't have new UTF-8 support, the legacy character sets are good enough and most of its internal operations are 8-bit clean so that it passed most of the EAI tests anyway. I do not think it would be a good use of time or money to add 6855 and 6866 support. R's, John
I'm really quite sad that their plans changed. They told me quite plainly the plans were tentative, so I have nothing to complain about. But I'm sad. FYI https://github.com/dovecot/core/pull/190 mostly works and reorganizes nothing. It's a clean and small patch, and enough to get Dovecot to the same level of support that Exchange 2019 has. The thing that doesn't work is the UTF8 modifier for the APPEND command, which is a syntactical problem, not a matter of substance. I wish I could charge $100k for such a thing. Arnt On 01/03/2023, 18:28, "UA-EAI on behalf of John Levine via UA-EAI" <ua-eai-bounces@icann.org on behalf of ua-eai@icann.org> wrote: It appears that Maria Kolesnikova via UA-EAI <masha@cctld.ru> said: >My apologizes for some off topic request but probably anyone of you knows >whether Dovecot is going to enhance their support of EAI? Or whether UASG >has an plans to motivate them to do it? I talked to the Dovecot people a while ago. They would be willing to add support for the new EAI features in RFCs 6855 and 6866 if someone paid them, but they won't do it on their own. It would require a lot of work reorganizing the internal structure of Dovecot so it would be quite expensive, I estimate on the order of $100,000. But when I did EAI conformance tests on Dovecot, I found the same things that Arnt did. Even though it doesn't have new UTF-8 support, the legacy character sets are good enough and most of its internal operations are 8-bit clean so that it passed most of the EAI tests anyway. I do not think it would be a good use of time or money to add 6855 and 6866 support. R's, John _______________________________________________ UA-EAI mailing list UA-EAI@icann.org https://mm.icann.org/mailman/listinfo/ua-eai _______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
Download Xgenplus from xgenplus.com, it's available free with full EAI support. On February 28, 2023 11:45:18 PM GMT+05:30, harsha wijayawardhana via UA-EAI <ua-eai@icann.org> wrote:
Dear Arnt, I implemented CentOS stream 9 as Operating System, Dovecot, and Roundcube as the Webmail client. To my surprise, CentOS works very well although it does not properly render Sinhala with ZWJ (It looks as if it does not strip off ZWJ, I still have to check and confirm). Roundcube has a bug when I type an email address with U-Label, it seems to drop the last character or a Vowel Modifier. If I type an email address with A-Label, it seems to work without a hitch.
Dovecot has partial UTF 8 support it looks like as Arnt pointed out. It does not allow me to authenticate either IMAP or POP. I'm in the process of implementing the MariaDB database as the email storage.
Thanks and regards Harsha Harsha
On Tue, Feb 28, 2023 at 11:01 PM Arnt Gulbrandsen via UA-EAI < ua-eai@icann.org> wrote:
Hi,
This may be slightly too long.
RFC 3501, dated March 2003, extends the folder naming rules of the previous versions of IMAP. In the new version there’s support for Unicode folder names, and it’s mandatory, and I believe that it’s also widely implemented. There are a couple of mandatory aspects of 3501 that software sometimes… but I digress. Unicode names aren’t one of those areas.
So support for Unicode folder names has been mandatory for almost 20 years. So far so good.
RFC 6855, the one that adds EAI support to IMAP, adds something new, namely UTF-8 support for folder names. Confused yet? Well, the older RFC had support for Unicode, but used a tortuous encoding called mUTF-7 instead of nice sensible UTF-8. mUTF-7 is bad. It’s a pain for people like myself, but for end-users mUTF-7 and UTF-8 are equivalent. You could have a folder called maßkrug before and you could have one afterwards, the only difference is that the SELECT command sent to the server is easier to read and debug for nerds to need to read and debug protocol traces. All praise nerds and let’s make their job simpler. It is however doubtful to me that IMAP.18 has any place in the Guide, given that the functionality it tests is mandatory in unextended IMAP.
Now to Harsha’s mailboxes.
Harsha uses Sendmail, Dovecot and Roundcube on Centos. Sendmail basically doesn’t use logins/passwords, it bases its decisions mostly on IP addresses and outsources the last question to either the system login database or to Dovecot’s database, which is often also the system login database. Roundcube makes no decisions, it does whatever Dovecot uses. Dovecot is very configurable. It can be configured to use the system’s logins/passwords or to have a database of its own. In addition, both Sendmail and Dovecot support aliases, so you can have an ASCII login, an email address that corresponds exactly to the login, and another email address that’s configured in a table.
The Centos system mostly supports Unicode for logins and passwords. I believe most scripts are perfectly supported, but it ought to have problems with a few, since it doesn’t do the transformations described in RFC
(RFC 4013 are generally not binding on Centos, but they exist for a reason, and the reason applies to Centos.)
Sendmail supports EAI well, except that if the system/Dovecot database has restrictions, Sendmail will have some of the same restrictions. This means that if you use just Sendmail+CentOS, EAI will tend to work, except for the few scripts where RFC4013 makes a difference. Using Sendmail+CentOS+Mutt is the same: Should work perfectly, as far as I understand.
Dovecot has only accidental support for EAI. If you configure it to use the system’s login database, it’ll partly work. It’s not exactly tested, but some things work anyway. Receiving and reading mail along the chain Sendmail Dovecot Mutt will generally work, replies… Harsha: I’m not at all surprised if you get a weird patchwork of features that work and don’t 😉
I did a three-line change to Roundcube to get it to work. That still hasn’t been merged, I have to explain and persuade. Without the patch it’ll be another of those patchworks, where some things work by accident. I have the impression that many things work by accident if you use Sendmail, Dovecot and Roundcube together. Folder names using Unicode ought to work, for example, but if a message arrives with From/To/Cc grå@xn--gr-zia.rg it’ll be displayed that way instead of a more sensible grå@grå.org.
--
Arnt Gulbrandsen
UA Technology Sr. Manager, ICANN
+32 492 374706 _______________________________________________ UA-EAI mailing list UA-EAI@icann.org https://mm.icann.org/mailman/listinfo/ua-eai _______________________________________________ By submitting your personal data, you consent to the processing of your personal data for purposes of subscribing to this mailing list accordance with the ICANN Privacy Policy (https://www.icann.org/privacy/policy) and the website Terms of Service (https://www.icann.org/privacy/tos). You can visit the Mailman link above to change your membership status or configuration, including unsubscribing, setting digest-style delivery or disabling delivery altogether (e.g., for a vacation), and so on.
-- Sent from my Android device with XGen Email.
participants (5)
-
ajay＠data.in -
Arnt Gulbrandsen -
harsha wijayawardhana -
John Levine -
Maria Kolesnikova