3. Changes to Message Header Fields

To permit non-ASCII Unicode characters in field values, the header definition in [RFC5322] is extended to support the new format. The following sections specify the necessary changes to RFC 5322's ABNF.

The syntax rules not mentioned below remain defined as in [RFC5322].

Note that this protocol does not change rules in RFC 5322 for defining header field names. The bodies of header fields are allowed to contain Unicode characters, but the header field names themselves must consist of ASCII characters only.

Also note that messages in this format require the use of the SMTPUTF8 extension [RFC6531] to be transferred via SMTP.

3.1. UTF-8 Syntax and Normalization

UTF-8 characters can be defined in terms of octets using the following ABNF [RFC5234], taken from [RFC3629]:

UTF8-non-ascii  =   UTF8-2 / UTF8-3 / UTF8-4

UTF8-2          =   <Defined in Section 4 of RFC3629>

UTF8-3          =   <Defined in Section 4 of RFC3629>

UTF8-4          =   <Defined in Section 4 of RFC3629>

See [RFC5198] for a discussion of Unicode normalization; normalization form NFC [UNF] SHOULD be used. Actually, if one is going to do internationalization properly, one of the most often cited goals is to permit people to spell their names correctly. Since many mailbox local parts reflect personal names, that principle applies to mailboxes as well. The NFKC normalization form [UNF] SHOULD NOT be used because it may lose information that is needed to correctly spell some names in some unusual circumstances.

3.2. Syntax Extensions to RFC 5322

The following rules extend the ABNF syntax defined in [RFC5322] and [RFC5234] in order to allow UTF-8 content.

VCHAR   =/  UTF8-non-ascii

ctext   =/  UTF8-non-ascii

atext   =/  UTF8-non-ascii

qtext   =/  UTF8-non-ascii

text    =/  UTF8-non-ascii
              ; note that this upgrades the body to UTF-8

dtext   =/  UTF8-non-ascii

The preceding changes mean that the following constructs now allow UTF-8:

1. Unstructured text, used in header fields like "Subject:" or "Content-description:".

2. Any construct that uses atoms, including but not limited to the local parts of addresses and Message-IDs. This includes addresses in the "for" clauses of "Received:" header fields.

3. Quoted strings.

4. Domains.

Note that header field names are not on this list; these are still restricted to ASCII.

3.3. Use of 8-bit UTF-8 in Message-IDs

Implementers of Message-ID generation algorithms MAY prefer to restrain their output to ASCII since that has some advantages, such as when constructing "In-reply-to:" and "References:" header fields in mailing-list threads where some senders use internationalized addresses and others do not.

3.4. Effects on Line Length Limits

Section 2.1.1 of [RFC5322] limits lines to 998 characters and recommends that the lines be restricted to only 78 characters. This specification changes the former limit to 998 octets. (Note that, in ASCII, octets and characters are effectively the same, but this is not true in UTF-8.) The 78-character limit remains defined in terms of characters, not octets, since it is intended to address display width issues, not line-length issues.

3.5. Changes to MIME Message Type Encoding Restrictions

This specification updates Section 6.4 of [RFC2045]. [RFC2045] prohibits applying a content-transfer-encoding to any subtypes of "message/". This specification relaxes that rule -- it allows newly defined MIME types to permit content-transfer-encoding, and it allows content-transfer-encoding for message/global (see Section 3.7).

Background: Normally, transfer of message/global will be done in 8-bit-clean channels, and body parts will have "identity" encodings, that is, no decoding is necessary.

But in the case where a message containing a message/global is downgraded from 8-bit to 7-bit as described in [RFC6152], an encoding might have to be applied to the message. If the message travels multiple times between a 7-bit environment and an environment implementing these extensions, multiple levels of encoding may occur. This is expected to be rarely seen in practice, and the potential complexity of other ways of dealing with the issue is thought to be larger than the complexity of allowing nested encodings where necessary.

3.6. Use of MIME Encoded-Words

The MIME encoded-words facility [RFC2047] provides the ability to place non-ASCII text, but only in a subset of the places allowed by this extension. Additionally, encoded-words are substantially more complex since they allow the use of arbitrary charsets. Accordingly, encoded-words SHOULD NOT be used when generating header fields for messages employing this extension. Agents MAY, when incorporating material from another message, convert encoded-word use to direct use of UTF-8.

Note that care must be taken when decoding encoded-words because the results after replacing an encoded-word with its decoded equivalent in UTF-8 may be syntactically invalid. Processors that elect to decode encoded-words MUST NOT generate syntactically invalid fields.

3.7. The message/global Media Type

Internationalized messages in this format MUST only be transmitted as authorized by [RFC6531] or within a non-SMTP environment that supports these messages. A message is a "message/global message" if:

• it contains 8-bit UTF-8 header values as specified in this document, or

• it contains 8-bit UTF-8 values in the header fields of body parts.

The content of a message/global part is otherwise identical to that of a message/rfc822 part.

If an object of this type is sent to a 7-bit-only system, it MUST have an appropriate content-transfer-encoding applied. (Note that a system compliant with MIME that doesn't recognize message/global is supposed to treat it as "application/octet-stream" as described in Section 5.2.4 of [RFC2046].)

The registration is as follows:

Type name: message

Subtype name: global

Required parameters: none

Optional parameters: none

Encoding considerations: Any content-transfer-encoding is permitted. The 8-bit or binary content-transfer-encodings are recommended where permitted.

Security considerations: See Section 4.

Interoperability considerations: This media type provides functionality similar to the message/rfc822 content type for email messages with internationalized email headers. When there is a need to embed or return such content in another message, there is generally an option to use this media type and leave the content unchanged or down-convert the content to message/rfc822. Each of these choices will interoperate with the installed base, but with different properties. Systems unaware of internationalized headers will typically treat a message/global body part as an unknown attachment, while they will understand the structure of a message/rfc822. However, systems that understand message/global will provide functionality superior to the result of a down-conversion to message/rfc822. The most interoperable choice depends on the deployed software.

Published specification: RFC 6532

Applications that use this media type: SMTP servers and email clients that support multipart/report generation or parsing. Email clients that forward messages with internationalized headers as attachments.

Additional information:

Magic number(s): none

File extension(s): The extension ".u8msg" is suggested.

Macintosh file type code(s): A uniform type identifier (UTI) of "public.utf8-email-message" is suggested. This conforms to "public.message" and "public.composite-content", but does not necessarily conform to "public.utf8-plain-text".

Person & email address to contact for further information: See the Authors' Addresses section of this document.

Intended usage: COMMON

Restrictions on usage: This is a structured media type that embeds other MIME media types. An 8-bit or binary content-transfer-encoding SHOULD be used unless this media type is sent over a 7-bit-only transport.

Author: See the Authors' Addresses section of this document.

Change controller: IETF Standards Process