| HTTP Working Group | P. Meenan, Editor |
| Internet-Draft | Google LLC |
| Intended status: Standards Track | Y. Weiss, Editor |
| Expires: January 25, 2025 | Shopify Inc |
| July 24, 2024 |
This specification defines a mechanism for using designated HTTP responses as an external dictionary for future HTTP responses for compression schemes that support using external dictionaries (e.g., Brotli (RFC 7932) and Zstandard (RFC 8878)).¶
This note is to be removed before publishing as an RFC.¶
Status information for this document may be found at <https://datatracker.ietf.org/doc/draft-ietf-httpbis-compression-dictionary/>.¶
Discussion of this document takes place on the HTTP Working Group mailing list (<mailto:[email protected]>), which is archived at <https://lists.w3.org/Archives/Public/ietf-http-wg/>. Working Group information can be found at <https://httpwg.org/>.¶
Source for this draft and an issue tracker can be found at <https://github.com/httpwg/http-extensions/labels/compression-dictionary>.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress”.¶
This Internet-Draft will expire on January 25, 2025.¶
Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
This specification defines a mechanism for using designated [HTTP] responses as an external dictionary for future HTTP responses for compression schemes that support using external dictionaries (e.g., Brotli [RFC7932] and Zstandard [RFC8878]).¶
This document describes the HTTP headers used for negotiating dictionary usage and registers media types for content encoding Brotli and Zstandard using a negotiated dictionary.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
This document uses the following terminology from Section 3 of [STRUCTURED-FIELDS] to specify syntax and parsing: Dictionary, String, Inner List, Token, and Byte Sequence.¶
This document also uses terminology from [HTTP] and [HTTP-CACHING].¶
When responding to a HTTP Request, a server can advertise that the response can be used as a dictionary for future requests for URLs that match the rules specified in the Use-As-Dictionary response header.¶
The Use-As-Dictionary response header is a Structured Field [STRUCTURED-FIELDS] Dictionary with values for "match", "match-dest", "id", and "type".¶
The "match" value of the Use-As-Dictionary header is a String value that provides the URL Pattern [URLPattern] to use for request matching.¶
The URL Pattern used for matching does not support using Regular expressions.¶
The following algorithm will return TRUE for a valid match pattern and FALSE for an invalid pattern that MUST NOT be used:¶
The "match" value is required and MUST be included in the Use-As-Dictionary Dictionary for the dictionary to be considered valid.¶
The "match-dest" value of the Use-As-Dictionary header is an Inner List of String values that provides a list of [FETCH] request destinations for the dictionary to match.¶
An empty list for "match-dest" MUST match all destinations.¶
For clients that do not support request destinations, the client MUST treat it as an empty list and match all destinations.¶
The "match-dest" value is optional and defaults to an empty list.¶
The "id" value of the Use-As-Dictionary header is a String value that specifies a server identifier for the dictionary. If an "id" value is present and has a string length longer than zero then it MUST be sent to the server in a "Dictionary-ID" request header when the dictionary is advertised as being available.¶
The server identifier MUST be treated as an opaque string by the client.¶
The server identifier MUST NOT be relied upon by the server to guarantee the contents of the dictionary. The dictionary hash MUST be validated before use.¶
The "id" value string length (after any decoding) supports up to 1024 characters.¶
The "id" value is optional and defaults to the empty string.¶
The "type" value of the Use-As-Dictionary header is a Token value that describes the file format of the supplied dictionary.¶
"raw" is the only defined dictionary format which represents an unformatted blob of bytes suitable for any compression scheme to use.¶
If a client receives a dictionary with a type that it does not understand, it MUST NOT use the dictionary.¶
The "type" value is optional and defaults to raw.¶
A response that contained a response header:¶
NOTE: '\' line wrapping per RFC 8792
Use-As-Dictionary: \
match="/product/*", match-dest=("document")
Would specify matching any document request for a URL with a path prefix of /product/ on the same Origin (Section 4.3.1 of [HTTP]) as the original request.¶
A response that contained a response header:¶
Use-As-Dictionary: match="/app/*/main.js"
Would match main.js in any directory under /app/ and expiring as a dictionary in one year.¶
When a HTTP client makes a request for a resource for which it has an appropriate dictionary, it can add a "Available-Dictionary" request header to the request to indicate to the server that it has a dictionary available to use for compression.¶
The "Available-Dictionary" request header is a Structured Field [STRUCTURED-FIELDS] Byte Sequence containing the [SHA-256] hash of the contents of a single available dictionary.¶
The client MUST only send a single "Available-Dictionary" request header with a single hash value for the best available match that it has available.¶
For example:¶
Available-Dictionary: :pZGm1Av0IEBKARczz7exkNYsZb8LzaMrV7J32a2fFG4=:
To be considered as a match, the dictionary resource MUST be either fresh [HTTP-CACHING] or allowed to be served stale (see eg [RFC5861]).¶
When a dictionary is stored as a result of a "Use-As-Dictionary" directive, it includes "match" and "match-dest" strings that are used to match an outgoing request from a client to the available dictionaries.¶
Dictionaries MUST have been served from the same Origin (Section 4.3.1 of [HTTP]) as the outgoing request to match.¶
To see if an outbound request matches a given dictionary, the following algorithm will return TRUE for a successful match and FALSE for no-match:¶
When there are multiple dictionaries that match a given request URL, the client MUST pick a single dictionary using the following rules:¶
When a HTTP client makes a request for a resource for which it has an appropriate dictionary and the dictionary was stored with a server-provided "id" in the Use-As-Dictionary response then the client MUST echo the stored "id" in a "Dictionary-ID" request header.¶
The "Dictionary-ID" request header is a Structured Field [STRUCTURED-FIELDS] String of up to 1024 characters (after any decoding) and MUST be identical to the server-provided "id".¶
For example:¶
Available-Dictionary: :pZGm1Av0IEBKARczz7exkNYsZb8LzaMrV7J32a2fFG4=: Dictionary-ID: "/v1/main.js 33a64df551425fcc55e4d42a148795d9f25f89d4"
This specification defines the 'compression-dictionary' link relation type [WEB-LINKING] that provides a mechanism for a HTTP response to provide a URL for a compression dictionary that is related to, but not directly used by the current HTTP response.¶
The 'compression-dictionary' link relation type indicates that fetching and caching the specified resource is likely to be beneficial for future requests. The response to some of those future requests are likely to be able to use the indicated resource as a compression dictionary.¶
Clients can fetch the provided resource at a time that they determine would be appropriate.¶
The response to the fetch for the compression dictionary needs to include a "Use-As-Dictionary" and caching response headers for it to be usable as a compression dictionary. The link relation only provides the mechanism for triggering the fetch of the dictionary.¶
The following example shows a link to a resource at https://example.org/dict.dat that is expected to produce a compression dictionary:¶
Link: <https://example.org/dict.dat>; rel="compression-dictionary"
The "dcb" content encoding identifies a resource that is a "Dictionary-Compressed Brotli" stream.¶
A "Dictionary-Compressed Brotli" stream has a fixed header that is followed by a Shared Brotli [SHARED-BROTLI] stream. The header consists of a fixed 4 byte sequence and a 32 byte hash of the external dictionary that was used. The Shared Brotli stream is created using the referenced external dictionary and a compression window that is at most 16 MB in size.¶
The dictionary used for the "dcb" content encoding is a "raw" dictionary type as defined in Section 2.1.4 and is treated as a prefix dictionary as defined in section 9.2 of the Shared Brotli Compressed Data Format draft. [SHARED-BROTLI] ¶
The 36-byte fixed header is as follows:¶
Clients that announce support for dcb content encoding MUST be able to decompress resources that were compressed with a window size of up to 16 MB.¶
With Brotli compression, the full dictionary is available during compression and decompression independent of the compression window, allowing for delta-compression of resources larger than the compression window.¶
The "dcz" content encoding identifies a resource that is a "Dictionary-Compressed Zstandard" stream.¶
A "Dictionary-Compressed Zstandard" stream is a binary stream that starts with a 40-byte fixed header and is followed by a Zstandard [RFC8878] stream of the response that has been compressed with an external dictionary.¶
The dictionary used for the "dcz" content encoding is a "raw" dictionary type as defined in Section 2.1.4 and is treated as a raw dictionary as per section 5 of RFC 8878.¶
The 40-byte header consists of a fixed 8-byte sequence followed by the 32-byte SHA-256 hash of the external dictionary that was used to compress the resource:¶
The 40-byte header is a Zstandard skippable frame (little-endian 0x184D2A5E) with a 32-byte length (little-endian 0x00000020) that is compatible with existing Zstandard decoders.¶
Clients that announce support for dcz content encoding MUST be able to decompress resources that were compressed with a window size of at least 8 MB or 1.25 times the size of the dictionary, which ever is greater, up to a maximum of 128 MB.¶
The window size used will be encoded in the content (currently, this can be expressed in powers of two only) and it MUST be lower than this limit. An implementation MAY treat a window size that exceeds the limit as a decoding error.¶
With Zstandard compression, the full dictionary is available during compression and decompression until the size of the input exceeds the compression window. Beyond that point the dictionary becomes unavailable. Using a compression window that is 1.25 times the size of the dictionary allows for full delta compression of resources that have grown by 25% between releases while still giving the client control over the memory it will need to allocate for a given response.¶
When a compression dictionary is available for use for a given request, the encoding to be used is negotiated through the regular mechanism for negotiating content encoding in HTTP through the "Accept-Encoding" request header and "Content-Encoding" response header.¶
The dictionary to use is negotiated separately and advertised in the "Available-Dictionary" request header.¶
When a dictionary is available for use on a given request, and the client chooses to make dictionary-based content-encoding available, the client adds the dictionary-aware content encodings that it supports to the "Accept-Encoding" request header. e.g.:¶
Accept-Encoding: gzip, deflate, br, zstd, dcb, dcz
When a client does not have a stored dictionary that matches the request, or chooses not to use one for the request, the client MUST NOT send its dictionary-aware content-encodings in the "Accept-Encoding" request header.¶
If a server supports one of the dictionary encodings advertised by the client and chooses to compress the content of the response using the dictionary that the client has advertised then it sets the "Content-Encoding" response header to the appropriate value for the algorithm selected. e.g.:¶
Content-Encoding: dcb
If the response is cacheable, it MUST include a "Vary" header to prevent caches serving dictionary-compressed resources to clients that don't support them or serving the response compressed with the wrong dictionary:¶
Vary: accept-encoding, available-dictionary
IANA is asked to update the "Hypertext Transfer Protocol (HTTP) Field Name Registry" registry ([HTTP]) according to the table below:¶
| Field Name | Status | Reference |
|---|---|---|
| Use-As-Dictionary | permanent | Section 2.1 of this document |
| Available-Dictionary | permanent | Section 2.2 of this document |
| Dictionary-ID | permanent | Section 2.3 of this document |
IANA is asked to update the "Link Relation Type Registry" registry ([WEB-LINKING]):¶
To minimize the risk of middle-boxes incorrectly processing dictionary-compressed responses, compression dictionary transport MUST only be used in secure contexts (HTTPS).¶
The security considerations for Brotli [RFC7932], Shared Brotli [SHARED-BROTLI] and Zstandard [RFC8878] apply to the dictionary-based versions of the respective algorithms.¶
The dictionary must be treated with the same security precautions as the content, because a change to the dictionary can result in a change to the decompressed content.¶
The dictionary is validated using a SHA-256 hash of the content to make sure that the client and server are both using the same dictionary. The strength of the SHA-256 hash algorithm isn't explicitly needed to counter attacks since the dictionary is being served from the same origin as the content. That said, if a weakness is discovered in SHA-256 and it is determined that the dictionary negotiation should use a different hash algorithm, the "Use-As-Dictionary" response header can be extended to specify a different algorithm and the server would just ignore any "Available-Dictionary" requests that do not use the updated hash.¶
The compression attacks in Section 2.6 of [RFC7457] show that it's a bad idea to compress data from mixed (e.g. public and private) sources -- the data sources include not only the compressed data but also the dictionaries. For example, if you compress secret cookies using a public-data-only dictionary, you still leak information about the cookies.¶
Not only can the dictionary reveal information about the compressed data, but vice versa, data compressed with the dictionary can reveal the contents of the dictionary when an adversary can control parts of data to compress and see the compressed size. On the other hand, if the adversary can control the dictionary, the adversary can learn information about the compressed data.¶
If any of the mitigations do not pass, the client MUST drop the response and return an error.¶
To make sure that a dictionary can only impact content from the same origin where the dictionary was served, the URL Pattern used for matching a dictionary to requests (Section 2.1.1) is guaranteed to be for the same origin that the dictionary is served from.¶
For clients, like web browsers, that provide additional protection against the readability of the payload of a response and against user tracking, additional protections MUST be taken to make sure that the use of dictionary-based compression does not reveal information that would not otherwise be available.¶
In these cases, dictionary compression MUST only be used when both the dictionary and the compressed response are fully readable by the client.¶
As with any usage of compressed content in a secure context, a potential timing attack exists if the attacker can control any part of the dictionary, or if it can read the dictionary and control any part of the content being compressed, while performing multiple requests that vary the dictionary or injected content. Under such an attack, the changing size or processing time of the response reveals information about the content, which might be sufficient to read the supposedly secure response.¶
In general, a server can mitigate such attacks by preventing variations per request, as in preventing active use of multiple dictionaries for the same content, disabling compression when any portion of the content comes from uncontrolled sources, and securing access and control over the dictionary content in the same way as the response content. In addition, the following requirements on a server are intended to disable dictionary-aware compression when the client provides CORS request header fields that indicate a cross-origin request context.¶
The following algorithm will return FALSE for cross-origin requests where precautions such as not using dictionary-based compression should be considered:¶
Since dictionaries are advertised in future requests using the hash of the content of the dictionary, it is possible to abuse the dictionary to turn it into a tracking cookie.¶