[Proposal] EXIF / image-resolution: (auto?) and from-image

I want to open a discussion about EXIF and resolutions.

The use case - changing the image-resolution in servers/CDN. Some CDN providers like Cloudinary do this today, and use an HTTP header that hasn’t reached consensus with implementors (Content-DPR).

The idea is to use existing EXIF attributes to hint the user agent that an image has an intrinsic size different from its pixel data dimensions, something like a pixel-ratio-multiplier.

The relevant EXIF attributes would be Resolution-X and Resolution-Y, and the idea is to have them work hand in hand with the image-resolution CSS attribute.

My initial concrete proposal would be:

• If image-resolution is from-image:
  • If EXIF Resolution-X/Resolution-Y are defined:
    • If EXIF Resolution-Unit is Inch: compute intrinsic size based on 96dpi
    • If EXIF Resolution-Unit is Pixel: compute intrinsic size based on the x/y units
    • Otherwise, use the image srcset value or 1dppx.
• If image-resolution is auto (which would be the default value, and a new value):
  • If EXIF Resolution-X/Resolution-Y are defined and Resolution Unit is Pixel:
    • compute intrinsic size based on the x/y units
  • Otherwise, use the image srcset value or 1dppx.

There are probably alternative ways to express it in CSS, the main idea is to allow that EXIF to be respected without breaking current images on the web.

Some alternative would be:

• Use the Pixel-X-Dimension and Pixel-Y-Dimension EXIF attributes
• Only respect Pixel resolution-unit, and avoid adding the new auto value
• Use the existing Inch resolution units (usually 72 is the default though while the web’s DPI is 96…)
• Create a new attribute for this purpose (pixel-density? Preffered-Display-Dimension?).

Thoughts please!

I like the general idea. I wonder about the compatibility of setting a default of auto with current web images:

• How many images on the web today have their EXIF data include Resolution-{X-Y} or Pixel-{X,Y}-Dimension attributes?
• How many of those contain values that would result in breakage if auto is applied?

Another concern is that many image optimizers blindly strip EXIF data today. That’s fine, but becomes less fine once browsers start to rely on EXIF data for the image’s rendering.

I don’t think anybody uses Pixel as a Resolution-Unit, it’s common to use Inch. So auto as it is in my proposal would not affect current images, but from-image might.

Regarding optimizers - this is also the case for image-orientation… If the standard about how user-agents do this is clearer, the image optimizer can also adjust.

I’d naively expect from-image to be sparsely used. Would be great to gather data both on from-image usage as well as on EXIF attributes used in the wild.

I agree this shouldn’t be a blocker, and didn’t seem to cause a lot of breakage in the orientation case.

from-image is not used yet because the image-resolution property is not yet implemented in user agents

I will try to gather information regarding EXIF attributes, I think such research was done for image-orientation.

A new unit type sounds promising. What does it represent, though? My guess is CSS ddpx, e.g.:

800x600@2x.jpg
Resolution-Unit: Pixel
Resolution-X: 2

(…then Resolution-Y, if undefined, defaults to the same as Resolution-X)

Also, if we used Pixel-X/Y-Dimension, those would be CSS pixels? e.g.,

800x600@2x.jpg
Pixel-X-Dimension: 400
Pixel-Y-Dimension: 300

Yes, exactly - since Inch stands for DPI, it would make sense that Pixel would stand for dppx. I believe there is no current use for Pixel as an EXIF Resolution Unit.

Yes. I tend to like the second one because it doesn’t have float rounding issues; However the first one might introduce fewer backward compatibility issues (maybe non at all). A study of current existence of EXIF in web-exposed image is probably a good next step

Continuing my research, I’m thinking that adding an XMP attribute might work better than EXIF… EXIF is less extensible and is more photography-oriented, also the mentioned EXIF attributes are used by image viewers.

Some more info and more detailed alternatives after some research:

Potential metadata formats

• EXIF: used mainly for photography data. Very concise and in wide use. Browsers today recognize it for image-orientation.
• XMP: very extensible, used for things like photo-tagging. Verbose.
• IPTC: used mainly for rights management, not very relevant.

Current resolution metadata identifiers:

See definitions

• ImageWidth/ImageHeight: usually refer to the original size of the image when it was taken, before it was scaled by any program. Many photos on the web have different ImageWidth/ImageHeight values from their actual pixel dimensions.
• XResolution/YResolution/ResolutionUnit: used almost exclusively for DPI, as a printing directive,. The default is 72dpi. The different resolution units are defined in the EXIF standard.
• PixelXDimension/PixelYDimension: intended for a different use, padding of compressed image data.

Proposed alternatives

1. Use ImageWidth / ImageHeight EXIF to override intrinsic dimensions

   These attributes are quite common in the wild, and are often different from the image pixel dimensions. That’s because they don’t change when a program changes the image. See explanation. Using them to compute intrinsic dimension as an override of the pixel data dimensions would make a lot of sense, but is also likely to modify the presentation of a small amount of existing images on the web.

2. Use ImageResolution EXIF with Pixel as ResolutionUnit

   This maybe makes the most sense spec-wise, and would not break any existing web site, but would require a revision to the Exif standard, as the different values for ResolutionUnit are defined in that spec.

3. Use ImageResolution EXIF with Inch as ResolutionUnit, and treat 72dpi as 1dppx

   This would not require any spec change, as Inch is already a defined ImageResolution value. However, it would change intrinsic size for some images currently on the web, where the dpi is defined with a value other than 72. That is not uncommon in the web today.

4. Use ImageResolution or ImageWidth/Height EXIF, with CSS image-resolution

   Either (1) or (3) can work without “breaking the web”, by allowing it as a non-default value for image-resolution CSS attribute.

5. Add a new EXIF attribute

   Maybe PreferredPixelDensity or DisplayWidth/DisplayHeight. Will require changing the EXIF spec, but would not introduce any backwards compatibility issues.

6. Define an XMP property for this

   This can be done without too much standardization overhead, as it can be done in an XMP namespace. However, it would require user agents to incur the overhead of parsing inline XMP in image data, which they currently ignore.

I see no reason to restrict this to still image formats. It appears to apply to video as well. Hence other container and tagging formats would need to be considered.

XMP is supported in video files, and it can be used to embed EXIF properties. So I believe that what we come up with here can be translated to video without a big effort.

Some details from the research:

• Created a crawler that goes to many websites (starting the crawl from here, downloads the images, and parses their EXIF. The source code for it is here.
• Ran the crawler on my computer, capped at 3000. The results are here.
• The analysis:
  • Out of 3000 images sampled, 189 have any EXIF info (6.3%).
  • Out of those 189 images, 55 (29%) are displayed with their natural width & height.
  • Out of those 55 images, 10 (18%) have a resolution other than 72dpi, and 16 (29%) have EXIF width/height different from their pixel width/height.

So the effect on current images in the wild of using existing EXIF properties, (considering that the sample is representing of the wild) would be on less than 0.3% of images in the case of resolution, and on less than 0.5% in the case of ImageWidth/ImageHeight.

Well, 0.3% >> 0.01% https://github.com/whatwg/html/issues/4495#issuecomment-484084805

A more rigorous check, of 5000 images from 1190 site URLs, also testing whether an image explicitly sets width/height and manual examination of automatically-found affected sites, found around 5 images out of 5000 to be effected. That’s ~0.1% of images, but ~0.4% of sites. Note that “affected” in all of these case would mean that the image would shrink by a ratio 1.2-4.

Examples found by the crawler and verified manually: https://www.e-unwto.org/doi/book/10.18111/9789284421084 (the picture with the houses) https://www.accountingfoundation.org/jsp/Foundation/Page/FAFSectionPage&cid=1176164598953 (the picture of the meeting) http://bellschool.anu.edu.au/regions-countries/australia-new-zealand (the picture of the magnifying glass).

Raw data: https://gist.github.com/noamr/ec771e4f5d0fbb5abe8a68297adaa41d

Thanks @noamr! 0.4% of sites seems pretty high to change image density by default. 1190 sites is a small data set, though, so the percentage is a bit uncertain.

So if I understand your analysis correctly, the 5 images found are using EXIF ImageWidth / ImageHeight / XResolution / YResolution, and those would change the rendered dimensions of the image if they were honored by default. Correct?

Yea, it’s a bit high. It’s a pretty random dataset built from the list of most used sites. Creating a bigger sample set (or a better crawler ) would take some effort.

The 0.4% is for sites with images that have XResolution / YResolution and rely on intrinsic size for layout. The percentage for ImageWidth & ImageHeight is higher - a lot of photos mark those properties when taken, and then don’t modify them when e.g. the image is cropped or scaled. It’s like “original image dimensions”

Have you considered adding a custom metric to HTTP Archive? That should give you data for ~5 million pages, next time it runs a crawl (twice a month). See e.g. https://github.com/HTTPArchive/legacy.httparchive.org/commit/9939498e07a1190720cb082d44e1a7090c3d931c and https://almanac.httparchive.org/en/2019/media

nice! but the aforementioned github repo is said to be deprecated