[Proposal] Packaging for the Web: signed and indexed

We’d like to propose the updated packaging format for the Web to address multiple forms of distribution of pages and PWAs, needs of Physical Web, access to content while offline and hosting of content on CDNs…

It allows to pack multiple resources into a single file and sign it with publisher’s certificate. Digital signature allows to validate authenticity and integrity of the package on the client device regardless of the delivery mechanism, so the pages loaded form the package can be given the claimed origin’s treatment, together with green lock, scripting and network access.

The Content Index structure allows efficient load to resources by direct access to them w/o need to unpack the package, which is important for devices with limited storage.

Here is the initial Explainer, this builds on work of Jeni Tennison and others (Packaging on the Web)

WDYT?

Questions: can this work with fetch + streams + cache API already? That is, could we implement it with those?

The digital publishing community is very interested in this. We’re working on a use cases document, and packaging is a key issue. We will definitely be in touch!

Do we know how web packaging interfaces with resource prioritization, Resource Hints and concepts like link rel=“preload” or H/2 Server Push? Something unclear to me at the moment is whether every asset in a package is considered to have the same priority or whether we’ll have hooks for saying certain assets are more critical to get down early than others. Thoughts?

It’s 404ing. Where did the draft go?

The link is working again. I’m guessing it was a GitHub glitch.

We still are at early stage of figuring out how this fits architecturally into layers of loading/caching. Current thinking is that it probably best to be thought of as “shadow of the http cache”. As such, it should be transparent to fetch/streams/cache APIs. Effectively, loading a package would add a layer to cache. Subsequent fetches of fresher versions of the resources would still go into regular cache, ‘overwriting’ the versions in the package.

It might be interesting to expose this ‘layer’ of caching to Cache API in some form. For example, it might be useful to clear all caches but not the package-provided one, to quickly go to a consistent set of resources.

Ok, so it still sounds like there is some experimentation to do here. Would be great to exhaust the exploration of doing this fetch + streams + cache API (over H/2). As Dmitry mentions, this might expose some use cases not yet covered by the Cache API - which would be great.

I have written about this, but not just software for the web—all software. http://kevinsuttle.com/posts/propelling-dx-through-config

The explainer has evolved quite a bit, we have a rough tool to generate signed packages, and I think it’s time to start writing a spec and taking this format through the standards process.

I think the IETF is probably the right place to standardize the format itself. They have more experts on file formats than the W3C does, and we don’t want to miss out on that review.

The W3C should standardize the process of a browser loading the format. To that end, I’d like to move the repository under the WICG and start the loading spec shortly.

Thoughts?

Fetch+streams+window.caches are enough to use a packaging format to fill in an existing Service Worker, but they don’t help bootstrap a Service Worker that hasn’t already been fetched from the network. The browser needs some sort of native support for that. We do need to figure out whether:

1. that native support should take the form of just loading a Service Worker from the package, and firing an event with the package contents as an argument, vs
2. the browser should put the package’s contents directly into its HTTP cache so that future fetches find them there.

That, and the signature checking and error handling model are the main things the W3C spec will need to spell out.

Could maybe web manifest help in some way? It has a way of bootstrapping a service worker.

https://www.w3.org/TR/appmanifest/#serviceworker-member

Could we add other stuff to manifest to help? or would the manifest just be another resource in the package?

(web manifest my be totally off base - please feel free to disregard if it’s unhelpful… just sounded familiar, so I’d thought I’d mention it)

The Web Package explainer currently says that App Manifest keys can be used inside a Web Package’s manifest.metadata section, but I think I’m going to need to promote that to the top level in order to let unsigned packages declare dependencies, and it probably makes more sense at the top level anyway.

The serviceworker member is definitely how we’d identify which resource inside the package should be used as the SW script, if we pick option 1 from above, but it doesn’t work to only provide the app manifest because we’d like people to be able to install these packages while offline, and the steps for processing the serviceworker member involve fetching data from the network. We need to bundle the intended results of those fetches with the manifest, which is exactly what Web Packaging does.

I think we will need to add some fields to the app manifest to support packages (e.g. origin and packaging-date), and I’ll be sending issues and PRs when we know more clearly what those are.