Noodling on an idea: Projections for Web Apps

For the last few months I’ve been thinking about the many ways native applications can expose information and/or focused tasks within operating systems. Examples of this include Android Homescreen Widgets, macOS Dashboard and Today Panel Widgets, Windows Live Tiles, the Apple Touch Bar, and so on. When building app experiences as Progressive Web Apps, it could be very useful to be able to project aspects of the web app onto these surfaces (hence, “projections”).

Here are a few use cases:

• A streaming video service could offer access to all of the shows or movies you have in your queue that is distinct from the actual player. It might live in a widget on one device, but, with access to all of the plumbing of the PWA itself, could enable users to control the services’ PWA running on the user’s smart TV.
• A stock tracking app could offer a widget for viewing current stock prices for stocks you are watching.
• A calendar service could provide a daily agenda at a glance.
• A music identification service could have a button widget that, when clicked, would access the microphone and attempt to ID the currently playing song.

Given that the web is responsive, the actual projection surface’s dimensions should not matter much. The projection would merely adapt to the available real estate. Some might even be resizable based on user preference.

These are very similar to additional windows in the traditional browser context, but would probably need to be plumbed quite differently.

Anyway, that’s the quick and dirty pitch. What do y’all think? Would you use a feature like this? Is it worth me spending some more time fleshing it out?

Other use cases: Samsung Daily Cards and Mini App Widgets.

I really like this idea, and think it has potential!

Any idea how this would work in reality? I assume there would be a service worker component, so data can be updated in the background?

Is the concept similar to extending the functionality of PictureInPictureWindow?

Reminds me a bit of this old Blackberry 10 API:

https://developer.blackberry.com/html5/apis/v2_2/blackberry.ui.cover.html

As I envision it, there would be connectivity to the SW thread and maybe some limited JS API access. For obvious reasons, it would need to have a low memory & performance profile.

Insofar as it is an extension of a web page outside of the traditional browser chrome, yes.

Yes. WebOS had similar concepts, as did OS X Dashboard widgets, Windows Live Tiles, etc.

There is a somewhat similar need for the PIP as web developers desire, similar to their native application counterparts, to place arbitrary content into them. Now, that doesn’t mean there isn’t a need for a separate API that isn’t joined with the PIP since that has an understanding of video based content; but the functionality is somewhat similar if you don’t need the content to persist outside of the document.

An additional thought here, since there is may be a desire for this to persist outside of the central document that spawned widget (which is not how the PIP works) can’t this just be a PWA with no title bar that is installed from a PWA itself or am I missing something?

Whether it leverages the PIP or not, it seems like a worthwhile thing to explore and has many usecases both in media and elsewhere but we should ensure that they play well together if possible.

PIP issue here: https://github.com/w3c/picture-in-picture/issues/113

One aspect that just dawned on me is that there may be a desire for the PIP even with the focus of media content to continue playing outside of the tab being destroyed. Eg: I’m not watching the video in the tab anyways, allow me to close it but keep the video playing. We would want to understand if this makes sense to users or if they even want it but it may be worth considering that merely making the PIP persist is enough.

Many of these surfaces are too limited to display full web content. However, some simple schema like notifications could integrate well and give flexibility to the operating system to choose how information is displayed.

iOS widgets, Windows live tiles, and Android slices have a concept of a “timeline” where an array of items can be defined, and one or more is shown at a time. Other items are shown by cycling or scrolling.

Been thinking a bit more on this and I think there’s an opportunity for widgets as well as data feeds in general (for providing content to the OS, like contacts for instance):

{
  "name": "Productivity Suite",
  "background_color": "#466FA2",
  "display": "minimal-ui",
  "start_url": "/",
  "icons": [{
      "src": "/img/icon.svg",
      "type": "image/svg+xml"
    },
    {
      "src": "/img/icon-36x36.png",
      "sizes": "36x36",
      "type": "image/png"
    },
    {
      "src": "/img/icon-48x48.png",
      "sizes": "48x48",
      "type": "image/png"
    },
    {
      "src": "/img/icon-72x72.png",
      "sizes": "72x72",
      "type": "image/png"
    },
    {
      "src": "/img/icon-96x96.png",
      "sizes": "96x96",
      "type": "image/png"
    },
    {
      "src": "/img/icon-144x144.png",
      "sizes": "144x144",
      "type": "image/png"
    },
    {
      "src": "/img/icon-192x192.png",
      "sizes": "192x192",
      "type": "image/png"
    }
  ],
  "shortcuts": [
    {
      "name": "Inbox",
      "url": "/"
    },
    {
      "name": "Compose New Email",
      "url": "/compose/"
    },
    {
      "name": "Calendar",
      "url": "/calendar/"
    },
    {
      "name": "New Calendar Event",
      "url": "/calendar/create/"
    },
    {
      "name": "ToDos",
      "url": "/todos/"
    },
    {
      "name": "New ToDo",
      "url": "/todos/create/"
    }
  ],
  "widgets": [
    {
      "name": "Agenda",
      "type": "text/calendar",
      "widget": "agenda",
      "data": "/widgets/data/agenda.ical",
      "update": "900",
      "backgrounds": [
        {
          "src": "/widgets/images/agenda-background.svg",
          "type": "image/svg+xml"
        },
        {
          "src": "/widgets/images/agenda-background-1200x960.png",
          "sizes": "1200x960",
          "type": "image/png"
        },
        {
          "src": "/widgets/images/agenda-background-800x600.png",
          "sizes": "800x600",
          "type": "image/png"
        }
      ]
    },
    {
      "name": "Latest Emails",
      "type": "application/atom+xml",
      "widget": "mailbox",
      "data": "/widgets/data/inbox.xml",
      "update": "900",
      "backgrounds": [
        {
          "src": "/widgets/images/inbox-background.svg",
          "type": "image/svg+xml"
        },
        {
          "src": "/widgets/images/inbox-background-1200x960.png",
          "sizes": "1200x960",
          "type": "image/png"
        },
        {
          "src": "/widgets/images/inbox-background-800x600.png",
          "sizes": "800x600",
          "type": "image/png"
        }
      ]
    },
    {
      "name": "ToDo List",
      "type": "application/atom+xml",
      "widget": "task-list",
      "data": "/widgets/data/todos.xml",
      "update": "1800",
      "backgrounds": [
        {
          "src": "/widgets/images/todos-background.svg",
          "type": "image/svg+xml"
        },
        {
          "src": "/widgets/images/todos-background-1200x960.png",
          "sizes": "1200x960",
          "type": "image/png"
        },
        {
          "src": "/widgets/images/todos-background-800x600.png",
          "sizes": "800x600",
          "type": "image/png"
        }
      ]
    }
  ],
  "feeds": [
    {
      "name": "Inbox",
      "type": "application/atom+xml",
      "data": "/widgets/data/inbox.xml"
    },
    {
      "name": "Contacts",
      "type": "text/vcard",
      "data": "/widgets/data/contacts.vcf"
    }
  ]
}

In all cases, the PWA would handle auth of the various feed URLs and the widgets could probably use the Service Worker for all networking to ensure the session carries through. Multi-tenant apps would likely be more complicated if they wanted to offer unique per-tenant feeds, but this would be a pretty interesting idea for both widgets and participating in the host OS as a data provider as well.

This is interesting. I’m not sure I fully understand the difference between the “widget” entries and the “feeds” entries. Does the widget control its own UI? Does it use JS to read the data (so could use any format?). Maybe I am conflating with your original idea above.

I like the idea of declarative data feeds, in some well-defined existing data format, for UA/OS integration. Presumably the UA/OS would determine the UI and could do deep integrations and link back to the original app? It feels a bit like a separate proposal to the original projections idea though.

I’ve actually begun capturing further thoughts—and greater explanations—on this over here: https://github.com/aarongustafson/pwa-widgets/

In case folks are interested, this is shipping in Edge for Windows 11 now: Build PWA-driven widgets - Microsoft Edge Development | Microsoft Learn