[Proposal] Content Aggregation Technology

Tags: #<Tag:0x00007fdf730b8100>

Dear all,

I am delighted to announce that we are releasing the first public draft of Content Aggregation Technology (CAT), about which we invite feedback from the Web community at large.

This document is intended to be the first step on a journey. Over the past years, many proprietary content aggregation technologies have been developed and deployed (AMP, Apple News Format, Facebook Instant Articles, MIP, Web Bundles). They have, however, largely been produced without the involvement of the parties whose content they aggregate. The primary intent of this document is to rectify this situation.

As such, CAT focuses primarily on taking two steps towards defining aggregation technology through cooperative methods rather than unilaterally, by leveraging market power:

  • it offers a framework with which to evaluate various approaches to content aggregation; and
  • it outlines a general architecture and a set of high-level requirements meant to improve upon the current situation.

In line with its cooperative intent, CAT proposes no solutions. These will need to be defined as outcomes of this discussion. Assuming the high-level requirements are roughly correct, however, improvements to the situation can be made incrementally and with orthogonal changes.

CAT is brought to your attention jointly by:

The Web already has most of the ingredients needed to deliver amazing experiences that benefit all parties. We look forward to working with all of you to move away from unilateral and proprietary aggregators and to jointly develop the few additions that can improve the Web not just for powerful platforms, but for everyone.

4 Likes

I feel like a good answer to the problem space you’re exploring would very likely also be a good answer to a what’s a good web-centric format for ebooks.

EPUB for instance, is made of web parts, but the end result isn’t very web like, which I’d attribute in large parts to the fact that it isn’t part of the hyperlinked web. It also has an impedance mismatch with the web due to being a single document made of multiple HTML files, without have a terribly clear model for how the stitching of those parts happen.

I’d suggest expanding the range of technologies that are being considered to also cover EPUB (and things like it), and possibly to expand the criteria on which all technologies are being evaluated if it feels like the current ones are missing some important aspect exposed by the addition of ebook focused technologies. Maybe something about archivability, or content integrity.

1 Like

From a user’s perspective, my main concern with this draft is that it asserts that “optimal” user experience and privacy can be attained by giving news publishers unfettered access to the full power of the Web platform. In fact, a primary motivator behind both open solutions like RSS readers and proprietary solutions like AMP is that this is not the case.

The problems with news websites are extensively documented online, such as in https://idlewords.com/talks/website_obesity.htm. They can also be confirmed by visiting any major news website in a browser without an ad-blocker enabled. (I visited usatoday.com and left the homepage open for a few minutes, and it sent over two thousand requests to an astonishing variety of origins.) News websites are notorious for user-hostile patterns like autoplaying video popups and advertisements that cause UI elements to shift around just as you’re about to click on them.

It’s definitely possible to solve that problem through open standards. Here’s one idea: we could standardize on using something like Microformats to mark up article content so it can be extracted and displayed by user-agents without executing arbitrary publisher-provided JavaScript. This could work similarly to how RSS clients and platforms like Mastodon do today, by discarding all tags and attributes except for a small subset of acceptable semantic HTML. It could also be integrated with Transmissible Entitlement Tokens, so publishers can confirm the user has paid for the content before delivering the machine-readable version.

Unfortunately, I get the impression that work like this will not happen as part of this project. The draft describes displaying multiple articles in a carousel – a standard feature present in nearly all popular RSS readers – as “excessively hostile” and counter to the “decisively mutualistic and consensual philosophy of this document”. If even basic UI affordances like that provoke such a reaction, I can’t imagine that taking all decisions about content formatting out of publishers’ hands (and putting them into users’) would ever be considered acceptable.

2 Likes

You bring up good points, but I think that the answer is more complex than that.

A lot of the time the companies pushing for greater bloat in the ad system turn out to be the same companies also then offering a “solution” in the form of AMP or similar projects. Ads are also increasingly invasive because those same companies are using their market power to keep extracting more money from the ad system, leaving less to publishers.

This isn’t long-term sustainable. The ad system needs fixing and the aggregation system needs fixing, but the latter shouldn’t be a solution to the first.

The document doesn’t intend to rule out RSS as hostile. There is plenty that could work with RSS, even though if (as you describe) that’s only for static content then that will be necessarily limited. If you look at things that readers flock to massively, like Covid maps or election results, neither of those works well (or at all) with purely static content. RSS is also different: it’s a direct relationship between readers and publishers.

1 Like

I think we were trying to solve specifically the problem of aggregation and not extend it too much, for instance to describe properties of any data format on the Web.

That said, Dave has also brought up EPUB on Twitter and there may indeed be an interesting path forward for better ebooks and CAT to merge.

https://github.com/nytimes/std-cat/ currently returns a 404.

I disagree.

Some of the organizations creating proprietary news-article formats also contribute to bloat and privacy invasion in the Web ad ecosystem. Some of them do not. For the ones that don’t, the primary reason to develop a proprietary restricted format for news articles is to provide better privacy and user-experience than linking to news websites directly. Any standard that does not address this will not be adopted by platforms or users.

This isn’t unique to ads, either. First-party content is entirely capable of causing problems on its own – many people use features like Reader Mode even when they also have an ad-blocker installed.

It’s worth noting that this content is a small fraction of all the news published online, and it’s often closer to a web application than a traditional article. I don’t think it’d be unreasonable to create a solution that, say, allows publishers to include hyperlinks to those web applications in their static content.

Not necessarily. RSS is an open standard, and there are platforms like Feedly and Flipboard that use it to provide a more curated/aggregated experience.