“This is the story of the birth of the web, its loss of innocence, its decline, and what we can do to make it a bit less gross. Or if you prefer, this is the video in which I say the expression “barbecue sets” far too many times.”
“This is the story of the birth of the web, its loss of innocence, its decline, and what we can do to make it a bit less gross. Or if you prefer, this is the video in which I say the expression “barbecue sets” far too many times.”
“In the last week of 2020, I decided to export my Goodreads data to display it on my personal website. This post is about what I did and how.”
“I’m going to be talking about personal data warehouses, what they are, why you want one, how to build them and some of the interesting things you can do once you’ve set one up.”
First, some backstory. But feel free to skip to the good stuff!
With topics ranging from media and social critiques, to making and tech topics that I care about, to death itself, regular content from creators that post on YouTube have been a part of my daily life for the last several years.
This is enabled by three main features:
Over time, I feel that YouTube has been consistently chipping away at this experience for the sake of engagement.
In 2016, when I found the advertisements to be too invasive, I became a paid "YouTube Red" (now YouTube Premium) subscriber. With ads gone, and with so many content creators posting weekly or more, it was easy to let watching videos through YouTube become a regular habit. Turning off and clearing my YouTube viewing history helped mitigate some of the most creepy aspects of the suggestion system, at the cost of being able to track what I'd seen.
This replaced a lot of idle TV watching time. For several years!
"Progress" marches on, however, and the next thing to go was the experience of accessing the Watch Later playlist. I first noticed this after updating to a 4th generation Apple TV. From the (suggestion-cluttered) main screen of the YouTube app, you must make a series of precise swipes and taps down a narrow side menu to "Library", then to "Watch Later", then to the video that you'd like to start. Not long after, I noticed that the YouTube iOS app and the website itself had similarly moved Watch Later behind a "Library" option that was given the smallest of screen real-estate, overwhelmed by various lists of suggestions of "Recommended for You", "Channels You Might Like", and more.
Most recently, I noticed that YouTube has been changing the definition of a "subscription", where the iOS app will show a timeline of text posts and ephemeral "Moments" in between the actual video content that I am trying to see. Or they'll (experimentally?) try to chunk the subscription display by days or weeks.
All the while, this extra emphasis on recommended videos wore me down. I found myself clicking through to watch stuff that I had not planned to watch when sitting down. Sometimes this would be a fun waste of time. Sometimes I'd get dragged into sensationalized news doom-and-gloom. Regardless, I felt I was being manipulated into giving my time to these suggestions.
And hey, it's #Blocktober, so let's see if we can escape the algorithm a bit more.
What I would like to achieve is what I described at the top of my post:
Did you know that YouTube provides an RSS feed for every channel? It's true! In fact, if you visit your Subscription manager page, you'll find a link that the bottom to download a file containing the feed URLs for all of your subscriptions in a format called OPML.
My YouTube subscriptions download had more than 80 feeds (yikes!) so I didn't want to load these into Aperture by hand. Thankfully, there's a command-line tool called ek that could import all of them for me. I had a small issue between ek's expectations and YouTube's subscription file format, but was able to work around it pretty easily. Update 2020-10-04: the issue has already been fixed!
With Aperture taking care of checking these feeds, I can now look at a somewhat minimal listing of new videos from my subscribed channels whenever I want. For any new video I can see the channel it came from, the title of the video, and when it was posted. Importantly, I can click on it to open the video in the YouTube app to watch it right away or save it for later.
This feels like a lot of work to avoid the mildly-annoying experience of opening the YouTube app and browsing the subscriptions page.
We must go further.
In addition to fetching and parsing feeds, Aperture also has a bit of a secret feature: each channel has an API, and you can generate a secret token which lets you push content into that channel, via an open protocol called Micropub.
So in theory, I could browse through the list of new videos in my YouTube Subscriptions channel, and — somehow — use Micropub to save one of these posts in a different channel, maybe named Watch Later.
This is where we introduce a super handy service called Indiepaper. It's a bit of web plumbing which essentially takes in a URL, gets all the nice metadata it can figure out (what's the title of this page? who's the author? etc.), and creates a Micropub post about it, wherever you want.
The real ✨magic✨ of Indiepaper comes in the form of utilities that making adding an item as few clicks as possible.
For your desktop web browser, Indiepaper can take your channel's Micropub URL and key and generate a bookmarklet which will send the current page you're looking at straight to your Watch Later channel. Add it to your browser's bookmark toolbar, load a YouTube video, click "Watch Later", and you're done!
For an iOS device, Indiepaper also provides a Shortcut that works the same way. Share a YouTube video URL (from the YouTube app, or straight from your reader) to the Shortcut and it adds the item to the channel right away.
For example, I can load up this YouTube video by Aaron Parecki about making a DIY Streaming Bridge with a Raspberry Pi for the ATEM Mini and OBS in my browser and click the "Watch Later" bookmark in my bookmarks toolbar. After a brief delay, I'll see a notification that it "Saved!", and can check my Watch Later channel (marked with the television emoji 📺) to see that it's there now.
At this point I can:
However, something is missing. I still want to be able to watch these, distraction-free, on my TV.
This is where things get ugly, folks. It is also where I admit that this project is not finished.
As far as I'm aware there are no apps for any "smart" TV or media appliance that can browse a Microsub channel. Much less one that can identify a video URL and send it off to the appropriate app for playback.
However, there are some existing ways to save media on your home network and play it back on your TV, such as Plex or Kodi.
So, here are some highlights:
Little details:
If this feels like a lot of work just to avoid some engagement temptation, that's because it is! It may sound simple to say that someone should build a competitor to YouTube that focuses on creators and viewers. One that doesn't seem to spend all its time pushing ads and pulling on you for engagement and all the other things that go into funding a corporate surveillance-driven behemoth.
But no matter how easy it feels to browse a slickly animated user interface that pushes carefully coached eye-grabbing thumbnails of videos with carefully coached compelling titles, there is a lot about video - even watching video - that is not easy!
It's good to stay mindful of what these services make easy for you, what they make hard, and what they make impossible. Trying to take charge of your own consumption is barely a first step.
What aspects of social media are you shutting down for yourself in #Blocktober?
I have a great fondness for IndieWeb building blocks, and Webmention is a wonderful meta-building-block that connects so many individual websites together.
Obligatory "what is Webmention?": it's a specification that describes a way to "tell" a website that some document out on the web links to one of the pages on that site.
Sound simple? It is! Perhaps even suspiciously simple. Webmention enables whole new kinds of interactions between sites (there are some great examples in this A List Apart piece). Unfortunately, almost all of the coordination to support these interactions happens outside of the "Webmention" spec itself.
So when I see blog post titles like these I am not sure exactly what to expect:
There are many more like this! I just grabbed a handful off the last few pages of news.indieweb.org.
These examples and many more are referring to the most talked-about use of Webmention: enabling site-to-site comments and responses, like a souped-up independent alternative to terrible embeds like Disqus.
This may sound like a simple feature! We might expect it to look like this:
However, for a webmention to "succeed", a lot of coordination needs to happen.
On your side:
From here, it's pretty much out of your hands. On their side:
So then they have the webmention, but to actually display it, their site must:
When I see folks posting "I added Webmentions to my site" I want to believe that they have some version of all of the bullet points above.
Unfortunately, there are lots of incompletes.
Folks in this category are collecting webmentions, probably by signing up for a receiving service like webmention.io. Their posts advertise webmention.io as the place to send mentions, webmention.io dutifully checks and stores them, and ... that's it.
This kind of "Webmention support" is often announced alongside a sentence like "Next up I'll figure out how to display them!" For me, this conjures up images of the warehouse at the end of Raiders of the Lost Ark, or a house filled to the ceiling with stacks of moldy newspapers.
There are, I recognize, lots of good reasons not to display webmentions, beyond some of the technical speedbumps and pitfalls I talk about below. For example there are a lot of unanswered questions and not-yet-built tools and services for dealing with moderation and abuse.
Static sites are back and I love it. But if there's one thing that static sites do extremely poorly it is responding dynamically to outside events. Some static sites (including my own!) will save webmentions as they come in, but won't display them until the next time a post is added or modified on the site.
From services to command line tools to fancy build hooks on your fancy hosting service, there are many ways to automate sending webmentions. But unless the receiving end can understand the content of your post, it might not show up as you intended (or at all) on the receiving end. Current best practices would have you add some extra markup to your post so that the receiver can know that your post is a reply, or a like, or an RSVP, or something else. This is a hard coordination problem between your site and theirs! In fact, there is a whole community and standards process for figuring out how to do these things!
Making sure your posts have the "correct" markup to look like you want can be difficult even for developers writing their own HTML. Tools like indiewebify.me, Monocle's preview, and microformats.io can help if you are getting your hands dirty. It's much harder for folks that just redesigned their site with a new WordPress theme.
Bridgy is an absolutely incredible suite of services provided by Ryan, also for free, for the community.
With the power of Bridgy Backfeed you can use Webmention to feed replies, likes, and reposts from your Twitter tweets to their corresponding post on your own site! This works despite the fact that twitter.com does not link to your website because Bridgy generates little "bridge" pages for which to send webmentions. And it's just a little bit of tweaking to have your Webmention display code handle the quirks.
With the power of Bridgy Publish you can use Webmention to automatically copy posts from your website directly to social media silos like Twitter! You do this by hiding a link to Bridgy in your post, which sends a Webmention to Bridgy, and then Bridgy parses your post to understand it and figure out which bits to tweet. And then Bridgy responds with info about your new tweet. And it's just a little bit of tweaking to have your Webmention sender handle those quirks and update your post with that link.
With the power of Bridgy Fed you can use Webmention to automatically copy posts from your website directly into The Fediverse where yadda yadda yadda. And it's just a bit of tweaking to have your Webmention yadda yadda yadda yadda. 😩 I've tired myself out.
These are all fantastic things that are built on top of Webmention but that I often feel are conflated with Webmention.
This one is a bit... unfair on my part. In fact, I think this setup is the best you can get for the least effort, and I encourage folks to go for it. It looks like this:
I love webmention.io and use it myself. It is an amazing community resource run by Aaron at no charge! Kevin's mention.tech is another great tool, as is VoxPelli's webmention.herokuapp.com. By configuring one of them to accept webmentions on your behalf, you save yourself a lot of self-hosting trouble. Each of these services provides APIs that let you pull out the mentions for pages across your site.
Similarly, webmention.js is a really great tool developed by fluffy (at no charge!) that hides a lot of complexity and forethought about how to display webmentions with a single JavaScript include.
All that said, I have some issues with this particular combo long-term because all the fetching and display of webmentions happens in the browser of the person viewing your post.
If 1,000 people visit your post, that's 1,000 requests to webmention.io, putting load on a service being run by one individual for free.
This setup also means that the webmentions for a post aren't included in the original HTML. So, if your site sends a webmention and wants to check back automatically to see if it's shown up, but their site only displays webmentions via JavaScript, then your site will never see it. Likewise, it becomes much harder to keep track of reply chains, for example.
Despite, apparently, being a bit salty today, I really do get excited about Webmention, how it's being used in so many ways to connect independent sites, and new ways it can be used in the future.
I'm worried, a bit, that "Webmention" is starting to lose its meaning in conversation. It's starting to feel like a shorthand that hides important details.
Maybe Webmention can be thought of as less of a "building block" and more like a glue. You can do so many things with glue, like combining a bunch of planks into a table, or building a parade float sculpture with papier-mâché, or doctoring the photo in a passport!
You wouldn't call them all "glue".
Yesterday I joined in the West Coast Homebrew Website Club via Zoom. I was worried I'd be too tired to join - it was held 9-10:30pm my time - but after a nice dinner and wind-down from the day I was in a pretty good place.
Jacky shared some demos, reading, and brainstorming (go check 'em out). And, as captured in Tantek's notes, we also got into some good discussion about community onboarding prompted by Sarah.
A recurring theme as new folks join the IndieWeb community is that the wiki is an amazing resource but also a source of crippling information overload. In recent years we have discussed ways to (re-)organize content for folks geting their own site started. On his own, Jason has begun writing guided intros for developers on topics like Webmention. But as Sarah pointed out, IndieWeb is much more than just developer documentation or guides for folks setting up their own websites. IndieWeb is also an active community with frequent live events, 24/7 discussions, and a long-term memory of observations and ideas and projects and participants. So, perhaps it's worth thinking about onboarding from a perspective of introducing new folks to this community, what they can expect, and how to successfully engage.
For my part, I continued research into the cursed project that I've been noodling on for years and finally started during IndieWebCamp 2020 West. My goal is to give folks a free way to dip their toes into the IndieWeb without needing to understand the building blocks first, but with on-ramps to understanding, customizing, and improving any part of it. While I am learning a lot from projects like Postr, I think I will end up actually making my own version of building blocks like authorization, token, and micropub endpoints. By way of being careful, I would like to be able to test that each one works properly. So that's how I ended up volunteering to help build out the IndieAuth.rocks test suite. 😅
I really enjoy and appreciate these meetups and I look forward to joining the next one!
If you've ever seen your browser automatically fill in your shipping address, or seen your iPhone offer to scan a credit card on an e-commerce site, you're seeing Autofill in action.
Autofill has been part of the WHATWG HTML Standard for some years now. This 2016 write-up by Jason Grigsby gives a pretty good sense of what can be done with it.
The spec describes ways that an HTML <input> element can use the "autocomplete" attribute to hint to the browser that it should offer to fill it with specific Autofill data, if the browser has it and if Autofill is enabled. There's a long list of values related to names, addresses, phone numbers, dates, and more. Additionally, since users might have more than one of a thing, these can be scoped with values like "home", "work", etc. It's possible to further group addresses by "shipping" and "billing", and even to group larger chunks of forms by named sections.
An example might look like:
<input name="home-street-address" autocomplete="shipping home street-address">
Web sign-in is a very IndieWeb concept where you sign into websites using your personal web address, rather than an email address or username.
As with any repetitive tasks, typing my site's URL into these login forms gets annoying. My main browser (Firefox) is pretty smart. Autocomplete kicks in after I type a few characters from my URL and it will offer to fill in URLs that I've typed before. However, since most URLs start with "https://", autocomplete suggestions aren't very useful until I've typed out 9 or more characters (or if I start typing from somewhere in the middle).
Helpfully, "url" is one of the many attributes in the WHATWG Autofill spec! It's described like so:
Home page or other Web page corresponding to the company, person, address, or contact information in the other fields associated with this field
In theory, it should be possible for sites with Web Sign-in to improve this process with the help of the browser and Autofill. For example:
<input name="url" type="url" autocomplete="url">
Or more specifically, use your "home" (personal) URL:
<input name="url" type="url" autocomplete="home url">
It's my thinking that, with this in place, a browser should automatically suggest my URL without me typing anything at all!
I tried this out by setting up url autocomplete suggestions on two different apps with Web Sign-in. (Specifically, my personal instance of Aperture, and the IndieWeb webring).
I then tried signing in and out several times to both sites, using the same URL each time. Browsers tested include Firefox, Chromium, and iOS Safari, all with Autofill enabled.
I am sad to report that none of the tested browsers attempted to automatically fill in the URL value. The extra autocomplete attribute didn't break the default autocompletion, but I still see it suggest every URL it knows about rather than learning one.
I have had trouble finding documentation on how specific browsers implement Autofill. One note in Jason's 2016 article suggests that browsers may need multiple "hints" before it will decide that a particular input is part of a group which should be auto-filled.
Another hint comes from Chromium's settings for managing Autofill data. This is what the form looks like for adding a new address:
Notice a field that isn't there?
IndieWeb NYC's meetup for May 2020 (also virtual Homebrew Website Club US East), met on Zoom from 6pm - 8pm EDT on May 20th.
Here are some notes from the meeting!
jmac.org — Moved to NYC and started attending local HWCs just before The Lockdown. Very interested in Webmention as a technology to connect websites to one another - but wants to improve the focus on how it can be used for real human interaction!
martymcgui.re — Long-time IndieWeb and NYC organizer. Been working on lots of random projects. Currently thinking about turning his pile of "read" posts, largely dumped from GoodReads, into more usable pages about what he's reading now, reading next, and read recently.
david.shanske.com — Long-time IndieWeb and NYC organizer. Been working on location features, aspirationally. Looking forward to a time when he can thrill audiences by reporting being in a location other than "Home".
Sandro — Has been building CMSes and more for years. Been working on some new stuff that works even offline!
Kevin — Worked w/ sandro at Limewire some years ago! Attended past HWC SF meetings in the Before Times. Has been calling into the West Coast virtual meetings, first time calling into East Coast. Wants his site (zootella.com) to publish to 3 places - webserver, local files, and Beaker Browser (just went 1.0!)
Mike — Did web development for years, then stopped, now back again! Learned about IndieWeb through Micro.blog community. Interested in making sure the software he develops is a "good IndieWeb citizen". Working on a kind of Meetup competitor.
jgregorymcverry.com — Long time IndieWeb and NYC organizer. Currently working on a website for a poetry radio show. Folks from all over the world can submit clips. It's wiki-based, but can also receive webmentions.
gRegorLove.com — In San Diego, but joining East Coast to see some new faces. Been out of IndieWeb-land for a bit but dipping toes back in.
kevinmarks.com — Long-time IndieWeb. Microformats co-founder with Tantek and has been blogging forever!
Colin (vonexplaino.com) — From the east coast... of Australia! Been working in the web forever, new to IndieWeb. Recently been working on a lightbox that works without JavaScript.
Other topics of discussion:
Thanks to all who joined us! We will see you all again at our next IndieWeb NYC (aka vHWC US East) meetup online on June 17th! Keep an eye on indieweb.nyc or events.indieweb.org/tag/nyc for the exact date, time, and online location!
Thanks to gRegor for the IndieWebCamp swag for Animal Crossing!
IndieWeb NYC's meetup for April 2020 (also virtual Homebrew Website Club US East, met on Zoom from 6pm - 8pm EDT on April 15th. It was mostly NYC folks, with one joining briefly from Portland!
Here are some notes from the meeting!
jmac.org — writing about webmentions on fogknife.com. Learned a lot about them through sending a "strange" RSVP to an event at events.indieweb.org. The RSVP was a longer post, which also included an RSVP inside it. events.indieweb.org treated the post as both an RSVP and a blog post about the event. Jason enjoyed learning about moderating webmentions through that experience! He also accidentally reposted a bunch of links to IndieNews because his site moved from http to https, and those are "new links". Eek!
aaronpk.com — had some time between giving a workshop and speaking at allthetalks.online. can now post photos from his Nintendo Switch to his website by making a new Twitter account that is just for posts from his Switch, then setting up a cron task to pull and post new tweets to special tags on his site, like the #ACNH tag for posts from Animal Crossing New Horizons.
david.shanske.com — now a member of NOAA's Citizen Weather Observing Program (CWOP)! Making him an official weather source. Working on a large project of archiving IndieWeb event videos from YouTube to archive.org. Hopes to find interested in re-watching some older IndieWebCamp videos to reconstruct notes that were lost when Mozilla recently shut down their Etherpad instance.
martymcgui.re — Still posting one cat GIF per day during the COVID-19 shutdown in NYC. Recently started work on a "consolidated watch later" list using IndieWeb building blocks - mostly with microsub channels in Aperture as a queuing mechanism.
Other topics of discussion.
Thanks to all who joined us! We will see you all again at our next IndieWeb NYC (aka vHWC US East) meetup online on May 20th! Keep an eye on indieweb.nyc or events.indieweb.org/tag/nyc for the exact date, time, and online location!