This is probably relevant only to very few people and likely only for myself the next time I think up an idea along these lines.

Obligatory background

I'm a heavy daily user of IndieWeb-style social readers. For my setup, I run my own copy of Aaron Parecki's Aperture, a backend service which manages my subscriptions, fetches feeds, and organizes everything into "channels". On my reading devices, I use an app like Aaron Parecki's Monocle, which talks to Aperture to fetch and display my channels and the entries for each, mark things as read, and more.

These tools speak a protocol called Microsub, which defines a simple HTTP protocol for all those things Monocle does. It specifies how a client can ask a server for channels, list entries in a channel, mark things as read, delete things, add new subscriptions, and so on.

One bonus feature that Aperture has, but that is not part of the Microsub (with an "s") spec, is that in addition to subscribing to feeds, you can also push content into channels, using a different protocol called Micropub, Though they are off by one letter, they do very different things! Micropub (with a "p") is a specification for authoring tools to help you make posts to your personal site, with extensions that also allow for searching posts, updating content, and much more. In Aperture's case, Micropub support is quite minimal - it can be used to push a new entry into a channel, and that's it. It's designed for systems that might not have a public feed, or that create events in real time.

Okay but what's the problem?

I use Aperture to follow some YouTube channels, so I don't have to visit the distraction-heavy YouTube app to see when my favorite channels update. This is possible because every YouTube channel has an RSS feed! What matters is that a good feed reader can take the URL for a YouTube channel (like the one for IndieWebCamp on YouTube) and parse the page to find its feed (in this case, https://www.youtube.com/feeds/videos.xml?channel_id=UCco4TTt7ikz9xnB35HrD5gQ).

YouTube also provides feeds for playlists, and maybe more! It's a fun way to pull content, and they even support push notifications for these feeds via a standard called WebSub .

But! (of course there's a but!) YouTube's feeds encode some useful information, like the URL for a video's thumbnail image, and the description for the video, using an extension of RSS called Media RSS. This isn't recognized by Aperture, and it also isn't recognized by my go-to feed munging service Granary. As a result, while I can see when a new video is posted by the channels I follow, they... don't look like much!

All I can see is that a given channel posted a video, and the title of the video.

Okay can we get to the point?

I'd like to fix this, and my first (wrong) thought was: since Aperture already has these not-very-good entries, maybe I can make an automated system that:

• acts like a Microsub client to fetch each entry from my YouTube Subscriptions channel
• look at each to see if it's missing information like the thumbnail
• for each entry with missing info, look up that info directly from YouTube, maybe via their API
• somehow update the entry with this info.

Again, this is ... the wrong mental model. But why? The docs for Aperture, the Microsub backend, gives us a hint when it covers how to write Microsub clients.

Aperture has implemented the following actions in the Microsub spec:

• GET action=timeline - retrieve the list of items in a channel
• POST action=timeline - mark entries as read, or remove an entry from a channel
• POST action=search - search for a new feed to add
• GET action=preview - preview a feed before following it
• GET action=follow - retrieve the list of feeds followed in a channel
• POST action=follow - follow a new feed in a channel
• POST action=unfollow - unfollow a feed in a channel (existing items from that feed are left in the channel, like IRC/Slack)
• GET action=channels - retrieve the list of channels for a user
• POST action=channels - create, update, and delete channels, or set the order of the channels

Nowhere in that list is the ability to update or even create entries. Those things are outside the scope of the spec. The spec is intentionally narrow in describing how clients can manage channels, subscriptions, and mark read or delete entries pulled from those subscriptions. That's it! And that's good!

Remembering that the "write API" I was thinking of was actually Micropub (with a "p"), I took a look at the source for Aperture that handles Micropub requests and it does refreshingly few things. It allows creating new entries from a Micropub payload, and it supports uploading media that would go along with a payload. That's it. And that's good!

At this point, I thought I could still continue down my wrong-idea road. The automated system would:

• act as a Microsub (with an "s") client to fetch each entry from my YouTube Subscriptions channel
• look at each to see if it's missing information like the thumbnail
• for each entry with missing info, look up that info directly from YouTube, maybe via their API
• use Microsub to delete the original entry
• use Micropub (with a "p") to create a new entry with all the new details

This approach... should work! However, it certainly felt like I was working against the grain.

I brought this up in the IndieWeb dev chat, where Aaron and Ryan cleared things up. Microsub is intentionally simple, and adding general operations to let clients treat the server like a general data store is way out of scope. Similarly, while Aperture supports some of Micropub, that's a choice specific to Aperture.

Have we learned anything?

The general consensus was that entries should get into a Microsub server like Aperture via feeds. And if the feeds I'm looking at don't have the content I want, I should make a feed that does! I should be able to make a proxy service that:

• accepts a URL for a YouTube channel or playlist feed,
• fetches the feed,
• extracts everything I want from each entry, including thumbnails, and maybe even uses the YouTube API to get info like video length,
• rewrites that in a feed format that Aperture likes better. Probably just HTML with microformats2 to make an h-feed

For each of my YouTube subscriptions, I'll swap out the YouTube RSS for the new proxied feed - something that the Microsub API is intended to let me automate.

One thing I mentioned in the chat discussion I linked above: I default to thinking of feed services like this proxy as "public infrastructure" by default - something that has to be on the public web, with all the maintenance and security issues that go along with that.

However, as I self-host my own Aperture instance, I can set up this proxy on the same server and ensure that it only accepts local requests. Fewer public endpoints, fewer problems.

Anyway, maybe I'll get that done and posted about in the near future, but today I just wanted to get these thoughts out of my head and close some tabs!

tl;dr – Kapowski is a simplified tool for finding and posting reaction GIFs to your personal website. It works without a sign-in and gives you HTML to copy-paste into whatever posting interface you use for your website. It's "progressively enhanced" with IndieWeb building blocks, so if your site supports them it becomes faster and easier to use. Search and content are currently powered by Gfycat.

IndieWeb building blocks, you say?

Kapowski supports IndieAuth and Micropub. If your site supports these, you can sign in using your website as your identity and then use Kapowski to post the image to your site directly.

But that's not all! If your site supports sending Webmentions and markup for reply posts, you can use Kapowski to make your post a photo reply!

Still reading? Here's some (too much) history.

I really like Micropub as an IndieWeb building block. As a developer, it's easy to understand on the wire. It's very extensible because the spec provides very few constraints over what you can post with it.

However, that flexibility comes at a coordination cost. I had (have!) a dream that being able to rapidly iterate on special-purpose Micropub clients will let many ways of posting bloom. I also loved (love!) Glitch as a place to build web apps in the open where other folk can see how they work and remix them to make them their own.

So, I built and stealth-launched Kapowski built on Glitch back in 2018. I didn't use it myself very much (but definitely some), and even shared it around some IndieWeb events, like when I talked about mobile posting at the 2019 IndieWeb Summit.

I kept Kapowski intentionally simple, hoping that some other IndieWeb folks might use it, give feedback, and iterate on the idea of what a good user experience might be for posting reply GIFs on the IndieWeb.

I ... didn't get much feedback! As far as I know, few people have used it. However, I very much did notice that it kept breaking.

Giphy, the original service Kapowski used, was bought by Facebook with the intention of, I don't know, tracking all the GIFs people posted. I didn't like that, so I switched things over to Gfycat. (Although with Facebook (Meta) being forced by UK regulators to sell off Giphy, maybe it'll be time soon to bring it back.)

The biggest problem, though, was that server-side Javascript bits rot. I want to be able to stand up a hobby project and forget it for months or years at a time. For a project of this pace, especially one that I think of as being very simple, the Javascript stack moves fast. I would get notices every week or so that this or that dependency had a required security update. Sometimes apply what looked like a small point update would cause a breaking change in an API (Axios!). Eventually, it became not-fun to think about keeping up Kapowski.

Multiplying this maintenance across a number of other Micropub clients I had managed to barely knock together on Glitch led to me burning out on the idea. So, I stopped maintaining it and at some point it stopped working.

Reviving the embers

I still want to see a thousand Micropub-powered flowers bloom, but I don't have the personal project bandwidth to build the tool set on Glitch that I thought would make that possible. I'm just not that fluent in server-side Javascript development and project management, and it's too far of a road right now to git gud.

That said, there are styles of web app development I am much more comfortable working in! I think I can take this stuff a lot further by sharpening the knives I already know how to use.

So, I've spent a good chunk of free time this year quietly porting some of my IndieWeb projects to PHP and hosting them on a virtual private server. That's stuff I know how to do! As I've re-built each one, I've also looked to extract the common points of similarity and complexity into a kind of "Micropub kit", with a common-but-extensible engine. That's made each client easier to build and deploy, and that's very exciting.

(This "micropub kit" isn't ready for public consumption at all but it is available for looking-at if you want. Here's the micropubkit source.)

What's next?

Since it's IndieWeb Gift Calendar season, I think I'll spend the next month polishing up and posting more about this work.  If you have thoughts about Kapowski, "micropubkit", or posting weird stuff on the IndieWeb in general, I'd love to read them! Just reply to this post on your own site and send me a Webmention.

If you aren't ready to reply in words... maybe a GIF would do? :)

First, some backstory. But feel free to skip to the good stuff!

With topics ranging from media and social critiques, to making and tech topics that I care about, to death itself, regular content from creators that post on YouTube have been a part of my daily life for the last several years.

This is enabled by three main features:

• Subscriptions, to let me check in for new videos from creators I want to follow.
• The Watch Later playlist, to let me save videos I wanted to include in my regular watching.
• A YouTube app connected to my TV to let me play through my Watch Later list.

Over time, I feel that YouTube has been consistently chipping away at this experience for the sake of engagement.

In 2016, when I found the advertisements to be too invasive, I became a paid "YouTube Red" (now YouTube Premium) subscriber. With ads gone, and with so many content creators posting weekly or more, it was easy to let watching videos through YouTube become a regular habit. Turning off and clearing my YouTube viewing history helped mitigate some of the most creepy aspects of the suggestion system, at the cost of being able to track what I'd seen.

This replaced a lot of idle TV watching time. For several years!

"Progress" marches on, however, and the next thing to go was the experience of accessing the Watch Later playlist. I first noticed this after updating to a 4th generation Apple TV. From the (suggestion-cluttered) main screen of the YouTube app, you must make a series of precise swipes and taps down a narrow side menu to "Library", then to "Watch Later", then to the video that you'd like to start. Not long after, I noticed that the YouTube iOS app and the website itself had similarly moved Watch Later behind a "Library" option that was given the smallest of screen real-estate, overwhelmed by various lists of suggestions of "Recommended for You", "Channels You Might Like", and more.

Most recently, I noticed that YouTube has been changing the definition of a "subscription", where the iOS app will show a timeline of text posts and ephemeral "Moments" in between the actual video content that I am trying to see. Or they'll (experimentally?) try to chunk the subscription display by days or weeks.

All the while, this extra emphasis on recommended videos wore me down. I found myself clicking through to watch stuff that I had not planned to watch when sitting down. Sometimes this would be a fun waste of time. Sometimes I'd get dragged into sensationalized news doom-and-gloom. Regardless, I felt I was being manipulated into giving my time to these suggestions.

And hey, it's #Blocktober, so let's see if we can escape the algorithm a bit more.

A Plan

What I would like to achieve is what I described at the top of my post:

• I want a way to check for new videos from creators I follow (no notifications, please).
• I want a way to add those to a list for later viewing.
• I want to view items from that list on my TV.

I have some tools that can help with each part of that plan.

RSS is (still) not dead

Feeds are already part of my daily life, thanks to an indie social reader setup. I run Aperture, a Microsub server that manages organizing source feeds in various formats, checking them periodically for new content, and processing them into items grouped by channel. I can browse and interact with those items and channels via Microsub clients, like Monocle which runs in the browser and on my mobile devices with an app called Indigenous.

Did you know that YouTube provides an RSS feed for every channel? It's true! In fact, if you visit your Subscription manager page, you'll find a link that the bottom to download a file containing the feed URLs for all of your subscriptions in a format called OPML.

My YouTube subscriptions download had more than 80 feeds (yikes!) so I didn't want to load these into Aperture by hand. Thankfully, there's a command-line tool called ek that could import all of them for me. I had a small issue between ek's expectations and YouTube's subscription file format, but was able to work around it pretty easily. Update 2020-10-04: the issue has already been fixed!

With Aperture taking care of checking these feeds, I can now look at a somewhat minimal listing of new videos from my subscribed channels whenever I want. For any new video I can see the channel it came from, the title of the video, and when it was posted. Importantly, I can click on it to open the video in the YouTube app to watch it right away or save it for later.

This feels like a lot of work to avoid the mildly-annoying experience of opening the YouTube app and browsing the subscriptions page.

We must go further.

Save me (for later)

In addition to fetching and parsing feeds, Aperture also has a bit of a secret feature: each channel has an API, and you can generate a secret token which lets you push content into that channel, via an open protocol called Micropub.

So in theory, I could browse through the list of new videos in my YouTube Subscriptions channel, and — somehow — use Micropub to save one of these posts in a different channel, maybe named Watch Later.

This is where we introduce a super handy service called Indiepaper. It's a bit of web plumbing which essentially takes in a URL, gets all the nice metadata it can figure out (what's the title of this page? who's the author? etc.), and creates a Micropub post about it, wherever you want.

The real ✨magic✨ of Indiepaper comes in the form of utilities that making adding an item as few clicks as possible.

For your desktop web browser, Indiepaper can take your channel's Micropub URL and key and generate a bookmarklet which will send the current page you're looking at straight to your Watch Later channel. Add it to your browser's bookmark toolbar, load a YouTube video, click "Watch Later", and you're done!

For an iOS device, Indiepaper also provides a Shortcut that works the same way. Share a YouTube video URL (from the YouTube app, or straight from your reader) to the Shortcut and it adds the item to the channel right away.

For example, I can load up this YouTube video by Aaron Parecki about making a DIY Streaming Bridge with a Raspberry Pi for the ATEM Mini and OBS in my browser and click the "Watch Later" bookmark in my bookmarks toolbar. After a brief delay, I'll see a notification that it "Saved!", and can check my Watch Later channel (marked with the television emoji 📺) to see that it's there now.

At this point I can:

• Browse new videos from my subscriptions in my feed reader.
  
• Save videos on demand to a separate watch later channel in my feed reader

However, something is missing. I still want to be able to watch these, distraction-free, on my TV.

The Last (and Longest) Mile

This is where things get ugly, folks. It is also where I admit that this project is not finished.

As far as I'm aware there are no apps for any "smart" TV or media appliance that can browse a Microsub channel. Much less one that can identify a video URL and send it off to the appropriate app for playback.

However, there are some existing ways to save media on your home network and play it back on your TV, such as Plex or Kodi.

So, here are some highlights:

• Assuming you've got a Plex server with a library called "myTube". Your TV (maybe through an appliance) can run a Plex app that lets you browse and play that local media distraction-free.
  
• An automated task on that server could act as a Microsub client, periodically looking in your Watch Later channel for new entries.
• For each new entry, the automated task could fetch the video with a tool like youtube-dl and save it to the myTube folder, where Plex will find it.

Little details:

• To prevent repeated downloads, the automated task should either delete or mark new entries as "read" once they've been downloaded.
• Plex doesn't have an easy way to delete media from the TV interface. Perhaps the automated task can check with Plex to see if a video has been watched and, if so, remove it from myTube. Or maybe save it to a "watched" list somewhere!
  

If this feels like a lot of work just to avoid some engagement temptation, that's because it is! It may sound simple to say that someone should build a competitor to YouTube that focuses on creators and viewers. One that doesn't seem to spend all its time pushing ads and pulling on you for engagement and all the other things that go into funding a corporate surveillance-driven behemoth.

But no matter how easy it feels to browse a slickly animated user interface that pushes carefully coached eye-grabbing thumbnails of videos with carefully coached compelling titles, there is a lot about video - even watching video - that is not easy!

It's good to stay mindful of what these services make easy for you, what they make hard, and what they make impossible. Trying to take charge of your own consumption is barely a first step.

What aspects of social media are you shutting down for yourself in #Blocktober?

This post gives more technical detail for the recent talk that I gave at Bring-a-Hack NYC. In it, I describe a system that copies posts from Ghost Party's Instagram automatically to the Ghost Party Website at ghostparty.today.

This roughly works like so:

• OwnYourGram periodically checks our Instagram for new images
  
• For each new post, it makes a Micropub request to a service that I wrote for this purpose.
• The endpoint accepts the image files and uploads them to our site using the Neocities API. Then it creates some new HTML from the micropub post data, edits the site's main index.html page to include the content, then uploads it back to Neocities.

Authorization with IndieAuth

One missing step in that rough outline is the bit where we tell OwnYourGram *how* to post to our site via Micropub, as well as making sure that OwnYourGram can prove that we have given it permission to post there. Many Micropub clients support IndieAuth as a way to do this permission-granting step (known as authorization). With a typical personal website, I'd delegate this process to indieauth.com, which lets you offload the step of "proving" that you own the website where you're trying to post by instead letting you log into an existing social network account.

ghostparty.today is a shared website, so I thought a simpler way to approach this would be to create a custom authorization endpoint that accepts a password. I could give this password to other GHOST PARTY folks and they can post whatever they like.

Some nice folks in the IndieWeb community had already created an authorization endpoint that works this way, called selfauth. I created a pretty simplistic port of selfauth from PHP to JavaScript using NodeJS on the awesome Glitch social code hosting platform. It needs some more robust error checking, and some micropub clients seem to have trouble with it, but it works with OwnYourGram and Quill, so it's Good Enough for now!

If you're interested, you can find the source for this endpoint here: https://glitch.com/edit/#!/befitting-price

Micropub Endpoint

Finally, there's the Micropub endpoint itself. I based mine off of this Glitch micropub project, created by Adam Bachman at IndieWebCamp Baltimore this year. Specfically, I took the bits that verify the access token. From there, it was a matter of stumbling through various bits of handling a Micropub post, pulling in extra libraries as I needed them. I slowly added support for saving uploaded files to our Neocities site via the Neocities API, then for generating an HTML snippet for the new post, and finally for adding the post to the site's main page.

The process for adding the generated post content to the site itself is extremely retro. The Micropub endpoint actually downloads the HTML from https://ghostparty.today/index.html. Inside there is a comment that looks like:

<!-- NEW POSTS BELOW -->

So the new contents for index.html is everything before the comment, the comment, the new content, and then everything after the comment. Once that's constructed, the endpoint uploads index.html back to Neocities with the new contents.

You can find the source for the Micropub endpoint here: https://glitch.com/edit/#!/prism-dirt

It should be possible to fork the project on Glitch and configure it to work with your own Neocities site! You'll need to use cURL with the Neocities API to generate an authorization token to configure the app, rather than saving username and password info in the project!

Making ghostparty.today "IndieWeb-Readable"

The GHOST PARTY site follows an extremely old-school publishing model where each post only exists as a small section of HTML on a single long page. This complicates things a bit because interacting with posts on the IndieWeb typically requires three things:

• A machine-readable "identity" h-card that says "who" this website is, including a logo, the site name, canonical URL.
  
• A machine-readable feed that contains new posts, allowing people to follow our updates in their favorite indie reader.
• A separate permalink URL for each post, containing machine-readable info about that post, allowing other IndieWeb sites to "understand" the content when they want to RSVP to events, repost or like photos, and more.

Microformats2 parsers like X-Ray do a great job at parsing HTML page into meaningful data, but the programs that use the parsed data usually stop at the first thing they find. So, if I ask X-Ray to parse https://ghostparty.today/, this is what it sees:

{
  "data":{
    "type":"card",
    "name":"GHOST PARTY",
    "url":"https://ghostparty.today/",
    "photo":"https://ghostparty.neocities.org/ghost-party-logo.png"
  },
  "url":"https://ghostparty.today/",
  "code":200
}

This is great in that it contains the "identity" information that we want the page to have. However, that page also has a feed! How can I get at that? By telling X-Ray to only look at a fragment of the page! Here's how X-Ray sees https://ghostparty.today/#feed: 

{
  "data":{
    "type":"feed",
    "items":[
      {
        "type":"entry",
        "published":"2018-03-07T16:44:16+00:00",
        "url":"https://ghostparty.today/#2018-03-07-181020",
        "photo":["https://ghostparty.today/uploads/2018-03-07/181020-ig0LN1JG.jpg"],
        "content":{"text":"snow day selfie #snowday #snowfie #ghoststories"} 
      },
      {
        "type":"entry",
        ...
      },
      ...
    ],
    "url":"https://ghostparty.today/#feed",
    "code":200,
    "info":{"found_fragment":true}
  }
}

Data! Nice! So if we want to follow ghostparty.today with an indie reader, we use https://ghostparty.today/#feed as the URL.

Looking at the content of the feed, you'll notice that the individual items have a url property which also has a fragment ID in the URL. This allows IndieWeb sites to interact with a particular post, even though it's one of many on the same page. For example, I was able to create an RSVP post to the most recent Ghost Party show, and (by looking just at the content in the fragment at https://ghostparty.today/#show-2018-02-14) my site was able to pick up the machine-readable details about the event to display them on my own site.

The microformats markup for the h-card and h-feed are built into the index.html file itself and don't change very often. However, for each new micropub post we need to convert those properties into a chunk of HTML for the index.html page that also contains the microformats h-entry markup. To do this, I made this Handlebars template:

{% raw %}<div class="h-entry" id="{{ mp-slug }}">
  <a class="u-author" href="/"></a>
  <a class="u-url" href="#{{ mp-slug }}">{{#if name}}<h2 class="p-name">{{ name }}</h2>{{/if}}</a>
  <time class="dt-published" datetime="{{ published }}"></time>
  {{#if video}}
    {{#each video}}
      <video controls loop class="u-video" src="{{ . }}"></video>
    {{/each}}
  {{ else }}
    {{#each photo}}
      <img class="u-photo" src="{{ . }}" />
    {{/each}}
    {{#each audio}}
      <audio class="u-audio" src="{{ . }}"></audio>
    {{/each}}
  {{/if}}
  <div class="e-content">{{ content }}</div>
</div>
<img src="ghosts.gif" />
{% endraw %}

Not so bad!

Resources

Here's a collection of some of the other libraries and docs I used to pull this thing together:

• Neocities API
  • Includes details on using cURL to get an authorization token.
  • There’s a Node module for the Neocities API, but it uses username and passwords, so I just used regular HTTP requests to handle talking to Neocities.
• Node Request HTTP Client
  • Verifies the authorization token by querying the token endpoint (I used tokens.indieauth.com for this)
  • Handles uploading files like images to Neocities via their API.
  • Used to fetch index.html from the site and later to writes it back via Neocities API.
• ES6 Promises
  • There's a lot of asynchronous code that waits for network requests. This code is very Promise-heavy to handle that.
• Handlebars templating
  • Made it easy to take the Micropub properties for a new post and rendered them into HTML content with microformats.
• Express 4 API
  • Glitch uses Node and Express 4 by default, and I got to learn lots about it!
• Multer Express Middleware
  • For handling uploaded files
• Body Parser Express Middleware
  • For handling regular Micropub properties
• Validator.js
  • Half-using this for the authorization endpoint, need to clean it up and finish it up!
• OwnYourGram API docs
  • Shows what properties are sent over Micropub, so you know exactly the kind of data your Micropub endpoint will need to handle.

This post is a rough summary of a lightning talk that I gave on February 28th at the Bring-A-Hack NYC with Hackaday + Ultimaker + New Lab. Many thanks to Shayna Gentiluomo for organizing and sometimes co-conspirator Matt Griffin for nudging me to speak!

---

I have a lot of distrust for social silos like Facebook and Twitter which draw us in with the promise of connecting with family, friends, and interesting people, but I sometimes have trouble articulating that in a succinct way. Thankfully, I saw a perfect example this morning:

Posting on the @facebook website is not working in Mac OS today. JavaScript eats CSS & HTML, making data entry field disappear except for a spookily disembodied close-box X. Been this way a few hours. — @zeldman, 9:53 AM - 28 Feb 2018

Some Facebook team pushed an (untested?) update which broke the website, preventing some users from posting. "Move fast and break things". These silos don't value us or our content. So, we should get out. But what is the alternative?

The IndieWeb helps us take our content back, by posting to our own websites on our own domains. With some work, we can even get the benefits of these social networks and take our conversations with us. There’s a whole lot of great stuff on the IndieWeb website, so go spend a couple of days reading it. But I’m not here to talk about all those good things today. I’m here to talk about how I used them to do a dumb hack.

ghostparty.today is a website for Ghost Party – an improv project that has a ... certain aesthetic. Specifically the website was pitched as “exactly like the Heaven’s Gate website” but we compromised on “extreme Geocities”.

My Ghost Party co-host (co-ghost) built the first version of the site on Neocities.org, a lovely hosting provider that recaptures the feel of editing raw HTML directly in your browser. However, hand-editing HTML gets exhausting and sometimes you just want to post things to Instagram (and, of course,  Ghost Party has an Instagram).

And Instagram is fine as a marketing platform! Lots of people look at it and like the pictures. But what about our website? Our aesthetic?

Micropub is a standard (and now W3C Recommendation) designed over some years of practice from the IndieWeb community that defines a way for apps and clients to easily make new posts to any website (even yours) if it supports Micropub.

I used Glitch, an amazing community and JavaScript-server-app hosting service, to make a simple micropub service for ghostparty.today.

It accepts posts via micropub, including file uploads, and uses the Neocities API (hey, it’s not the 90s! Neocities has an API!) to upload the files to our website. Then, it fetches our site’s index.html, adds the content of the new post, and uploads it back. You can find the source here on Glitch and fork it for your own purposes!

Now that our site supports micropub, we can make use of a wonderful service called OwnYourGram. Once we give OwnYourGram permission to post to our site and read our Instagram feed, it will poll our Instagram account and copy each new post back to our website, where we can display it however we want.

We can go a step further, using an IndieWeb building block called Webmention and a service called brid.gy, to bring all of those responses, those Instagram and Facebook likes and comments, Twitter retweets, and more, back to our site for display.

And to show how it works, here’s a micropub client call Quill that works for any micropub-enabled site. Let’s make a quick post, now!

Once we hit the "Post" button and refresh the homepage, we can see the new post on the site!

This was the end of time time. I would have like to further encourage folks to check out Homebrew Website Clubs in their own communities, or to start one if they don't exist. They're part show-and-tell, part support group, and part mad science lab for people who love the web!

Jonathan Prozzi and I have challenged one another to make a post about improving our websites once a week. This one should have gone up last week!

A few weeks ago I posted some thoughts about my IndieWeb setup called "Easier POSSE with Micropub Edits?" in which I wished for a tool that would let me take a given post from my site, syndicate it to silos like Twitter and Facebook (tweaking the content if I want), and updating the post on my site to show the links to those syndicated copies.

Why?

I failed to make at least one important thing clear in my original post – why do I care about syndication links? There are many reasons.

• I have set up a system to notify me when my posts get webmentions from other sites.
• Brid.gy will backfeed comments and reactions from sites like Twitter and Facebook to my site via webmention, but only if brid.gy knows which posts on those silos correspond to a post on my site.
  

If I decide that a post should be syndicated to a silo, it's because I want it to reach the people who follow me there and, if that is true, I also want their interactions to come back to my site. So, in these ways, a post isn't "done" unless it is on my site, with syndicated copies on the silos I care about, and with syndication links for brid.gy to feed the interactions back.

Starting at the End

I decided to start by making my site's Micropub server support Micropub Source Content Queries and Micropub Updates. Any tool that helped automate syndication would need this plumbing to operate.

When implementing a new feature, it always helps to have something to test against. So, I went looking for a Micropub client which supported queries and edits. The test suite for Micropub at micropub.rocks includes a lovely implementation report grid, showing which Micropub clients support what features of the spec.

Of the clients listed, two of them were web-based and Open Source. I had played with and liked Inkstone in the past, but its edit features are currently considered a work-in-progress. So, I tried out Micropublish.net, and it was exactly what I was looking for.

Micropublish has a feature to let you enter a URL for a post on your site to edit. It will use Micropub source content queries to get the source data for that post and let you edit the content and other properties of the post. It can then send a Micropub update to save the updated version of the post back to your site, if your server supports updates. It even has a great feature for developers - a "Preview" button will show you exactly what request will be sent to your server for the update.

Micropublish.net is a great tool for testing out Micropub query and update support, but my Micropub server is bespoke, hastily-written, hand-rolled Python. So, while it was easy enough to add query support, it took me a while to get my code structure cleaned up, write some tests, and actually implement updates.

A New Workflow

I am pleased to say that it works and, with the help of Micropublish.net, I now have a functioning workflow for publishing to my site and syndicating to silos like Twitter and Facebook, even from my phone, without having to open my laptop, edit YAML data, and push git repositories around. It looks like this.

• Make a new post to my site with a micropub client like Quill.
  
• Open the post for editing in micropublish.net (I use Url Forwarder for Android to make this super easy on my phone, a bookmarklet makes it easy on my laptop).
• In a new tab, log in to Twitter and make a similar post, copy the URL to the new tweet into the Syndication field on my post.
  
• Repeat the steps to make posts on Facebook, Mastodon, etc., copying their URLs into the Syndication field.
• Finally, hit "Update" in micropublish.net to update my post with the syndication links.
  

This is still a very manual process, but it now makes it possible to finish a post in a way that I couldn't before. In the spirit of manual until it hurts, I will use this for a while and see what existing pain points remain, and what new ones appear, to help decide what comes next.

Thanks to Barry Frost for micropublish.net and to Tantek for the nudge to write an update!

Jonathan Prozzi and I have challenged one another to make a post about improving our websites once a week. I'm late with this one!

Most of the features on my website are experiments in learning new things. Sometimes I learn a better way of doing something that I've already built into the site and it's time to migrate!

Moving Media files from Git LFS to a Media Endpoint

I build my site with Jekyll, and I store my site's configuration and text content via Git. One of the things that most folks avoid with Git is storing text content (which fits into Git's model of efficiently storing differences over time) with large binary files like images, etc. (which Git cannot manage as efficiently).

When I first set up my site, I made use of Git LFS ("Large File Storage") for managing anything that wasn't text. Any images, video, or audio that I added to my site was stored in an _assets/ folder in a way that matched uploaded files to the posts they were a part of. Git LFS would transparently ship those files off to a secondary server rather than include their content in the Git repository itself. I had to go through some hoops to set up my local GitLab server to support Git LFS and to set up Git LFS with the server that handles receiving new posts via Micropub, compiling and deploying the site.

It turns out that there are many reasons that a site would want to handle media files separately from the text content that refers to them. In fact, it is a common enough pattern that the Micropub standard includes a definition for a separate "media endpoint" to handle file uploads. I shared a Micropub media endpoint implementation that I built called Spano a while back, and it has been working well with support from tools like Quill. So the text content of my site is served from https://martymcgui.re/, and my media files from https://media.martymcgui.re/. With a couple of changes in my code and my workflows, this has become the way I handle all media files for my site.

However, I still had a bunch of files in site being handled by Git LFS, and some of my Jekyll code (plugins and templates) for showing embeds expected files to be on the local filesystem. This past week I took some time to write some scripts to find all references to those local files, migrate them to my media server, and update the outgoing links. I also updated my embed handling so it didn't rely on local files. This let me delete a lot of local metadata I was keeping but not using, like all the EXIF tags in uploaded photos. I am now Git LFS free and it feels like one less thing to worry about.

Better Caching for Mentions from Webmention.io

When I finally started displaying webmentions, I had a very simple model for how to cache all the info from webmention.io. Basically: I stored all mentions in a big array and, when my site went to fetch new mentions, it would keep fetching until it saw the "last" mention again. This led to a bit of a bug where someone might send me a mention, update their page, and send the mention again. My site would not be able to recognize the "last" mention, so it would fetch all my mentions again, leading to everything appearing twice.

This past week I rewrote my mention handling to avoid this problem by replacing this array and storing mentions in a hash based on the source and target. The new code also checks to see if the verification date of the mention has changed (giving me a way to detect and notify about changed mentions in the future). I also reorganized my mention cache to include an index by the target URL on my site. This makes it a bit quicker to find mentions for a given page when rendering out the site.

Neither of these changes are really visible to readers of my site, but they have been useful for cleaning things up. The webmention.io handling in particular has brought my plugin a lot closer to being something I could release for other people to use!

Earlier this week I announced Screech, a simple client for posting audio content (such as podcasts) to your own site using Micropub.

I’m pleased to announce that Screech is now available publicly. You can check it out at https://screech.schmarty.net/.

Screech also now has an adorable logo, thanks to a couple of pieces of public domain art.

• Owl from LumoSpark
• Microphone from OpenClipArt-Vectors

I've been working on my idea of what IndieWeb podcast publishing looks like for some months now, both with the improvised We Have to Ask comedy podcast that I make with Jonathan Monroe and more recently with the This Week in the IndieWeb Audio Edition that I started producing last month.

In addition to following standard IndieWeb-friendly practices like using microformats2 feeds, backfeeding social interactions from Twitter and Facebook with bridgy, and exploring other interesting audio markup tricks, I wanted a tool that made it easy for me to publish new content to my sites via the Micropub protocol, which supports sending audio media files.

I didn't see another micropub client in the wild that supported audio files in the way that I wanted, so I made my own.

Screech is an audio-publishing-focused micropub web client with a Python server component built on Flask. Screech supports logging in with your own site using indieauth and posts to your site's micropub endpoint.

It's still a work-in-progress, but the basic flows work well enough for my needs.

One fun feature of Screech is that once you select an MP3 file for upload, it uses the jsmediatags library to pull out information about the track, such as its duration, track title, album and artist info, etc. This info is outside the scope of the Micropub standard, but if you want to add support to your server, you'll see those properties arrive with names like "id3-title", "id3-artist", etc.

Edit, March 17, 2017 — Screech is available at screech.schmarty.net. If you'd like to run it yourself, add features, or fix bugs, you can find the source code and instructions on GitHub.

Edit, November 30, 2022 — Screech has been rewritten in PHP. The new version is available at screech.bayside.pub. If you'd like to run it yourself, add features, or fix bugs, you can find the source code and instructions on my Git server.

There are many TODOs yet on my plate for Screech before I'd consider it "done", such as micropub media endpoint support, syndication support, adding a photo to the post as a "poster" image, and more.

I'd love to hear feedback from the IndieWeb community! What do you think it means to be an "IndieWeb podcaster"? What features would make Screech useful for you?

Micropub is an open API standard to create posts on one's own domain using third-party clients  and currently a W3C Candidate Recommendation. One of the (semi-) recent additions is the idea of a Micropub Media Endpoint. The Media Endpoint provides a way for Micropub clients to upload media files to a Micropub service, receiving a URL that is sent along in place of the file contents when the post is published.

Some of the things I like about Micropub media endpoints include:

• The spec allows the media endpoint to be on a completely separate domain from the "full" micropub endpoint.
• The spec doesn't specify anything about how the files are stored or their final URLs or filenames.
  
• They make it easy to separate the handling of (large) media files from the (presumably much smaller) content and metadata of a post.
• They enable Micropub clients to upload multiple files without creating multiple posts. This makes it simpler to create posts that contain multiple images, like a gallery.

Personally, I wanted a Micropub media endpoint server with a few extra properties:

• It should be able to run completely separately from, and therefore work in conjunction with, any other micropub server implementation.
  
• It should not store duplicate files. If the same file is uploaded twice, the same URL should be returned both times.
• It should not allow overwriting files. If two images of the same name are uploaded, both are kept and receive different URLs.

Enter HashFS

My extra features above essentially describe a content-addressable storage storage system. CAS is a way of storing and accessing data based on some property of the actual content, rather than (potentially arbitrary) files and folders.

HashFS is a Python implementation of a content-addressable file management system. You give it files, it will put them in a directory structure based on a cryptographic hash function of the contents of that file. In other words - HashFS can take any file and give back a unique path to that file which will never change (if you later upload a new version of the file, it gets a different path).

To add the the fun of HashFS, there is a Flask extension called Flask-HashFS which makes it easy to expose a HashFS file store on the web via the Python Flask framework.

Introducing Spano

Spano is a Micropub Media Endpoint server written in Python via the Flask framework which combines Flask-HashFS for file storage with Flask-IndieAuth (introduced earlier) to handle authentication and authorization.

Spano is a server-side web app that basically does one thing: it accepts HTTP POST requests with a valid IndieAuth token and a file named "file", stores that file, and returns a URL to that file. The task of serving uploaded files is left to a dedicated web server like nginx or Apache.

Using Spano

Once Spano has been set up and configured for your domain, uploading is a matter of getting a valid IndieAuth token. IndieAuth-enabled Micropub clients will do this automatically. For testing by hand I like to log in to Quill and copy the access token from the Quill settings page. With token in hand, uploads are as easy as:

curl -D - -F "file=@myfile.jpg" \
  -H"Authorization: Bearer xxxx..." \
  https://media.example.com/micropub/

Which should output a response like:

HTTP/1.1 100 Continue

HTTP/1.0 201 CREATED
Content-Type: text/html; charset=utf-8
Content-Length: 108
Location: https://media.example.com/cc/a5/97/7c/2004..2cb.jpg
Server: Werkzeug/0.11.4 Python/2.7.11
Date: Thu, 26 Jan 2017 02:40:05 GMT

File created: https://media.example.com/cc/a5/97/7c/2004..2cb.jpg

Integrating Spano with your Micropub Endpoint

If you want Micropub clients to use Spano as your Media Endpoint, you need to advertise it. This is handled by your "main" Micropub server using discovery. Essentially, a client will make a configuration request to your server like so:

https://example.com/micropub?q=config

And your server's response should be a JSON-formatted object specifying the "media-endpoint". A bare minimum example:

{
  "media-endpoint": "https://media.example.com/micropub/"
}

In addition to advertising the media-endpoint, your Micropub server must be able to handle lists of URLs in places where it would normally expect a file.

For example, when posting a photo from Quill without a media endpoint, your Micropub server will receive a multipart/form-data encoded file named "photo". When posting from Quill with a media endpoint, your Micropub server will instead receive a list of URLs represented as "photo[]=https://media.example.com/cc/...2cb.jpg". Presumably this pattern would hold for other media types such as video and audio, if you are using Micropub clients that support them.

This particular step has been an interesting challenge for my site, which is a static site generated by Jekyll. My previous Micropub file-handling implementation expected all uploaded assets to live on disk next to the post files, and updating my Jekyll theme and plugins to handle the change is a work in progress. I eventually plan to move all my uploads out of the source for my project in favor of storing them with Spano.

Feedback Welcome!

Spano is probably my second public Python project, so I'd love feedback! If you try it out and run into issues, please drop me a line on GitHub. Or you can find me in the #indieweb chat on freenode IRC.

I'd also like to thank Kyle Mahan for his Woodwind Flask server application, which inspired the structure of Spano.