Jonathan Prozzi and I have challenged one another to make a post about improving our websites once a week. This is me getting back on the train!

In a previous site update I wrote about setting up a system to notify me whenever my site received webmentions. Essentially, this meant that I could now get notifications on my phone and desktop whenever somebody interacted with my site, such as: replying to one of my posts on their own site, retweeting or favoriting one of my posts, or even RSVPs to my Facebook events.

One thing I didn't super like about this system is that it used the Pushbullet service which, while great, is not under my control.

I've been running a Matrix chat server at home for a while now. I primarily use it to chat with people in my household in IRC channels. I use a really nice client for Matrix called Riot, which runs in the browser, but is also available on Android and iOS, and is capable of sending notifications about chat events, which I have found really handy.

Recently, I've added a chatbot to my Matrix server named Hubot, thanks to the Hubot-Matrix adapter. Hubot is super neat because it is fairly easy to script up new behaviors, and it has nice built-in support for the web - both for making web requests, but Hubot also runs a server for accepting web requests. Once I realized this, it occurred to me that I could replace my previous notification system that uses Pushbullet with one that goes through Hubot.

First, a note on security. Exposing a chatbot's HTTP listener interface to the great wide internet comes at some risk! I made sure to the following:

• I run Hubot behind a firewall, so no plain HTTP traffic can come directly across the internet.
  
• Using another home server, I set up nginx to act as a secure HTTPS proxy, using a certificate from Let's Encrypt to encrypt all traffic that goes over the internet.
• I decided that any behaviors I write for Hubot that use the HTTP listener will use some kind of secret token to ensure that the request is valid. I don't want spammers blowing up my chatrooms!

I decided that the bot should:

• Allow a user to request webmention.io notifications for a given site into any room.
  
• Generate and store a "callback secret" to work with webmention.io's Web Hook system and tell the user the URL and callback secret to configure over on the Webmention.io Dashboard.
• Accept HTTP requests from webmention.io at something like <HUBOT_HOST>/hubot/wmio/notify
• Verify that the request contains the callback secret
• Generate a nice text summary of the notification based on its contents
• Send the notification to the room that the user was in when they made the follow request.

With that in mind, I began learning lots about testing Hubot scripts, refreshing myself on Coffeescript, and so on.

I am now happy to introduce this first (janky) release of my Hubot Script, hubot-webmentionio-notify!

Once installed, you can start a conversation with your hubot and ask it to follow a site:

  you> hubot wmio follow mycoolsite.biz
  

  hubot> @you OK! Use this as your Web Hook: <HUBOT_URL>/hubot/wmio/notify
And use this as your callback secret: 1a2b3c4d5e6f7890000

The string "mycoolsite.biz" can actually be anything and should be something easy to remember in case you want to unfollow notifications later. Hubot doesn't check incoming mentions against it at the moment.

You can enter the URL and callback secret in the Webmention.io dashboard, and future webmentions will be sent to your Hubot and output into the room of your choice.

I don't know how useful hubot-webmentionio-notify will be for other folks at the moment, but I am excited be getting these notifications via services that I control. I look forward to building more fun things with Hubot!

Jonathan Prozzi and I have challenged one another to make a post about improving our websites once a week. I'm late with this one!

Most of the features on my website are experiments in learning new things. Sometimes I learn a better way of doing something that I've already built into the site and it's time to migrate!

Moving Media files from Git LFS to a Media Endpoint

I build my site with Jekyll, and I store my site's configuration and text content via Git. One of the things that most folks avoid with Git is storing text content (which fits into Git's model of efficiently storing differences over time) with large binary files like images, etc. (which Git cannot manage as efficiently).

When I first set up my site, I made use of Git LFS ("Large File Storage") for managing anything that wasn't text. Any images, video, or audio that I added to my site was stored in an _assets/ folder in a way that matched uploaded files to the posts they were a part of. Git LFS would transparently ship those files off to a secondary server rather than include their content in the Git repository itself. I had to go through some hoops to set up my local GitLab server to support Git LFS and to set up Git LFS with the server that handles receiving new posts via Micropub, compiling and deploying the site.

It turns out that there are many reasons that a site would want to handle media files separately from the text content that refers to them. In fact, it is a common enough pattern that the Micropub standard includes a definition for a separate "media endpoint" to handle file uploads. I shared a Micropub media endpoint implementation that I built called Spano a while back, and it has been working well with support from tools like Quill. So the text content of my site is served from https://martymcgui.re/, and my media files from https://media.martymcgui.re/. With a couple of changes in my code and my workflows, this has become the way I handle all media files for my site.

However, I still had a bunch of files in site being handled by Git LFS, and some of my Jekyll code (plugins and templates) for showing embeds expected files to be on the local filesystem. This past week I took some time to write some scripts to find all references to those local files, migrate them to my media server, and update the outgoing links. I also updated my embed handling so it didn't rely on local files. This let me delete a lot of local metadata I was keeping but not using, like all the EXIF tags in uploaded photos. I am now Git LFS free and it feels like one less thing to worry about.

Better Caching for Mentions from Webmention.io

When I finally started displaying webmentions, I had a very simple model for how to cache all the info from webmention.io. Basically: I stored all mentions in a big array and, when my site went to fetch new mentions, it would keep fetching until it saw the "last" mention again. This led to a bit of a bug where someone might send me a mention, update their page, and send the mention again. My site would not be able to recognize the "last" mention, so it would fetch all my mentions again, leading to everything appearing twice.

This past week I rewrote my mention handling to avoid this problem by replacing this array and storing mentions in a hash based on the source and target. The new code also checks to see if the verification date of the mention has changed (giving me a way to detect and notify about changed mentions in the future). I also reorganized my mention cache to include an index by the target URL on my site. This makes it a bit quicker to find mentions for a given page when rendering out the site.

Neither of these changes are really visible to readers of my site, but they have been useful for cleaning things up. The webmention.io handling in particular has brought my plugin a lot closer to being something I could release for other people to use!

Jonathan Prozzi and I have challenged one another to make a post about improving our websites once a week. I'm a little late with this one!

I recently added support for displaying mentions, such as likes, reposts, comments, etc. from around the web that refer to the posts on my site. One thing the update didn't do is catch another type of mention, such as when someone mentions me in a tweet (example). These get fed to my website by brid.gy, but weren't displayed anywhere.

So, I created a /mentions page for displaying these mentions. In the future, when a post mentions my homepage, the result will show up on the mentions page.

My mentions still don't yet update in real time - they are compiled into my site whenever I make a new post. That's coming up in the future, but I have taken one more step towards real-time interactions with notifications!

Webmention.io, the service that I use for accepting and storing webmentions, has a WebHook option that can notify your site whenever a new webmention has been received. I wrote up a simple Python service using Flask that will listen for these messages from webmention.io and send them to me via PushBullet, a notification service that I've been using for a while for other projects.

Now, I'll see a notification on my phone and laptop when another site sends me a webmention!

Update: Here's an example notification!

Webmentions are one of the most interesting and powerful technologies floating around the IndieWeb. At their most basic, they sites on the web to interact by sending a notification when a page on one site links to a page on another. When combined with machine-readable metadata like microformats2, they enable really neat social interactions between sites, feeding back likes, comments, bookmarks, shares, event RSVPs, and plenty more.

Receiving Webmentions

A site doesn't have to do all its own Webmention handling, and there are a few services that will handle them for you. I set up my website with the Webmention.io service back in August 2016 (so long ago!) and it's been accepting mentions from other sites since then. And, while there aren't a lot of websites that send Webmentions natively, there are services like Bridgy which uses Webmentions to backfeed social interactions to my site from sites like Facebook and Twitter. Pretty neat!

Sending Webmentions

When I publish a post with a link to a site that support Webmentions, I still need to actually send that notification. I haven't yet built a tool that does that for my own website, but I have been able to make use of Aaron Parecki's Telegraph, which will take in a link to one of my posts and parse it for outgoing links, find out of the targets of those links support Webmentions, and allow me to send them with the press of a button. It's ridiculously easy to use and has the added benefit of letting me pick-and-choose which links go out as Webmentions.

Displaying Webmentions

Webmention.io has been collecting mentions for my site for something like 6 months, but they don't just magically show up on my site! Webmention.io provides an API for fetching the mention data for individual pages, or all mentions for my domain.

My site is built on Jekyll, a static site generator, and I like that so far it doesn't rely on JavaScript for folks to read it. I didn't want to require JavaScript for displaying mentions, so I needed a way to "bake in" my mentions for each post. I was inspired by Aaron Gustafson's jekyll-webmention_io, but found that I didn't like some of the choices in markup or the way that it stored the mention data, so I went ahead and wrote my own. It's still heavily a work-in-progress, but I do hope to release it for other folks to use once it's more stable.

What works? Let's see!

Here's an example post with some Likes and RSVPs (both "yes"es and "maybe"s):

And an example post with some replies backfed from Facebook:

All of these are being displayed with the data that Webmention.io provides with its API, and there are some types of post that I would like to handle differently such as the ❤️ above (which was a Facebook "heart" reaction), and I'd like to include a JavaScript enhancement that will show any new mentions, so they aren't sitting in "limbo" until I make a new post.

Overall, I'm really excited to finally be showing these on my site! I think Webmention is a pretty critical part of bringing the "social web" into the IndieWeb and back out of the silos. I am grateful to all the folks that have made this possible with their work on standards and tools!