Posts Tagged micropub

Mon Mar 12

Micropub for a static Neocities website

This post gives more technical detail for the recent talk that I gave at Bring-a-Hack NYC. In it, I describe a system that copies posts from Ghost Party's Instagram automatically to the Ghost Party Website at ghostparty.today.

This roughly works like so:

  • OwnYourGram periodically checks our Instagram for new images
  • For each new post, it makes a Micropub request to a service that I wrote for this purpose.
  • The endpoint accepts the image files and uploads them to our site using the Neocities API. Then it creates some new HTML from the micropub post data, edits the site's main index.html page to include the content, then uploads it back to Neocities.

Authorization with IndieAuth

One missing step in that rough outline is the bit where we tell OwnYourGram *how* to post to our site via Micropub, as well as making sure that OwnYourGram can prove that we have given it permission to post there. Many Micropub clients support IndieAuth as a way to do this permission-granting step (known as authorization). With a typical personal website, I'd delegate this process to indieauth.com, which lets you offload the step of "proving" that you own the website where you're trying to post by instead letting you log into an existing social network account.

ghostparty.today is a shared website, so I thought a simpler way to approach this would be to create a custom authorization endpoint that accepts a password. I could give this password to other GHOST PARTY folks and they can post whatever they like.

Some nice folks in the IndieWeb community had already created an authorization endpoint that works this way, called selfauth. I created a pretty simplistic port of selfauth from PHP to JavaScript using NodeJS on the awesome Glitch social code hosting platform. It needs some more robust error checking, and some micropub clients seem to have trouble with it, but it works with OwnYourGram and Quill, so it's Good Enough for now!

Interface displaying text 'You are attempting to login as https://ghostparty.today/ with client https://quill.p3k.io/' along with a password prompt.
Enter the password to get authorized!

If you're interested, you can find the source for this endpoint here: https://glitch.com/edit/#!/befitting-price

Micropub Endpoint

Finally, there's the Micropub endpoint itself. I based mine off of this Glitch micropub project, created by Adam Bachman at IndieWebCamp Baltimore this year. Specfically, I took the bits that verify the access token. From there, it was a matter of stumbling through various bits of handling a Micropub post, pulling in extra libraries as I needed them. I slowly added support for saving uploaded files to our Neocities site via the Neocities API, then for generating an HTML snippet for the new post, and finally for adding the post to the site's main page.

The process for adding the generated post content to the site itself is extremely retro. The Micropub endpoint actually downloads the HTML from https://ghostparty.today/index.html. Inside there is a comment that looks like:

<!-- NEW POSTS BELOW -->

So the new contents for index.html is everything before the comment, the comment, the new content, and then everything after the comment. Once that's constructed, the endpoint uploads index.html back to Neocities with the new contents.

You can find the source for the Micropub endpoint here: https://glitch.com/edit/#!/prism-dirt

It should be possible to fork the project on Glitch and configure it to work with your own Neocities site! You'll need to use cURL with the Neocities API to generate an authorization token to configure the app, rather than saving username and password info in the project!

Making ghostparty.today "IndieWeb-Readable"

The GHOST PARTY site follows an extremely old-school publishing model where each post only exists as a small section of HTML on a single long page. This complicates things a bit because interacting with posts on the IndieWeb typically requires three things:

  • A machine-readable "identity" h-card that says "who" this website is, including a logo, the site name, canonical URL.
  • A machine-readable feed that contains new posts, allowing people to follow our updates in their favorite indie reader.
  • A separate permalink URL for each post, containing machine-readable info about that post, allowing other IndieWeb sites to "understand" the content when they want to RSVP to events, repost or like photos, and more.

Microformats2 parsers like X-Ray do a great job at parsing HTML page into meaningful data, but the programs that use the parsed data usually stop at the first thing they find. So, if I ask X-Ray to parse https://ghostparty.today/, this is what it sees:

{
  "data":{
    "type":"card",
    "name":"GHOST PARTY",
    "url":"https://ghostparty.today/",
    "photo":"https://ghostparty.neocities.org/ghost-party-logo.png"
  },
  "url":"https://ghostparty.today/",
  "code":200
}

This is great in that it contains the "identity" information that we want the page to have. However, that page also has a feed! How can I get at that? By telling X-Ray to only look at a fragment of the page! Here's how X-Ray sees https://ghostparty.today/#feed

{
  "data":{
    "type":"feed",
    "items":[
      {
        "type":"entry",
        "published":"2018-03-07T16:44:16+00:00",
        "url":"https://ghostparty.today/#2018-03-07-181020",
        "photo":["https://ghostparty.today/uploads/2018-03-07/181020-ig0LN1JG.jpg"],
        "content":{"text":"snow day selfie #snowday #snowfie #ghoststories"} 
      },
      {
        "type":"entry",
        ...
      },
      ...
    ],
    "url":"https://ghostparty.today/#feed",
    "code":200,
    "info":{"found_fragment":true}
  }
}

Data! Nice! So if we want to follow ghostparty.today with an indie reader, we use https://ghostparty.today/#feed as the URL.

Looking at the content of the feed, you'll notice that the individual items have a url property which also has a fragment ID in the URL. This allows IndieWeb sites to interact with a particular post, even though it's one of many on the same page. For example, I was able to create an RSVP post to the most recent Ghost Party show, and (by looking just at the content in the fragment at https://ghostparty.today/#show-2018-02-14) my site was able to pick up the machine-readable details about the event to display them on my own site.

The microformats markup for the h-card and h-feed are built into the index.html file itself and don't change very often. However, for each new micropub post we need to convert those properties into a chunk of HTML for the index.html page that also contains the microformats h-entry markup. To do this, I made this Handlebars template:

{% raw %}<div class="h-entry" id="{{ mp-slug }}">
  <a class="u-author" href="/"></a>
  <a class="u-url" href="#{{ mp-slug }}">{{#if name}}<h2 class="p-name">{{ name }}</h2>{{/if}}</a>
  <time class="dt-published" datetime="{{ published }}"></time>
  {{#if video}}
    {{#each video}}
      <video controls loop class="u-video" src="{{ . }}"></video>
    {{/each}}
  {{ else }}
    {{#each photo}}
      <img class="u-photo" src="{{ . }}" />
    {{/each}}
    {{#each audio}}
      <audio class="u-audio" src="{{ . }}"></audio>
    {{/each}}
  {{/if}}
  <div class="e-content">{{ content }}</div>
</div>
<img src="ghosts.gif" />
{% endraw %}

Not so bad!

Resources

Here's a collection of some of the other libraries and docs I used to pull this thing together:

  • Neocities API
    • Includes details on using cURL to get an authorization token.
    • There’s a Node module for the Neocities API, but it uses username and passwords, so I just used regular HTTP requests to handle talking to Neocities.
  • Node Request HTTP Client
    • Verifies the authorization token by querying the token endpoint (I used tokens.indieauth.com for this)
    • Handles uploading files like images to Neocities via their API.
    • Used to fetch index.html from the site and later to writes it back via Neocities API.
  • ES6 Promises
    • There's a lot of asynchronous code that waits for network requests. This code is very Promise-heavy to handle that.
  • Handlebars templating
    • Made it easy to take the Micropub properties for a new post and rendered them into HTML content with microformats.
  • Express 4 API
    • Glitch uses Node and Express 4 by default, and I got to learn lots about it!
  • Multer Express Middleware
    • For handling uploaded files
  • Body Parser Express Middleware
    • For handling regular Micropub properties
  • Validator.js
    • Half-using this for the authorization endpoint, need to clean it up and finish it up!
  • OwnYourGram API docs
    • Shows what properties are sent over Micropub, so you know exactly the kind of data your Micropub endpoint will need to handle.

A Ghost (Party) is Haunting the Silo'd Web

This post is a rough summary of a lightning talk that I gave on February 28th at the Bring-A-Hack NYC with Hackaday + Ultimaker + New Lab. Many thanks to Shayna Gentiluomo for organizing and sometimes co-conspirator Matt Griffin for nudging me to speak!


I have a lot of distrust for social silos like Facebook and Twitter which draw us in with the promise of connecting with family, friends, and interesting people, but I sometimes have trouble articulating that in a succinct way. Thankfully, I saw a perfect example this morning:

Posting on the @facebook website is not working in Mac OS today. JavaScript eats CSS & HTML, making data entry field disappear except for a spookily disembodied close-box X. Been this way a few hours. — @zeldman, 9:53 AM - 28 Feb 2018

Some Facebook team pushed an (untested?) update which broke the website, preventing some users from posting. "Move fast and break things". These silos don't value us or our content. So, we should get out. But what is the alternative?

The IndieWeb helps us take our content back, by posting to our own websites on our own domains. With some work, we can even get the benefits of these social networks and take our conversations with us. There’s a whole lot of great stuff on the IndieWeb website, so go spend a couple of days reading it. But I’m not here to talk about all those good things today. I’m here to talk about how I used them to do a dumb hack.

ghostparty.today is a website for Ghost Party – an improv project that has a ... certain aesthetic. Specifically the website was pitched as “exactly like the Heaven’s Gate website” but we compromised on “extreme Geocities”.

Ugly large text says 'Welcome to GHOST PARTY'
We have an aesthetic.

My Ghost Party co-host (co-ghost) built the first version of the site on Neocities.org, a lovely hosting provider that recaptures the feel of editing raw HTML directly in your browser. However, hand-editing HTML gets exhausting and sometimes you just want to post things to Instagram (and, of course,  Ghost Party has an Instagram).

And Instagram is fine as a marketing platform! Lots of people look at it and like the pictures. But what about our website? Our aesthetic?

Micropub is a standard (and now W3C Recommendation) designed over some years of practice from the IndieWeb community that defines a way for apps and clients to easily make new posts to any website (even yours) if it supports Micropub.

I used Glitch, an amazing community and JavaScript-server-app hosting service, to make a simple micropub service for ghostparty.today.

Screenshot of code in the Glitch UI

It accepts posts via micropub, including file uploads, and uses the Neocities API (hey, it’s not the 90s! Neocities has an API!) to upload the files to our website. Then, it fetches our site’s index.html, adds the content of the new post, and uploads it back. You can find the source here on Glitch and fork it for your own purposes!

Now that our site supports micropub, we can make use of a wonderful service called OwnYourGram. Once we give OwnYourGram permission to post to our site and read our Instagram feed, it will poll our Instagram account and copy each new post back to our website, where we can display it however we want.

Super simplified Instagram photo on Ghost Party website
Aesthetic

We can go a step further, using an IndieWeb building block called Webmention and a service called brid.gy, to bring all of those responses, those Instagram and Facebook likes and comments, Twitter retweets, and more, back to our site for display.

And to show how it works, here’s a micropub client call Quill that works for any micropub-enabled site. Let’s make a quick post, now!

Posting interface with 'Hello from New Lab' typed into note field
Quill's note interface

Once we hit the "Post" button and refresh the homepage, we can see the new post on the site!

'Hello from New Lab' text with ghost separator image
Aesthetic

This was the end of time time. I would have like to further encourage folks to check out Homebrew Website Clubs in their own communities, or to start one if they don't exist. They're part show-and-tell, part support group, and part mad science lab for people who love the web!

Mon May 8

Site-Updates: Easier POSSE with Micropub Edits!

Jonathan Prozzi and I have challenged one another to make a post about improving our websites once a week. This one should have gone up last week!

A few weeks ago I posted some thoughts about my IndieWeb setup called "Easier POSSE with Micropub Edits?" in which I wished for a tool that would let me take a given post from my site, syndicate it to silos like Twitter and Facebook (tweaking the content if I want), and updating the post on my site to show the links to those syndicated copies.

Why?

I failed to make at least one important thing clear in my original post – why do I care about syndication links? There are many reasons.

If I decide that a post should be syndicated to a silo, it's because I want it to reach the people who follow me there and, if that is true, I also want their interactions to come back to my site. So, in these ways, a post isn't "done" unless it is on my site, with syndicated copies on the silos I care about, and with syndication links for brid.gy to feed the interactions back.

Starting at the End

I decided to start by making my site's Micropub server support Micropub Source Content Queries and Micropub Updates. Any tool that helped automate syndication would need this plumbing to operate.

When implementing a new feature, it always helps to have something to test against. So, I went looking for a Micropub client which supported queries and edits. The test suite for Micropub at micropub.rocks includes a lovely implementation report grid, showing which Micropub clients support what features of the spec.

Of the clients listed, two of them were web-based and Open Source. I had played with and liked Inkstone in the past, but its edit features are currently considered a work-in-progress. So, I tried out Micropublish.net, and it was exactly what I was looking for.

Micropublish has a feature to let you enter a URL for a post on your site to edit. It will use Micropub source content queries to get the source data for that post and let you edit the content and other properties of the post. It can then send a Micropub update to save the updated version of the post back to your site, if your server supports updates. It even has a great feature for developers - a "Preview" button will show you exactly what request will be sent to your server for the update.

Screenshot of micropublish.net preview for an update to add a syndication link to a post

Micropublish.net is a great tool for testing out Micropub query and update support, but my Micropub server is bespoke, hastily-written, hand-rolled Python. So, while it was easy enough to add query support, it took me a while to get my code structure cleaned up, write some tests, and actually implement updates.

A New Workflow

I am pleased to say that it works and, with the help of Micropublish.net, I now have a functioning workflow for publishing to my site and syndicating to silos like Twitter and Facebook, even from my phone, without having to open my laptop, edit YAML data, and push git repositories around. It looks like this.

  • Make a new post to my site with a micropub client like Quill.
  • Open the post for editing in micropublish.net (I use Url Forwarder for Android to make this super easy on my phone, a bookmarklet makes it easy on my laptop).
  • In a new tab, log in to Twitter and make a similar post, copy the URL to the new tweet into the Syndication field on my post.
  • Repeat the steps to make posts on Facebook, Mastodon, etc., copying their URLs into the Syndication field.
  • Finally, hit "Update" in micropublish.net to update my post with the syndication links.
Screenshot of micropublish.net with new syndication links

This is still a very manual process, but it now makes it possible to finish a post in a way that I couldn't before. In the spirit of manual until it hurts, I will use this for a while and see what existing pain points remain, and what new ones appear, to help decide what comes next.

Thanks to Barry Frost for micropublish.net and to Tantek for the nudge to write an update!

Thu Apr 13

Site updates: simplifying media, complicating mentions

Jonathan Prozzi and I have challenged one another to make a post about improving our websites once a week. I'm late with this one!

Most of the features on my website are experiments in learning new things. Sometimes I learn a better way of doing something that I've already built into the site and it's time to migrate!

Moving Media files from Git LFS to a Media Endpoint

I build my site with Jekyll, and I store my site's configuration and text content via Git. One of the things that most folks avoid with Git is storing text content (which fits into Git's model of efficiently storing differences over time) with large binary files like images, etc. (which Git cannot manage as efficiently).

When I first set up my site, I made use of Git LFS ("Large File Storage") for managing anything that wasn't text. Any images, video, or audio that I added to my site was stored in an _assets/ folder in a way that matched uploaded files to the posts they were a part of. Git LFS would transparently ship those files off to a secondary server rather than include their content in the Git repository itself. I had to go through some hoops to set up my local GitLab server to support Git LFS and to set up Git LFS with the server that handles receiving new posts via Micropub, compiling and deploying the site.

It turns out that there are many reasons that a site would want to handle media files separately from the text content that refers to them. In fact, it is a common enough pattern that the Micropub standard includes a definition for a separate "media endpoint" to handle file uploads. I shared a Micropub media endpoint implementation that I built called Spano a while back, and it has been working well with support from tools like Quill. So the text content of my site is served from https://martymcgui.re/, and my media files from https://media.martymcgui.re/. With a couple of changes in my code and my workflows, this has become the way I handle all media files for my site.

However, I still had a bunch of files in site being handled by Git LFS, and some of my Jekyll code (plugins and templates) for showing embeds expected files to be on the local filesystem. This past week I took some time to write some scripts to find all references to those local files, migrate them to my media server, and update the outgoing links. I also updated my embed handling so it didn't rely on local files. This let me delete a lot of local metadata I was keeping but not using, like all the EXIF tags in uploaded photos. I am now Git LFS free and it feels like one less thing to worry about.

Better Caching for Mentions from Webmention.io

When I finally started displaying webmentions, I had a very simple model for how to cache all the info from webmention.io. Basically: I stored all mentions in a big array and, when my site went to fetch new mentions, it would keep fetching until it saw the "last" mention again. This led to a bit of a bug where someone might send me a mention, update their page, and send the mention again. My site would not be able to recognize the "last" mention, so it would fetch all my mentions again, leading to everything appearing twice.

This past week I rewrote my mention handling to avoid this problem by replacing this array and storing mentions in a hash based on the source and target. The new code also checks to see if the verification date of the mention has changed (giving me a way to detect and notify about changed mentions in the future). I also reorganized my mention cache to include an index by the target URL on my site. This makes it a bit quicker to find mentions for a given page when rendering out the site.

Neither of these changes are really visible to readers of my site, but they have been useful for cleaning things up. The webmention.io handling in particular has brought my plugin a lot closer to being something I could release for other people to use!

Fri Mar 17

Earlier this week I announced Screech, a simple client for posting audio content (such as podcasts) to your own site using Micropub.

I’m pleased to announce that Screech is now available publicly. You can check it out at https://screech.schmarty.net/.

Screech also now has an adorable logo, thanks to a couple of pieces of public domain art.

Tue Mar 14

Screech - a micropub client for podcasting

I've been working on my idea of what IndieWeb podcast publishing looks like for some months now, both with the improvised We Have to Ask comedy podcast that I make with Jonathan Monroe and more recently with the This Week in the IndieWeb Audio Edition that I started producing last month.

In addition to following standard IndieWeb-friendly practices like using microformats2 feeds, backfeeding social interactions from Twitter and Facebook with bridgy, and exploring other interesting audio markup tricks, I wanted a tool that made it easy for me to publish new content to my sites via the Micropub protocol, which supports sending audio media files.

I didn't see another micropub client in the wild that supported audio files in the way that I wanted, so I made my own.

Screech is an audio-publishing-focused micropub web client with a Python server component built on Flask. Screech supports logging in with your own site using indieauth and posts to your site's micropub endpoint.

It's still a work-in-progress, but the basic flows work well enough for my needs.

Screenshot of main Screech posting interface with form fields.
Screech interface before posting an episode of This Week in the IndieWeb Audio Edition

One fun feature of Screech is that once you select an MP3 file for upload, it uses the jsmediatags library to pull out information about the track, such as its duration, track title, album and artist info, etc. This info is outside the scope of the Micropub standard, but if you want to add support to your server, you'll see those properties arrive with names like "id3-title", "id3-artist", etc.

Edit, March 17, 2017 — Screech is available at screech.schmarty.net. If you'd like to run it yourself, add features, or fix bugs, you can find the source code and instructions on GitHub.

There are many TODOs yet on my plate for Screech before I'd consider it "done", such as micropub media endpoint support, syndication support, adding a photo to the post as a "poster" image, and more.

I'd love to hear feedback from the IndieWeb community! What do you think it means to be an "IndieWeb podcaster"? What features would make Screech useful for you?

Thu Jan 26

Spano - a minimum-viable Micropub Media Endpoint

Micropub is an open API standard to create posts on one's own domain using third-party clients  and currently a W3C Candidate Recommendation. One of the (semi-) recent additions is the idea of a Micropub Media Endpoint. The Media Endpoint provides a way for Micropub clients to upload media files to a Micropub service, receiving a URL that is sent along in place of the file contents when the post is published.

Some of the things I like about Micropub media endpoints include:

  • The spec allows the media endpoint to be on a completely separate domain from the "full" micropub endpoint.
  • The spec doesn't specify anything about how the files are stored or their final URLs or filenames.
  • They make it easy to separate the handling of (large) media files from the (presumably much smaller) content and metadata of a post.
  • They enable Micropub clients to upload multiple files without creating multiple posts. This makes it simpler to create posts that contain multiple images, like a gallery.

Personally, I wanted a Micropub media endpoint server with a few extra properties:

  • It should be able to run completely separately from, and therefore work in conjunction with, any other micropub server implementation.
  • It should not store duplicate files. If the same file is uploaded twice, the same URL should be returned both times.
  • It should not allow overwriting files. If two images of the same name are uploaded, both are kept and receive different URLs.

Enter HashFS

My extra features above essentially describe a content-addressable storage storage system. CAS is a way of storing and accessing data based on some property of the actual content, rather than (potentially arbitrary) files and folders.

HashFS is a Python implementation of a content-addressable file management system. You give it files, it will put them in a directory structure based on a cryptographic hash function of the contents of that file. In other words - HashFS can take any file and give back a unique path to that file which will never change (if you later upload a new version of the file, it gets a different path).

To add the the fun of HashFS, there is a Flask extension called Flask-HashFS which makes it easy to expose a HashFS file store on the web via the Python Flask framework.

Introducing Spano

Spano is a Micropub Media Endpoint server written in Python via the Flask framework which combines Flask-HashFS for file storage with Flask-IndieAuth (introduced earlier) to handle authentication and authorization.

Spano is a server-side web app that basically does one thing: it accepts HTTP POST requests with a valid IndieAuth token and a file named "file", stores that file, and returns a URL to that file. The task of serving uploaded files is left to a dedicated web server like nginx or Apache.

Using Spano

Once Spano has been set up and configured for your domain, uploading is a matter of getting a valid IndieAuth token. IndieAuth-enabled Micropub clients will do this automatically. For testing by hand I like to log in to Quill and copy the access token from the Quill settings page. With token in hand, uploads are as easy as:

curl -D - -F "file=@myfile.jpg" \
  -H"Authorization: Bearer xxxx..." \
  https://media.example.com/micropub/

Which should output a response like:

HTTP/1.1 100 Continue

HTTP/1.0 201 CREATED
Content-Type: text/html; charset=utf-8
Content-Length: 108
Location: https://media.example.com/cc/a5/97/7c/2004..2cb.jpg
Server: Werkzeug/0.11.4 Python/2.7.11
Date: Thu, 26 Jan 2017 02:40:05 GMT

File created: https://media.example.com/cc/a5/97/7c/2004..2cb.jpg

Integrating Spano with your Micropub Endpoint

If you want Micropub clients to use Spano as your Media Endpoint, you need to advertise it. This is handled by your "main" Micropub server using discovery. Essentially, a client will make a configuration request to your server like so:

https://example.com/micropub?q=config

And your server's response should be a JSON-formatted object specifying the "media-endpoint". A bare minimum example:

{
  "media-endpoint": "https://media.example.com/micropub/"
}

In addition to advertising the media-endpoint, your Micropub server must be able to handle lists of URLs in places where it would normally expect a file.

For example, when posting a photo from Quill without a media endpoint, your Micropub server will receive a multipart/form-data encoded file named "photo". When posting from Quill with a media endpoint, your Micropub server will instead receive a list of URLs represented as "photo[]=https://media.example.com/cc/...2cb.jpg". Presumably this pattern would hold for other media types such as video and audio, if you are using Micropub clients that support them.

This particular step has been an interesting challenge for my site, which is a static site generated by Jekyll. My previous Micropub file-handling implementation expected all uploaded assets to live on disk next to the post files, and updating my Jekyll theme and plugins to handle the change is a work in progress. I eventually plan to move all my uploads out of the source for my project in favor of storing them with Spano.

Feedback Welcome!

Spano is probably my second public Python project, so I'd love feedback! If you try it out and run into issues, please drop me a line on GitHub. Or you can find me in the #indieweb chat on freenode IRC.

I'd also like to thank Kyle Mahan for his Woodwind Flask server application, which inspired the structure of Spano.

Thu Oct 20

Excited that I can now more easily post events to my own website from quill.p3k.io. #indieweb #micropub