Spano - a minimum-viable Micropub Media Endpoint

Micropub is an open API standard to create posts on one's own domain using third-party clients  and currently a W3C Candidate Recommendation. One of the (semi-) recent additions is the idea of a Micropub Media Endpoint. The Media Endpoint provides a way for Micropub clients to upload media files to a Micropub service, receiving a URL that is sent along in place of the file contents when the post is published.

Some of the things I like about Micropub media endpoints include:

• The spec allows the media endpoint to be on a completely separate domain from the "full" micropub endpoint.
• The spec doesn't specify anything about how the files are stored or their final URLs or filenames.
  
• They make it easy to separate the handling of (large) media files from the (presumably much smaller) content and metadata of a post.
• They enable Micropub clients to upload multiple files without creating multiple posts. This makes it simpler to create posts that contain multiple images, like a gallery.

Personally, I wanted a Micropub media endpoint server with a few extra properties:

• It should be able to run completely separately from, and therefore work in conjunction with, any other micropub server implementation.
  
• It should not store duplicate files. If the same file is uploaded twice, the same URL should be returned both times.
• It should not allow overwriting files. If two images of the same name are uploaded, both are kept and receive different URLs.

Enter HashFS

My extra features above essentially describe a content-addressable storage storage system. CAS is a way of storing and accessing data based on some property of the actual content, rather than (potentially arbitrary) files and folders.

HashFS is a Python implementation of a content-addressable file management system. You give it files, it will put them in a directory structure based on a cryptographic hash function of the contents of that file. In other words - HashFS can take any file and give back a unique path to that file which will never change (if you later upload a new version of the file, it gets a different path).

To add the the fun of HashFS, there is a Flask extension called Flask-HashFS which makes it easy to expose a HashFS file store on the web via the Python Flask framework.

Introducing Spano

Spano is a Micropub Media Endpoint server written in Python via the Flask framework which combines Flask-HashFS for file storage with Flask-IndieAuth (introduced earlier) to handle authentication and authorization.

Spano is a server-side web app that basically does one thing: it accepts HTTP POST requests with a valid IndieAuth token and a file named "file", stores that file, and returns a URL to that file. The task of serving uploaded files is left to a dedicated web server like nginx or Apache.

Using Spano

Once Spano has been set up and configured for your domain, uploading is a matter of getting a valid IndieAuth token. IndieAuth-enabled Micropub clients will do this automatically. For testing by hand I like to log in to Quill and copy the access token from the Quill settings page. With token in hand, uploads are as easy as:

curl -D - -F "file=@myfile.jpg" \
  -H"Authorization: Bearer xxxx..." \
  https://media.example.com/micropub/

Which should output a response like:

HTTP/1.1 100 Continue

HTTP/1.0 201 CREATED
Content-Type: text/html; charset=utf-8
Content-Length: 108
Location: https://media.example.com/cc/a5/97/7c/2004..2cb.jpg
Server: Werkzeug/0.11.4 Python/2.7.11
Date: Thu, 26 Jan 2017 02:40:05 GMT

File created: https://media.example.com/cc/a5/97/7c/2004..2cb.jpg

Integrating Spano with your Micropub Endpoint

If you want Micropub clients to use Spano as your Media Endpoint, you need to advertise it. This is handled by your "main" Micropub server using discovery. Essentially, a client will make a configuration request to your server like so:

https://example.com/micropub?q=config

And your server's response should be a JSON-formatted object specifying the "media-endpoint". A bare minimum example:

{
  "media-endpoint": "https://media.example.com/micropub/"
}

In addition to advertising the media-endpoint, your Micropub server must be able to handle lists of URLs in places where it would normally expect a file.

For example, when posting a photo from Quill without a media endpoint, your Micropub server will receive a multipart/form-data encoded file named "photo". When posting from Quill with a media endpoint, your Micropub server will instead receive a list of URLs represented as "photo[]=https://media.example.com/cc/...2cb.jpg". Presumably this pattern would hold for other media types such as video and audio, if you are using Micropub clients that support them.

This particular step has been an interesting challenge for my site, which is a static site generated by Jekyll. My previous Micropub file-handling implementation expected all uploaded assets to live on disk next to the post files, and updating my Jekyll theme and plugins to handle the change is a work in progress. I eventually plan to move all my uploads out of the source for my project in favor of storing them with Spano.

Feedback Welcome!

Spano is probably my second public Python project, so I'd love feedback! If you try it out and run into issues, please drop me a line on GitHub. Or you can find me in the #indieweb chat on freenode IRC.

I'd also like to thank Kyle Mahan for his Woodwind Flask server application, which inspired the structure of Spano.