Recent Posts

Thu May 23
β˜… Liked https://www.jvt.me/posts/2019/05/22/ical-events-hugo/
post from Adding iCalendar Feeds for Events in Hugo
Using Hugo's custom output formats to automagically create an iCalendar feed for events.
Tue May 21
πŸ”– Bookmarked https://www.newyorker.com/tech/annals-of-technology/can-indie-social-media-save-us
Sun May 19
β˜… Liked https://werd.io/2019/pro-choice-is-pro-freedom
post from Pro-choice is pro-freedom
A woman has the right to choose what to do with her own body. This should be obvious to anyone who cares about freedom.It's ironic to me (but not surprising) that the people who are most in favor of subjugating women are also the people who advocate most strongly for what they …
Thu May 16
β˜… Liked https://www.manton.org/2019/05/16/following-other-blogs.html
post from Following other blogs on Micro.blog
After launching support for Mastodon on Micro.blog, I blogged about how Micro.blog is evolving to support 3 types of usernames: normal Micro.blog users, Mastodon users, and IndieWeb-friendly domain names. This last type of username is where I think we can bring more social …
β˜… Liked https://dribbble.com/shots/5954454-Burrowing-owl
Mon May 13
β˜… Liked http://brendandawes.com/blog/permission-to-write-anything
Fri May 10
β˜‘ RSVP'd to an event https://indieweb.org/events/2019-05-11-homebrew-website-club-nyc
post πŸ—½ Homebrew Website Club NYC
Join us for an afternoon of IndieWeb personal site demos and discussions!
I'm going!

Let’s try a Saturday afternoon!

From 1-3pm, join us for an IndieWeb Meetup at Think Coffee on 8th Av at 14th St in Manhattan!

Come work on your personal website, whether it exists yet or not!

This post from Calum finally brought out the FOMO I had been suppressing for IndieWebCamp Berlin.

Really looking forward to the 2019 IndieWeb Summit June 29-30th in Portland!

https://calumryan.com/blog/indiewebcamp-berlin-2019/

post from
IndieWebCamp was back in Berlin again this month for a weekend of talks, discussion and making, along with a meeting for IndieWeb organisers the day before.

Archiving rooms from a Matrix.org Homeserver (including end-to-end encrypted rooms)

I'm in the middle of a Forever Project, migrating stuff and services off of an old server in my closet at home onto a new (smaller, better, faster!) server in my closet at home.

One such service is a Matrix.org Synapse homeserver that was used as a private Slack-alternative chat for my household, as well as a bridge to some IRC channels. I set it up by hand in haste some years ago and made some not-super-sustainable choices about it, including leaving the database in SQLite (2.2GB and feelin' fine), not documenting my DNS and port-forwarding setup very well, and a few other "oopsies".

I had been keeping the code up to date via "pip install" and the latest "master" tarballs, but when the announcement came about needing valid TLS for federation starting in 0.99.X, I wasn't sure if I was good to upgrade. (I later found out that I was okay, ha!)

I found some docs on the most recent ways to set up Matrix on a new server, and even on how to migrate from SQLite to PostgreSQL. However, I don't know if I'll be able to set aside the time to do it all at once, or if it'll be easier just to set it up fresh, or even if I need a homeserver right now. So, I decided to figure out how to make archives of the rooms I cared about, which included household conversations, recipes, and photos from around the house and on travels.

Overview

The process turned out to be pretty involved, which is why it gets a blog post! It boils down to needing these three things:

  • osteele/matrix-archive - Export a Matrix room message archive and photos.
  • matrix-org/pantalaimon - A proxy to handle end-to-end encrypted (E2EE) room content for matrix-archive
  • matrix-org/Olm - C library to handle the actual E2EE processing. Pantalaimon relies on this library and it's Python extensions.

Getting all the tools built required a pretty recent system, which my old server ain't. I ended up building and running them on my personal laptop, running Ubuntu 19.04.

Since both matrix-archive and pantalaimon are Python-based, I created a Python 3.7 virtualenv to keep everything in, rather than installing everything system-wide.

Olm

The Olm docs recommend building with CMake, but as someone unfamiliar with CMake I could get it to build and run tests, but could not actually get it installed on my system.

I ended up installing the main lib with:

  make && sudo make install

The Python extensions were a challenge and I am not sure that I remember all the details to properly document them here. I spent a good amount of time trying to follow the Olm instructions to get them installed into my Python virtualenv.

In the end, the pantalaimon install built its own version of the Python Olm extensions, so I'm going to guess this was enough for now.

Pantalaimon

The pantalaimon README was pretty straightforward, once I installed Olm system-wide. I activated my virtualenv and ran:

  python setup.py install

That resulted in a "pantalaimon" script installed in my virtualenv's bin dir, so I could (in theory) run it on the command line, pointing it at my running Synapse server:

  pantalaimon https://matrix.example.com:8448

That started a service on http://127.0.0.1:8009/ which matrix-archive would connect over, with pantalaimon handling all the E2EE decryption transparently.

matrix-archive

The matrix-archive setup instructions suggest using a dependency manager called "Pipenv" that I was not familiar with. I installed it in my virtualenv, then ran it to setup and install matrix-archive:

  pip install pipenv
  pipenv install

Pipenv "noticed" it was running in a virtualenv, and said so. This didn't seem to be much of a problem, but any command I tried to run with "pipenv run" would fail. I worked around this by looking in the "Pipfile" to see what commands were actually being run, and it turns out it was just calling specific Python scripts in the matrix-archive directory. So, I resolved to run those by hand.

MongoDB

matrix-archive requires MongoDB. I don't use it for anything else, so I had to "sudo apt install mongodb-server".

Running the Import

First, I set the environment variables needed by matrix-archive:

  export MATRIX_USER=<my username>
  export MATRIX_PASSWORD=<my password>
  export MATRIX_HOST=http://127.0.0.1:8009

Then confirmed it was working by getting a list of rooms with IDs:

  python list_rooms.py

I set up the list of room IDs in an environment variable:

  export MATRIX_ROOM_IDS=!room@server,!room2@server,...

And slurped in all the messages with:

  python import_messages.py

At the end, it said it had a bunch of messages. Hooray!

Running the Export

This is where things kind of ran off the rails. In trying to export messages I kept seeing Python KeyErrors about a missing 'info' key. It seems like maybe the Matrix protocol was updated to make this an optional key, but the upshot was that matrix-archive seemed to assume that every message with an image attached would have an 'info' with info about a thumbnail for that image.

Additionally, the script to download images had some naive handling for turning attachment URLs like "mxc://example.com/..." into downloadable URLs. Matrix supports DNS-based delegation, so you can say "the Matrix server for example.com is matrix.example.com:8448, and this script didn't handle that.

I did some nasty hacks to only get full-sized images, and from the right host:

  • updated the schema to return the full image URL instead of digging in for a thumbnail
  • added handling to export_messages.py to handle missing 'info', which was used to guess image mimetypes
  • added some hardcoding to map converted "mxc://" URLs to the right host.

Afterwards I was able to do an export of alllllll the images to a "images/" folder:

  python download_images.py --no-thumbnails

And could then export a particular room's history with:

  python export_messages.py --room-id ROOM-NAME --local-images --filename ROOM-NAME.html

Note that the "--room-id" flag above actually wants the human-readable room name, unless it's actually a room on the main matrix.org server.

Afterwards, I could open room-name.html in my browser, and see the very important messages and images I worked so hard to archive.

a message exchange. marty asks maktrobot to 'pug me'. maktrobot responds with an image of a pug.
Screenshot from the (very minimal) HTML export, including an image (of a pug, sourced by a chat bot).

What's Next?

For now, I'll be putting these files and images in a safe backup and not worrying about them too much, because I have them. I've already stopped my old Synapse server, and can tackle setting up the new one at my leisure. We've moved our house chats to Signal, and I've moved my IRC usage over to bridged Slack channels.

Running a Matrix Synapse homeserver for the past couple of years has been quite interesting! I really appreciate the hard working community (especially in light of their recent infrastructure troubles), and I recognize that it's a ton of work to build a federating network of real-time, private communication. I enjoyed the freedom of having my own chat service to run bots, share images, and discuss private moments without worrying about who might be reading the messages now or down the road.

That said, there are still some major usability kinks to work out. The end-to-end encryption rollout across homeservers and clients hasn't been the smoothest, and it can be an issue juggling E2EE keys across devices. I look forward to seeing how the community addresses issues like these in the future!

TL;DR - saving an archive of a room's history and files should not be this hard.

Mon May 6

πŸ—“οΈ True Colors - A Diverse Variety Show

Magnet Theater 254 W 29th St. New York, NY 10001
πŸ“† Add to Calendar: iCal | Google Calendar

True Colors is a diverse variety show that celebrates the Magnet community! It will showcase improv, character pieces, stand up, and storytelling and be performed by teachers, students, and performers in the community. Come celebrate the comedy of our community with us!

Sun May 5

πŸ—“οΈ UCB Improv 201 Showcase!

UCB Theatre, NY 555 w 42nd, New York, NY 10036
πŸ“† Add to Calendar: iCal | Google Calendar

We’ve been deep in the lab the past 8 weeks and are excited to share our findings with you!

Come on out for a super-fun improv grad show!

Sat May 4

Cheers! 🍻

πŸ“ Checked in at Kopitiam, New York, NY.
Thu May 2
πŸ”– Bookmarked Component frameworks and web standards https://hiddedevries.nl/en/blog/2019-02-28-component-frameworks-and-web-standards

“There are lots of organisations out there using Angular 1 in projects, and I see them struggle to find developers willing to do the work. They’ve since moved on to newer frameworks. The orgs are stuck with outdated paradigms that are incompatible with the now.”

Sometimes you boop the snoot, and sometimes the snoot boops you.

πŸ‘‰πŸˆ

https://autoboop.com/

Tue Apr 30
β˜‘ RSVP'd to an event https://indieweb.org/events/2019-05-01-homebrew-website-club-nyc
post

Location:

www.thebeannyc.com
πŸ—½ Homebrew Website Club NYC
Join us for an evening of IndieWeb personal site demos and discussions!
I'm not attending.

Whoops! Due to several “no” RSVPs from HWC NYC regulars, we’re going to skip this week.

Stay tuned for the next HWC NYC on Saturday May 11th. Details to come!

Sun Apr 28
β˜… Liked https://marco.org/2019/04/27/overcast-clip-sharing
Thu Apr 25
β˜… Liked https://beesbuzz.biz/blog/4534-ICANN-seeking-comments-regarding-gTLD-pricing-deregulation
πŸ”– Bookmarked The luxury of opting out of digital noise Β· Woman. Legend.Blog http://blog.vickiboykis.com/2019/04/25/digital-noise/

“I miss more and more events targeted at my daughter’s age level that we could have attended. I miss small observations that my friends wouldn’t make over text that they do via Facebook posts that I no longer discuss with them. I miss parenting conversations that are extremely relevant to my local school district. I miss birthdays that I should have written down in my paper calendar, but didn’t. I miss discussions the Jewish community at large, which I am connected to digitally instead of physically, is having. By opting out of performing emotional labor on Facebook and going into my own sort of media hibernation, I miss the steady background hum of β€œhaving my finger on the pulse” as it relates to me and my family.”

Wed Apr 24

With Jonathan away, it’s time for you to play! Can you solve the mysterious Facebook comment thread in this week’s We Have to Ask Podcast?

https://wehavetoask.com/episodes/2019-04-24/

post from Who Is This For? β€’ Ep 310 - What the Comments Meant
Jonathan is out this week so Marty presents a mystery for you. Who well-wishes for whom?
Sat Apr 20

πŸ—“οΈ Magnet Improv Level 2 Showcase

Magnet Training Center & Rehearsal Space 22 W 32nd St, Fl 10th, New York, NY 10001
πŸ“† Add to Calendar: iCal | Google Calendar

We’ve been learning from the amazing Elana Fishbein for 8 weeks! Now come see us do a show!

Best of all: it’s free!

See you at:
Magnet Training Center & Rehearsal Space
22 W 32nd St, Fl 10th
New York, NY 10001

Thu Apr 18
↩ Replied to https://aaronparecki.com/2019/04/18/13/
post from
Your "elsewhere" links are much easier to read than mine

I’ve decided I don’t iterate on the pieces on my site enough, so decided it was okay to try something really different with one block. It’ll get out of control soon but I hope to learn more that way. :}

An unexpected quiet afternoon means time to work on the ol’ website.

Wed Apr 17

HWC NYC Wrap-Up 2019-04-17

New York City's second Homebrew Website Club of April met at The Bean at Cooper Union on April 17th, with me playing host.

We didn't really have a "broadcast" portion of the meetup, but we discussed some things and worked on personal projects!

mfgriffin.com (new!) β€” Has been to IndieWeb events in the past, but is looking to get re-started. Went over his past notes about some of his hopes and plans for his site, and how he wants to use it for personal notes, writing, drawing, and blogging projects, and more. Got started on a stripped-down system of using Hugo to track plaintext notes, with tags, so he can worry about how to organize it more later.

dmitri.shuralyov.com β€” Interested in being "done" with his notification system updates. In general, wants to be able to iterate faster on his site updates, and finds that a lot of his time is absorbed in HTML/CSS design processes. Started working on some text-only designs, which get a minimal amount of styling by being converted from Markdown to HTML. By using text and emojis, he was able to prototype a couple of quick things during the meetup.

martymcgui.re β€” Recently decided that he disagrees with how Granary processes his site's feed (which is HTML+Microformats2) into JSON Feed and Atom. The result is that his feeds often have missing info and show up as weirdly empty posts on micro.blog or in Atom feed readers. At the meetup, finished writing and adapter that will take his main feed and spit out a JSON Feed. Managed to break his site's build process trying to integrate it, but will get it working soon enough.

Other discussion:

  • A couple Dmitri's cool game projects
  • The recent release of the original source for Zork and other Infocom games, as they would have been worked on originally, for a compiler that no longer exists. Really neat to see how that info is organized.
  • We want to start later! Folks often need until 6 or 6:30pm to arrive.
  • We're also open to venue options!
Dmitri, Marty, and Matt smile for the camera in front of a background of a busy coffee shop.
Left-to-right: dmitri.shuryalov.com, martymcgui.re, mfgriffin.com

Thanks to everyone who came out! We look forward to seeing you at our next meetup on Wednesday, May 1st from 6:30pm - 8:30pm!

Photo for tonight’s HWC NYC!

β˜… Liked https://blog.adafruit.com/2019/04/17/new-guide-pyportal-thingiverse-viewer-adafruit-johnedgarpark-adafruit/
Tue Apr 16
↩ Replied to https://martymcgui.re/2019/04/16/115702/
post from
I'm going!Looking forward to another Homebrew Website Club tomorrow! I plan to fix my feeds as they appear for Atom and micro.blog.

I’m also going to this Caveat show tonight: Flying University: Is AI Racist?

Looking forward to it because Anil Dash will be speaking! Also because AI can totally be racist. And I love Caveat shows.

I’d love to talk Glitch and IndieWeb with Anil but I will probably just play the part of polite audience member. πŸ˜…

β˜‘ RSVP'd to an event https://indieweb.org/events/2019-04-17-homebrew-website-club-nyc
post

Location:

www.thebeannyc.com
πŸ—½ Homebrew Website Club NYC 2019-04-17
Join us for an evening of IndieWeb personal site demos and discussions!
I'm going!

Looking forward to another Homebrew Website Club tomorrow! I plan to fix my feeds as they appear for Atom and micro.blog.

Semi-bonus: the following day is an Internet Explorers comedy show about MEMES at Caveat. I plan to go to that as well!

πŸ”– Bookmarked https://www.theatlantic.com/science/archive/2019/04/katie-bouman-black-hole/587137/
β˜… Liked https://mxb.dev/blog/webring-kit/