Great post from @aaronpk about his #IndieWeb setup. I should make one of these!

https://aaronparecki.com/2016/12/12/9/my-website

post from
A Brief Intro to My Website Architecture
RSVP'd to an event https://martymcgui.re/2016/12/06/212756/
post from
Homebrew Website Club Baltimore Meetup 12/14

Digital Harbor Foundation Tech Center Baltimore, MD

I am goingto Homebrew Website Club Baltimore on 12/14!

Join us to work on your own website and take control over what you post online!

Can’t make it out physically? Homebrew Website Club also has a virtual meetup this time around!

Great writeup by http://jonathanprozzi.net/ about the positive effects of setting aside even an hour a month to learn and work on new things.

http://jonathanprozzi.net/2016/12/homebrew-website-club-reflections/

post from
Homebrew Website Club: Reflections

2017: The Year of Peak Sloth - Live Podcasting Event

Single Carrot Theatre 2600 N Howard St, BaltimoreMD, 21218

Join us for a night of live podcast recordings from We Have to Ask, The Digression Sessions and Hobo Radio and improv brought to you by the Baltimore Improv Group.

2016 has been a rough year filled with tragic celebrity deaths and awful headlines. It’s time to say so long so we can look to the future. It’s time to look to 2017: The Year of Peak Sloth.

So before you head out to drink and party 2016 away, join the Peak Sloth Podcast Network for a live podcast event at the Single Carrot Theatre!

Admission is free! Beverages may be available! Live music may be part of the show! Celebrity guests may be interviewed! The only way to find out is to be there!

Homebrew Website Club Baltimore Meetup 12/14

Digital Harbor Foundation Tech Center 1045 Light St., BaltimoreMD, 21230

Create or update your personal web site!

  • Finish that blog post you’ve been working on.
  • Demos of recent IndieWeb breakthroughs.
  • Share what you’ve gotten working.
  • Ask the experts questions.

Join a community with like-minded interests. Bring friends that want a personal site!

Any questions? Ask in chat: http://chat.indieweb.org/today#bottom

Optional quiet writing hour starts at 6:30pm. Meetup begins at 7:30pm.

More information: http://indieweb.org/events/2016-12-14-homebrew-website-club

Facebook event: https://www.facebook.com/events/230344914069104/

Replied to https://twitter.com/mkphenicie/status/806314368464015360
post from Kristen is a great podcast guest and real-life gem!

@mkphenicie It’s so true! I can’t believe she’s in our improv troupe!

Thanks to our guest this week Kristin L. McKenzie for teaching us that there are always more ways to share.

https://wehavetoask.com/episodes/2016-12-06/

post from
Do You Care Enough To Share? • Ep 72- LeeAnn of P.E.T.I.

Rescuing my tiny videos from Vine

Ah, Vine. I loved the idea of a platform for sharing tiny video moments. It was truly a platform for some really amazing things.

Personally, I didn't make very many posts. Most that I did centered around my involvement with the BlinkyTape Kickstarter that we ran at the height of my awareness of Vine. So, while I had not thought about Vine for quite some time, I was disappointed to hear that it was shutting down the ability to upload.

Announcement like these tend to trigger my site-deaths reflex, so although they have announced plans for some kind of easy export mechanism, I thought: why wait?

This post documents the haphazard, poke-it-with-a-stick-and-see-what-happens method by which I rescued my Vines and their metadata to post them on my own site.

Getting all my Vine URLs

First up, I needed a way to get all of the permalinks to my Vines. The first stumbling block here was that Vine uses an "infinite scroll" technique that shows only the latest vines until you scroll down the page, triggering it to load more for display. Checking out the source of the page, I realized that I could simply scroll to the bottom of my List view to get all the Vine metadata loaded into the page, and I could then open up the Developer Console for Chrome, and drop in a short one-liner to print out all the URLs for my permalinks:

> console.log($('li.permalink a').map((i,el) => el.href));
["https://vine.co/v/hUt0rDdrlK6", "https://vine.co/v/hDIMuqQgtU0", ...

After copying this to a text file and editing out all the quotes and commas, I had a simple text file with the permalink for one of my Vines on each line.

Fetching the HTML

Next up, I grabbed the HTML for each Vine's permalink with a simple script:

#!/usr/bin/env bash
# Usage: ./fetch_htmls.sh < some_file_full_of_vine_urls.txt
while read url
do # each vine has a unique id after the last '/' character
id=`echo "${url}" | cut -d'/' -f 5`
html_file="htmls/${id}"
if [ ! -f "${html_file}" ]; then
wget -O "${html_file}" "${url}"
sleep 5
fi
done

Extracting the Useful Data

It's often "enough" to have all the HTML on a page - you can write a script to parse the HTML, select out various attributes and text values from the elements on the page, and grab whatever data you need. In the case of Vine, they make it a little easier. Each HTML page has an embedded <script> element in JSON-LD format:

After playing with the data in there a bit, I wrote a quick script using the pup command line tool to do parse the HTML and extract the (valid parts of the) JSON-LD chunks from each HTML page.

#!/usr/bin/env bash
# Usage: ./extract_ld_json.sh htmls/*
# Extract <script type="application/ld+json"> content from Vine
# page, filtering out attributes that are not valid JSON.
#
# - description - invalid quoting and redundant with author and name values
# - duration - invalid non-value
for html_file in "$@"
do
id=`basename "${html_file}"`
jsonld_file="json_lds/${id}"
if [ ! -f "${jsonld_file}" ]; then
pup 'script[type="application/ld+json"] text{}' < "${html_file}" \
| grep -v '"description":' \
| grep -v '"duration" :' \
> "${jsonld_file}"
fi
done

These JSON-LD chunks gave me all the metadata I cared about, including URLs to the media that I wanted to save: the videos, their thumbnail images, and the avatar image that accompanied each post. So, it was time to write tools to extract those, too. One of my recent favorite command line tools is jq aka "sed for JSON", so I used that to extract the URLs I needed to fetch all the media.

The videos:

#!/usr/bin/env bash
# Usage: ./fetch_videos.sh json_lds/*
for jsonld_file in "$@"
do
id=`basename "${jsonld_file}"`
video_file="videos/${id}.mp4"
if [ ! -f "${video_file}" ]; then
video_url=`jq -r '.contentUrl' < "${jsonld_file}"`
wget -O "${video_file}" "${video_url}"
sleep 6
fi
done

The thumbnails:

#!/usr/bin/env bash
# Usage: ./fetch_thumbs.sh json_lds/*
for jsonld_file in "$@"
do
id=`basename "${jsonld_file}"`
thumb_file="thumbs/${id}.jpg"
if [ ! -f "${thumb_file}" ]; then
thumb_url=`jq -r '.thumbnailUrl' < "${jsonld_file}"`
wget -O "${thumb_file}" "${thumb_url}"
sleep 6
fi
done

And finally, the avatars. I really could have just manually saved my one avatar image, but that is not my way. Instead, this script finds every avatar URL, hashes that URL to get a filename-safe name, and fetches it (once):

#!/usr/bin/env bash
# Usage: ./fetch_avatars.sh json_lds/*
for jsonld_file in "$@"
do
avatar_url=`jq -r '.author .image' < "${jsonld_file}"`
id=`echo "${avatar_url}" | shasum - | awk '{ print $1; }'`
avatar_file="avatars/${id}.jpg"
if [ ! -f "${avatar_file}" ]; then
wget -O "${avatar_file}" "${avatar_url}"
sleep 6
fi
done

Finally, it was time to synthesize all of this info to put these Vines on my own site. I currently use Jekyll to generate my website, so I wrote a pair of scripts to convert the JSON-LD data into Jekyll posts. First, a Python script that converts a single Vine into a Jekyll post with YAML metadata:

#!/usr/bin/env python
import json
import sys
import yaml
import os
with open(sys.argv[], 'r') as f:
jsonld = json.load(f, )
vineid = os.path.basename(jsonld['url'])
post_meta = {
'date': jsonld['datePublished'],
'files': [ vineid + ".mp4" ],
'filemeta': {
(vineid + ".mp4"): {
'poster': (vineid + ".jpg")
}
},
'h': 'entry',
'syndication': [ jsonld['url'] ],
'vine_metadata': jsonld,
'updated': jsonld['datePublished']
}
content = jsonld['name']
print "---"
print yaml.safe_dump(post_meta)
print "---"
print ""
print content

And finally a shell script to run the Python converter over each JSON-LD file, save the resulting Jekyll post output in the appropriate place, and copy over the related media assets:

#!/usr/bin/env bash
# Usage ./vines_to_jekyll.sh json_lds/*
BASEDIR=~/me/martymcgui.re
for vine in "$@"
do
vineid=`basename "${vine}"`
vinedate=`jq -r '.datePublished' < "${vine}"`
filename="${BASEDIR}"/_posts/`date -j -f "%FT%H:%M:%S" "${vinedate}" +"%F-%H%M%S"`.md
asset_dir="${BASEDIR}"/_assets/posts/`date -j -f "%FT%H:%M:%S" "${vinedate}" +"%Y/%m/%d/%H%M%S"`
echo "${vine}"
mkdir -p "${asset_dir}"
cp videos/"${vineid}.mp4" "${asset_dir}"
cp thumbs/"${vineid}.jpg" "${asset_dir}"
python vine_to_jekyll.py "${vine}" > "${filename}"
done

At last, I created a Jekyll include to render the Vines out as HTML5 video elements, with the thumbnail as the poster image, visible play controls, and the obligatory feature of looping forever once started. My Jekyll theme is a bit out of scope for this post, but it pretty much looks like this:

{% capture poster %}
  {% if include.filemeta and include.filemeta.poster %}
    poster="{{ page | asset_path | append: include.filemeta.poster }}"
  {% endif %}
{% endcapture %}
<video loop controls style="width: 100%">
<source class="u-video" type="video/mp4" src="" />
</video>

What is the end result? How about some example posts?

While I am not particularly happy that Vine is seemingly closing down, I do enjoy data liberation projects like these. I learned a bit about how Vine was built, got to play with some neat tools, and I now own my Vines, having brought them into my own site. While I'm not in control of whether Vine works in the future, I do have control over the future of the videos that I posted there.

Was this useful to you? Let me know! I don't have my #indieweb comments system in place, yet, but you can hit me up on Twitter or Facebook and I'd love to read your thoughts!

Almost went for the Nitro cold brew at 3 Bean. Made a more relaxed choice of tea at the Bun Shop.

HWC Baltimore 11/30 Wrap-Up


Baltimore's third Homebrew Website Club meet up happened at the Digital Harbor Foundation Tech Center on 11/30/2016. We had a nice quiet writing hour followed by some quick introductions, lots of progress on projects, and some new attendees!

Notes from the "broadcast" portion of the meetup:

Marty McGuire (co-organizer) - Been working on displaying reply contexts and manually POSSE-ing, sending webmentions. Also set up a self-hosted copy of Woodwind to keep following other IndieWeb sites. 2017 commitment to improve handling of images so posting to my site doesn't take forever and each page load doesn't include huge full sizes image by default.

Jonathan Prozzi (co-organizer) - finished setting up site on DigitalOcean, powered by Hugo. Has front page h-card working and logged in to IndieWeb wiki to RSVP to this event. Created a 2017 commitments post tonight.

Brian Young - Captured a ton of ideas for what to do next as Google Tasks. Started with IndieWeb Wordpress plugins and getting the feel for them at brianey.com. Using it as a test site for playing with indieweb features. Also started a new site imnotwrong.com, a humor site featuring a bad/fake/satirical advice column. Also has POSSE working on uncannycreativity.com to Tumblr and Medium.

Adam Bouhmad - Started working on a site for collecting malware samples and other interesting code specimens for security research.

Derek (no site, yet) - Teaching front-end web development to folks recently released from prison. Is pretty new to web development himself, so picking up new skills to stay ahead. Been working on converting a video tutorial about HTML+CSS menu bars into a workshop for students. Digging into gridview for the next class topic.

Here's the customary HWC group selfie:

Left-to-right, front-to-back: Derek, Brian, Jonathan, Adam, Marty

We had a great discussion and were all grateful for the dedicated time to working on our sites.

We hope you'll join us for our next HWC Baltimore on 12/14!