Making The Long Good Read "Data Paper" Covers.

All Six Issues of The Long Good Read

For the previous 6 weeks I've been working on a project with the @guardian and @newspaperclub to revive @thelonggoodread in the form of a "Data Paper". I shan't go over the reasons again as they're covered well enough here: The Guardian experiments with a robot-generated newspaper with The Long Good Read and here: On Algorithmic Newspapers and Publishing. But I did want to cover the covers (heh) quickly because they damn well took a bit of effort, and all good efforts should have words thrown at them.

The weird thing about the covers is that they're actually a failure, in that I didn't manage to do with them what I was planning to at all. The contents of the paper is algorithmically created from a big pool of articles and variables, what I wanted the covers to be was a visualisation of the algorithm and variables that ended up producing the whole paper, while also being aesthetically appealing as a piece of cover art in itself.

Each week the plan was for the cover to be stylistically similar, holding the "brand" from week to week, but visually effected by the content. Each one could stand on its own as an image, but if a reader learnt the language of they cover they'd be able to decipher it and see at a glance what the contents of the paper was likely to be.

A quick example; pretend the paper ran for a few years, when the Olympics arrived the stories algorithmically picked would most likely be sports related, which have their own bunch of related tags and represented in the Guardian by the colour green. Then a while later the Paris Fashion Week hits and so there are different tags, relationships and colours. Some weeks nothing much "Newsy" happens, others it's a new breaking news story each day. There are cycles and explosions to the flow of news that I've learnt over the years of measuring this stuff, to the point of it being a background hum to everything else I do.

It was that which I was hoping to bring to the covers. But alas while I have my hands on the data, I just never found the time to make the data pretty in the way I wanted.

They were also supposed to be fully automated, allowing a user to visit a "paper control panel" preview the algorithmically selected stories, decide on a cover "theme" and then press a button which would throw all the stories into Newspaper Club's ARTHR tool and go off and generate the cover.

Meanwhile I'm happy with what I ended up with instead especially given that the whole project was to produce a paper each week in a tiny amount of time. The covers were more representative of the newspaper and all the data as a whole. There just wasn't quite enough time to get to where I wanted to be.

But anyway, a quick rundown on what we have instead.

Issue #001

Issue One

This one was a bit short notice, we had some designs for the front cover, but they were things looking like data, but not actually data. In much the same way as computer interfaces in movies look like (HUGE) interfaces but aren't actually real. As I didn't have anything lined up instead I wanted to represent the process of putting together what's inside the paper on the outside. The original Long Good Read did a similar thing, the title/logo at the top and headlines and short descriptions of each story in the bottom half. This was a more visual version of that.

Tools: InDesign, Photoshop.

Issue #002 & #003

The week came round too fast for the next issue so I dug up some code I'd used previously, which produced a "barcode" type of image, which I rotated this time to have more of a DNA chart look about it, as though it was showing you the underlaying building blocks for life (of a newspaper).

Turns out that's not too much of a stretch, there are 7 strips each representing a 24 hours day, Monday on the left, Saturday & Sunday on the right. Midnight to midnight runs down from the top to the bottom. That's roughly how many articles and when the Guardian publishes.

All Six Issues of The Long Good Read

From a code point of view it's written JavaScript, which calls the Guardian API and draws to a huge canvas element. I then use the Chrome extension Full Page Screen Capture to grab it and shove it into Photoshop to add the title over the top.

For Issue #003 I added the colour of each section that the articles were published into. I also started messing around with the title/logo. Normally you'd take a while establishing the logo before you mess around with it but given our compressed timeline it seemed worth just getting on with blowing it away.

Tools: Guardian API, javascript.

Issue #004

I had a bit more time to produce the cover this time and had been wanting to throw slabs of colour at it to see how they looked.

Issue Four

This is probably the most popular cover as it happens, misusing Treemaps to show the number of words published by section for the previous 7 days. This is one of those covers that works well once but probably doesn't make much sense to use regularly.

I didn't have a chance to write any custom code for this one, instead using Many Eyes to create the treemap. Which was then taken into Photoshop again and essentially drawn of the top of. Also once more the Guardian API was used to grab the word counts for each section which spat out a CSV file used as the source data.

Tools: Many Eyes, Guardian API, javascript & Photoshop.

Issue #005

Probably my favourite cover and the closest to what I was aiming for, a semi abstract design that's actually based on real hard data.

Issue Five

This was playing with some graphing code (and d3js) I'd already written which calculates the strength and clusters between tags used on the "top" stories over the last 7 days. I used d3 to position all the nodes, which then dumps out the positions and sizes once it's finished. I then took those positions and re-rendered the image at a much higher resolution with ProcessingJS, because I wanted gradated lines which I couldn't hack into the SVG of d3.

Out of all the designs this is the one closest to being totally automated with the code creating and outputting a high-resolution image.

Oh and at some point add back the tag names onto the image.

Tools: Guardian API, javascript, d3js, ProcessingJS.

Issue #006

Similar to #004 but instead of word counts it's using article counts instead

Issue Six

I actually spent a bit more time with this one poking at something I'd only read about, which is getting Processing via various messy javascript hackery to play with Illustrator. I managed to get it to the point of drawing the correctly sized and colours circles and then ran out of development time.

Then, because I don't really use Illustrator and do use Photoshop, I pulled it into PS to move them around and put the text on top.

On the automation side I'd managed to get enough done to see that getting Processing to build the whole file, text and everything in Illustrator would be possible… just not by the end of the day.

Tools Guardian API, javascript, Processing, Illustrator, Photoshop.

Lessons Learnt

Mainly that to have a tool that generates something for you automatically in a manner of minutes or seconds, takes days to create.

That and weeks come around really quickly. Each week I thought "I know, I'll have some time to code a thing before next week, then it'll be easy" and before you (i.e. me) know it, next week has arrived and I'm all "Agggh, it's newspaper making day again today, what am I going to do?". Repeat.

And indeed repeat as I get to do it all over again next year.


Photos for issues #001 & #004 by Newspaper Club used under a Creative Commons, Attribution NonCommercial NoDerivs License.

Minimum Viable Ur, modestly rebuilding Glitch an update of sorts.

Lots of people, yay!

Since TinySpeck released a whole bunch of Glitch (the game) assets into the Public Domain I've been grabbing evenings here and there to see what I could build as a proof-of-HTML-version-concept. I always knew it was going to be rough and ready as I was travelling at speed, rather than being "proper".

I believe there's an actual proper attempt to revive the game, getting the server & client up and running over at ElevenGameMMO.

Meanwhile I've been aiming a lot lower. In my head I wanted to quickly test three main pillars…

  • Something that can take the .xml files that describe all the locations in Glitch and turn them into something I can load up in javascript and paint onto <canvas> objects, so we can actually see the locations.

  • The ability to "walk" around the locations, moving from one to the other, with a basic "Glitch" character sprite that you can control with the WASD & cursor keys. So that it feels like we're in the locations that we've drawn.

  • Multiplayer, having everyone in the same "world", being able to see where other people are walking and the ability to chat, both in the local location and globally. Essentially turning each room into a chat room.

Yesterday I managed to get all those things together, and achieve what I'm calling Minimum Viable Ur ("Ur" is the name of the world in Glitch). Loading locations, "walking" (flying) round them, seeing other people and chatting. A few friends even popped in to help me check that it was working, which was lovely. There's always something fun about sharing a virtual space with people you know.

You can experience it running, in a fashion, as there are many things wrong and likely to break, here, at MVURXI.com, although there's not likely to be anyone around (invite a friend to join in).

Oh and to celebrate it getting to "shipped" I finally gave the whole thing a name…

MVURXI

The original Glitch game used the number Eleven as a reoccurring thematic element, so given that four letter .com domains are impossible to get, I've tagged the roman numerals version of eleven "XI" onto the end of Minimal Viable Ur (MVUR) to get the easy to remember and pronounce MVURXI.

What Next

Well, there's still an awful lot to be done. As I said the code is pretty rushed, I've cut corners all over the place. I've used socket.io for the first time (which is awesome as far as I can tell). There's a whole bunch of stuff that isn't done properly.

Which means my first task is to clean all that up, add error checking, logging, sort my file structure out, testing it on various browsers and so on. I'm pretty sure there's a good Christmas month of evenings just doing that and not really moving forwards other than good clean code.

After that the plan is to focus on just improving the three "pillars".

  • Find all the missing image assets for locations. Make sure the client doesn't die if it can't find a location. Add the signposts back into the game and make them work. Basically improve the world builder. Oh and get all that to work on the iPad.

  • Import the "paths" and "walls" and "boxes" for each location that describe where the player can and can't walk. Add gravity, physics and the ability to jump, and climb up and down ladders. There's a whole heap of code just in that alone.

  • Allow the user to import in their old Glitch character from the old game into the new one. Then rebuild the "Wardrobe", so people and new "players" can customise their character. Again there's a pile of work there. Maybe even re-introduce houses to people can have personal private rooms they can chat in, with an invite/key system.

There's other stupid stuff going on as well that anyone who's built games will spot right away (I've got movement tied to the frame rate, rather than time, which is a no-no). I trust the client to keep track of where the player is, which is another no-no, as it allows walking through wall hacks and flying and such like.

But, a lot of that stuff is only a problem if it was an actual game, and one player getting to point X before another was a real problem. Because I'm keeping this as a friendly chat room I can safely-ish ignore a whole bunch of issues.

To keep things tidy and not to flood my personal blog too much I've set up a Dev Diary blog and a twitter account @MVURXI to track updates and general MVURXI news.

Code

Because this is a general learning process as I go along, but some people may find it interesting to poke around at how it all works, the code is up on GitHub, because I'm nice like that :)

Lessons learnt

A whole bunch of stuff about socket.io, and just how lovely it is to use with Node (obviously). I'm glad I had this fun project to give me a reason to try it out. There's something very wonderful about how Node works and socket.io fits into all that, the idea that a backend can just be doing stuff and letting the front-end know when it needs to.

I can feel some art coming on.

Oh, and coding in public is fun, even if your code is a mess, people are kinda forgiving. And always be shipping, ropey code beats theoretically perfect but not running code every time.

Yay, the satisfaction of the mundane, administration.

Today I managed something very satisfying, but ultimately dull and also temporary. A long while back I decided I wanted to move my blog away from being hosted by WordPress to being hosted on my own server.

Only I didn't really manage to do that properly, I had a look round and picked MediaTemple, 'cause they liked designers and seemed (and are) friendly, helpful and cool, which had this thing called "Grid Service" (gs), which gave you one-click setup of WordPress blogs and various other bits and bobs. Enough to get me up and running.

You could also FTP files up, including PHP.

So over the years I hosted various stupid simple sites and projects there, but kept stopping short because I didn't want to have to deal with PHP too much.

But recently I've been wanting to play with Node more, slightly drunk on the power of being able to very quickly hack together backend stuff in a language that I actually enjoy. Which means getting a server that runs Node.

Which I also got from MediaTemple, so now I had two servers, a (gs) server and a DEV server, and of course paying for both.

Slowly, slowly over the last few months I've been moving sites from one to the other. Porting this blog away from WordPress to a Node blogging platform, figuring out how the hell to actually serve static sites and lost at sea with email because the DEV box is bare bones while the (gs) just did email.

Email is a post for another day.

Today I moved the last site off the old (gs) server onto the new one meaning I could close down the old box and save myself $20 a month.

It is deeply satisfying.

Also because now that I've gone through the pain of setting up the bare bones DEV box I can think about moving everything again away from a US hosted server to one in a different county. But at least I now feel that's possible and I'm a little less locked in.

See, mundane, dull and the whole process was fairly tedious, glad I've done it now though.

Rebuilding Ur, bringing back the beauty of Glitch locations

Double Rainbow

Last year, to much sadness from everyone, TinySpeck shut down the rather lovely game of Glitch. A beautiful quirky game loved deeply by not quite enough of us. A lot of the game's pleasure for me was just walking around the place, growing trees & herbs in my back garden and generally chatting to friends.

There was a definite sense of loss on the last day, that last hour when it all shut down, knowing that I wasn't going to be able to "walk" through those curious landscapes again, that it was all being packed up into a box, tucked away, and would simply cease to exist.

Much like geocities going away, but prettier.

Last week however TinySpeck did something lovely, they took that box, opened it up and tipped the contents all over the GitHub floor. Now, admittedly the world was pretty much all in pieces, but a quick poke around showed that all the bits and bobs needed to stick it back together in a semblance of what we had before were there.

And so, over the course of the next 24 hours I got stuck in, poked through the .xml files that described each location, dusted off some Flash Javascript horror (thanks to twitter for the pointers) to convert the chunks of Flash scenery into .pngs and had a pretty workable Glitch world viewer.

You can experience it here and find out more about the code in this GitHub repo.

Then two waves passed over me. The first of gentle joy, that I was seeing the world I thought had been lost. I tweeted out the link and a lot of the feedback was similar, "Oh, I never thought I'd see this place again", and general happiness. Bring something back from the dead felt good.

The second wave was of rough sickness, brought about by kids with their germs from school, bringing something up felt a lot less good. So I took it easy for the next few days.

What I am doing with Glitch

glitch-layers

I suppose an easier way around this is to say what I'm not doing with Glitch which is rebuilding the whole thing.

What I've just been doing though is getting my html game "client" to render all the scenery to <canvas> elements to make controlling the scene easier (and faster), paving the way for allowing a single player to walk a Glitchen avatar around the world. Moving from location to location.

And after that, bolting on chat with a backend Node server. With, fingers crossed, global chat and "local" chat.

At which point I'll probably spend the rest of the time just tidying up those two elements of the "game". Turning it essentially into a pretty multi-user, multi-chatroom. Which is generally what I used it for in the first place.

Then we'll once more have a place to walk around, wonder, and chat.

And I absolutly will not have any feature creep and start adding in other stuff. Totally and utterly no!

Not even a pig.

Maybe.

General blogging maintenance

So I kinda half moved the blog over to Ghost a few weeks ago, while also moving DNS and goodness knows what else. Since then I've been meaning to blog things but didn't want to until I'd "properly" finished patching up the blog. Tonight I finally got it to a point where I'm happier.

I had a whole bunch of Bootstrap CSS where I didn't need about 97% of it, there was a pile of JS and a couple of webfonts loading in from Ghost that I didn't need, as I'm using TypeKit. Most of the time was spent cutting down all the CSS into just what I use, and killing off all the javascript. Annoying but done now.

I also had to hack around with the Ghost code a tiny amount, which I know is going to be a pita whenever I have to update Ghost, hopefully it's worth the trouble though.

There's two main reasons why I moved...

  1. I fancy hacking around with the code base a bit and I feel better about doing that in Node than PHP for WordPress, and Ghost has pretty much everything I need... because I don't need much from a blogging platform.

  2. I've moved servers on MediaTemple from their "you don't need to know anything, 1-button pressing WordPress hosting" which was nice but only really offered FTP, to a "proper" DEV box which has nothing on it and I need to install everything. I'd already installed Node and nginx, and didn't fancy then having to install PHP just to run a WordPress instance. There's only so much installing crap I can take.

So now here I am, running Ghost on nginx on a dev box, which hopefully means migrating off onto a non-MediaTemple box should be easier in the future.

And now I can get on with the proper blog posts.

Sorry to the RSS readers, moving over to Ghost blogging, breaking stuff.

I'm guessing that if you read this blog in an RSS reader you may end up with my last 10 posts all over again. There's also going to be all sorts of other stuff broken (beyond the normal brokeness). As I've just moved from Wordpress, the well established blogging platform now at version 3.7.1 to Ghost... currrent version 0.3.

There's just over 400 blog posts pulled in from Wordpress, nearly all of which point to images and photos stored in whatever folder wordpress stores them in. I've been through about 20 posts re-uploading them to this new Ghost's content directory.

And of course I've blown away Flickr, so there's that too.

Ah, and I've done a horrible hack to try and preserve the URL structure so I don't break old links to the blog.

Anyway. I'll go into the reasons for moving over in another post. This is just to note that it's happened, I'm also moving servers so DNS is all messed up.

Always fun to break stuff in the name of progress.

(Oh, and I also need to roll-up the CSS and .js files, sorry about the hefty load of files, just easier to debug this way to start with)