The one in which I like big fonts and bright colours.

The above is a slide from a presentation I did on Friday, which was quite fun, possibly interesting and most certainly delivered in the wrong order. This one sits between tracking a single word across the site over two years (not data journalism) to tracking the top keywords all the time and connecting them to the stories of day.

Basically it was attempting to explain spotting patterns in data and how we can then use that (short version, we’re a bit more sweary this year than last).

In the meantime, back to the weeknotes.

After the success of getting That Omniture up and running at the end of last week I spent Monday (once all the email, meeting alerts and various other cruft was sorted, <3 Monday mornings) and Tuesday setting up the Omniture report fetching framework type thing. Which mainly consisted of admin panels that tell me what’s going on and what’s going to happen next. As previously mentioned it’s so other people can step in and see what’s happening.

Sadly my Granddad passed away on Monday night/Tuesday morning so a lot of Tuesday was spend sorting things out on that front. Not really weeknote stuff but obviously has an effect on work, which is why it’s in here.

Then, fun times! A few days down at the office, a nice early start to catch the 5:43am train and it was so cold too, I could probably do without those kind of days, we have Teleport yet? Anyway, as I’ve not been into the office for a few weeks a lot of Wednesday was spent catching up and talking to people.

I kind of have a habit of making out that Working-From-Home (WFH) is when I actually get the work done and going into the office slows things down. That’s only half true, the WFH is great for uninterrupted concentration but you really do need the walking around and chatting to allow ideas to flow (kinda like the spice in Dune, maybe, having never read Dune that’s probably a really bad analogy), and keeping up with what’s happening.

Thursday, the Day of Slides.

Presentations always mean lots of preparation, this normally consists of trying to mentally construct a narrative and play it over and over until everything seems to flow sensible. Not quite remembering the logic behind some jumps is a bit of a pain when you’re suddenly presented with a slide of a kitten and can’t recall exactly the point you were trying to make. But heck, if those things are going to happen, better it be a kitten than anything else.

I told myself that I’d only do 49 slides, it’s good to limit yourself. Here’s the planning on Scrivener’s cork-board (that I swear I’ll write a blog post about).

101 slides in 34 minutes and a stream of consciousness later, I think but can’t quite remember that I may have claimed that the Editors should stop worrying about what things look like so much, apart from the bits where they should and we aught to just run everything off WordPress apart from the parts where we shouldn’t.

An exercise in clear clarity then.

I then spent the afternoon replying to emails :)

Bit of an easy week to keep track of week really.

Slides Planned: 49
Slides Created: 101
References to Kittens: 14
Temperature last night: -10°

Filed under: guardian, weeknotes by Reverend Dan Catt

Wait, I’m supposed to be writing blog posts between WeekNotes, this isn’t good, otherwise this’ll just be a WeekNotes blog and where’s the fun in all of that? Note to self: Fill in some of the gaps by writing about things raised in these WeekNotes, surely I must do something that isn’t work related? Actually this week no, because…

This is the week that Omniture ate.

Well not strictly Omniture, but also JSON, python, Google AppEngine and myself ate. But ever had one of those days/weeks where something that should just work, didn’t and doesn’t. Well this was that!

I’ll explain, the Zeitgeist, you know that thing I keep mentioning, it’s run off our Analytics package Omniture as well as what’s happening out there in internet land. Omniture is great, tracks everything and more importantly tracks all the stuff I need it to track, however getting reports out automatically and in a format I could use was tricky. Mainly in that the Omniture API consisted of making SOAP requests.

Now I’m sure SOAP makes sense in many cases, but frankly SOAP and I just don’t get along (no jokes at the back please). That aside I still needed the data and there was another way to get it. You can set Omniture up to email reports to you at regular intervals.

Those regular intervals are normally daily or weekly. I however wanted those reports hourly, so that’s what I did, set up hourly reports.

Initially 9 of them, just to cover the main sections of the Guardian. Then 18 to cover all the main sections of the Guardian.

This is the point where things get a little scary, 18 reports emailed hourly. They’d tend to arrive all at once around 20 minutes past the hour, the Google App Engine instance that was set up to receive the emails would get them, file them, parse them and then sort through all the 50 records in each one.

It would take about 23 minutes to get thru the whole lot.

Then

I needed to start consuming more sections as they came online at the Guardian. I think the system is now getting around 31 emailed reports, which then get crunched through.

Which is still fine until Google AppEngine decides to go down for a couple of hours or the emails from Omniture back up, or the moon is aligned in just the wrong way. And crunching through the reports all crash together causing a vortex of task queues to collide, fall over, multiply and generally get out of control.

In short, I need to baby sit the task queues (Google AppEngine has no programatic way to monitor the task queues) checking in on them every couple of days to make sure they haven’t suddenly spiked up to 1,000+ tasks.

Omniture REST API to the rescue

So it’s fortunate then that Omniture have released a nice REST API.

The theory being that now I can POST requests to reports whenever I feel like it, ask the queue how those requests are getting along and then fetch the reports when they are ready. Which for the record appears to be in seconds, which is nice.

The obvious advantages is that I can tune the speed at which those reports are asked for. If the AppEngine is having trouble getting through all the rows in the record (for whatever various reasons) I can get it to back-off asking for the reports. Previously the best I could do was force it to drop the reports on the floor when they got emailed in. I can request different reports at different rates depending on the amount of traffic each section of the website gets. In short, I can tune the engine.

It was with good energy on Tuesday that I started poking the Omniture API Explorer, a while later I could use it to request and fetch reports … “easy” I thought, this is going to be awesome!

Wednesday I spend building up the foundations for Zeitgeist Version 3. This time with proper ground up admin controls and reporting to make it easier for anyone not just me to manage and debug. Setting up a new project is always time consuming making sure all the files are there, checked in and so on. Everything is quite nice, jquery running the monitoring tools and to on, all good stuff.

Thursday is where is all started to go wrong. First, I couldn’t authenticate. There’s a special way you authenticate with the Omniture API, you take a self-generated one time token, the date/time of your request and the secret API key, concatenate them all together, SHA1 the result and then base64 encode that to send off, along with the one-time token and your username.

Did, not, work.

I stepped through everything. I used online tools to compare my results with other SHA1 and base64 encoders to make sure they matched … they did. I grabbed sample code off the forum, which matched mine, but didn’t authenticate. I read the instructions and followed them to the letter, still didn’t authenticate.

In the end I used firebug to step through the javascript on the Web Explore page comparing it’s results with mine step by step.

Turns out that when you’ve taken your one-time token, timestamp and secret, catted, SHA1 and base64 encoded the result to send off, you don’t send it along with the one-time token you originally used, but you also base64 encode that, so the one-time token you originally use is different to the one you end up sending.

Makes sense? Possibly. Documented with the Authentication instructions? Nope.

So that put me into a grumpy mood.

Then

With authentication work I posted a report request, it authenticated YAY! It also failed, telling me I was requesting an invalid Report Suite ID. Although I had the Omniture Report requests working in the API Explorer, getting python to POST the requests was hair-pullingly awful. Not matter what I tried I got just one of two error messages back.

I tried all sorts of different ways to POST the request… you’re supposed to POST a JSON object over to the API REST endpoint. I posted the JSON object directly, I posted it serialised, URL encoded, serialised and URL encoded. As a hand build string, base64 encoded, simplejson.dumps(), json.dumps() … every… thing.

That took me to Friday afternoon. Turns out the way I managed to get it to work in the end was hand building the string but swapping all the “ for ‘ or perhaps it was the other ways round, I forget now (thanks to @ad0815 for that). Anyway it was the way round as specified in the JSON standards thingy. Which is why I’m at a loss for why it didn’t work when simplejson dumped the results down to a string.

Anyway, by mid afternoon on Friday I finally had some code POSTING a report request :) But really from starting to work with it on Tuesday to getting it to actually work on Friday, that’s annoying!

I did however managed to get a few other things done along the way, so not a total loss.

The Week of Other Things

Along with the usual adminsitrivia that I try and avoid as much as possible, there’s a new “Most Viewed” page out, which hopefully looks like the old “Most Viewed” page. Only this time the Viewcounts are coming from somewhere I must try not to break :) and I fixed up the links at the bottom of the “Most Viewed” widgets to point to the right place.

I worked a bit more on Keyword tracking. I have code that keeps an eye on keywords over time, you can see a graph of a bit of it up at the top of this post tracking Culture related keywords over time. I changed the way that it detected how these things are moving up and down so that information can be fed off to other things that want that data.

That, and the Omniture stuff above means I can now pave the way for tracking arbitrary keywords over time, and the stories related to them. Hopefully that’ll be useful at some point :)

I also did a bit more work on mystery project, which involved looking at a number of different mapping tile providers, and a nice cork-board texture :)

And that was the week that was. Hopefully this week will feel more productive but I guess last week was really, I just feel better when pushing out interesting code rather than good solid foundation code. Conversely I feel happier knowing there’s good solid foundation code than just interesting code.

Filed under: guardian, weeknotes by Reverend Dan Catt

I suppose the thing with weeknotes, is you have to do them every week, so here we go.

Last week involved a lot of playing with the Guardian’s API, the work I do tends to involved a lot of digging around with what the API can produce. Often it’s just fetching (an awful lot) the content of an article, or checking its tags. The problem with this is that’s it’s very easy to get something up and running and then pay no more attention to it.

In the meantime features keep getting added… see the posts titled “Content api build xxx relased” over in the Guardian API Talk Google group, to see what’s changing and new. Which is why it’s important to go back and poke them frequently.

The most recent thing I’ve been looking at is trying to keep track of what’s happening on the Guardian site, right-at-this-very-moment. Which involved comparing this page: global stories [day] [month] which shows the latest stories on the site to this API call…

content.guardianapis.com/search?page-size=5
&order-by=newest
&format=json

… which you’d expect to be the same thing.

However, due to the way the news flows around our internal systems the API is often around 5ish minutes behind the content getting published on the site. There are also interesting effects seen when refreshing the API where articles appear to pop-in-and-out, which I assume depends on which server I happen to be hitting at the time, as caches clear out and suck like.

Anyway, to that end I threw together http://www.whatisthelatestguardianheadline.com/ which scratched several itches for me and putting a really big FONT frontend onto data is always fun. Russell Davies as always has some great thoughts on screenification and polite media.

The first half of the week also involved the lovely administrative duties of email, following up email, arranging and moving meetings, writing a “short bio about yourself” again, which I should really get sorted just the once and be done with it. I think it should read:

Daniel Catt: Programmer

The rest of administrative duties involved stupid Google App Engine and stupid Google Domains and getting one to look at the other. For all the good stuff Google has going, they’re terrible at joining the dots between their various services.

Taking us into the second half of the week that involved a lot of kicking Google App Engines when the cron and tasks queues starts to fall over (Google’s fault) and backup (probably my fault). So that needs some more monitoring tools that will …

1. alert me
2. make it really easy to drill down to what the actual problem is

Which is getting to be a touch more important now that the data I’m calculating is starting to get used in more and more places. Which was the last couple of days of the week. More building feeds that various things that I can’t talk about can use … but it’s fair to say that they are things that want to automatically highlight interesting content.

Bringing us nicely to the launch of The Long Good which happened midweekish …

http://thelonggoodread.com/

… that I posted about over here: Introducing The Long Good Read – For when people want longer articles to read. Which itself is another example of how number crunching can be used to help people surface good articles and information.

If the focus on numbers, views, trends and so-on shows us anything, it’s that automation is all very well and good, but it’s far better when it’s used to present people with a selection of things the computer thinks is good, and allow the users to filter the final selection, even if that selection is for other users.

The aim of having “Computers pick all the news specifically tailored to you, using maths and science” is all very well and good as long as people don’t hide the part where humans were involved, for that’s the most interesting bit.

Finally I did some more work on the-project-that-cannot-be-named.

The project-that-cannot-be-named has shown me that the setup I blogged about last week is working pretty well and having files synced across machines and easy access to those machine is a real life saver when working from home. Which brings me to…

Working from Home, the Good: Commute is shorter, less meetings, lovely yellow leaves as the biting autumn frost kicks in.

Working from Home, the Bad: My (work) iPad has turned up! At work, where I’m not, I have to wait until next week to get my hands on it.

Photo by Russell Davies

Filed under: guardian, weeknotes by Reverend Dan Catt

I’ve been running The Long Good Read for about 4 months now, it’s probably time I wrote about it. Trouble is, the longer it’s taken the more I have to say about it. As I’m already tragically bad at keeping posts short I’ve split this one down into three parts, just jump to the bit you want.

1. The (lost?) Art of Long-form reading
2. Open vs Closed, Onsite vs Offside news is all very exciting at the moment
3. Technology, the bits and bobs that make it all happen

The (lost?) Art of Long-form Reading

I’m sure it’s entirely possibly to make a distinction between “long form journalism” and “reading long articles” but this isn’t the place for that, just wanted to get that out of the way at the start :) For this post I’m going to horribly mix up the two concepts.

There’s been an amount of talk around the subject of long-form journalism recently with Long-form journalism starts a new chapter, the Media Blog: Size does matter: Can long-form journalism thrive online? and the Nieman Journalism Lab Post: Center for Public Integrity’s HTML5 product aims to make long-form journalism readable on any device all discussing the topic.

Sites such as longreads.com, longform.org and givemesomethingtoread.com and twitter accounts like @IfYouOnly & @longreads have all popped up to hunt out and supply the apparently growing demand for the reading of longer pieces.

Meanwhile Instapaper, Readability and that weird button in Safari have been around for a while now helping people consume the articles free from the detritus of webpages.

The problem for reading long articles it seems that no-one buys newspaper anymore and no-one likes to read for long periods of time from the screen. Mobile phones, tablets and specific reading devices like the Kindle are bringing that back to the breakfast table, sofa and commute on the train. The demand for long, in-depth, interesting articles never really went away, the demand for attention on our time and the crapness of computer screens is wot did it g’vnor.

Anyway The Long Good Read is my attempt to surface longer articles from the Guardian that people may have otherwise missed.

Putting long articles next to short ones works well in print, the eye can quickly scan over them and decide which to read based on headline, length, free time, mood and so on. It’s hard to translate this over to the screen for several reasons. When arriving at or browsing the Guardian website it’s difficult to judge where the long articles are going to be, if that’s what you’re in the mood for.

This is an attempt to chip away at that problem by gathering a number of “long good reads” together in one place.

As already mentioned reading on the screen can suck, so the blog isn’t really trying to be the new-place-to-read-the-content but rather a hub that goes on to supply the content in a number of ways. The user can point their favourite RSS reading tool at it, such as The Early Edition, Reader or Pulse.

There are Instapaper buttons on each article ready to send them to whatever Instapaper reading setup the user has. Of course there’s also the obligatory twitter account @thelonggoodread, which isn’t really there for people to follow in the normal sense but rather so it can get pulled into tools that aggregate what your network is linking to.

Open vs Closed, Onsite vs Offside news is all very exciting at the moment

As well as talk about long-form stuff, there’s also a lot of chatter going on at this very second about Very Open Data vs Paywalls. Maybe that’s just the people I’m hanging around with now though :)

There are so many discussions around already about why Open is good it’s hard to know where to start, here’s one of the Guardian’s own from Mike Bracken: From Publisher to Platform: 14 ways to get benefits from social media that has a slideshow worth watching in full screen (even at 110 slides).

And talk about why Closed is bad too, for which we’ll turn to Clay Shirky: The Times’ Paywall and Newsletter Economics

tl;dr

Either way, what the Long Good Read is doing is re-framing articles from the Guardian, placing them into a different context. From “We’re a News Organisation” to “Here are some posts that are interesting and will take a while to read”. It can do this because the Guardian offers it’s content via the Open Platform and it’s API.

Why is this good for the Guardian?

Well, first, who actually knows? But it’s a good hunch.

Secondly it allows the Guardian’s own content to move from it’s website out into the internet, into apps, onto blogs and whatever else crops up.

If that were the end of it, then perhaps you could say “well it increases brand awareness” but there’s [possibly] more going on that just that.

When these offsite blogs, tools and sites start to curate the content or present the news in different ways then they are delivering to a different more specific audience. The people who come to The Long Good Read, however they choose to consume it, are coming because they are interested in the type of content it’s providing, articles that take time to read, that are more in-depth, that aren’t necessarily time-critical, interviews and so on.

A very different site could concentrate on delivering the very latest American political articles. Yet another could be presenting all of the Iraq articles from the last 10 years in a timeline format, making connections and telling a story. A third all about music reviews, another about cooking, so on and so on and so on.

And along with these articles the Guardian is suppling an advert. Currently they appear to be fairly general.

To throw in some Weasel words, if I as an outside consumer of the API were to think about how adverts could be targeted toward users of sites that make successful use of the Guardian API to deliver content and gain a good targeted audience, then I may come to the conclusion that sometimes it could be worth it for the Guardian to direction users to their own content offsite. More so if you also reach the conclusion that open is to encourage your content to spread as far and wide as possible.

Disclaimer of Doom: I work for the Guardian and I think about these things (and others) quite a bit, although it’s not specifically my job. The above views don’t reflect any insider knowledge, or necessarily those of the Guardian. I have no idea what the plans for those adverts are although I could probably ask (but that takes the fun out of it). Etc. etc. etc.

Anyway, the Free/Open vs Paid/Close will rumble on and each case is different. In some cases free is a good model, in others paid works best … and all points in-between depending on the situation. I happen to think a bit of both is good and on the Free/Open side I’m (obviously) all for pumping content out from an API :)

Technology, the bits and bobs that make it all happen.

Ok, so all that up there is roughly, kinda of, a wrap-up of the Why, now some of the How.

First a very quick recap on the Zeitgeist.

The Zeitgeist is an experiment I’m involved with at the Guardian. It’s monitoring what people are reading on the site, based on our own analytics and comparing that to normal expected behaviour for the various sections and content type. It also looks out into the internet to see what discussions are going on around Guardian content out there. There’s a more detailed post over here: Behind the curtains of Zeitgeist, the Open Platform, APIs and Google App Engine

In the same way as the Zeitgeist keeps an eye on all the Guardian content, the Long Good Read is keeping an eye on the Zeitgeist.

In effect I’m essentially screen scraping my own data ;) (Zeitgeistyness will make its way into the API at some point).

The Long Good Read backend (running on Google App Engine) checks the Zeitgeist for new content, stuff that’s being surfaced as popular, interesting or otherwise unusual.

It the calls the Guardian’s API for each new item it sees to find out more about it. It rejects things like Galleries, Podcasts, Videos and such like right away. It also gets rid of the “Live Blogs” at this stage. Then does a word count rejecting anything under 1,000 words.

Anything that survives gets put into the review queue, which turns up on an admin panel like this …

Computer rules can only get you so far, at this point I can click on any of the stories to review them and then choose to approve or reject them. I can also fiddle around with the order a bit. Approved stories move over to the “Queued stories” column ready to be posted.

At around 3am and 3pm each day the backend then posts the top story to the longgoodreads.com WordPress blog, which includes a Twitter PlugIn that in turn posts a “New Post” tweet to twitter.

And that, is pretty much that. It takes me about 3-4 minutes a day to quickly review the “Review Queue” and I always try and make sure the to-be-posted-queue has enough to keep going without me for a couple of days. The articles I try and pick are non-time-dependent ones, items that can be picked up and read for several days after the date of publication. Also not too UK centric, it isn’t often you’ll find a long post about the latest government policies on the site.

The last thing for the backend to do is re-download all the content once every 24 hours, this is to adhere to the Terms and conditions and to make sure any updates, retractions and so on get dealt with in a timely fashion. It’s a bit of a pain, but doesn’t take a prohibitive amount of time and I can’t see it becoming a problem for quite a few years :)

A quick recap:

The Editors pick articles to commission and write, that reflect the voice of the Guardian. People and the internets find some more interesting than others. The Zeitgeist crunches the numbers. The Long Good Read crunches the Zeitgeist. I crunch the Long Good Read Review Queue.

The stories that get posted are the end result of:

Editors -> Readers -> Robots -> Curator

Which, if I had any sense I would have put at the start of the blogpost and left it at that :)

Filed under: api, guardian, hack by Reverend Dan Catt

File under stupid quick API hacks.

I’ve been taking an even closer look at our API over the past few days for various reasons and in the process threw this together.

http://www.whatisthelatestguardianheadline.com/

Which does pretty much what it says on the tin, aside the fact that the API lags a few minutes behind content going live on the site. The indexes for the API are re-fed every few minutes, the very latest stories are of course over here (probably).

The code is very basic, it’s also on github if you want to pull apart/copy my hastily constructing python.

In essence though it does this API call…

content.guardianapis.com/search?page-size=5
&order-by=newest
&format=json

Which gets the most recent 5 headlines from the site, it’s also an example of our “Keyless” API, which as the name suggests can be called without needed to bother with a key. Making it ideal for throwaway things like this.

A quick look at the fragment returned shows us we have this to play with…

We get the headline, URLs, section and publication date. We can also see the API has as of this moment just over 1.2 million articles in.

The backend code grabs that and shoves it into the database, doing only a tiny bit extra to work out how long ago it first saw the article and the colour to use. All with some extra random caching for no particular reason other than making everything a possible further 60 seconds out of date :)

Front end Ajax loads new (or not-new as the case may be) headlines once a minute. So in theory just leaving it should give you the latest(ish) headlines throughout the day.

I did go back and forth a bit on wether I should make a specific iPad style sheet and so on but decided that was probably too much effort for what it essentially is. I may get round to wrapping it up as a PhoneGap app at some point but only to learn more about PhoneGap.

Idea generally kicked off by, but unrelated to BERG’s Media Surfaces: The Journey, big font and a single page inspired by the many other single serving sites out there. On a black background because everyone complains at me (understandably) when I put Life and Style yellow on a white background, because I’m not particularly keen on the shade of orange they use.

Filed under: api, guardian, hack by Reverend Dan Catt