Feeds:
Posts
Comments

#563 in a series of Things That Have Arrived In The Post - Field Notes Pouch 6

I missed the tradition end of year round-up for 2011 so I thought I’d get in quickly for 2012. Also this post is going to be self indulgent and rambling much like all my other ones, it also isn’t really about anything.

Once more I’m battling with writing, in that I want to write but utterly hate the process and find it soul crushingly difficult. Actually I have no idea why I want to write other than it seems a good way of tracking what I’m thinking and looking back at all the plans I had that didn’t come to anything. From various people I’ve heard that getting over the horror of writing is just a case of sitting down and doing it regularly and after 8 to 10 years it gets a little bit easier. Looking forward to that.

Week Notes worked well for me last year and I was going to start again this year, a project deadline put pay to that but now it’s out the way I could get back into the swing of things.

@rooreynolds is trying out a (Weekly? Daily?) newsletter thing and @FinalBullet has a monthly one both* of which seem like scary commitments to me. I can just assume that no-one reads this blog, if I knew people actually read it then I’d have to make more of an effort. Meanwhile @currybet keeps a gruelling blog posting schedule going with near daily posts, for which I could totally poke him in the eye if I wasn’t too busy being in awe. And most of my other friends seem to be able to knock out a decent blog post once a week or so. Anyway, somehow I’m going to try to write more often, again.

Other things that I’m planning…

A list so I can laugh at it in 2013

  • Write more often
  • Clean up my hard drive of folders like “Temp3″ and “Desktop Junk.old” and keep it tidy
  • Keep all my personal projects public on GitHub, even though I haven’t run them through a Linter yet and the code is terribly embarrassing
  • Build the Guardian News Ambient 24 Hour Radio Station that I’ve been thinking about for the last 2 years (more on this in another post)
  • Fix the other Guardian toys I’ve been writing
  • Make new ones
  • Rewrite Ant Attack in javascript
  • Publish all my Snakes & Ladders code and results, finally
  • Maybe record some podcasts/video things
  • Actually create the 2012 Lunar Wheel Calendar I should have done by now
  • Start running again
  • Start swimming again
  • Do some more work with The Long Good Read and update it’s code
  • Learn how to use apostrophes
  • Pick up the work I was doing on the Zeitgeist at the Guardian
  • Build a friendly/simple interface to the Guardian Content API
  • Make more Books
  • Make more Magazines
  • Make more Newspapers
  • Make some things with Moo cards
  • Revive the Liveblog-o-matic

Things I have done so far…

  • Moved work GitHub repositories over to work’s git thingy, so I can now remove them (they were private) from my own account. This was a function of not having VPN access to work, really glad I have work code in the correct place now o_O
  • Finished version 1 of sekrit Guardian project, that has potential to be awesome if pushed right
  • Written some blog posts about Instagram and Flickr that were quite popular
  • Put up some shelves

Errr, and that’s it.

And thus ends the blog post in which I once more use too many words to say too little that too far too long to write.

 

*That both Roo & Leila should be doing the newsletter thing should come as no surprise considering they live together in the Shift Run Stop house, which is like the Big Breakfast house but with less Paula Yates and more Dave Green.

A week! I admit it’s been kind of a crazy week, even more so than usual, which is why I haven’t gotten round to recording (lightly) last Friday. And really there’s not much to do other than drop a whole bunch of photos in one after the other. Anyway, I’m now heading down to London every other Friday and to get there in time for breakfast, it’s starts at around, well here…

5:07

Catching the train from Shrewsbury to Wolverhampton.

6:00

Standing around at Wolverhampton looking for the London train, always a bit weird this one as I’m generally booked into the Quiet coach which tends to be at one end of the train or the other. Which involves a walk down to the end of the platform but it’s never possible to tell which end until it arrives, sometimes I worry that the doors will close before I even reach the end of the train.

Stupid long trains!

8:19

Euston – Underground – The Shepherdess for breakfast

9:00

BERG, getting a sneak peek at the new office space, which is lovely and full off things that could end up in the background of photos that aught not to, so best not to risk it and just take photos of the sign instead.

9:35

Finally, nearly five hours after getting up, made it into work. This is my Friday from now on (well every other Friday anyway).

13:01

Too busy to take photos or eat until now. The downside of not being in the office that much, is that when you are your time tends to feel up with meetings pretty quickly.

Clearly cake was in order.

15:36

Inexplicably at the British Museum, as you do.

16:47

Back in the office just as the sun is setting, always really hard to get a photo of the sunset as the whole office is lit up and reflects in the glass as you shoot out, grrrr

19:11

Back to Euston to get the train home, time for some podcasts

20:47

Connection at Birmingham New Street, just enough time to play the which platform is my train going to be on this time game.

22:11

Back into Shrewsbury station, here’s the fire alarm system.

22:29

Finally home, 18 hours later, making it somehow a theoretical 8 hour work day, with 10 hours of faffing around getting to work and back. But in this case some of that “faffing” is having breakfast and talking to people, and hanging around a bit to get a cheaper train on the return home.

sleep.

 

And so the final part of the Instagram trilogy, that of Social Graph, register and where’s my sodding profile page?!

Instagram has two interesting features, no privacy controls and no straightforward website or profile page. The latter means it essentially lives on your phone (although web & desktop API driven clients exist), at a pinch there’s a web presence where you can link to an image but it’s not really designed to be used like that. The lack of privacy controls, also by design, is just keeping things as simple as possible. Instagram is pretty much all about getting the photo from the camera into the ‘stream’ as quickly and simply as possible. Which is why I like it.

But not having a “website” for it feels odd for a web-native (rather than a mobile-native) like me.

A while back Maciej wrote an interesting essey on the Social Graph, the way in which different sites enforce relationships with other people, I’ll get back to that in a second though.

First, Privacy…

The whole essay Maciej wrote is worth a read and will probably give you a better background for where I’m coming from. But here’s the pull quotes…

“The funny thing is, no one’s really hiding the secret of how to make awesome online communities. Give people something cool to do and a way to talk to each other, moderate a little bit, and your job is done. Games like Eve Online or WoW have developed entire economies on top of what’s basically a message board. MetaFilter, Reddit, LiveJournal and SA all started with a couple of buttons and a textfield and have produced some fascinating subcultures. And maybe the purest (!) example is 4chan, a Lord of the Flies community that invents all the stuff you end up sharing elsewhere: image macros, copypasta, rage comics, the lolrus. The data model for 4chan is three fields long – image, timestamp, text.”

Instagram is that, something cool to do (taking photos), a way to talk to each other (by following someone & leaving comments) and three fields (image, timestamp, title).

While other networks force you into odd constructs of your social network, who are friends, close friends, family, co-workers, maybe they’re dead, you’ve split up and so on. And the more edge-cases are found the more fields engineers put into databases to account for them (or indexes) until you end up with Google+ or Facebook. Maciej continues…

“Now tell me one bit of original culture that’s ever come out of Facebook.”

There’s none of this in Instagram, however privacy (for me) at its very basic level works like this; because the App exists in the phone, only other people with the App on their phone who have then chosen to follow me will see my photos.

I know this is a lie: http://statigr.am/revdancatt

I know really my photos are public (in the same way I assume all my tweets are public, even DMs), but generally people aren’t going to stumble over them.

But this effects the Flickr/Instagram difference. With Flickr I’ll maybe take a whole bunch of photos at an event (recently Christmas) then upload maybe 12-20 of my favourite. This isn’t a problem because that’s what it’s for, uploading and sharing photos, and by its nature it doesn’t shove all 20 of those photos down my followers’ throats. This is good, I like it.

I wouldn’t however do that with Instagram, the register of conversation is wrong for that. People, or at least those I follow tend to upload 1 or 2 photos of what they are currently doing, I don’t think I’ve seen anyone produce a flood of photos. In return I don’t upload 20 photos in one go as I would with Flickr.

My behaviour on Instagram is different because the society constructed there, by my friends, appears to have defined the common rules by which we’ll play in this Instagram space.

This difference is down pretty much purely to the presentation of the ‘your contacts’ photos, one a constant stream that is the interface, and one that has been much discussed here: http://blog.timoni.org

I don’t for the record agree with everything stated there but the comments are an interesting read.

That’s one of the main reasons my use of Flickr & Instagram is different.

But back to the Social Graph...

This is how I use Instagram, I suspect its probably not like this for most people.

When you upload a photo with Instagram you can very quickly chose other services to send the photo to, this isn’t an extra step its on the same page as adding the title and telling the thing to fly – be free.

I have 3 services set up;

  1. Flickr
  2. Twitter
  3. Facebook

I already have 3 different Social Graphs on 3 different networks, Instagram doesn’t need to replicate that Social Graph yet again, thank goodness. But I’m free to use the ones I have.

I said yesterday that I wasn’t worried too much about the apparently temporal nature of Instagram photos. That isn’t strictly true, if I take a shot I particularly like and want to be able to easily find again in the future (or it’s a good one where all three kids happen to be smiling at the same time!) I send it off to Flickr. By doing so I’m essentially also sending it to my relatives as that’s where they go to find out what I’m up-to and look at photos of the kids smiling.

I also have two pretty distinct groups of friends, Facebook friends who tend to be less ‘techie’ are mainly friends from back at Uni. Twitter are my techie friends plus a whole bunch of people who’ve chosen to follow me for whatever reasons they have.

Some Instagram photos I also send off to Facebook when I think those friends will find it interesting. Other photos I’ll send to Twitter, which are similar to those that stay on Instagram but I think are of interest to a wider ‘audience’ (for want of a better word) or support or bring context to something I’ve been tweeting about anyway.

This way I can control the register of photo conversation, some are for family and some for friends who don’t use Instagram, some are special and I want to keep and others are ‘private’ between myself and those who also play the Instagram game and follow me.

Again I understand this is somewhat of an illusion, although one that seems to work as a loosely connected web of services and contexts, without the need for database enforced friends & contacts stuff.

It’s also why I don’t think there should be only one photo service to rule them all, without all the other Social Networks Instagram possible wouldn’t work for me. Since they exists though and Instagram intergrates them so seamlesly I’m free to use them (and it) as suits me.

And I think that brings me pretty much to the end of this unexpected Instagram trilogy and all I want to write on the Flickr/Instagram subject for the moment. Or maybe I’ll convert the whole thing to slides and turn it into a 10 minute talk or something ;)

[Previously: Instagram and Flickr, the one where I refine my argument]

So, part 2 of a trilogy of posts it seems, and a response to the responses. Yesterday pulled up a couple of view-points about Flickr and “proper” photography, I’m going to pull the same whole damn quote from Aaron that Paul did (sorry Paul)…

“We did a lot of stuff wrong during my time at Flickr but if I had to highlight one thing we fucked up it was somehow creating an environment where people started to believe that their photos were not good enough for Flickr. I mean, really, how did we ever let that happen? I was speechless the first time a friend said that me and for the record: It was never part of the plan. How did we ever let people think that there is one measure of photography? How did we let people imagine that a medium which gave the world both Ansel Adams and Garry Winogrand (a photographer who died with a reported 10, 000 rolls of undeveloped film in his studio and who said that every time you take a picture you are hopefully risking failure) and everyone else in between was about any other than the joy and the discovery of the possible, foofy equipment and technique and measures of “good”-iness be damned?”

/via http://notes.husk.org originally from http://www.aaronland.info. And the line from Dan W, that Paul also pulls (sorry Paul)…

“Why I never got into flickr: it’s for ‘proper’ photographers only and my friends weren’t on there.”

In an attempt at originality I’m going to pull an additional quote from Aaron from the same post, which was actually totally about something else and so is utterly out of context here (sorry Aaron)…

“We are hard-wired for photos or more precisely still images (I’ve got six thousand years worth of claims that “painting is dead” to back up that outrageous claim, by the way) and they come with enough emotional baggage and social triggers that they deserve to be treated with care and thought. At the very least they deserve to be guaranteed some degree of permanence.”

…emphasis mine.

I’m not, for the record switching to Instagram from Flickr, I have room for both, yesterday’s post was about how my focus on seeing what my friends were upto had shifted from one to the other. To mangle someone else’s phrase into my own:

Flickr is for the story I want to remember, Instagram is for the story I want to tell now.

To me Flickr isn’t just a dumping ground for images, it’s a narrative, something I want to look back at to remember and remind myself what was happening. And for that to work I attempt to keep roughly to correct chronological order. Because these memories are important to me, and I want them to be treated with care and thought, I got myself a decent camera that can handle shooting fast moving kids in dark badly lit rooms (I don’t intentionally keep kids in dark badly lit rooms, but rather the reality of English weather and energy saving bulbs). I take quite a few photos with it. The trouble is that it can take a few days to find the time to sort through all those “proper” photos, I’m pretty sure I’m not the only one who feels like they have a backlog of photos they want to upload. It isn’t that I can only upload “Proper” photos to Flickr, rather that Flickr is the only place for me to upload them, and while I have a backlog I want to avoid posting quickly snapped photos out of order, although sometimes I do anyway.

Instagram is the narrative of now, I don’t feel particularly precious about them, I don’t tend to go back and look through old photos. It doesn’t matter that the shots I shoot with it are of lower resolution with a cameraphone lens, that they have filters, these are my transient photos (I understand this isn’t the case for everyone and Instamatic photos can be an art). I don’t need a central place where I can always find them, instead it’s a pooled visual stream of consciousness of myself and the people I know.

The problem for me never was that Flickr was for “Proper” photos, but rather it never exchanged the immediacy of the moment with friends particularly well (see the Contacts Page passim). But for years it was the best damn thing and that’s where the conversations of “now” revolved. Instagram is just that though; Insta, in a way Flickr isn’t. But Flickr is “proper” in a way Instagram isn’t.

It’s not that I’m going to use one over the other, that’s crazy talk, it’s that one was more magical in the moment of Christmas morning in a way the other wasn’t. I would hope that Christmas 2011 is more magical on Flickr in 10 years time though :)

[Update: Flickr, Instagram, the Social Graph and Interfaces effecting behaviour]
[Previously: My first Instagram Christmas, a nervous step away from Flickr]

A curious thing happened this holiday season, I kinda fell in love with Instagram just a little bit, and out of love with Flickr, just a little bit. Which, given that I used to work at Flickr way back in the day is an odd feeling.

When Instagram first came out I scoffed at it, filters, borders, faux retro and all that, me, I was sticking with Flickr. Flickr had all my friends on, it’s where the comments and conversations took place, everyone was over at Flickr, I’d invested years into it so why switch (or augment it with other tools)?

And then Instagram rolled out version 2, allowing much larger images to be stored and the ability to get rid of the sodding borders. People I follow on twitter started mentioning good things about it, I of course ignored them, ’cause Flickr is where it’s at don’t cha know! Then I caved in and downloaded it.

Suddenly I discovered that most of my Flickr friends were already on it, and they were having conversations on Instagram, sharing jokes, punning photos around hash-tags, generally having a good time, even though it wasn’t Flickr. Even though they were posting the photos to Flickr, the party it seems had mainly moved to Instagram.

It’s like they’d all been having photo fun behind my back… bastards!

How dare they be using something else while I was faithfully still using Flickr.

So ignoring that fact that I’d help build Flickr I started snapping away with Instagram, it was easy, frictionless, and dare I say it, I even liked some of the filters. Everything that the Flickr iPhone app wasn’t.

Then Christmas happened…

The days before Christmas, friends were sharing photos of the build-up, putting up the tree, wrapping presents, sitting on trains getting to parents houses, cooking hams. To me there was a real sense of flow, connection, joining in of everyone’s experience, a bit like a pictorial version of twitter to some degree.

Christmas morning was almost magical.

The previous year on Flickr was almost as magical, once people had a chance to sort through the 100s of photos they’d taken with their dSLR, pick out the best ones, run them through lightroom and then get a chance to upload them. It was nice to look back on the Christmas mornings that people had, rather than having.

Don’t get me wrong, I still enjoy Flickr. It makes me think about photography, inspires possible projects to play with, upload proper photos taken with my proper camera. But when there’s something happening (often involving the kids, a cat or a visual joke I know my friends will get) I find myself reaching for the iPhone and uploading to Instagram.

This Christmas I kinda fell in love with Instagram just a little bit, and out of love with Flickr, just a little bit.

[Update: Instagram and Flickr, the one where I refine my argument]

SecondScreen

I’ve been playing with GitHub Hosted Pages, by porting whatisthelatestguardianheadline.com over to pure JavaScript and setting it up to run on GitHub. You can see the end result (maybe, I’ve only tested it on Chrome) over here…

http://revdancatt.github.com/whatisthelatestguardianheadline/

…and the GitHub page for it is here…

https://github.com/revdancatt/whatisthelatestguardianheadline

…should you want to pick apart my code, or even better make it work on things that aren’t Chrome.

The GitHub Hosted Pages Part

I didn’t realise until just the other day that you could actually host pages on GitHub. I’d seen it done a couple of time but never really paid attention until Michael Brunton-Spall posted his HTML5 + JavaScript slides fork.

I often (well often-ish) throw together quick JavaScript only hacks to play around and end up throwing them up onto a Google AppEngine instance for no particularly good reason other than habit, which of course burns up an instance quota (for various tedious reason I don’t currently self host).

Now, by making a magic branch called gh-pages hosting on GitHub is easy. Setup an index.html file, stick it into your gh-pages branch and (after a short while) there it is being hosted. Here’s the GitHub page for it with the normal folders and files for a minimal webpage.

Github

The nice thing is that I was planning to put more of my javascript (and processingjs) experiments on GitHub anyway, now I get the hosting with it too (or the other way round, by hosting on GitHub I have versioning thrown in). Having version control and hosting that tightly coupled makes me very happy for some reason.

The Cloud9IDE coding part

Now, right off the bat, I’m not really an IDE kinda guy, I like my colour syntax highlighting but I’m not that bothered about all the magic autocompletion stuff, a simple (colourful) text editor is pretty much all I need. With that in mind I gave Cloud9IDE a spin.

It’s just what you’d expect, a text editor in a browser with all the files held on the server rather than your local machine. The magic part comes when you connect it to GitHub, any of your project can be cloned into Cloud9 and then edited, here’s my project in cloud9…

Cloud9IDE

By connecting the IDE to GitHub I can pull over any project I like, work on it, and then push it back over when I’m ready. There’s a Preview option which launches the current page in a browser and because I’m dealing with just JavaScript this is fine.

Creating a new project is easy, go over to GitHub, make a new repository, pull it into the Cloud9IDE and start bashing away. Not a single file hits the local machine, all the files are *handy wavy* over there.

Obviously this pretty much only works for anything that only needs JavaScript and doesn’t need API keys you want to keep hidden, but that’s good enough for me*

I like the idea that I can sit down at any machine with a modern browser, login and start working on my files. Again the tight integration with GitHub and Gloud9′s own collaboration tools make it feel like a good way of working. It makes me feel more like I wish all the project I’m working on could work this way.

Problems and general feelings of woolliness

I have 3 general problems with this setup.

#1

The GitHub integration once you’ve done the initial clone is all (browser based) command line stuff. So I’m still typing ‘git commit -am "some message"‘ and switching branches by hand. I’m used to using GitBox and Kaleidoscope for all my Git and Diff needs, because I like a bit of luxury now and then.

It’d be great if this was more UI based rather than jumping down to type wordy stuff in.

#2

Image upload, or any binary file uploading. You can do it via commands from the annoying virtual command line but it’s a big faff. I ended up (for a different project) pulling the project down to my local machine, adding the images and pushing it back up. Which kind of defeats the whole it’s-all-in-the-cloudness about it.

Both points above are on the Cloud9IDE roadmap so I’m not that worried about it.

#3

What I am worried about though is the whole losing edits and changes bit. Locally the files I’m working on sit within my Dropbox folder. So every time I hit save, which I do often, up they go to Dropbox where they are safe, or at least safer than just on my machine.

If my machine were to die, I could just pop over to another one of mine and pickup from where I left off. In the worst case I’d need to install Dropbox on a new machine and grab the files.

With Cloud9IDE I’m only as safe as my last push command, if I spent the afternoon working, previewing and committing, but not pushing and cloud9IDE went away, I’m only as good as my last version on GitHub.

I’m not yet sure quite how I feel about that.

 

*Well you can do NodeJS stuff and stick things in a private account, but where’s the fun in that?

FRSTEE the Twitter Snowman

I was going to attempt to construct a blog post about the shear cleverness of the little fellow pictured above, FRSTEE the Twitter Snowman. But it turns out Bobbie has already done a pretty good job of that over at Giga Om: Meet Frstee, the snowman built from Twitter.

The very short version is that the Really Interesting Group set-up a site (http://www.frstee.com/) where you can enter your (or someone else’s) twitter name, and via the magic of 3D printers get a snowman sent to you where the head represents the number of people following you, a button down the front for each year on Twitter and your twitter name on the base.

Instead, I’ll just write a quick blog post about how damn cute he is. And in particular how pleased I am that I joined Twitter 5 years ago and have just the right number of followers to make my frstee nicely proportioned ;) (all those years of culling spam-bot followers has finally paid off).

And when I say “cute” I kind of mean “cute and small”. The smallness isn’t really a problem, it’s worth pointing out that sometimes people are surprised by how much such a small thing costs. I’ve seen this a couple of times with My Robot Nation, a site where to get to design a robot who then gets 3D printed and sent to you & FigurePrints for World of Warcraft. Each time you’re paying not a trivial amount of cash for a figure a few inches high. Robots and Warriors are supposed to be BIG.

But in the case of FRSTEE this happens to be a strength rather than weakness as it’s supposed to hang from your Christmass/Yule tree and is the perfect size for doing so. There’s no slight feeling of sadness of it being small, but rather a that’ll-fit-nicely-on-the-tree-ness about it.

Oh, I also wanted to write this blog post to attempt to coin(ish) the word Ornametrics on the grounds that I couldn’t get Decoration and Data to mash together quite so well (Datoration sounds a bit rubbish). Instead here’s something ornamental that has some numbers behind it. And frankly I’m looking forward to seeing more Ornametrics appear in day to day life as the technology gets cheaper and faster. Maybe something that prints out each morning, and then you feed it back into the machine just before going to bed, so it can be broken down, re-absorbed and reprinted ready for when you wake up again.

[Filed under too long for a quick tweet, unless that tweet was Playlistify is awesome]

So a while ago last.fm removed the ability to play just your loved tracks. This made me sad because I rather liked listening to my loved tracks, what with loving them and all. They’re over here…

http://www.last.fm/user/RevDanCatt/library/loved

…all 430 of them. The first one Tips For Teens by Sparks both loved and banned on the 9th Nov 2005 – I was testing out what loving and banning a track did. Which I guess means the first real loved track was Night of the Swallow by Kate Bush, aaaaand I see that I just missed my 6th last.fm birthday, ho hum.

Anyway, over in the left corner last.fm (good for discovering new music), over in the right Spotify (good for listening to music that, errr, you already know you want to listen to) with playlists letting you listen to the same damn track over and over and over again.

Somehow, there must be a way to get loved tracks from over there to over umm, there.

Playlistify!!!

Discovered today, by googling for “How to play my loved tracks from last fm on Spotify”. I know, I’m a genius!

The site is ugly, just about functional and clever. You stick your last.fm username in at one end and out the other pops a Spotify playlist. It takes a while to crunch thru all the tracks, checking them one at a time to see if they exists on Spotify. Out of 430 loved tracks 337 came out the other side, final playlist is…

http://open.spotify.com/user/revdancatt/playlist/21zQPYCBbK1VuXTnepSc9H

…that’s 6 years of music listening distilled down to 24 hours of bestest best tracks. Of course when I say “best” I mean Depeche Mode, Kate Bush and other stuff from the 80s.

And then it all goes onto the phone incase I get stuck on a plane or train for a day (surprisingly plausible now it seems). Yay, faffing around with online services, much easier than actually owning music ;)

Latertodayguardian appspot com(screenshot from: http://latertodayguardian.appspot.com/)

Dan Roberts has been running an experiment over on the Guardian for the past two weeks, and it’s an experiment I wholely approve of, the Open News Lists. The idea being that we have our Open Platform API which pushes out the news we’ve already posted to the website, but this is displaying what stories the Guardian is thinking about and publishing soon.

I like to think of it as open sourcing the Guardian, even if not technically correct its a good philosophical standpoint.

The idea of seeing what a newspaper is planning (along with the chance to get involved) is I think, pretty exciting. It’s also a position that’s been made more understandable by the explosion in Twitter. You often see the comment that people saw the news on Twitter first and somehow this means that news orginisations are too slow or don’t respond quickly enough. The obvious truth is more along the lines of checking facts & trying to explain the background for context more than 140 characters will allow, which of course takes time.

An Open News List is an interesting hybrid, it’s an acknowledgement that we know something has happened or are aware of it coming up and encouraging a discussion to take place on twitter while the editors and reports put the piece together, including that feedback and those tips from twitter.

The experiment was quickly put together using the tools at hand, trying things out at it’s best! Google Spreadsheets and it looks like this

Open newslist | News | The Guardian

Which is great’n'all for the raw data-ness of it, but possible not that easy to read or more importantly engage with. I figured it’s be good to have the data presented in a more approachable fashion. But being busy I did what any good developer does, build a JSON endpoint for the data and hope that someone else does all the hard work ;)

A couple hours later with Google AppEngine and an endpoint is born:

http://opennewstracker.appspot.com/api/gu.get.newslist

The output looks roughly like this…

Opennewstracker appspot com api gu get newslist

…the same data from the spreadsheet but turned into a format that’s really easy for computer programming languages to understand and handle. In short, it’s still not pretty or good for humans to read, but a really useful mid-point between the source spreadsheets and data that can more easily be turned into something human readable.

Chris Thorpe took the JSON endpoint, worked a bunch of coding magic and turned it into the later today guardian, from which the top screenshot is taken…

“It hauls in the newslists using the very nice JSON feed created by Dan Catt and it then makes a column layout. One thing is it gives you a chance to interact with the journalists. If the story is in the “help” phase you can send a tweet tagged with #opennews to the journalist using a Twitter web intent

This is now the part where I wrap up with comments about how this reflects the distributed network of ideas, journalists, editors, developers & users and an example of small pieces loosely joined. Along with the stressing of the “It’s all very experimental” part, Dan Roberts is experimenting with open news lists in the simplest quickest form possible, I created an experimental scraper to convert it to JSON and Chris created an experimental version 0.1 webpage that formats the JSON.

But all those rough experimental bits put together is pretty good, none of us had to do the whole thing from start to finish and no-one was in-charge & orchestrating the whole thing.

More about the Open News Lists:

This Week in Review: An open-newsroom experiment, and News Corp.’s troubles spread to the WSJ
Memo to newspapers: Let your readers inside the wall
Guardian to publish daily newslist online
Guardian: Lessons from our open news trial
Is An Open News Approach The Way To Go?
Guardian to Readers: Here’s Our Upcoming Stories ListLessons from our open news trial

*DayNotes, otherwise known as a ‘blogpost’ I believe.

Outside the Guardian at Kings Place

So today has been mainly organising Google AppEngine Applications. As anyone who’s dealing with AppEngine is aware the billing method is changing… and slightly radically at that.

Before the change we were getting charged for CPU usage per hour, incoming and outgoing bandwidth, storage and so on. Which was great as you could just create a new App to do something trivial and not worry about it too much. There was also this great philosophy behind it, which was roughly… build something and we’ll scale it for you… even it it gets slashdotted or whatever we’ll scale up to meet the demand, worry about backend infrastructure no more!

From a coding point of view this is great, I don’t need to worry about what’s actually running the code, it’s all “cloudy”. I do have to worry about how I code my app as some tasks “cost” more CPU cycles than others.

A general rule of thumb was; if it serves the data faster, the CPU cost is greater, so a database hit ‘costs’ less CPUs but took longer to serve up the results to the end user. A memcache hit served up the data faster, but cost more CPU cycles. You could end up in the weird situation of a page serving in 500ms but the CPU cost used to deliver that page could be in the order of 2500ms.

You were basically picking a point somewhere between serving fast or costing less. For applications that poll for information and sometimes do something with the result, serving quickly wasn’t much of a problem and therefore even easier to keep the price down. I’m mainly thinking Twitter Bots here.

Now it’s all changed, the change makes sense but rather goes away from the original core philosophy. We’re now charged for “instance hours”, it goes somewhat like this…

If a request comes in, an “instance” is spun up to service that request, the instance will that stick around for 15 minutes ready to serve any new requests. The act of spinning up an instance takes some so having it ready and waiting is theoretically good.

The down side is if you have a task that runs once every 10 minutes to check something, that instance stays alive forever, it never has a chance to die.

In one case I’m actually running an instance 24/7 for a total of 4.6mins CPU time.

I understand what they’re doing and it makes sense, but at the same time I’m now no-longer care-free of how my app works and happy just to let Google scale it, I’m now managing ‘instances’. Even though that ‘instance’ may be spread over several CPUs in various locations, meaning I don’t have to worry about maintaining a specific box. I am now thinking in terms of running my code on x number of servers, which is totally what I didn’t want to be thinking about.

If I wanted to be in that mindset I could rent a server, or use one or more Amazon EC2 instances. This, AppEngine, wasn’t supposed to be that.

Anyway…

I have several trivial AppEngine Apps running along quite happily, each using up a small handful of CPU cycles, but each keeping a whole ‘instance’ alive. The answer seems to be to put them all into one singlemeta App with a small amount of management, allowing them to all run off one ‘instance’ rather than several.

It’s not the end of the world and I’m certainly not as much up in arms about it as some others. But it is a shame to be thinking in terms of ‘boxes’ and ‘instances’ rather than just CPU cycles, bandwidth and DB reads/writes and I think AppEngine has lost some of its charm for that.

Addendum

There are some technical problems with the new Billing though. Above I talked about a single instance staying alive to serve any request… having an instance alive makes perfect sense and indeed the new billing model allows you to have 24 hours of instance running for free, you only pay for more than one instance.

Which sounds (more than) fair, keep everything running on just one box and then it’s free.

However, AppEngine feels free to spin up extra idle instances if it thinks you’re going to need them, if left to its own devices it’ll spin up several instances and keep them there. You don’t get charged for any that stay idle for over 15 minutes, Google just hasn’t got round to killing them yet. But you do get charged for the first 15mins that it’s there just-in-case it gets used.

On an App by App basis you can tell it how many idle instances (idle instances are the number above the ‘main’ instance) you want, in this case the lowest number is 1. But I’ve ended up with my main instance up and  running for 24 hours and then Google firing up the extra now and then, causing me to use 27 hours of instance time in a 24 hours window, even though the main instances had more than enough capacity to handle the minimal requests.

So no matter how careful I am in writing my application, Google can just fire up an extra instance for no accountable reason and then charge be for it :/

Older Posts »