Ah Weeknotes how I have neglected you :)

Two things appear to go hand in hand; being too busy to write and the stuff I have to write about being not very interesting. This is no great surprise, the hard slogs aren’t much fun for reading, writing or doing, while the gaps and spaces between the ‘hard stuff’ is where interesting things happen.

Although, to tell the truth I quite like the hard slogs too.

Anyway, last week was going swimmingly (more on that later) until my desktop iMac died. The space taken up by the laptop on the photo above, normally there’s an iMac there :( However, due to modern computer wonderfulness, that I’ll list in a second, this wasn’t as horrific as it could have been. After smacking the iMac around a bit, resetting it’s PRAM or whatever the troubleshooting told me to do, I shoved it into it’s box and chucked the laptop onto a stand.

Things that saved my coding ass, in order of probably importance:

Dropbox

This one probably goes without saying, but for someone who doesn’t really trust Time Machine and hasn’t found Dolly Drive or other back-up to the cloud with Time Machine things to work with FileVaulted drives, Dropbox is teh awesume.

I have overnight offsite backup anyway (Mozy) but Dropbox just works for the I-hit-Save-and-off-it-goes-to-Dropbox. And I tend to hit save a lot when I’m working for various reasons.

Which of course means the moment I hit save on the desktop iMac, the MacBook immediately downloads it. So when the iMac died, the MacBook had everything it needed, woot!

GitHub + GitBox

Obviously Dropbox isn’t really a source control/backup solution, it’s just really really handy to have around. All the projects I work on end up on GitHub, some public and many private. I tend to commit and push whenever I’ve finished a small “atom” of work. “Atoms” tend to range from 15mins to a couple of hours of work. So without DropBox I’d be loosing around that much work, not terrible but just enough to be annoying.

GitBox is just me being plain old lazy and getting fed up of the command line, yeah I’m not 1337 anymore, whatever. Nice shiny Mac interface to Git, yeah! Shows me which files have changed and lets me easily add commit messages, totally.

It also integrates with Kaleidoscope ’cause I’m lazy about diffs too…

Kaleidoscope

This is possibly overkill, if you consider that you get FileMerge as part of XCode and Git or the command line or something has diff somewhere, but as mentioned above as I get old and lazy I like my tools to be nice looking and easy to use :)

Anyway, this is how Kaleidoscope fits into my workflow, I like to make small changes often and deploy them as soon as possible. This means that if anything goes wrong it’s easy to spot and easy to roll-back. Rather than making big releases where most things work and one thing doesn’t, giving you the choice between rolling the whole lot back or fixing the one thing.

Unit tests are probably supposed to fix that, but honestly do you know anywhere that’s implemented unit tests and now doesn’t have any bugs in their bug tracker what-so-ever because of it, nope, didn’t think so.

So, right. I make my changes, test them locally, look at GitBox to grock which files have been changed. And then Kaleidoscope to sanity check what’s actually changing where. Really good for spotting when you’ve left a sys.exit() in there by mistake and so on.

In theory you don’t need this step, in reality I find it a handy sanity check.

Google AppEngine

Google AppEngine, Python, hosted. Code is up “there” somewhere as well as “down here”. I think there’s now even an option to download the source, which of course we won’t need to to because it’s all in GitHub, Dropbox and over night backups in Mozy.

Anyhow, most running code I deal with is up “there” now, it’s been a long time since the code was complied and run on the desktop :)

The App Store

Finally, tiny mention for the App Store, some people may not like it, but I certainly do. Even more the sooner all the apps I use move into the App Store the better. There’s simply not much better (from my point of view) than having a single place where all the apps I’ve bought and downloaded are stored.

As machines get old and slow, and I upgrade to faster laptops and so on, it’s kinda nice to know I can log into the App Store, click the “Purchased” tab and install all my tools.

Meanwhile…

I’m probably as sick as writing this as anyone who reads this is of reading it. The grand Zeitgeist switch over, part 2…

The main Zeitgeist page is now running off the new backend, just 1 more item to move off and the old one is history.

I gave the front-end a quick update while I was about it, story titles moves to the bottom of each “tile” and the colours in the CSS are generated from an admin-panel rather then hard coded into the page. Which means when someone adds a new section they can defined the colours there, rather than me having to open up a file to edit.

Little things but makes a difference.

Weeks since I last wrote a Week Notes: 3
Night that I wrote this blogpost: Sunday
Why: Because I’m excited about the stuff I’m doing on Monday and don’t want this to get in the way.

Length of time I forgot to “live” this post: Most of the day.

Filed under: guardian, weeknotes by Reverend Dan Catt

Here’s a Monday morning update to the previous post about the Kaiser Chiefs’ internet album project. Plus some other bits and bobs after the table below.

In summary, Problem Solved is still the most popular track and Things Change is least popular getting put on around half as many albums. Which shows there is some sort of aesthetic decision making going on. The number of albums created and sold doubled over the weekend.

Total albums created and sold: ~2,500 (as of Monday morning 11pm)

And with bars!

But wait, there’s more!

So as I mentioned before, the code that monitors albums being created is up and running, sorting the tracks to create the Most Statistically Correct Kaiser Chiefs album. With tongue in cheek I suggested that by using Maths and programming we could come up with the “best” album.

However, as we all know there’s more to albums that that. You can’t just open the album with the best track and then progressively have the less popular ones. If the KC’s new album had all 20 tracks ordered as shown above, you’d finish on officially the worst track on the album, and that doesn’t seem like a good idea.

Albums are about the narrative, the structure. Some bands have gone as far as saying they don’t want the whole iTunes thing where people can buy the odd track here or there. The want people to experience the whole package as they intended.

Here though, the Kaiser Chiefs are passing over that challenge to the user, who now gets to create their own package. It’s an interesting idea, the package is still important but the arrangement is up for grabs.

So here’s two other ways of coming up with Most Statistically Correct Kaiser Chiefs Albums.

#1 The What Goes Where Method

As well as just counting up the tracks put on albums, I’ve been recording where on the album the tracks have been put.

You can actually see all this data over here: Kaiser Chiefs Track Tracker

Although Problem Solved is the most popular track, Little Shocks is most likely to be the opener on peoples’ albums. So we could look at each track that’s most likely to appear in each position on the Album. If we do that we end up with this track listing instead…

1. Little Shocks
2. When All Is Quiet
3. Problem Solved
4. Fly On The Wall
5. Child Of The Jago
6. Back In December
7. Long Way From Celebrating
8. My Place Is Here
9. Dead Or In Serious Trouble
10. Man On Mars

As we can see Problem Solved appears third on the track, even though it’s more popular than both the tracks above it. Eight of the tracks are still from the top 10, but two of them Fly on the Wall (#4) and Long Way From Celebrating (#7) wouldn’t have appeared using our initial method.

And here’s yet another way…

#2 The What Follows What Method

Another consideration when creating an album is which tracks work well together, I’ll just say Pink Floyd and leave it at that. Back over at the Kaiser Chiefs Track Tracker we have that information too.

If we picked the most popular track (Problem Solved) and then put the track that most often follows it, and so on and so on, we’ll end up with an album with the tracks on in the order than most people put add them. Lets give that a try…

1. Problem Solved
2. If You Will Have Me
3. Can’t Mind My Own Business
4. Cousin In The Bronx
5. Saying Something
6. Coming Up For Air
7. Heard It Break
8. Fly On The Wall
9. Things Change
10. I Dare You

This is a far different bunch to the other Most Statistically Correct albums, only 2 of the top 10 tracks are there; Problem Solved (because we started with that one) and Can’t Mind My Own Business. The remaining 8 tracks are all from the bottom 10.

There is however a problem with this method that I’ll get onto later.

So, there we go 3 different “Best” albums.

A Quick Analysis of the Project

The most obvious question, possibly even before which is the most popular track, is how many albums have they sold?

As of writing 2,345 albums have been created, some of which have been sold to more than just the original creator, around 156. Each album sells for £7.50, when they are sold a 2nd time, £1 goes to the seller, so they “make” £6.50 instead. Giving a grand total of around £19,000.

Here’s a quick breakdown of the three stages a user can get to when creating an album…

1. Listening to tracks and selecting them, making a cover but not created an account to save it.
2. The above but also creating an account but not buying it.
3. Actually clicking the Paypal button and buying it.

We have no real way to know how many people mess around with the album creation and go no further.

The second though we can figure out. When a user signs up it appears their album gets allocated an ID, but it only goes public when they buy it (yes I bought 3 Kaiser Chiefs albums testing this theory!) The earliest ID was 1,064 and the most recent (as of writing) is 4,333, maths-is-hard = 3,269 albums with signed up users.

2,345 of those went on to actually order the album, which involves having or setting up a paypal account. Making for a conversion rate of around 72% which sounds pretty impressive to me. The 72% has been pretty consistent over the weekend.

Is That Number High Or Low?

When people hear about things going “Viral” on the internet it’s often a YouTube video with 15 Million views. And so we start to get the impression that good/viral stuff gets millions of hits. Obviously other things like a Flash animation or some such can also get massive hits. However that number drops off sharply as you move out of the top tier of successes.

Twitter for example has gazbillions of users, with things trending and getting retweeted all over the place, again giving the impression of BIG numbers. But using sites such as Topsy.com we can see that it only take 8 retweets to end up in the top 1,000 tweets in the whole world. There’s a whole article which talks about how relatively few users it takes (1,270) to effect the “Most e-mailed” story list (out of the 50 million visitors per month) on the New York Times website. The people who actually take an action as opposed to just consuming is tiny, we should expect low numbers most of the time.

Given that it was launched pretty much over a weekend (the slowest internet time of the week) and that saving an album involves registering with the site, then I’d say that 3,269 albums created isn’t bad at all. Particularly as a) I’ve got some nice stats out of it and b) Simon Rix (bass player of the band) says this…

“hello. fact is we knew we wouldn’t sell millions this way. that wasn’t the interesting bit to us. we did it cause we wanted to do something different and we are enormously pleased with how it has gone so far. not comparing directly but even itunes took years to establish itself. thanks for the support though”

You can read a similar viewpoint from Ricky Wilson over here: From Ricky Wilson Of The Kaiser Chiefs

Personally I think the numbers and results are pretty much what I’d expect (i.e. quite good) and it’s an interesting a novel way of doing things. Something that possibly counts against it though is that you actually have to work to create an album. If you really wanted to sell digitally direct to the user then you’d make the whole buying process as simple as possible.

Which brings me to…

The Thing That I Don’t Think Worked…

Is the idea of selling the album you created onto other people.

I understand the logic, to help things go “Viral” you want people to tell other people about the whole project. And to do this you want to encourage tweets and facebooky stuff and so on. By setting up a system where a user get £1 for each copy they sell it seems as though a user would want to try and sell as many as possible. Thus helping the band shift albums.

If I make a frankly awesome selection of tracks, which I buy for £7.50 and then convince 8 other people to buy my version then I’ve gotten the album for free and made £0.50. If I push many more then I make even more money.

This would seem to make sense, you’re paying people to promote your stuff and giving them banners and web badges to do it. However there’s three problems…

1) Other people will want to make their own version rather than just accept yours, although this isn’t really a big deal.

2) People aren’t really that keen on selling to their friends or other people. While someone may be happy to recommend an album, as soon as money’s involved there’s an effect. Are the people I try and promote my album to going to think I genuinely like it, or that I’m trying to make money off them. Which also ties in with the 3rd problem…

“numerous scientific studies have shown, compensation typically decreases motivation to engage in activities we would otherwise freely enjoy”
Reality is Broken – Jane McGonigal

…when money is involved, getting paid for something, it turns into a job – it’s work rather than something you do for pleasure. While I’m happy to spread the love for a KC album I’ve just created, that addition of a single £1 suddenly turns it into something else, if I manage to get to 8 albums sold, I’ve broken even, if I can sell 10 more I’ve made a tenner.

See: “A Meta-Analytic Review of Experiments Examining the Effects of Extrinsic Rewards on Intrinsic Motivation.” (short version, work suck, hobbies are fun, until they become your job at which point they suck again – not me though I happen to love work :)

Out of all the albums, just 49 people have sold any additional copies. 37 out of those 49 extra-selling copies have only sold one additional album. The top 3 “resellers” have sold 69, 38 and 15 copies. The remaining 8 albums have sold around 3-6 copies.

Which seems to support the hypothesis that reselling an album isn’t that fun or interesting.

Going back to Ricky’s email…

But we wanted to try something a bit different with that too. Now here’s our favourite bit (also the most unbelievably complex thing to build).

Once you’ve bought your album, you can sell it on. For every album you sell, we’ll give you a pound (or a dollar in your case). We’re giving people posters, banner adverts, a website, Facebook tools etc. etc. All to help them sell these albums. If you sell 8 you make a profit.

So it looks like the most unbelievably complex thing to build is the least used and probably works in opposition to how it was intended. (note: not complex from a technical point, but from an accounting & rights point of view… money is generally supposed to go one way).

But… at the same time breaking new ground always costs money, so while I think it doesn’t appear to have paid off this time maybe the work done will make things easier and bigger in the future with a different approach to the re-selling part.

Finally, Things Wrong With My Methodology

When we look at the data for which track follows which other track, we can see that there’s always one that strongly out performs all the others. There’s a spike of around 350 “follows”, something that should set bells ringing for anyone dealing with data.

Do that many more people all think that a certain track should follow other tracks?

Well as it turns out there’s an explanation, which also explains to a less extent a bias in the track positions. Here’s the interface when you choose the tracks…

The objects on the bottom two shelves represent each track. The first object on the top shelf is for Little Shocks which despite not being the most popular track mainly appears in 1st position on the album. The track that most often follows Little Shocks is When All Is Quiet (which also occupies the 2nd track position on most albums and second position on the shelf). You can see where this is going.

The numbers look like around 10% of people put the tracks down in order, when looking at the albums that get converted into sales it appears that these people buy two albums to get all 20 tracks. The first 10 on one album the 2nd on the other.

To get better data for which tracks really follow other tracks, and which tracks get put into which position then we’d probably want to remove the albums that are just ordered 1-10 and 11-20. Or we can just look at the second highest values for similar (but not the same) results.

Another observation is that the tracks on the top shelf are generally added to albums more that those on the bottom shelf. Maybe people start listening and adding to those first. Or maybe the band put the tracks they thought were the best up on the top shelf and people are naturally picking them.

Interestingly though the top track Problem Solved is the 7th object on the top shelf which suggests that even though it’s not in pole position people are actually picking it out as the best track.

If you’re going to release a single, pick that one (or Little Shocks).

The least popular track is the two headed bear on the bottom shelf. I think the simple reason for this is that it sounds like a 80s song ;)

Addendum

I wrote this blog-post last night and updated the figures just now to make sure they were correct at time of publishing, I’ve noticed that since this morning that the rate at which albums are being created has increased somewhat, which is to be expect compared to the weekend. I’ll keep an eye on the numbers to see if anything super special happens or if it just carries on as normal.

Also, turns out I quite like the new Album, all 3 different versions of it I created. While I was working on the code to track them I had my first version of the album on repeat. The two later version while writing the blog post.

The concept is good and certainly interesting enough to make me spend a day hacking away at code to try and follow what was going on. The music being good, well that’s just a bonus.

I’ll probably try and see a Kaiser Chief concert if I get the opportunity.

Filed under: hack by Reverend Dan Catt

The Kaiser Chiefs have an interesting project over on their homepage. It invites fans (or people like me) to make their own version of the album to download.

Here’s mine and it’s absolutely perfect for reasons I’ll explain in a second…

Now I’m not the biggest fan of the KC but their tracks come up now and then on last.fm, so how to pick the tracks? Worse, you only get to chose 10 out of 20 tracks (unless you buy two albums, which I guess is the point to a degree).

Clearly this is a problem best solved with a computer program. Something that’ll count up the number of times each track has been selected by everyone else.

Cutting to the chase here’s the results so far, as of 10pm London time on Friday the 3rd June, the totals for tracks placed onto users’ albums. I’ll update these numbers over the next few days…

Total albums sold when I ran the code: 1,159 (10pm Friday night)

For those of a more visual nature…

The How and the Code

I needed a way to get all of the albums.

There’s a fairly unsatisfying (from a screen-scraping point of view) Album H.Q. page that lists the top-5 albums, but as we all know leader-boards suck. Now those are in the top 5 they’ll generally stay there, feedback loops and all that.

You can also search by username, but you need to know the full usernames, which also sucks.

Fortunately, they have a Latest section of the page, which is always a good candidate for scraping. Better still rolling over an album we can see that the albums are actually sequentially numbered. According to the website the first album was created by Alex and is number #1113

http://www.kaiserchiefs.com/album/1113

As of the time of writing that number was upto #2702

http://www.kaiserchiefs.com/album/2702

There’s a few albums missing between those numbers, but the grand total at this point is 1,159 sold (£8,695.50).

We can also use this to monitor the rate at which they sell the albums and how new album generation responds to social media experiments and so on. I will of course be tracking this :)

Anyway, I threw some code together in python to run locally on my desktop, it also uses Beautiful Soup to parse the HTML. You can find the code on github over here…

https://github.com/revdancatt/kaiserchiefstrackcounter

Feel free to fork/improve it, or use it to track the Kaiser Chiefs yourself.

I left it running (you need to keep an eye on it running off the end of available albums) for a short while to grab all the tracks and then put them into the spreadsheet stuff you can see above.

And then made the album. The Statistically Best Kaiser Chief Album. Which presumably gets slightly worse as you moved through the tracks. But at least you have the best 10 out of the bunch!

Go, Buy: http://kaiserchiefs.com/statisticallyso/2

Filed under: hack by Reverend Dan Catt

http://kottke.org/11/05/the-most-important-page-on-flickr:

The recent uploads by your contacts is the most important page on Flickr and it’s broken. Timoni West is a designer at Flickr and she wrote a brief post on that page’s problems.

The page fails on a fundamental level — it’s supposed to be where you find out what’s happened on Flickr while you were away. The current design, unfortunately, encourages random clicking, not informed exploration.

The page isn’t just outdated, it’s actively hurting Flickr, as members’ social graphs on the site become increasingly out of sync with real life. Old users forget to visit the site, new sign ups are never roped in, and Flickr, who increased member sign-ups substantially in 2010, will forego months of solid work when new members don’t come back.

Many of my friends have switched their photo activities to Instagram and, more recently, Mlkshk. And Flickr’s broken “what’s new from your friends” page is to blame. Both of those sites use a plain old one-page reverse-chronological view of your friends’ photos…just scroll back through to see what’s going on. The primary advantage of that view is that it tells a story. Ok, it’s a backwards story like Memento, but that kind of backwards story is one we’re increasingly adept at understanding. The Flickr recent uploads page doesn’t tell any stories.

As long as we’re talking about what’s wrong with Flickr — and the stories thing comes in here too — the site is attempting to occupy this weird middle ground in terms of how people use it. When Flickr first started, it was a social game around publishing photos. You uploaded photos to Flickr specifically to share them with friends and get a reaction out of them. As the service grew, Flickr became less of a place to do that and more of a place to put every single one of your photos, not just the ones you wanted friends to see. Flickr has become a shoebox under the bed instead of the door of the refrigerator or workplace bulletin board. And shoeboxes under beds aren’t so good for telling stories. A straight-up reverse-chron view of your friends’ recent photos probably wouldn’t even work on Flickr at this point…you don’t want all 150 photos from your aunt’s trip to Kansas City clogging up the works. Instagram and Mlkshk don’t have this problem as much, if at all.

Yet.

This post is licensed under the Attribution-ShareAlike 2.0 Creative Commons license

Filed under: flickr, general by Reverend Dan Catt

A short while ago alongside Meg Pickard I spoke at FutureEverything, we wanted to talk about the rolls of Editors, Robots, Strangers & Friends in the act of curation and overall online experience. However more about that some other time because I want to work through a small section of that talk, a bit I mentioned briefly about “Data Scientists”.

I don’t believe in them.

Data Science I don’t have a problem with, it’s the rise in “Data Scientist” jobs recently that I think are missing an important aspect, the dance, the rhythm. Example, read and view this, come back here once you’ve gotten over the urge to throw-up and stab things, I’ll wait…

Data Science & The Role of the Data Scientist

If you haven’t torn your own eyes out I’ll try to explain more.

Firstly Programmers (or Software Engineers if you like)… in a nutshell programming consists of Loops, if statements, Lists, Sorts, Ordered Database Queries, maybe some binary searches and a bunch of common sense. A variety of algorithms, message queues and endpoints if you want to get fancy but anything more and you’re generally overcomplicating the problem.

The one thing programmers didn’t have until recently, mainly because of limitations in computing was the ability to deal with large, huge, massive datasets easily. Data the size of a city or a country, highly populated dynamic data.

Projects such as MapReduce and Hadoop are now allowing that to happen. You get to throw a whole bunch of data into a distributed system and make it bend to your will using cunning and guile.

A Data Scientist basically appears to be a Programmer + Big Data + Statistics.

However it’s not something that generally needs to be done in our web world, organizations such as Facebook, Twitter, Nokia, FourSquare and so on who have the large data sets are the ones to gain the most from it. Anything smaller (which is most things) and you can carry on with your more traditional Loops, Sorts, Message Queues, DBs and indexes.

And because it’s fairly new it’s often these organizations that are inventing or contributing to the technologies used. See The Engineering Behind Twitter’s New Search Experience to get a good feel for that.

Which is why when another organization starts dealing with “Big Data” they advertise for a Data Scientist, but there’s just not that many people with the experience. I know many programmers, I know two maybe three people who can handle big data.

So is it as simple as being a programmer and getting Hadooped up to fill this market niche?

Nope and here’s why…

“in order to grasp and analyze rhythms, it is necessary to get outside them, but not completely: be it through illness or a technique. A certain exteriority enables the analytic intellect to function. However, to grasp a rhythm it is necessary to have been grasped by it; one must let oneself go, give oneself over, abandon oneself to its duration. Like in music and the learning of a language (in which one only really understand the meanings and connections when one comes to produce them, which is to say, to produce spoken rhythms).

“In order to grasp this fleeting object, which is not exactly an object, it is therefore necessary to situate oneself simultaneously inside and outside.”

- Henri Lefebvre from Rhythmanalysis: Space, Time and Everyday Life.

Lefebvre in his 1992 collection of essays talks about the rhythm of cities. To me this is the flow of the people, the morning coffee routine, the lunchtime decisions, the evening meandering, the beat of the bar on a Friday night, the sweat dripping off the ceiling of a tiny club, the sun coming up late on a Saturday night. Strangers exchanging stolen kisses under umbrellas, the race across the road in a gap in the traffic, the sudden surprising green park round the corner, the hidden entrance to the underground stations.

How people shape the city, the pulse as agents gather together to form a temporary autonomous zone before collapsing back to being shaped by the city. To be not just in the city, but of the city.

I’m not a fan of cities, I lived in one, San Francisco and avoid another where I work, London. I can’t design for cities as I don’t understand them.

Lefebvre’s writing suggests that to analyze a city you need to have been consum/mat/ed by it.

This to me is the same as Big Data. You can’t just turn your Data Scientist eye onto something and say “Oh we’ll throw this into MapReduce, it’ll be awesome”, you need to have been part of that data, to have lived it. We don’t have Big Data where I work at the Guardian, we have lots-of-data, we look at Big Data out there and attempt to consume the signals. I came from Flickr which had fairly big fast data, the Guardian is positively quaint in comparison (in terms of what it generates). I set myself the task of getting immersed in the flow of news, trying to understand how the organization worked, the signals, the input, the output. The difference between news on a Monday to news on a Friday, the waves that Google and other sites can throw at you and so on. Living in the data, watching its rhythms, the pulse, the flow. I’m getting there, it takes a while, maybe I’m just old :)

To deal with big data you have to have been in it, not a Scientist but as a Dancer. I would at this point direct the dear reader to the music track Chime by Orbital [spotify] but I fear it may be pushing the analogy a step to far.

But that isn’t enough, many people are already immersed in the data, here our journalists know all this stuff inside out. Getting carried away by the rhythms is as easy as getting in and letting yourself go. According to Lefebvre you then have to get back out again.

And that’s the trick…

“in order to grasp and analyze rhythms, it is necessary to get outside them, but not completely: be it through illness or a technique. A certain exteriority enables the analytic intellect to function.”

You can’t have someone who’s a “Data Scientist” just turn up and apply their tools, clusters and statistics. They haven’t been in-it enough. And you can’t have someone who’s within the company, who understands and feels the flow of data everyday, unless, unless they know how to separate themselves, to get outside. When people grow with a company, love the company, understand everything that company could be, getting outside it is a hard won skill. The “Scientist” needs to be able to remove themselves and apply clear analytical skill, but with the fundamental understanding of the subject.

So all those companies advertising for a Data Scientist, I think I have this to say…

1. You want a Dancer not a Scientist.
2. Good luck with that!

As for the future of (data driven) journalism…

“In order to grasp this fleeting object, which is not exactly an object, it is therefore necessary to situate oneself simultaneously inside and outside.”

The “fleeting object, which is not exactly an object” that’s your story. The flow of data will gather together now and then forming a tangible shape for you to spot and grasp before it collapses back into the stream. You have to be in it to understand it, and outside to spot it. Just one or the other wont do.

And for not-journalists, it’s your competitive advantage.

Further Reading

When the Data Scientist/Dancer has sufficiently honed their skill to identify useful shifts, patterns and rhythms in the data, they can then set up algorithms to spot these on their behalf.

To understand the usefulness of algorithms we should first fully understand Golems and Robots.

http://en.wikipedia.org/wiki/Golem
http://en.wikipedia.org/wiki/R.U.R._(Rossum%27s_Universal_Robots)

Filed under: api, general, guardian, hack, tools by Reverend Dan Catt