Last week The Royal Society put on display Robert Boyle's wish list of things that science might achieve in the future, written in the 1660s.
What did the acclaimed scientist want? "The Attaining Gigantick Dimensions." Blue sky thinking, far in advance of anything modern ideas people come up with. As the films would have it - "In an age before anesthesia, in a time of brutal civil war, when the black death stalked the land, one man had a dream - to be taller".
There's something about the spelling and the school-boy ambition of it which I love. When Gulliver's Travels was published 35 years after Boyle's death I hope the book was inspired by his desire for greater physical heft. Swift - pleasingly described by Will Self, who has himself toyed with scale in his stories, as satire's Shakespeare - and Boyle were fellow Anglo-Irish theologians.
But how big did Bobby Boyle want to be? Gigantick! Is that the size of a house? So big he had to be accommodated in a cathedral, as with Gulliver? Robert Hook, Boyle's scientific protege, was instrumental in the rebuilding of London after the great fire. Now we know why - he wanted St Paul's as a hanger for his newly upscaled friend to be birthed in.
Perhaps, again like the story, he wanted to be large enough to wade out into the sea and engage in naval warfare by overturning ships. After all, contemporaneously with the publication with the list many worried that the dutch were going to sail up the Thames and take tactical advantage of the recent fire.
I don't think pV = k, the gas law most school children know Boyle for, really does justice to his flights of imagination. Could we have a film from the "Honey I shrunk the kids" franchise to honor him? I think that more accurately reflects his spirit. It's what he would have wanted, when he wasn't pondering "The Emulating of Fish without Engines by Custome and Education". What a guy...
Thursday, June 17, 2010
Wednesday, June 16, 2010
Local by Social - Where sexy social media and bashful local government go to flirt
When you're a nerd like me you it's easy to think that a website is the solution to everything, so I try to remind myself that it probably isn't.
When I went to Local by Social yesterday I was determined to maintain a detached skepticism - either I was seduced by the confrency world of balancing cups and saucers of pump-action coffee while trying to avoid conversational lapses with people I've never met before (unlikely), or I need to be bit more optimistic. Only briefly did the event feel like pie-in-the-sky geekery.
Local by Social was a discussion of the ways in which local governments can utilise social media, taking an extraordinarily broad definition of that term. Topics ranged from the stratospheric stature of reports complied for the Havard Kennedy School of Government to the more prosaic (and eternal) question of how to get doubting officials to engage with social media.
From the more philosophical end of the spectrum I was surprised discover an aspect of the web which I've not really come across before. My assumption was that debate would all be around some variety of communication between government and people, probably something to do with eschewing the broadcast model and adopting a many-to-many, responsive approach to social media - a message we've all heard before.
But Local by Social was way ahead of me with the concept of "social innovation", or "Public Service 2.0" (I've never heard the "2.0" suffix used so much). The concept is to close the loop of Official-Public-Official conversation by having, to varying degrees, the public actually solve problems themselves.
Examples included Washington's "Snowageedeon", where citizens used Google maps to allocate the work of clearing snow between themselves when the authorities were overwhelmed and Brighton Council using Twitter to find volunteer van drivers for meals on wheels during another bout inclemency. Or Fix My Tweet, which allows you to tell the council where pot holes are. The difference here is that work that would once have been carried out by an authority is being done for free through social media magic.
Instead of hulking, snail-paced governmental organisations facilitating this processes we were asked to imagine social entrepreneurs setting up not-for-profits to delivering these types of service. The question of what would happen when someone tried to monetise their successful website was left hanging in the air. Being new to the world of e-Government I was surprised these stories don’t seem to be more prevalent in the general web-trend reporting press, but perhaps that’s because the sites I follow tend to come from the marketing angle.
Another theme was adversity as a catalyst. The above examples were not the only times that snow precipitated social innovation. But, as Dominic Campbell (founder f FutureGov) explained in literally Churchillean terms ("We will innovate on the beaches..."), the most important opportunity in adversity was the coming cuts in public spending. Referencing Schumpeter's creative destruction he pointed out that more than ever authorities are receptive to novel cost-saving ideas. Big stupid bureaucracies listening to exciting web start ups to good to be true? Well, if you listen to David Cameron's TED Talk and his Big Society rhetoric (which I willing to concede might be just slightly more than electioneering) then perhaps there is room for a chink of optimism.
Am I maintaining detached skepticism? I'm trying... honest. Here's a dose: Twitter doesn't seem to me like it's important for local councils. In a break-out session at the end of the day a lot of local government people were bemoaning the fact that councillors wouldn't Tweet, but I don't think they really need to.
It might be useful as a means of disseminating press releases to journalists, but that doesn't mean that you can garner the ear of the populous using Twitter. Only 30,000 Twitter users tweeted about the election on election night. That's one in 2,000 people, and a close run general election is a lot more interesting than local government.
Much more exciting to me is the possibility of local government making itself known on existing localised communities. Filippo Ciampini, who is writing a masters on the subject of government public relations, told me that there are lot's of ethnically based online communities and forums - citing Islington's Chinese forum as a vibrant example - which seems to me the perfect place to use the web to communicate with hard-to-reach people. Filippo added that although they are mainly used by second and third generation immigrants, these are the people who probably have most contact with first generation immigrants - a group that the council traditionally has great difficulty reaching.
None of the local government mandarins in our group said their council was using existing message boards as a means of outreach, one indicated that there was a perception that to engage on these forums might somehow undermine the legitimacy of local government's voice.
But it's such a missed opportunity. Hugh Flouch of Harringay Online told us that his hyper-local community has 3000 members (though we might question how many are active), while the area it covers has a population on 17,000. As he pointed out, that's more penetration that BBC Two.
Little tidbit from our break out group: apparently Coca Cola's social media principles are a great place to start if you need to write guidelines for an organisation that is precious about it's brand, on which front I'm sure Coca Cola is unimpeachable.
We also discussed the fear many higher up the organisational chain have of saying the wrong thing online. I don't think there is anywhere to hide from the internet's wrath. Recently a representative of Hackney council gave duff information on the phone (decidedly an old medium), only to find a recording posted on Private Eye's website (incidentally, also a Luddite publication which dislikes the internet). So it's time for local councils to take a dip their toes into the wide social media sea, it might not look enticing, but sooner or later someone is going to throw you in anyway.
Have I trodden the line of disinterested rationality? I hope so, but to end on a positive note, the more you engage with people on the web the more they love you. Paul Hodgkin of Patient Opinion told us that his service which allows patients to... give their opinion receives only around 20% purely negative feedback. Amazingly, after they have been moderated, about 10% of feedback leads to some kind of change is hospital practice, and the 20% of feedback which was completely positive was a huge boost to staff morale.
Research from Hugo Flouch (he of Harringay Online) suggests that the more officials engage with online communities the better users perceptions of them are. That was true for MPs, council workers, councillors themselves and police.
So, to sum up in a completely equivocal manner: budget cuts lead to a world where municipal work is carried out for free by the people who do it out the goodness of their own hearts and then thank each other through feedback services provided by not-for-profit social entrepreneurs. And no one who isn’t completely comfortable with Twitter will be forced to use it as part of a box-ticking drive to engage with Interweb 2.0.
When I went to Local by Social yesterday I was determined to maintain a detached skepticism - either I was seduced by the confrency world of balancing cups and saucers of pump-action coffee while trying to avoid conversational lapses with people I've never met before (unlikely), or I need to be bit more optimistic. Only briefly did the event feel like pie-in-the-sky geekery.
Local by Social was a discussion of the ways in which local governments can utilise social media, taking an extraordinarily broad definition of that term. Topics ranged from the stratospheric stature of reports complied for the Havard Kennedy School of Government to the more prosaic (and eternal) question of how to get doubting officials to engage with social media.
From the more philosophical end of the spectrum I was surprised discover an aspect of the web which I've not really come across before. My assumption was that debate would all be around some variety of communication between government and people, probably something to do with eschewing the broadcast model and adopting a many-to-many, responsive approach to social media - a message we've all heard before.
But Local by Social was way ahead of me with the concept of "social innovation", or "Public Service 2.0" (I've never heard the "2.0" suffix used so much). The concept is to close the loop of Official-Public-Official conversation by having, to varying degrees, the public actually solve problems themselves.
Examples included Washington's "Snowageedeon", where citizens used Google maps to allocate the work of clearing snow between themselves when the authorities were overwhelmed and Brighton Council using Twitter to find volunteer van drivers for meals on wheels during another bout inclemency. Or Fix My Tweet, which allows you to tell the council where pot holes are. The difference here is that work that would once have been carried out by an authority is being done for free through social media magic.
Instead of hulking, snail-paced governmental organisations facilitating this processes we were asked to imagine social entrepreneurs setting up not-for-profits to delivering these types of service. The question of what would happen when someone tried to monetise their successful website was left hanging in the air. Being new to the world of e-Government I was surprised these stories don’t seem to be more prevalent in the general web-trend reporting press, but perhaps that’s because the sites I follow tend to come from the marketing angle.
Another theme was adversity as a catalyst. The above examples were not the only times that snow precipitated social innovation. But, as Dominic Campbell (founder f FutureGov) explained in literally Churchillean terms ("We will innovate on the beaches..."), the most important opportunity in adversity was the coming cuts in public spending. Referencing Schumpeter's creative destruction he pointed out that more than ever authorities are receptive to novel cost-saving ideas. Big stupid bureaucracies listening to exciting web start ups to good to be true? Well, if you listen to David Cameron's TED Talk and his Big Society rhetoric (which I willing to concede might be just slightly more than electioneering) then perhaps there is room for a chink of optimism.
Am I maintaining detached skepticism? I'm trying... honest. Here's a dose: Twitter doesn't seem to me like it's important for local councils. In a break-out session at the end of the day a lot of local government people were bemoaning the fact that councillors wouldn't Tweet, but I don't think they really need to.
It might be useful as a means of disseminating press releases to journalists, but that doesn't mean that you can garner the ear of the populous using Twitter. Only 30,000 Twitter users tweeted about the election on election night. That's one in 2,000 people, and a close run general election is a lot more interesting than local government.
Much more exciting to me is the possibility of local government making itself known on existing localised communities. Filippo Ciampini, who is writing a masters on the subject of government public relations, told me that there are lot's of ethnically based online communities and forums - citing Islington's Chinese forum as a vibrant example - which seems to me the perfect place to use the web to communicate with hard-to-reach people. Filippo added that although they are mainly used by second and third generation immigrants, these are the people who probably have most contact with first generation immigrants - a group that the council traditionally has great difficulty reaching.
None of the local government mandarins in our group said their council was using existing message boards as a means of outreach, one indicated that there was a perception that to engage on these forums might somehow undermine the legitimacy of local government's voice.
But it's such a missed opportunity. Hugh Flouch of Harringay Online told us that his hyper-local community has 3000 members (though we might question how many are active), while the area it covers has a population on 17,000. As he pointed out, that's more penetration that BBC Two.
Little tidbit from our break out group: apparently Coca Cola's social media principles are a great place to start if you need to write guidelines for an organisation that is precious about it's brand, on which front I'm sure Coca Cola is unimpeachable.
We also discussed the fear many higher up the organisational chain have of saying the wrong thing online. I don't think there is anywhere to hide from the internet's wrath. Recently a representative of Hackney council gave duff information on the phone (decidedly an old medium), only to find a recording posted on Private Eye's website (incidentally, also a Luddite publication which dislikes the internet). So it's time for local councils to take a dip their toes into the wide social media sea, it might not look enticing, but sooner or later someone is going to throw you in anyway.
Have I trodden the line of disinterested rationality? I hope so, but to end on a positive note, the more you engage with people on the web the more they love you. Paul Hodgkin of Patient Opinion told us that his service which allows patients to... give their opinion receives only around 20% purely negative feedback. Amazingly, after they have been moderated, about 10% of feedback leads to some kind of change is hospital practice, and the 20% of feedback which was completely positive was a huge boost to staff morale.
Research from Hugo Flouch (he of Harringay Online) suggests that the more officials engage with online communities the better users perceptions of them are. That was true for MPs, council workers, councillors themselves and police.
So, to sum up in a completely equivocal manner: budget cuts lead to a world where municipal work is carried out for free by the people who do it out the goodness of their own hearts and then thank each other through feedback services provided by not-for-profit social entrepreneurs. And no one who isn’t completely comfortable with Twitter will be forced to use it as part of a box-ticking drive to engage with Interweb 2.0.
Thursday, May 13, 2010
We know that David Cameron doesn't like Twitter - what does it think of him?
Over the course of the General Election I recorded 1000 random tweets every hour and sent them to tweetsentiments.com for sentiment analysis.
Tweetsentiment have a service which gives one of three values to each tweet. '0' means a negative sentiment (unhappy tweet), '2' a neutral or undetermined sentiment and '4' positive (happy tweet). Similar technology is used to detect levels of customer satisfaction at call centres by monitoring phone calls.
Obviously it's difficult for a machine to detect the emotional meaning of a sentence, especially with the strange conventions used on Twitter. Despite this Tweetsentiment seems to be fairly reliable - tweets always which express happy emotions tend to be rated as such, and vice verse. More accurately, if Tweetsentiment does make a classification it tends to get it right. Sometimes an obviously positive / negative tweet gets a '2', but that shouldn't affect things here.
My hypothesis was that the Twitterati would be less happy if there was a Conservative victory. Of course I can’t prove that Twitter has a bias to the left, but I would presume that young, techy, early adopters are more likely to be left leaning. The reaction to the Jan Moir Stephen Gately article perhaps supports this.
David Cameron famously noted that Twitter is for twats, I wondered if Twitter would reciprocate...
The graph indicates that usually Twitter is just slightly positive, with a mood value of 2.1 on average. As predicted, as a conservative victory becomes apparent on Thursday evening there is a decline in mood which lasts until Saturday lunchtime. Then everyone cheers up, presumably goes down the pub, and is pretty chirpy for Sunday lunch. Sentiment only returns to average for the beginning of work on Monday morning.
In short, it does look like the election result was a disappointment to Twitter.
Obviously we need to know what normal Twitter behaviour is over the course of the week to draw very much information from the graph, and this is something that I’m going to try and produce a graph for soon.
It does look as though the size of negative reaction to a once-a-decade change in government is about the same magnitude as the positive mood elicited by the prospect of Sunday lunch - which I think is fairly consistent with the vicissitudes of Twitter as I experienced them.
I used Twitter’s API to gather the data, and frankly, it’s not particularly great, particularly if you want to get Tweets from the past. I was surprised to discover that any Tweets more than about 24 hours old simply disappear from the search function on Twitter.com – in effect they only exist in public for a day. For this reason the hourly sample size wasn’t always exactly 1000, but it was on average.
I’ll post again when I have some more data on normal behaviour. I’m also curious to find out if different countries have different average happiness levels on Twitter, but I think finding a Tweetsentiment-style service for other languages might prove difficult.
Tweetsentiment have a service which gives one of three values to each tweet. '0' means a negative sentiment (unhappy tweet), '2' a neutral or undetermined sentiment and '4' positive (happy tweet). Similar technology is used to detect levels of customer satisfaction at call centres by monitoring phone calls.
Obviously it's difficult for a machine to detect the emotional meaning of a sentence, especially with the strange conventions used on Twitter. Despite this Tweetsentiment seems to be fairly reliable - tweets always which express happy emotions tend to be rated as such, and vice verse. More accurately, if Tweetsentiment does make a classification it tends to get it right. Sometimes an obviously positive / negative tweet gets a '2', but that shouldn't affect things here.
My hypothesis was that the Twitterati would be less happy if there was a Conservative victory. Of course I can’t prove that Twitter has a bias to the left, but I would presume that young, techy, early adopters are more likely to be left leaning. The reaction to the Jan Moir Stephen Gately article perhaps supports this.
David Cameron famously noted that Twitter is for twats, I wondered if Twitter would reciprocate...
The graph indicates that usually Twitter is just slightly positive, with a mood value of 2.1 on average. As predicted, as a conservative victory becomes apparent on Thursday evening there is a decline in mood which lasts until Saturday lunchtime. Then everyone cheers up, presumably goes down the pub, and is pretty chirpy for Sunday lunch. Sentiment only returns to average for the beginning of work on Monday morning.
In short, it does look like the election result was a disappointment to Twitter.
Obviously we need to know what normal Twitter behaviour is over the course of the week to draw very much information from the graph, and this is something that I’m going to try and produce a graph for soon.
It does look as though the size of negative reaction to a once-a-decade change in government is about the same magnitude as the positive mood elicited by the prospect of Sunday lunch - which I think is fairly consistent with the vicissitudes of Twitter as I experienced them.
I used Twitter’s API to gather the data, and frankly, it’s not particularly great, particularly if you want to get Tweets from the past. I was surprised to discover that any Tweets more than about 24 hours old simply disappear from the search function on Twitter.com – in effect they only exist in public for a day. For this reason the hourly sample size wasn’t always exactly 1000, but it was on average.
I’ll post again when I have some more data on normal behaviour. I’m also curious to find out if different countries have different average happiness levels on Twitter, but I think finding a Tweetsentiment-style service for other languages might prove difficult.
Tuesday, May 4, 2010
Michael Jackson is more important than Jesus. Fact.
My last post used Wikipedia’s list of dates of births and deaths to build a timeline showing the lifespans of people who have pages on Wikipedia. There are a lot of people with Wikipedia pages, so I limited it to only include dead people.
That still leaves you with a lot of people to fit on one timeline, so I wanted to prioritise ‘important’ or ‘interesting’ people at the top and show only the most 'important' 1000. Some have been confused by my method for doing this, and others have questioned its validity, so this post will address both issues. I’m also going to suggest an improvement. It turns out that whatever I do Michael Jackson is more important than Jesus. I'm just the messenger.
Explaining the method
To get a measure of ‘importance’ I used work done by Stephan Dolan. He has developed a system for ranking Wikipedia pages which is very similar to the PageRank system which Google uses to prioritise its search results.
Wikipedia’s pages link to one another, and Stephan Dolan’s algorithm gives a measure of well linked to all the other Wikipedia pages a particular page is. If we want to know how well linked in the page about Charles Darwin is the algorithm examines every other page in Wikipedia and works out how many links you would have to follow to get from the page it is examining to the Charles Darwin page using the shortest route.
For example, to get from Aldous Huxley to Charles Darwin takes two links, one from Aldous to Thomas Henry Huxley (Aldous’s father) and then another to Darwin (TH Huxley famously defended evolution as a theory). Dolan’s method calculates the average number of clicks from every page in Wikipedia to the Charles Darwin page, and then takes an average value. To get to Charles Darwin takes an average 3.88 clicks from other Wikipedia pages.
Equivalently, Google shows pages that have many links pointing to them nearer the top in its search results.
This method works OK, but it could be better. For example Mircea Eliade ranks as the fifth most important dead person dead person on Wikipedia, taking on average 3.78 clicks to find him. But Mircea Eliade is a Romanian historian of religion - hardly a household name. We can take this as a positive statement, perhaps Mircea Eliade is a figure of hither to unrecognised importance and influence. On the other hand it seems impossible that he can be more ‘important’ than Darwin.
Testing the validity of the Dolan Index
I decided it would be interesting to compare what I’m going to call the Dolan index (the average number of clicks as described above) with two other metrics that could be construed as measuring the importance of a person. Before we do that, here is a Graph of what the Dolan index of dead people on Wikipedia looks like.
The bottom axis shows the rank order of pages, from Pope John Paul II, who is has the 275th highest Dolan index on Wikipedia, to Zi Pitcher, who comes 430900th in terms of Dolan index. It makes a very tidy log plot.
As I mentioned previously, the Dolan index is very similar to a Google PageRank, so lets compare them.
The x axis is the same as the first graph, Wikipedia pages from highest to lowest Dolan index. A well linked page has a low Dolan index, but a High PageRank, so I used the reciprocal of PageRank for the y axis. I've also added a log best fit line.
Comparing with PageRank seems to indicate there is a reasonable correlation between Dolan index and PageRank, which is indicated by the fact the first and second graphs have a similar shape.
PageRank is only given in integer values between 1-10 (realistically, all Wikipedia pages have a PageRank between 3-7), so I’ve smoothed the curve using a moving average.
This seems to lend some weight to the Dolan Index as a measure.
I’ve also made a comparison between the Dolan index the number of results returned when searching for a person’s name (without quotes) in Google search. It should be noted that this number seems to be quite unstable - a search will give a slightly different number of results from one day to the next. I’ve used a log scale because of the range of results.
There is barely any correlation here, except a very low values of Dolan index. Despite this, it’s still possible for the number of Google results to be useful, as becomes in apparent when trying to improve my measure of ‘importance’.
A suggestion for improvement
The problem with all the measures seems to be the noise inherent in the system. While Dolan Index, PageRank and number of Google results all provide a rough guide to ‘importance’ or ‘interest’ overall, each of them frequently gives unlikely results. How about using a mixture of all three? Here is a table comparing the top 25 dead people by Dolan index and using a hybrid measure of importance constructed from all three metrics.
To get the hybrid measure I just messed around until things felt right. Here is the formula I came up with:
Hybrid measure = ((1/Dolan index)x 20) + (PageRank x0.6) + (log(number of results)x 0.6)
For some reason additive formulas give better results than multiplicative ones.
Using the hybrid measure seems to have removed the surprises (like Peter Jennings) although you might still argue that Oscar Wilde or Jimi Hendrix are much too high. Michael Jackson comes out as bigger than Jesus, but then he is an exceptionally famous person, and he died much more recently than Jesus. Timur (AKA Tamerlane) is a bit of a curiosity.
I considered ignoring Number of Google results because its such a noisy dataset, however it’s the only reason that Jesus appears at all in this list, he gets a very low ranking (4.01) from the Dolan Index. Any formula which brings Jesus out on top (which I think you could make a reasonable case for his deserving, at least over Michael Jackson!), gives all kinds of strage results elsewhere.
I am a bit suspicious of "number of google results" metric. In addition to volatility Number of results fails to take into account that occurrences of words such as "Newtonian" should probably count towards Newton's ranking, but that people called David Mitchell will benefit artificially from the fact that at least two famous people share the name.
Any further investigation would have to consider what made a person ‘important’ – would it simply be how prominent they are in the minds of people (Michael Jackson and Jimi Hendrix) or would it reflect how influential they were (Charles Darwin for example, or the notably absent Karl Marx)?
I love the idea that the web reflects the collective conciousness, a kind of super-brain aggregation of human knowlege.
Just this week the idea of reflecting the whole of reality in one enormous computer system was promoted by Dirk Helbing, although my formula doesn't rate him as very important, so I'm unsure as to how seriously to take this.
That still leaves you with a lot of people to fit on one timeline, so I wanted to prioritise ‘important’ or ‘interesting’ people at the top and show only the most 'important' 1000. Some have been confused by my method for doing this, and others have questioned its validity, so this post will address both issues. I’m also going to suggest an improvement. It turns out that whatever I do Michael Jackson is more important than Jesus. I'm just the messenger.
Explaining the method
To get a measure of ‘importance’ I used work done by Stephan Dolan. He has developed a system for ranking Wikipedia pages which is very similar to the PageRank system which Google uses to prioritise its search results.
Wikipedia’s pages link to one another, and Stephan Dolan’s algorithm gives a measure of well linked to all the other Wikipedia pages a particular page is. If we want to know how well linked in the page about Charles Darwin is the algorithm examines every other page in Wikipedia and works out how many links you would have to follow to get from the page it is examining to the Charles Darwin page using the shortest route.
For example, to get from Aldous Huxley to Charles Darwin takes two links, one from Aldous to Thomas Henry Huxley (Aldous’s father) and then another to Darwin (TH Huxley famously defended evolution as a theory). Dolan’s method calculates the average number of clicks from every page in Wikipedia to the Charles Darwin page, and then takes an average value. To get to Charles Darwin takes an average 3.88 clicks from other Wikipedia pages.
Equivalently, Google shows pages that have many links pointing to them nearer the top in its search results.
This method works OK, but it could be better. For example Mircea Eliade ranks as the fifth most important dead person dead person on Wikipedia, taking on average 3.78 clicks to find him. But Mircea Eliade is a Romanian historian of religion - hardly a household name. We can take this as a positive statement, perhaps Mircea Eliade is a figure of hither to unrecognised importance and influence. On the other hand it seems impossible that he can be more ‘important’ than Darwin.
Testing the validity of the Dolan Index
I decided it would be interesting to compare what I’m going to call the Dolan index (the average number of clicks as described above) with two other metrics that could be construed as measuring the importance of a person. Before we do that, here is a Graph of what the Dolan index of dead people on Wikipedia looks like.
The bottom axis shows the rank order of pages, from Pope John Paul II, who is has the 275th highest Dolan index on Wikipedia, to Zi Pitcher, who comes 430900th in terms of Dolan index. It makes a very tidy log plot.
As I mentioned previously, the Dolan index is very similar to a Google PageRank, so lets compare them.
The x axis is the same as the first graph, Wikipedia pages from highest to lowest Dolan index. A well linked page has a low Dolan index, but a High PageRank, so I used the reciprocal of PageRank for the y axis. I've also added a log best fit line.
Comparing with PageRank seems to indicate there is a reasonable correlation between Dolan index and PageRank, which is indicated by the fact the first and second graphs have a similar shape.
PageRank is only given in integer values between 1-10 (realistically, all Wikipedia pages have a PageRank between 3-7), so I’ve smoothed the curve using a moving average.
This seems to lend some weight to the Dolan Index as a measure.
I’ve also made a comparison between the Dolan index the number of results returned when searching for a person’s name (without quotes) in Google search. It should be noted that this number seems to be quite unstable - a search will give a slightly different number of results from one day to the next. I’ve used a log scale because of the range of results.
There is barely any correlation here, except a very low values of Dolan index. Despite this, it’s still possible for the number of Google results to be useful, as becomes in apparent when trying to improve my measure of ‘importance’.
A suggestion for improvement
The problem with all the measures seems to be the noise inherent in the system. While Dolan Index, PageRank and number of Google results all provide a rough guide to ‘importance’ or ‘interest’ overall, each of them frequently gives unlikely results. How about using a mixture of all three? Here is a table comparing the top 25 dead people by Dolan index and using a hybrid measure of importance constructed from all three metrics.
| Dolan index | Hybrid measure |
| Pope John Paul II | Michael Jackson |
| Michael Jackson | Jesus |
| John F. Kennedy | Ronald Reagan |
| Gerald Ford | Jimi Hendrix |
| Mircea Eliade | Abraham Lincoln |
| Peter Jennings | Adolf Hitler |
| John Lennon | Albert Einstein |
| Adolf Hitler | William Shakespeare |
| Harry S. Truman | Charles Darwin |
| Rold Reagan | Oscar Wilde |
| J. R. R. Tolkien | Woodrow Wilson |
| James Brown | Isaac Newton |
| Anthony Burgess | Elvis Presley |
| Elvis Presley | Walt Disney |
| Christopher Reeve | John Lennon |
| Susan Oliver | George Washington |
| Franklin D. Roosevelt | John F. Kennedy |
| Winston Churchill | Timur |
| Ernest Hemingway | Martin Luther |
| Theodore Roosevelt | Voltaire |
To get the hybrid measure I just messed around until things felt right. Here is the formula I came up with:
Hybrid measure = ((1/Dolan index)x 20) + (PageRank x0.6) + (log(number of results)x 0.6)
For some reason additive formulas give better results than multiplicative ones.
Using the hybrid measure seems to have removed the surprises (like Peter Jennings) although you might still argue that Oscar Wilde or Jimi Hendrix are much too high. Michael Jackson comes out as bigger than Jesus, but then he is an exceptionally famous person, and he died much more recently than Jesus. Timur (AKA Tamerlane) is a bit of a curiosity.
I considered ignoring Number of Google results because its such a noisy dataset, however it’s the only reason that Jesus appears at all in this list, he gets a very low ranking (4.01) from the Dolan Index. Any formula which brings Jesus out on top (which I think you could make a reasonable case for his deserving, at least over Michael Jackson!), gives all kinds of strage results elsewhere.
I am a bit suspicious of "number of google results" metric. In addition to volatility Number of results fails to take into account that occurrences of words such as "Newtonian" should probably count towards Newton's ranking, but that people called David Mitchell will benefit artificially from the fact that at least two famous people share the name.
Any further investigation would have to consider what made a person ‘important’ – would it simply be how prominent they are in the minds of people (Michael Jackson and Jimi Hendrix) or would it reflect how influential they were (Charles Darwin for example, or the notably absent Karl Marx)?
I love the idea that the web reflects the collective conciousness, a kind of super-brain aggregation of human knowlege.
Just this week the idea of reflecting the whole of reality in one enormous computer system was promoted by Dirk Helbing, although my formula doesn't rate him as very important, so I'm unsure as to how seriously to take this.
Sunday, April 11, 2010
Wikipedia Book of the Dead
DBpedia mashup: the most important dead people according to Wikipedia
The timeline below shows the names of dead people and their lifespans, as retrieved from Wikipedia. They are arranged so that people nearer the top are the best linked in on Wikipedia, as measured by the average number of clicks it would take to get from any Wikipedia page to the page of the person in question.
I had imagined that Wikipedia 'linkedin-ness' would serve as a proxy for celebrity, which it kind of does - but only in a lose way.
Values range from 3.72 (at the top) to 4.04 (at the bottom). This means that if you were to navigate from a large number of Wikipedia pages, using only internal Wikipedia links, it would take you, on average, 3.72 clicks to get to Pope John Paul II. This data set was made by Stephan Dolan, who explains the concept better than me. Basically, it's the 6 degrees of Kevin Bacon on Wikipedia.
I looped through the data set and queried DBpedia to see if the Wikipedia article was about a person, and if so retrieved their dates of birth and death.
The timeline does show a certain amnesia on the part of Wikipedia, Shakespeare and Newton are absent, while Romainian historian of religion Mircea Eliade comes 5th. If I had included people who are alive tennis players would have dominated the list (I don't know why) - Billie Jean King is the second best-linked article on wikipedia, one ahead of the USA (the UK is number one!).
Any mistakes (I have seen some) are due to the sketchiness of the DBpedia data, though I can't rule out having made some mistakes myself...
There results are limited to the top 1000, and they only go back to 1650. Almost no names previous to 1650 appeared, the exceptions being Jesus (who was still miles down) and Guy Fawkes.
In case you were wondering 'Who's Saul Bellow below?', the answer is Rudolph Hess.
The timeline below shows the names of dead people and their lifespans, as retrieved from Wikipedia. They are arranged so that people nearer the top are the best linked in on Wikipedia, as measured by the average number of clicks it would take to get from any Wikipedia page to the page of the person in question.
I had imagined that Wikipedia 'linkedin-ness' would serve as a proxy for celebrity, which it kind of does - but only in a lose way.
Values range from 3.72 (at the top) to 4.04 (at the bottom). This means that if you were to navigate from a large number of Wikipedia pages, using only internal Wikipedia links, it would take you, on average, 3.72 clicks to get to Pope John Paul II. This data set was made by Stephan Dolan, who explains the concept better than me. Basically, it's the 6 degrees of Kevin Bacon on Wikipedia.
I looped through the data set and queried DBpedia to see if the Wikipedia article was about a person, and if so retrieved their dates of birth and death.
The timeline does show a certain amnesia on the part of Wikipedia, Shakespeare and Newton are absent, while Romainian historian of religion Mircea Eliade comes 5th. If I had included people who are alive tennis players would have dominated the list (I don't know why) - Billie Jean King is the second best-linked article on wikipedia, one ahead of the USA (the UK is number one!).
Any mistakes (I have seen some) are due to the sketchiness of the DBpedia data, though I can't rule out having made some mistakes myself...
There results are limited to the top 1000, and they only go back to 1650. Almost no names previous to 1650 appeared, the exceptions being Jesus (who was still miles down) and Guy Fawkes.
In case you were wondering 'Who's Saul Bellow below?', the answer is Rudolph Hess.
Sunday, February 7, 2010
Smoke Stacks to Apple Macs
The digital revolution will not be televised - to the contrary, is it possible that no artist or medium can be said to have adequately addressed the information age?
Zizek once sumerised Marx as having said that the invention of steam engine caused more social change than any revolution ever would. Marx himself doesn't seem to have provided a useful soundbite to this effect (at least not one that I can find though Google), so I'm afraid it will have to remain second hand. It's a powerful sentiment, whoever originated it - which philosopher's views cannot be analyzed as the product of the social and technological novelties of his day?
It's easy to see that the technology that is most salient in our age is the internet, which has been made possible by consumer electronics. Have our philosophers stepped forward to engage with the latest technological crop?
Moving on from philosophers, what of our artists? Will Gompertz recently posted to share an apparently widely held view that no piece of art has yet spoken eloquently from or about the internet. He cites Turner prize winning Jeremy Deller describing "a post-warholian" era, presumably indicating that Warhol was last person to adequately reference technological change in the guise of mass production. I wonder if the Saatchi-fueled infloresence has also captured something of marketing-led landscape we currently live in, but whatever the last sufficient reflection on cultural change afforded by art was, I think we may be on safe ground in stating that the first widely accepted visual aperçus of the digital era is still to come.
Which is some surprise when you consider, for example, how engaged the news agenda is with technology: I was amazed to see that Google's Wave technology (still barely incipient) got substantial coverage on BBC news.
With my employment centering on the web, and my pretensions at cultural engagement, this weekend I visited the Kinetica Art Fair. Kinetica is a museum which aims to 'encourage convergence of art and technology'. The fair certainly captured one aspect of contemporary mood - a very reasonably priced bar was a welcome response to our collective financial deficit.
Standout pieces included a cleverly designed mechanical system for tracing the contours of plaster bust onto a piece of paper and a strangely terrifying triangular mirror with mechanically operated metal rods. It looked like a Buck Rogers inspired torture device designed to inflict pain by a method so awful that you'd have to see it in operation before its evil would be comprehensible. The other works included a urinal which provided an opportunity for punters to simulate pan-global urination (sadly not with real urine) by providing a jet of water and a globe in a urinal. I would defy anyone not to be entertained by spending time wondering round the the fair.
However, Will Gompertz's challenge was not answered at Kinetica - the essence of the technological modernity was distilled into any of work - not even slightly.
I've been mulling over various possible reasons for this failure, and quite a few suggestions spring to mind. Do computers naturally alienate artists? Is information technology to visually banal to be characterised succinctly?
I'd like to suggest that its the transitory nature of our electronic lives that makes them so hard to pin down. Mobile phones, web sites, computers and opperating systems from a decade ago all look ludicrously dated - it's almost impossible to capture the platonic form of these items because they have so little essential similarity. Moreover, their form is almost an accident, and not connected with their more profound meaning in any way. The boats of the merchantile age and the smoke stacks of the industrial age all seem to denote something broader - how can communism be separated from its tractors? Yet the form factor of my computer is trivial. Form and functional significance are of necessity separated by digital goods, their flexibility is the source of their power.
In someway I think films give us tacit acknowledgment of the contingent nature of the digital environment that we spend much of our lives in: no protagonist is ever seen using Windows on their computer, in films computer's interfaces are always generic. When we see a Mac in a film it impossible to see it as anything other than product placement.
So, the Kinetica Art Fair may not have been able to help society understand its relationship with technology, but actually, despite their rhetoric, I think it was a little unfair to expect it to. Really the fair was about works facilitated by technology, rather than about it.
But, in case you think I've picked a straw man in Kentica, let me say that the V&As ongoing exhibition Decode really does no better, though its failures and successes are another topic.
Zizek once sumerised Marx as having said that the invention of steam engine caused more social change than any revolution ever would. Marx himself doesn't seem to have provided a useful soundbite to this effect (at least not one that I can find though Google), so I'm afraid it will have to remain second hand. It's a powerful sentiment, whoever originated it - which philosopher's views cannot be analyzed as the product of the social and technological novelties of his day?
It's easy to see that the technology that is most salient in our age is the internet, which has been made possible by consumer electronics. Have our philosophers stepped forward to engage with the latest technological crop?
Moving on from philosophers, what of our artists? Will Gompertz recently posted to share an apparently widely held view that no piece of art has yet spoken eloquently from or about the internet. He cites Turner prize winning Jeremy Deller describing "a post-warholian" era, presumably indicating that Warhol was last person to adequately reference technological change in the guise of mass production. I wonder if the Saatchi-fueled infloresence has also captured something of marketing-led landscape we currently live in, but whatever the last sufficient reflection on cultural change afforded by art was, I think we may be on safe ground in stating that the first widely accepted visual aperçus of the digital era is still to come.
Which is some surprise when you consider, for example, how engaged the news agenda is with technology: I was amazed to see that Google's Wave technology (still barely incipient) got substantial coverage on BBC news.
With my employment centering on the web, and my pretensions at cultural engagement, this weekend I visited the Kinetica Art Fair. Kinetica is a museum which aims to 'encourage convergence of art and technology'. The fair certainly captured one aspect of contemporary mood - a very reasonably priced bar was a welcome response to our collective financial deficit.
Standout pieces included a cleverly designed mechanical system for tracing the contours of plaster bust onto a piece of paper and a strangely terrifying triangular mirror with mechanically operated metal rods. It looked like a Buck Rogers inspired torture device designed to inflict pain by a method so awful that you'd have to see it in operation before its evil would be comprehensible. The other works included a urinal which provided an opportunity for punters to simulate pan-global urination (sadly not with real urine) by providing a jet of water and a globe in a urinal. I would defy anyone not to be entertained by spending time wondering round the the fair.
However, Will Gompertz's challenge was not answered at Kinetica - the essence of the technological modernity was distilled into any of work - not even slightly.
I've been mulling over various possible reasons for this failure, and quite a few suggestions spring to mind. Do computers naturally alienate artists? Is information technology to visually banal to be characterised succinctly?
I'd like to suggest that its the transitory nature of our electronic lives that makes them so hard to pin down. Mobile phones, web sites, computers and opperating systems from a decade ago all look ludicrously dated - it's almost impossible to capture the platonic form of these items because they have so little essential similarity. Moreover, their form is almost an accident, and not connected with their more profound meaning in any way. The boats of the merchantile age and the smoke stacks of the industrial age all seem to denote something broader - how can communism be separated from its tractors? Yet the form factor of my computer is trivial. Form and functional significance are of necessity separated by digital goods, their flexibility is the source of their power.
In someway I think films give us tacit acknowledgment of the contingent nature of the digital environment that we spend much of our lives in: no protagonist is ever seen using Windows on their computer, in films computer's interfaces are always generic. When we see a Mac in a film it impossible to see it as anything other than product placement.
So, the Kinetica Art Fair may not have been able to help society understand its relationship with technology, but actually, despite their rhetoric, I think it was a little unfair to expect it to. Really the fair was about works facilitated by technology, rather than about it.
But, in case you think I've picked a straw man in Kentica, let me say that the V&As ongoing exhibition Decode really does no better, though its failures and successes are another topic.
Monday, February 1, 2010
Internet Protocol and HRH the Queen
Whatever we end up using the web for, don't the world's citizens lead more equal lives if they are all mediated by the same technology?
The queen tweets. She's commissioned a special jewel embossed netbook and a bespoke Twitter client with skinned with ermine and sable.
I made that up. For starters, she hasn't actually started tweeting - there is a generic royal feed which announces the various visits and condescensions of Britain's feudal anachronism, but nothing from miss fancy hat herself. Perhaps royal protocol means she can only use it if her followers can find a way of curtsying in 140 characters?
The feed does give an insight into how boring the Royal's lives might actually be - opening wards and meeting factory workers - when they aren't having a bloody good time shooting and riding. However, as a PR initiative it breaks the rule that states for a Twitter account to be of any interest then tweets must emanate from the relevant horse's mouth, if you'll forgive the chimerical metaphor. If you can't have the lady herself, I don't really think there is much point in bothering. But that's not the point I'm here to make.
I'm more interested in the fact that, should any of us choose to, Bill Gates, Sir Ranulph Twisleton-Wykeham-Fiennes, 3rd Baronet OBE, Osama Bin Laden and I will have exactly the same experience when we use Twitter (assuming it's available in the relevant language).
I suppose Bin Laden might have quite a slow connection in Tora Bora, and probably Bill Gates has something faster than Tiscali's 2meg package. Details aside, everyone is doing the same thing.
Actually, not only will we be using the same website, we'll be using very similar devices. Bill probably doesn't have a Mac like me (he may be the richest man in the world, he can still envy me one thing), but all our computers will be very similar.
The reason for this is that for both websites and computer technology have very high development costs, and low marginal costs per user. Even the Queen can't afford to develop an iPhone, but everyone can afford to buy one.
If a lot of your life is mediated by technology then this is going to be very important to you. While there is healthy debate about the web's democratisation of publishing, I think we might reasonably add to the web's egalitarian reputation its ability to give people of disparate incomes identical online experiences.
That doesn't sound like a blow against inequality and tyranny in all its forms - but none the less I think its important . Even people using OLPC computers [low priced laptops aimed at the third world] have basically the same experience of the internet as you or I. That's to say Uruguayan children will quite possibly spend a good part of their day doing exactly the same things as New York's office workers and Korea's pensioners. When you consider that only very recently there were probably no major similarities in these disparate lives I think it does constitute a significant development.
Of course, for all I know a line of luxury websites will come along and exclude some strata of the social pile. In a way it's already happened - we've seen the thousand dollar iPhone app - but its hard to see this one off as part of a pattern. This is not to say that the 'freemium' business model [basic website for free, pay to get the premium version] couldn't exclude certain people, it's more that this model can only exist when there isn't much pressure for a free version. At the moment, there aren't any widely used web applications that aren't available at zero cost - of course this may change if your audience is sufficiently well off to attract paid advertising, but there again it may not.
This is a phenomena that's been observed before: technology tends to eliminate differences between cultures. It's been termed the Apparatgeist, and has been developed as a concept in response to the observation that mobile phone habits, one differentiated locally, are now more or less identical in all developed economies. As a concept it surely applies equally as well to class and income - leaving us us in a more equal experiential world. And perhaps also a monoculture - but then isn't that entailed in the new equalities that so many internet optimists evangelise?
The queen tweets. She's commissioned a special jewel embossed netbook and a bespoke Twitter client with skinned with ermine and sable.
I made that up. For starters, she hasn't actually started tweeting - there is a generic royal feed which announces the various visits and condescensions of Britain's feudal anachronism, but nothing from miss fancy hat herself. Perhaps royal protocol means she can only use it if her followers can find a way of curtsying in 140 characters?
The feed does give an insight into how boring the Royal's lives might actually be - opening wards and meeting factory workers - when they aren't having a bloody good time shooting and riding. However, as a PR initiative it breaks the rule that states for a Twitter account to be of any interest then tweets must emanate from the relevant horse's mouth, if you'll forgive the chimerical metaphor. If you can't have the lady herself, I don't really think there is much point in bothering. But that's not the point I'm here to make.
I'm more interested in the fact that, should any of us choose to, Bill Gates, Sir Ranulph Twisleton-Wykeham-Fiennes, 3rd Baronet OBE, Osama Bin Laden and I will have exactly the same experience when we use Twitter (assuming it's available in the relevant language).
I suppose Bin Laden might have quite a slow connection in Tora Bora, and probably Bill Gates has something faster than Tiscali's 2meg package. Details aside, everyone is doing the same thing.
Actually, not only will we be using the same website, we'll be using very similar devices. Bill probably doesn't have a Mac like me (he may be the richest man in the world, he can still envy me one thing), but all our computers will be very similar.
The reason for this is that for both websites and computer technology have very high development costs, and low marginal costs per user. Even the Queen can't afford to develop an iPhone, but everyone can afford to buy one.
If a lot of your life is mediated by technology then this is going to be very important to you. While there is healthy debate about the web's democratisation of publishing, I think we might reasonably add to the web's egalitarian reputation its ability to give people of disparate incomes identical online experiences.
That doesn't sound like a blow against inequality and tyranny in all its forms - but none the less I think its important . Even people using OLPC computers [low priced laptops aimed at the third world] have basically the same experience of the internet as you or I. That's to say Uruguayan children will quite possibly spend a good part of their day doing exactly the same things as New York's office workers and Korea's pensioners. When you consider that only very recently there were probably no major similarities in these disparate lives I think it does constitute a significant development.
Of course, for all I know a line of luxury websites will come along and exclude some strata of the social pile. In a way it's already happened - we've seen the thousand dollar iPhone app - but its hard to see this one off as part of a pattern. This is not to say that the 'freemium' business model [basic website for free, pay to get the premium version] couldn't exclude certain people, it's more that this model can only exist when there isn't much pressure for a free version. At the moment, there aren't any widely used web applications that aren't available at zero cost - of course this may change if your audience is sufficiently well off to attract paid advertising, but there again it may not.
This is a phenomena that's been observed before: technology tends to eliminate differences between cultures. It's been termed the Apparatgeist, and has been developed as a concept in response to the observation that mobile phone habits, one differentiated locally, are now more or less identical in all developed economies. As a concept it surely applies equally as well to class and income - leaving us us in a more equal experiential world. And perhaps also a monoculture - but then isn't that entailed in the new equalities that so many internet optimists evangelise?
Subscribe to:
Posts (Atom)