Comments on: Bringing the New York Times’ Cornucopia to All http://citmedia.org/blog/2007/10/22/bringing-the-new-york-times-cornucopia-to-all/ Mon, 08 Sep 2008 18:42:27 +0000 http://wordpress.org/?v=2.6.1 By: HorsesAss.Org» Blog Archive » Tuesday Morning Blews http://citmedia.org/blog/2007/10/22/bringing-the-new-york-times-cornucopia-to-all/#comment-151990 HorsesAss.Org» Blog Archive » Tuesday Morning Blews Tue, 23 Oct 2007 14:29:21 +0000 http://citmedia.org/blog/2007/10/22/bringing-the-new-york-times-cornucopia-to-all/#comment-151990 [...] those things that lead to places we simply cannot foresee (including a dead end). But Dan Gillmor has it right when he lauds The Times for opening up its data stream to outside resources — in the cause of [...] [...] those things that lead to places we simply cannot foresee (including a dead end). But Dan Gillmor has it right when he lauds The Times for opening up its data stream to outside resources — in the cause of [...]

]]> By: The New York Times river flows, but whereto? « Alexander van Elsas’s Weblog on new media & technologies and their effect on social behavior http://citmedia.org/blog/2007/10/22/bringing-the-new-york-times-cornucopia-to-all/#comment-151989 The New York Times river flows, but whereto? « Alexander van Elsas’s Weblog on new media & technologies and their effect on social behavior Tue, 23 Oct 2007 13:46:03 +0000 http://citmedia.org/blog/2007/10/22/bringing-the-new-york-times-cornucopia-to-all/#comment-151989 [...] it on his blog post announcement. I do see a lot of comments from people in the field (techies, here, here and here for example) that really like the work and the possibilities of the [...] [...] it on his blog post announcement. I do see a lot of comments from people in the field (techies, here, here and here for example) that really like the work and the possibilities of the [...]

]]> By: Scripting News for 10/22/07 « Scripting News Annex http://citmedia.org/blog/2007/10/22/bringing-the-new-york-times-cornucopia-to-all/#comment-151985 Scripting News for 10/22/07 « Scripting News Annex Tue, 23 Oct 2007 03:07:04 +0000 http://citmedia.org/blog/2007/10/22/bringing-the-new-york-times-cornucopia-to-all/#comment-151985 [...] Dan Gillmor: “Dave Winer has been exploring a superb news resource, exploring the depth and breadth of the New York Times‘ data-stream.” [...] [...] Dan Gillmor: “Dave Winer has been exploring a superb news resource, exploring the depth and breadth of the New York Times‘ data-stream.” [...]

]]> By: Jon Garfunkel http://citmedia.org/blog/2007/10/22/bringing-the-new-york-times-cornucopia-to-all/#comment-151984 Jon Garfunkel Mon, 22 Oct 2007 21:26:56 +0000 http://citmedia.org/blog/2007/10/22/bringing-the-new-york-times-cornucopia-to-all/#comment-151984 Irony of ironies. When RSS was being conceived in the summer of 2000, there were two basic camps. Rael Dornfest saw in it a way to make a true RDF query engine. Imagine: anybody could then build their own query engine to universally query blogs + news + anything RDF (by *anybody*, I don't mean *everybody* -- it's just under the notion that if anybody can build their own javascript libraries, you'll have a fierce Darwinian competition) But Dave wanted to make it simple. ("I find the activity towards 'modularization' to be dry and uninteresting.") So, things like keywords (and a bunch of useful metadata out of NewsML) were left out and were left as an exercise to the developer. Hence many different Atom extensions these days, none of which are wholly standard. And, here's the irony: the Times is not supplying the metadata in their RSS2.0 feeds-- they are putting them in the HTML, leaving Dave to scrape them from there! ha ha! And, of course, Google and Topix already do this clustering technique-- across thousands of sources. They scrape, because news organizations do not make metadata-rich NewsML feeds available... Not that pulling META tags out of an HTML document is hard. It's just curious that, if you wind the clock back 7 years, and tell a bunch of software engineers that the coolest news clustering applications today would still rely on HTML scraping, I don't know whether they'd laugh or cry. Irony of ironies.

When RSS was being conceived in the summer of 2000, there were two basic camps. Rael Dornfest saw in it a way to make a true RDF query engine. Imagine: anybody could then build their own query engine to universally query blogs + news + anything RDF (by *anybody*, I don’t mean *everybody* — it’s just under the notion that if anybody can build their own javascript libraries, you’ll have a fierce Darwinian competition)

But Dave wanted to make it simple. (”I find the activity towards ‘modularization’ to be dry and uninteresting.”) So, things like keywords (and a bunch of useful metadata out of NewsML) were left out and were left as an exercise to the developer. Hence many different Atom extensions these days, none of which are wholly standard.

And, here’s the irony: the Times is not supplying the metadata in their RSS2.0 feeds– they are putting them in the HTML, leaving Dave to scrape them from there! ha ha!

And, of course, Google and Topix already do this clustering technique– across thousands of sources. They scrape, because news organizations do not make metadata-rich NewsML feeds available…

Not that pulling META tags out of an HTML document is hard. It’s just curious that, if you wind the clock back 7 years, and tell a bunch of software engineers that the coolest news clustering applications today would still rely on HTML scraping, I don’t know whether they’d laugh or cry.

]]>