OpenID's hidden value in Application Integration

I haven't seen any articles discussing this subject - but let me know if you know of any and i'll link.

OpenID is a hugely valuable standard protocol for federated authentication and by the looks of things adoption is and will coninue to grow at a phenomenal rate.

The advantage of OpenID over traditional authentication systems is well known - it saves you having to remember multiple usernames and passwords. An additional advantage of OpenID over other Internet authentication systems is that it is completely decentralized and anyone can create their own authentication provider.

Consider an enterprise that has some internal applications that can't be seen by the web itself. That enterprise can set up an OpenID provider to let all their staff log into these internal applications. Additionally however they can allow that provider to be seen from the outside world and so allow staff to log onto applications on the web using that same OpenID.

Cool! However there is an additional benefit when integrating applications which isn't quite as explicit and you really only notice at development stage. In particular this is very important for creators of 3rd party software. That is:

If a 3rd party application supports OpenID then no matter the language or platform it can be easily integrated into an existing software site or portal.

What do i mean by this? Well recently I created a web site that used a 3rd party open source system that had traditional basic authentication as well as OpenID. Great. However later on i wanted to add a nice blogging feature using another 3rd party ... but the problem was they didn't support OpenID.

Now, let's ignore all the technical detail here such as Single Sign On between applications - sure this is cool but i'd be willing to bet that even if the user had to log in using their OpenID twice into these two applications it is miles ahead of the common current issue where you suddenly need to remember a username and password again (and all the management that goes with that). Consider the case where most of us may integrate 4 or 5 separate systems to create a solution and suddenly the fact that ONE of then uses OpenID but the other 4 require old style username/password authentication really isn't much of a benefit.

In my case i chose not to go for them. I simply didn't have time to write OpenID support for their products. I was quite happy for the applications to run independently on the server with a simple link bewteen each ... even if they had to log in to each - so long as i didn't have to manage duplicate accounts. The great thing though is that the managament of their login details etc could all be done centrally. I know from my experience supporting OpenID.ORG that the time taken to support users dwarfs almost everything else you do.

So although it may sound obvoius that an OpenID account can really help you log into multiple applications with the same details, it's not until you start pulling together 3rd party applications you realize how valuable this is even on a single website or portal.

Going back to our enterprise - they can now create a cool portal that integrates custom software, 3rd party products & remote services,  without worrying about user managament. And i know from current personal experience how big a win that will be.

xUnit & TeamCity

Having used nUnit for some time and Visual Team System's test architecture i was recently interested in putting this to work on my own project as part of an automated build & test architecture. The test architecture of Visual Team system isn't particualy suited to Continuous Integration as you need to reference assemblies that only seem to be part of Visual Studio and you're not allowed to distribute these. I considered nUnit as this is what i am used to but as it's my own projected decided to give xUnit a shot as it seems to be the next logical step in good testing practices.

I had started using ThoughtWords CI server but after banging my head on the table a few times in my failed attempts to integrate xUnit, and on the advice of Brad Wilson, decided to look at TeamCity.

With some help from Brad I managed to get this working but as it was still a little tricky (i only rarely change this infrastruture of this kind of thing once it it set up) i decided to document how to do it. Note that you can't just point TeamCity at the .sln file in this case because you need to TELL it that it must run your xUnit tests and clearly the .sln file doesn't know about them. So what you need to do is create an MSBuild file where you can tell TeamCity to (a) build your solution file AND (b) run your tests via xUnit.

1. Download and Insteall team city. Configure this to get it building your base code ( i won't document this as there are loads of places to find out more).

2. Go to the folder where your .sln file sites and create a file called "mysolution.msbuild" (or something similar).

3. Enter the following Xml:

<project defaulttargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<usingtask assemblyfile="[RelativePathToXUnit]xunit.runner.msbuild.dll" taskname="Xunit.Runner.MSBuild.xunit">

<target name="Build">
<msbuild projects="mysolution.sln" targets="Build" properties="Configuration=Debug">
<xunit assembly="[RelativePathToProject]bin\Debug\MyProjectTests.dll">
</xunit></msbuild></target>

</usingtask></project>


* Remember to replace [RelativePathToXUnit] with the relative path from your solution file to where the xunit runner assembly can be found. Also make sure [RelativePathToProject] is changed to point to the project where your tests can be found.

4. Now via the TeamCity GUI in the "Build Runner" tab of your project, select MSBuild and add the relative path to your "mysolution.msbuild" file relative to your checkout directory (in my case it is often "/mysolution.msbuild").

5. Now you can run your build and you'll find all your unit tests are run.

Microsoft Tags, QR Codes and OpenID

I have used Barcodes and QR Codes in the past on some projects and played around a bit. Well, this morning i saw Microsoft released "Microsoft Tags" as a beta at CES 2009.

Clearly from my own selfish perspective, what i'd like to see if OpenID (disclaimer, i run http://OpenID.ORG) go OPEN in the real world. So, at a conference you could put a QR Code or Microsoft tag on your badge, t-shirt, hat or what ever. Maybe on the back of a card, on your company documentation, promotional material and so on.

If your OpenID is encoded in there then people could save you as a contact direct from your t-shirt :)

Thoughts? Other ideas?

Here is my OpenID (http://weblivz.openid.org) as a Microsoft tag (which would be printed out of course):

... and here is my OpenID as a QR Code :

Update : Sure enough, i installed the Tag reader on my PDA and simply pointed the phone at the image and it took me to my OpenID page in the web browser. Very nice!



They say more formats may be supported in the future - hope they add QR Codes to that list!

Twitter phishing issues

I agree with this article in the main, but there are some things worth pointing out… baring mind the majority and most concerning of these phishing issues came from people giving their credentials out and the 3rd parties assuming their identities and the associated trust with that.

1. oAuth could have allowed me to provide access for 3rd party systems in a case by case basis - not providing the credentials that could be stored and re-used.

2. oAuth would allow me to disable access immediately for a 3rd party site.

3. oAuth, as used in GMail, could be used to restrict the levels of access you permit of 3rd party applications. Therefore i may allow a site to read my contacts (which most do) but not sent replies or DM’s on my behalf (in fact this is what GMail can allow).

... I totally get that anyone can @ you on twitter - but i would be unlikely to click a random link. It it came from a friend - especially DM I am MUCH more likely to click it.

Only question i have is why did the phishing sites not use tinyURL or something to mask the link?

In the zone

Every New Year we promise to change something about ourselves. Well, this year i'm THINKING of trying something new (for me). Starting Monday 5th Jan, I am going to change timezone.... well, for an experimental period anyway! I was inspired to try this by reading this article by @4entrepreneur ... and the fact that it's harder to work at home during the day when my daughter is around (not yet at nursery).

What does that mean? Well, for 2 weeks (gotta trial this and start somewhere!) i am going to work to PST time. So that means i will be working through the night, at the same time as most of the tech activity on Twitter. It also means i can see whether coding in the evenings really is more beneficial. I used to write all my code books into the early hours but when i had the kids things reversed and i starting getting up very early.

I will blog and tweet as i go.

But i need to get a few typical daily timetables from tech folk working in PST (when do you rise, work, go to sleep and so on?). If anyone can comment, email or DM on twitter it would be much appreciated!! I could easily subtract 8 from my normal day, but want something real.

Anyone NOT in PST already work to that timezone, perhaps out of necessity via remote working?

The Social Netbook

It is perhaps reasonably well known that i won a Netbook courtesy of Vodafone last month. As soon as i got it I started to think about how it could make its way into the everyday lives of the people i know. So, here are a couple of suggestions:

Social Software
The first thing is that these Netboos need to come pre-bundled with social software. Most people won't buy a Netbook to write code or even compile documents. Most will buy them to "get on the web" as Netbooks are a great opportunity to point them at things to do.

I imagine a wizard in startup that suggests the services you may like to use, takes you through a signing up process and installs and configures the appropriate software for you.

An example is ...

1. Configure an OpenID account
2. Create Twitter account (they should be doing (1) by now!) and install software
3. Create a flickr and twitpic account and configure appropriately.
4. Set up friendfeed.
5. Create a MySpace/Facebook account and store shortcuts.
6. Install UStream/Qik and configure an account.
7. Email account.
8. blah dee blah ...

(Other suggestions ??)

Cloud Services
We're talking about a lightweight laptop that has limited disk space and less than the typical laptop in terms of memory and processor speeds. It is pretty much reliant on you being online. So why not use the cloud??

An example ...

1. An Amazon/Mesh storage account direct from your desktop ... via your OpenID account ;)
2. Allow direct storage and management of pictures, video and so on on the clound (maybe on top of (1)).
3. Perhaps even EC2 could be somehow synchd with Netbooks to allow complex calculations to be carried out on the cloud. Idea here?

In short there is potentially a huge opportunity for services to get on board this new wave of laptop and whilst rolling out an empty laptop works fine for me, there are many others who could climb on board the social wagon if they were given some help along the way!

Who else uses a Netbook? What do you think?

Xtranormal - create your own 3D animation

Keeping the kids amuzed on a Sunday morning can be somewhat slow, but when @joecar mentioned Xtranormal to us this morning we tried it out... and how addictive it has become. To me, sure - but my 5 yr old loves it.

So we created a story (i helped him out but all the ideas, script, names and so on were his and you can perhaps guess someof his influences!).

Here is it below - you can also see it here and in youtube . I suggest you try it out - some things still to be worked on, but in general very, very well done.

OpenID "Friend" based Attributes

Something i have been thinking about in line with some recent discussion on the OpenID mailing lists has been techniques to verify someone's identity and the attributes that are associated with it.

My initial thoughts comes up with the following:

Reputation
In this case, the longer you use your OpenID, the more people will come to know it is associated with you. I’d be interested in how we could explicitly extend this concept to support a distributed reputation system where you can attach OpenID Reputation points which are assigned from other sites. These could even be broken down into types of reputation. The longer and more you use it the more reputation points. Even sending an email that isn’t spam etc via Gmail or Windows Live could “increment” your reputation score (like pagerank, but for OpenID).

Friend Verification
Slightly more explicit and extension of the above whereby friends can verify things you write down about yourself so that others can trust them more. John says he works at Google Inc and Brian has verified this etc.

Central Validation
Shibboleth model and probably needed for institutions who perhaps verify things such as “Yes, this is definitely Dr. Livingstone” etc.

More thinking
However, i can't think of a reason why OpenID AX couldn't support attributes from third parties that are signed and stored with the OpenID. This way an institution could be given a users OpenID, make statements about them and return the signed statements/attributes which can then be stored anywhere with that users OpenID profile.

Someone who wants to get those attributes can easily check via the insituions publically available key that thse attributes have not been altered in any way.

Extending this you could have a plethora of OpenID attribute information that is both unverified and strongly verified.

OpenSearch and JSON

In the limited spare time that i seem to have these days, one of the things i've been looking at is OpenSearch, "... a collection of simple formats for the sharing of search results".

OpenSearch is slowing gaining lots of traction and each of the major browsers now allow you to add an OpenSearch to your toolbar automatically (although the User Experience certainly needs work!).

In my current consultancy work for the NHS I was asked to extend the search API I wrote some months back which works against FAST, and enable any third party site to ended some script in their site and make Ajax style queries. Last week I wrote this functionality (which returns custom Xml and Json) and sure enough it works great.

However, I like open stuff because it makes it easy for others to pick up and run and so after I added OpenSearch support via the usual description file and tested this with Internet Explorer et al and it worked, i decided to look at whether i could use OpenSearch as the actual syndication format.

Now, if this all ran on the same domain there would be no problem in pulling back Atom or RSS and formatting it for display. But in a cross domain architecture you can't do this due to Same Origin Policy. So, using JSON-P we can enable OpenSearch queries over the wire.

To do that however, we need to have an OpenSearch format for JSON - in my case i simply took the Atom format as a starting point and mapped much of that and added in the OpenSearch elements and attributes.

So now you can take a few lines of code, add it to any page in the world and make an OpenSearch query using the template Uri and get an OpenSearch response which can be parsed and dislayed.

Early days but if it's useful to no-one else on the planet, it is useful to me :)

If interested, please try it out here. It queries over 100 million records so you will get results no matter what you type in, but the formats and data mappings are still being worked out.

PS. Would be *very* nice if we could query the OpenSearch discovery endpoint and ask for JSON to be returned. That way we could dynamically build all of the front end stuff that i can't do with the existing reponse (it returns an OpenSearch specific content type which is not trusted by browsers).

Grassroots rail support

I've travelled on trains, planes and automobiles for so many years now and one thing that has yet to be resolved is telling you what is happening as it is happening. You are often left to try and guess what the problem is, or when the train will arrive - and this is worse at night when there's often no-one around to ask. Well, for years I always intended to build a grassroots support network for transport as the profilferation of mobile devices enabled more people to become involved using SMS and mobile browsing.

Well, below is a rough sketch (it's not too detailed as my son is pushing me to watch an Indiana Jones puppet show he has created, but you'll get the idea).

Thoughts on the Cloud API

I have seen quite a few diagrams on cloud computing but not something like the one below - (please tell me if you know of more and i'll update!).




















You see what many leave out is that if you build your service on a third party service in the cloud and it closes down, you're screwed. Well, currently you are. You need to find a new provider, swap your data, change the api, modify your internal code. Which in simple terms, means your're screwed.

Now, it's not such a big deal if Amazon is your service (although this should still be a consideration) but if the POINT of all this open web stuff and open source and api's etc etc is to allow the little guy to provide services, then it IT a big deal. The little guy will just NOT get the trust for many businesses and it is a shame. Once their ideas are bought by a bigger company then they get exposure. But surely this is a shitty way to do things. Why not let the little guy in?

A good way for this to happen would be for the "cloud" (and in Scotland we have cloud computing every day of the year, with torrential rain computing thrown in) to support common (i hate saying standard!) API's. oAuth is arguably the most successful example of that to date (with PoCo to come), but OpenID is also another good example (oAuth is not used seamlessly by many a big company).

But surely we need to look at common API's for all services - or at least the service providers do. I want to use provider A today and provider B tomorrow. It should be a configuration change for the client (and a "swap provider" checkbox ;) ).

Internally however there should be a service provider data exchange API that allows swapping of data (and arguably hot swapping for back up - imagine service A and service B backing each other's data up whilst providing the same service!?). So when i ask to move, i can have all my data moved to the new provider (for a fee of course). Here's that API :

data.Transfer(newProviderURI);

OK, so it's a little more complex than that, high level is a good place to start. I also like the idea of a higher level cloud API that provides common services for all kinds of cloud services. I know there has been lots of work on various SOAP style services at W3 and so on (transations, instrumentation etc) - not sure how this thinking all fits in with the cloud stuff but would love to hear/read about it.

One Web, Many Cultures - Reaching Out

I'm not entirely sure where this post is going, nor do i have any conclusions in mind. I have simpy been thinking about this for some years and feel like writing it down. I'm not sure what tone it will have for the person reading, but it is intended to ask some questions rather than make any kind of statement! Let me first say that i am from Scotland and have live in England, Italy, Canada and Chile in the last 15 years - i wrote books and used the web constantly in each.

I listened to Kevin Marks interview with Jemima Kiss at the Guardian and other than Kevin sounding incredibly similar to a friend of mind, the part that jumped out of me was his comments that many of the big social networks have done well out of users they just didn't expect. US technology companies growing "execpectedly" from string user bases in Russia, Brazil, China and so on - in other words from users NOT in North America.

So, I wrote on Twitter :

"that was an interesting interview Kevin. Given the cultural luck many networks
have do you think we need to put more work in involving other cultures right
from the start rather than once the technology is ready?"

Kevin Marks of Google made the point to me on Twitter that :

"the technology is ready, and has been for a while. Finding people to
bridge app ideas to other cultures and language is still hard."

My wording wasn't great (140 characters isn't much!) so this is what i meant....

Much of the technology driving the web is coming from specific locations in the US - mostly California - there is some input from the rest of us through discussion groups and so on, but primarily it is driven offline. This is understandable on many points due to the huge concentration of top people over there who are much like myself - driven and excited by the potential of the web as a platform that can fit into the daily life of everyone on the planet - in the many different ways people will use it. Also, at the moment virtual working groups are still a work in progress.

Now, oftentimes i get frustrated that i'm not amongst it - i had hoped to some years back startig with OpenID - (every week on twitter there is another cool event somewhere in the US where all the guys and gals get together to push these things forward). However, something that must be more frustrating is little involvement other cultures in the rest of the world must feel - at least i speak (bad) English and can follow in the forums and blog posts (and every so often post in the forums). What if you speak Chinese and want to use oAuth? Or Spanish? Say you want to add to it or suggest a change? There are a few involved, but this is very much the exception and if you check the board membership for most of these initiatives you can see they are based in the US and often (but not always) part U.S. web companies.

I think it is a problem waiting for a solution - the problem is how to integrate other languages and cultures so that the open web can be something that fits for everyone. If Orkut surprisingly managed to get a dominant user base in Brazil, was there something different to what other social networks were offering which made this happen? Does anyone know? In Africa it is suggested the uptake of mobile phones will continue to be huge - how do these efforts fit into their plans for a social web?

There are many of these folks who do a lot of evangelizing for these efforts so perhaps they can get to know active people who could get involved in the core specifications from Europe, Asia, Africa, South America and so on. It's worth remembering this IS one web, but has many cultures and what fits one may not fit all.

Thoughts on Using Netflix’s New API

OK - I have read through Using Netflix’s New API: A step-by-step guide by Joseph Smarr and tried out a few of the things mentioned.

My only irritation in the excellent efforts Netflix have done are around the anomalies you can see in the API workflow.

i'm sure they have some good reasons for diverting from the "standard" process and perhaps some view to these may be useful.

In addition it may suggest some extension points are needed that could be added to oAuth so that rather than hacking these things we could simply add some optional extensions (which are generic such as "dynamic parameter" rather than "userid") and can be plugged into any oAuth library.

The fact this can even be done is excellent but in using it it would be nice to just have a graphical pipeline for every site that shows where they may have used some of the optional filters during the oAuth process.

Copy and Paste between OpenID's

In case you missed it i've added a new feature that allows you to Copy and Paste data between OpenID's.

More information can be found here.

Syndication in 2009

Had some thoughts on what could be done with Feed Aggregation sites such as http://weblivz.com/ and where it may all go. I decided to consider where we are, the problems we currently have and the inevitable move towards syndication for all.

This is a picture of where we stand today - a very much limited set of pre-defined feeds we can subscribe to (some such as twitter are doing some good work on improving this for their search).





















So where are we going? Well, everyone needs to open up - and there are a whole load of good advantages to doing so. Personally, i think the fact we will be able to write complex queries is a major advantage - users are just never going to learn anything more than AND/OR - but we can improve their results by employing the advanced capabilities of search engine logic and API's.

Additionally we want to start using the Semantic Web, but how a user would work with Google Base or FreeBase etc isn't entirely straightforward. It really comes down to either providing an interface they can build upon or writing the queries for them - perhaps in a templated more to substitute their specific query (ala OpenSearch).

Whatever method is used people want RSS/Atom. Feeds just now are starting to get out of control because even a reliable source have a small percentage of data you really want. We now need to be able to write custom queries to request exactly what we want. To join, filter, deduplicate and so on.

I fully expect in the next year or so that the advances that have already started in these areas will only gain momentum and make it a hell of a lot easier for us to get results.

Will be see a "RSS" button on each page of the Google Search results by the end of 2009?

Happy Belated Anniversary Google

I kinda wish i had spent more time on this the other day but i was busy doing other stuff. Thanks to a link from DeWitt Clinton (who now works at Google) on Google jobs back in 1999, i played around with archive.org for the first time in ages.

People may already have found this, but i had to blog anyway about some gems i found that blew me away.

1. The Google Beta Search Engine in 1998 - link
















2. The "Google Friends" mailing list which i suspect was the first public message about Google. It was written by Larry Page himself on 28 April 1998 at 10.28 PM (so must have been a late one!). I love message 5 "Google gets funding" !!

















The "Group Info" suggests 1 post a month. I like that either Larry or Sergey categorized the group as "Culture & Lifestyle : Gender : Women, Girls, Mothers, Daughters".




























I did actually send a "Happy Birthday" message to "google-friends-subscribe@makelist.com" - it's nice to be nice. We made a big deal of my son's 5th birthday and i reckon i see Google as much as him these days! It hasn't bounced so maybe i'll get a reply ;)
























3. This is the first public email (mentioned above) i am aware of by Larry Page himself about Google.
























4. There is also the first email from Larry Page and another in July 1998 where he talks about the new features. For anyone stressing out over their servers, here is surely one of the historic paragraphs of the Internet:






Combined "our server" (some suggest there is now > 64,000 servers) and "try back in a minute or two". Come on guys - how did you get funding with that line ;) If this kind of thing doesn't inspire you as an entrepreneur you're maybe in the wrong job!

Look around and let me know if you find other similar gems.

This reminds us these were two guys who were like every other entrepreneur at the start - they had no idea where they would be in 10 years.

Hey, they'll likely never ready this, but Well Done - i consider myself inspired !!

Think i'm starting to really think about writing a book on this kind of thing. Amazing!

PS. I forgot to mention the Google Stickers... check this.

The art of a search query

One thing that i don't see much coverage over is how non-technical users get the power afforded by advanced querying. Something that drive me to create something LIKE http://weblivz.com/ was through my experiences with querying in a project which involved many hundreds of millions of distributed records.

In that project we had a team over over 20 experts writing specific queries that were semantically relevant to the area they were in and the output of one search was actually a fairly complex backend query most of the time consisting of the union of multiple backend queries all reformatted for a specific output.

Today many sites - and emerging distributed query sites - are focused on simple queries, but this requires that you KNOW semantically what to look for and that you want to type it in all the time. In addition it assumes you know how to construct fairly complex (it's all relevant) queries.

So, YES, we need all these cool sites that do the keyword searching. But we ALSO need something a bit higher level. We need to hide the user from the complexity of searching and also make it easier for them to remember the kinds of searches they either constructed or used before.

A syndication formatting cache

I'm really thinking about this stuff just now, so this note is as much use for me as anyone else.

We have a ton of sources all working with Atom/RSS formats but being semantically different and in cases extending the same concepts in different ways (e.g. Digg has it's own namespace in its Atom feeds for authors).

Imagine a service that indexed and transformed these sources to normalized formats. So you could basically do XPATH style queries (the interface wouldn't be so complex of course) on the RSS/Atom sources and not only get the data in a given element, but be semantically accurate on what you are getting.

In addition, extension namespaces could also be queried, so you could ask for media items from youtube, flickr, meefedia and so on and get an accurate result.

This service may even be useful if we were all using the same format, but at a time where joining feeds is near impossible when you are thinking about the user, it may be useful to have a service helping out.

I've already written a bunch of tranforms to do this and to be honest had to write an Xslt for every single feed i got (i think delicious was ok), so i know the headache as others look for more advanced syndication feeds!

Long tail of atom extended formats

I picked up on a post by Andrew Turner on OGC Geospatial Search Summit

“Of course, a format can expand upon this and offer more complex formats that
conform to more complex specs. But by at least providing a common baseline means
that almost any service can easily interconnect with another service.”

I can see why we need to stop at some point! The issue is that in the long tail, these extended formats are quite prevalent and I’d like to see extended communities supporting people who want to extend. The reason I say that is that even in the rich media space I have numerous Xslt’s, function calls and so on to normalize what is essentially the same data. GeoRSS is an example of a specific community that does it well!

My thinking is that if (at the extreme) two companies in the world extended for a very specific topic, we could at least get some normalized view of the data for everyone as a response from an OpenSearch query.

Opening up advanced search syndication

[update: 24 Sept @ 12:07 PM : Google Chrome also supports OpenSearch]

[update: 24 Sept @ 11:45 AM : Through the OpenSearch group, I see MS IE 8 Beta supports OpenSearch so perhaps even more sites will realize more complex searches are needed]

As you may well now be aware, I just launched an Alpha release of weblivz.com and what I tried to do there is (in many cases) write intelligent query sets against sites that provide the results as RSS or Atom feeds. So rather than just pulling in every feed we can find, we actually create a query such as "Europe and technology" and so on. It really isn’t easy and requires a lot more work that isn't visible up front. Here are the five issues:

1. Search, no feed
2. Query syntax
3. RESTful
4. The commercial clause
5. Semantics of response formats

I will provide one example in each section, but there are many others I have come across.

Search, no feed
In many cases you can get RSS or Atom feeds from static pages, but as soon as it comes to searching and gathering the results as a feed, you’re in trouble. One example if MeetUp.com.

I can do a bunch of querying to get certain feeds but as soon as I want to something such as "languages and Glasgow" I’m out of luck. In short, you get exactly what you want with most search queries on some of these excellent sites, but they only work when you are ON the site.

This misses the opportunity of syndicating the results to third parties to allow them to point at YOUR site. You end up with having to use the limited feeds available and most of the time this isn’t much use to anyone – especially in the era of content overload and the increasing importance of providing the user with what they want.

Requirement
Ensure all the results from your search query can be syndicated as RSS or Atom. The extra queries against your server will be balanced against higher profile or extra hits on your site.

Query Syntax
In short, query syntax is all over the place. In some cases you can only search for one term and in other cases you can’t use AND or OR. There is a real lack of support for doing interesting things – if you really want to customize the feeds, in most cases you are seriously limited by what you can achieve. One example is http://eventbrite.com.

If I want to search for Tech events in Glasgow or Edinburgh I really need to do 4 separate queries – "tech glasgow", "technology Glasgow", "technology Edinburgh" and "tech Edinburgh". This is a waste of resources all round for something that is relatively simple to achieve.

In some cases typing "or" doesn’t mean the same as "OR" and in others typing "and" gets interpreted as part of the querying rather than ANDing the terms.

Requirement
If we are not going to use http://www.OpenSearch.org then the site needs to at least provide a powerful search interface – perhaps more powerful than any individual is going to use, but in cases such as http://weblivz.com we are actively creating powerful queries against your backend database – saving everyone resources!

Give us a REST
For the majority of queries, a RESTful URL should be enough to get the results. Granted many sites already support this but there are others that provide access only through a POST API XML based syntax. This is good for more advanced queries but sometimes you want to write something simple that returns results in a given format, without passing an extended collection of parameters.

Requirement

If I can’t type (something like):

http://www.yoursite.com/search.atom?query=tech+Glasgow

… then you really need to think about adding this functionality.

The commercial clause
Now this is one I simply DO NOT GET. Many sites say the data can’t be used in a commercial context, without really defining what that means. I have some sympathy for this when it is data you have collated and published – such as a postcode search facility or something... syndicating that would render visiting your site close to pointless.

However, when it comes to user generated content I just cannot understand it. Many sites allow searching, feeds and then insert a clause saying you cannot syndicate the data without a license. The point of these feeds however is to provide a "teaser" to bring the people TO your site… so even in a commercial context, surely allowing your feeds to be displayed can ONLY work for you.

http://www.mystrands.com but it’s got "commercial" all over it (it also doesn’t have RESTful URL access to feeds). So do we write YET ANOTHER MyStrands or do they just provide intelligent syndicated search feeds we can all use and drive business to our and their site.

Requirement
Remove this kind of clause.

Semantics of response formats
This particular part almost drove me to distraction. We now have RSS and Atom as the key formats of feeds – sure there are some variations in versions but we are pretty close to two general formats.

So where is the problem with this? Well, the problem is twofold. The first is different interpretation of what goes into each field and the other is the extensions used within the feeds and the variation on how these are semantically interpreted.

The first of these is particularly an issue with "content" and "summary". Some people put in a short description, others put in formatted html. Some don’t’ put a summary and only add content so you need to parse this somehow if you want to display some kind of a summary.

In addition to this you may find some sites (such as FriendFeed) provide much of the information that should be in atom fields (such as the author) embedded within the content so you would need to parse that to give any kind of standard view.

Now, the extensions is altogether more of an issue. Just try combining some of the feeds using the xmlns:media (http://search.yahoo.com/mrss/ ) namespace. Sometimes the link is in the atom elements, others it’s with the media player element, sometimes the author is in the media credit and others it’s in the atom author field.

You need to parse some of these to death just to get a standard output – in fields where the output should really be a specific extension of the core RSS or Atom specifications. This is a nightmare when applied to video, photo’s, music and so on and makes intelligent search and syndication very difficult.

Call to action
We really need to change some of this. It’s not like we need any scientific breakthrough to make this work – we just need to come to some kind of agreement on the points I outlines above – all the difficult technical stuff has already been done. It doesn’t require ripping software – just extending it.

If you provide an Atom feed you may not want to change that but adding a version parameter in an API is easy. That way you can provide the "new" improved feed. Really the best option is to look at http://opensearch.org but there are any number of options I would happily accept.

We also need to generally improve search and syndication and realize it is not something that takes people away from your site, but rather drives them to your site. The better your search APi the easier it will be for sites like http://weblivz.com to integrate your feed with specialist and content sensitive queries. Users will like that and they will come to you through all sorts of gateways!

Please take away the commercial stuff, or at least tone it down. In most cases people will come to your site if they see a teaser of what they want no matter what site they are on.

Feedback
Thanks for reading. This was based on my experiences creating the weblivz website. I’d love to hear feedback – good or bad. If you have pointers or want to point out anything right/wrong or additions, please suggest and I will update the article.