Monthly Archives: September 2006

Building the web with links.

Scott Savage is the guest blogger up at TechTalkBlogs. Last week he posted up a note about how online communities are built with links. This is one of the reasons that blogs seem to bubble to the top of search rankings because bloggers link to each other and have a converstation about a topical issue.

This was actually one of the reasons that I spent so much time ensuring that my old notgartner.com links continued to push through to my new URL.

Blogging Frequency

Darren is sitting across from me in the office hassling me about getting my blog up to date. Apparently twenty one posts per month isn’t enough. Of course, if he looked at the graph I’m not going too badly.

You’ll notice the tool that I am using here Darren – it’s your favorite 🙂 

Blogging is a high latency hivemind uplink.

I’m a huge fan of the Elliot.org newsletter. In the latest issue two links caught my attention:

The first one struck close to home because of some stuff that I am working on at the moment. I faced an ethical dilema this week when I discovered that the travel industry routinely overcharges the customer (and then refunds money) because of the challenges of aggregating the products of multiple vendors into a single travel package.

While this isn’t quite the same issue as what is described on the Elliot.org article, it does underscore the need for full disclosure.

The second article was just interesting in how bloggers start to have an impact because service providers are starting to realise that they can impact the bottom line. The interesting thing about it is how peoples experiences propogate around the blogosphere and how it can actually start to change peoples perceptions who are not bloggers themselves, and perhaps not even connected to the blogosphere at all.

I’m starting to think that what we are seeing in blogging is a first generation hivemind uplink (albiet a high latency one – new information doesn’t always get good absorbsion in a small period of time).

The blogosphere is more than just a loud speaker where a message is broadcast and quickly forgotten. Blog posts are largely durable (sure there is some information loss but it is a slow leak) which means that the message can actually spread further and affect a lot more people. If those people have similar experiences then they echo the post and you get a bit of an echo chamber effect.

Business is going to need to start recognising this because if they let the negative noise level around their company build up in the blogosphere it may be hard to recover. If that is true, how does a company combat the negative noise in the blogosphere? Do you need to have 1000s of bloggers like Microsoft?

Divide and Conquer

It started out with David Lemphers posting up about this article called The Expert Mind(pages 1, 2, 3, 4, 5, 6)  Darren chimed in and then Geoff added to the conversation (don’t you just love the way the blogosphere worked). Darren must have figured that I would be interested in the thread of conversation so he IM’d me a link to the David’s post – he was right.

The article spends a lot of time covering some of the techniques that chess players use to organise their memory such that they can play well. One of the techniques is chunking where the players use the board layout or fragments of the board layout as a key to access winning moves or at least winning behaviours.

David drew a parallel between this and programming and how quite possibly there are coding movies that we can apply to produce better code (if your memory is a hashtable then David is focusing on the value of the key value pair). Darren looked at it from the opposite angle (the key). They are both talking about different sides of the same coin – unfortunately it is a pretty abstract discussion – lets see if I can ground it in some general observations I’ve made over the years.

When I first _really_ started to learn to program Pascal was my tool of choice, and the person that was teaching me how to program basically used the same top level structure to every program that he wrote:

  1. Initialize
  2. Process
  3. Terminate

These would each be PROCEDURES and the idea was that this represented the lifecycle of pretty much every program ever written and from here you can start to break down the problem. It was a very structured and procedural way of thinking – but it worked.

One skill that this taught me was that the way to approach programming (and problem solving in general) is to divide the problem domain continuously until you find a solution. By dividing the problem domain and focusing on just that division then you remove a lot of the noise which slows you down (and eventually disables you).

I’ve found that this ability has saved me time and time again from getting stuck on one particular problem – all I have to do is remember to break it down. This is fundamentally the same skill that chess players are using (although their capacity for recall seems to be amazing). I think that the challenge that we face at Readify is finding the people that know how to break it down but also have the discipline to actively learn new moves so that they can recommend what the logical divisions in a problem are.

My moves?

I’ve got one for your Dave. One of the classic moves I have is around the double check in threading – the mental trigger for this in my head is actually working with a list in a multi threaded program where multiple threads might me working with this same list and manipulating it. But its only one possible code move, and it doesn’t always win (but it has reasonable probability).

I’m off to play some chess to see if I can exercise my brain a little (mind you, the article did say that the recall skills don’t necessarly span problem domains).

Bug 1894: TFS Integrator fails to kick off one or more builds when more than one are triggered.

This is a bug on TFS Integrator that I just logged on our internal TFS installation. I thought that I would post it here for those that have started using TFS. We will get a fix out for this soon – we may take the opportunity to use Windows Workflow Foundation (now that it is stable) so it could take a few weeks before someone has the time available. Bug follows.

I suspect that Team Build is the problem here in that it doesn’t support queuing on its side of the fence. I uncovered this with the CrmOnTime project. The CrmOnTime project consists of three seperate builds CrmOnTimeCommon, CrmOnTime and CrmOnTimeServices. When CrmOnTimeCommon builds dependencies are replicated to the other two projects. If these projects kick off their builds at the same time they seem to fail if they are using the same build server. If they each kick off on a different build server then there is no problem.

Unfortunately we still have to implement the fix in our code because we are a Continuous Integration engine our users should rightfully expect not to have to run multiple build servers just to get multiple dependent builds kicking off – we just need to queue the requests. I’m thinking that moving forward we should replace the internal state management with an event driven workflow or something like that.

Temporary Workarounds:

Structure the sleep periods such that you allow one build to complete before the other gets kicked off – or use multiple build servers.

Getting Started with TFS Integrator

ATTENTION: TFS Integrator is now obsolete. Please use TFS Continuous Integrator and TFS Dependency Replicator. For more information, read the post on why TFS Integrator is now obsolete.


This is a post that I’ve been meaning to write for a while. Earlier this year, when Chris Burrows joined our team we had the opportunity to spend a little bit of time building a TFS extension that Darren and I dreamed up whilst we were working on a client project together.

As you are probably no doubt aware, Team Foundation Server does not ship out of the box with a Continuous Integration capability. A lot of people, including myself consider this a glaring omission – but as a developer I have to appreciate the demands of tight delivery schedules.

What is Continuous Integration?

Continuous Integration (CI) is the process of continually scanning the source code repository for changes, and, once they have been detected automatically kicking off a build process to verify that the code compiles successfully. In many instances teams will tag on a round of build verification tests to ensure that not only does the code compile, but that it doesn’t catch on fire when you try to run it (smoke tests).

While not unique to agile methods, continuous integration is certainly one of the tools that agile development teams to keep a handle on their code base as it evolves throughout an iteration. It is no surprise then that when agile teams came to use TFS that there was a collective “what the” when it didn’t include a CI engine.

How did we build our TFS CI engine?

Fortunately for us, the team at Microsoft did build TFS with an eye on extensibility and they included in the system an “event service” which we can plug into to listen for events in the source code repository. The way it works is that whenever you do something significant in Team Foundation Server, such as checking in a file, that TFS sub-system notifies the TFS eventing system which then broadcasts that message to any subscribers. This is how you get an e-mail when another developer checks in some code.

This same eventing mechanism can actually be instructed to send a notification via the SOAP format to a web-service endpoint which might be hosted in something like ASP.NET, or event Windows Communication Foundation.

TFS Integrator uses this facility to kick off a build in the Team Build component of Team Foundation Server. When TFS Integrator initialises it reads a configuration file to identify which parts of the source tree it is interested in listening to then subscribes to the check-in event notifications that the eventing system sends out.

When it receives a notification it sleeps for a specified period of time then kicks off a build with the Team Build facility. Once the build is completed the eventing system sends how a notification to subscribers as to the outcome. The completed interaction between the system components looks like this:

 

All of this automation makes use of the tools available from the Visual Studio 2005 SDK which you can download from Microsoft.

How to configure TFS Integrator to support Continuous Integration?

Before we start you need to understand that the current build of TFS Integrator relies on the .NET 3.0 runtime (RC1) being installed. For this reason we don’t currently recommend installing it on the same computer as either your application or data tiers. Once you have downloaded and installed the .NET 3.0 runtime (RC1), the next thing that you are going to need to do is download the latest build of TFS Integrator.

The setup package itself does nothing more than drop a few files onto the file system, the rest of the configuration requires some manual intervention. The installation files are dropped into the “TFS Integrator” directory under program files.

The “TFS Integrator” directory contains three seperate configuration files:

  • TfsIntegrator.exe.config
  • ContinuousIntegration.xml
  • DependencyReplication.xml

The first file, TfsIntegrator.exe.config contains the bootstrap information required to get TFS Integrator talking to your Team Foundation Server installation. In this configuration file there are three main settings that you should be interested in. These are BaseAddress, RegistrationUserName and TeamFoundationServerUrl.

BaseAddress is the address of the server itself. We would have liked to have allowed the program to determine this itself, but it can get tricky in multi-homed systems. The value that you need to provide here is the address that Team Foundation Server will be able to communicate to TFS Integrator on via the event notification service. Provided this address maps to the local host, the TFS Integrator service will self register the port as it starts up.

The next setting, RegistrationUserName is the account that you want TFS Integrator to register subscriptions under. Typically this is the same account that is used to run the service (especially in a domain configuration). Finally, the TeamFoundationServerUrl is the address of the Team Foundation Server that TFS Integrator will need to talk to.

Once the initial configuration is done the service can then be installed. So that it can run headless. You do this by running the “TfsIntegrator.exe” program with the “-i” switch.

 

After the installer registers the service you just need to go into the Services MMC snap-in (services.msc) and specify the logon credentials that the TFS Integrator service is going to use. This account should have access to TFS, be the same account as specified in the RegistrationUserName and have sufficient rights to write files to the “TFS Integrator” directory in program files.

Now that the basic configuration for TFS connectivity is out of the way its time to actually modify the ContinuousIntegration.xml file. The ContinuousIntegration.xml configuration file contains all the information that TFS Integrator needs to trigger the build of a “team build type” in Team Foundation Server.

The root element in the document is ContinuousIntegration, and under that you can have one or more TeamProject elements. The TeamProject element defines the name of the team project to which TFS Integrator will listen to events for. This element in turn contains multiple Build elements. The build element defines what path in the version control store to listen for changes in, what build type to kick off, and on what build machine. The sleep attribute defines the settle period that TFS Integrator will wait for further check-in notifications before spinning off the build.

Once you have made the modifications to the above configuration file you should be able to start up TFS Integrator. Congratulations! But what about DependencyReplication.xml?

What the heck is Dependency Replication?

Dependency Replication is a process that most development teams have to do as the scope of what they are trying to achieve grows. The idea is that within your organisation you might have a set of common frameworks that you use and maintain. Rather than linking in the source code into each new project that you undertake you treat the framework like an internal product which you build seperately.

The problem you then have is integrating the updated binaries into the other projects that depend on them. This can be quite a time consuming process, so much so that teams will often give up and end up compiling hundreds of different versions of the same code into their code base – which creates a maintenance nightmare.

TFS Integrator includes a dependency replication engine which extends the continuous integration feature. It does this by listening for the build completion even from TFS and using that to trigger the execution of some code which picks up the compiled build output and checking it back into Team Foundation Server in a specified location.

The effect is that with dependency replication the feedback loop is complete and dependant components can be linked together.

In the case of the framework example we could build up a scenario where TFS Integrator gradually rebuilds your entire code base.

  1. Developer checks in some code in the framework code base.
  2. Check-in notification sent to TFS Integrator.
  3. TFS Integrator kicks off a build in Team Build.
  4. Build completes.
  5. Notification of successful build completion sent to TFS Integrator.
  6. TFS interator checks in some of the drop files into version control.
  7. Check-in notification sent to TFS Integrator.
  8. TFS Integrator kicks off a build in Team Build.
  9. Build completes.
  10. Notification of successful build completion sent to TFS Integrator.
  11. Rinse, repeat.

In the past I’ve spent significant amounts of time getting similar systems going on top of version control systems like Visual SourceSafe where sharing was used as the replication mechanism. I actually think that this approach is much cleaner and more obvious.

How to configure TFS Integrator to support Dependency Replication?

Like continuous integration, the configuration of dependency replication is controlled via a configuration file – in this case, DependencyReplication.xml.

The root element of the configuration file is DependencyReplication. Within it there are multiple TeamProject elements which define the TeamProject and what the root folder is (I’m not sure what the root folder configuration item is off the top of my head, but I suspect its the relative path to which check-ins will be made). Within each TeamProject element, one or more Dependency elements are defined which specify which build type the team  project depends on. Inside each Dependency element is a File element which specifies the drop location relative source file any dependencies that need to get checked back into source control. The destination is the fully specified path to put those files into version control.

In the configuration file above we are saying that SystemB depends on some files from SystemA (Output1.dll and Output2.dll) and that those files should be checked back into the $/SystemB/SystemB/Dependencies location in version control. The interesting thing to note is that the file name is also specified in the Destination path. This allows configuration managers to rename files as part of the replication process.

Once both continuous integration and dependency replication configuration is complete, you need to restart TFS Integrator to get it to pick up its latest settings.

Next Steps

If you are interested in trying out TFS Integrator in your environment, just go ahead and download it. We are very keen to see people use this tool because it is genuinely useful to us, and we think it will be useful for you too. We probaly won’t be giving away the source code at this stage but unless a future version of Team Foundation Server makes it redundant we will continue to make revisions to the code base and support it (as best we can).

If you want to get support for TFS Integrator then I recommend that you join the OzTFS mailing list and community which was recently setup by Grant Holliday, you can subscribe via e-mail. Once you are subscribed you can send an e-mail with your question. If you can’t post to that mailing list for any reason, just send an e-mail to me.

Thanks

Finally – before I sign off on this post, I want to thank a few people. TFS Integrator was not developed by me – in fact I can’t even claim sole responsibility for its design. Both Darren Neimke and Chris Burrows both helped get this software out the door. Chris in particular put in hours above and beyond the call of duty to get this out the door.