Getting Started with TFS Deployer

Earlier this year I announced the availability of TFS Integrator, a continuous integration and dependency replication engine for Team Foundation Server. Since then we haven’t been idle, in fact we have been working on another tool that will save you time in your release management processes.

Introducing TFS Deployer

TFS Deployer is installed as an agent on your test and production systems and supports the execution of PowerShell scripts when an event happens in TFS. In particular it listens fo build quality change notificaions which occur when you change the quality of a build under the Team Builds node of Team Explorer.

The way it works is that the release manager, using Team Explorer updates the build quality indicator in the build store via Team Explorer (1), the build store then notifies the event service (2) which in turn distributes notifications to all the event subscribers. In this case you can have one or more TFS Deployer installations listening across multiple machines (3). When TFS Deployer is notified it doesn’t initially know whether it needs to do anything. To determine this it grabs a deployment mapping file from a well known location in the version control store (4).

The deployment mapping file lists out each of the servers and what that server should do in the event that a build is transitioned from quality state to another. What happens is encapsulated in a PowerShell script file (*.ps1) and once that script file is identified TFS Deployer will instansiate a PowerShell host environment and execute the script (5).

Since we finished implementing TFS Deployer Readify has been using it internally along with TFS Integrator to enable us to rapidly modify, build and deploy our timesheeting system. In the remainder of this post I am going to walk through what you need to do to get up and running with TFS Deployer.

Installing TFS Deployer

Before installing TFS Deployer you first need to download and run the setup package. The setup package is a small MSI that simply drops the program files and the default configuration files into the C:\Program Files\TFS Deployer directory - from there you need to perform a few configuration steps which we haven’t fully automated yet.

The first step is to get the service to self register. This is done by executing the TFSDeployer.exe program with the -i switch. Once this is done, open the services MMC snap-in (services.msc) and scroll down to the TFS Deployer service. Change its logon behaviour to use an account that has access to Team Foundation Server.

Before starting the service review the settings in the TfsDeployer.exe.config configuration file, the relevant settings are:

• FromAddress; this is the address that all e-mails are sent from.
• SmtpServer; this is the address of the SMTP server that e-mails are sent via.
• ToAddress; this is the default address that TFS Deployer users to notify when a build either succeeds or fails.
• BaseAddress; this is the address that Team Foundation Server will use to contact this TFS Deployer instance. This address should be addressable from across the network so localhost is not acceptable.
• RegistrationUserName; this is the name of the account that the process is running under and which the subscriptions will be assigned against in Team Foundation Server.
• TeamFoundationServerUrl; this is the address at which TFS is addressable across the network.
• UseDefaultCredentials; should always be true unless you are using TFS Deployer in a cross domain configuration.

Once you have updated the configuration file you can start the TFS Deployer service. At this point in time TFS Deployer will connect to Team Foundation Server and subscribe to all build quality indicator change notifications.

Preparing Deployment Scripts

Deployment scripts are associated with specific Team Build Types and are designed to be triggered when the build quality indicator transitions from one state to another.

The configuration information which controls whether an instance of TFS Deployer springs into action is stored under version control in a Deployment sub-directory under each Team Build Type directory. For example, in the screenshot below we have a Team Build Type called CrmOnTimeContinuous which is part of the CrmOnTime team project, so the path that TFS Deployer will look for its configuration/scripts is $/CrmOnTime/TeamBuildTypes/CrmOnTimeContinuous/Deployment.

The DeploymentMappings.xml file quite simply lists out which script to execute on what computer when it transitions between two states. It also lists the e-mail address to send a notification to when the script succeds or fails. The screenshot below is a sample script.

TFS Deployer will download this file and if it has a match it will pull down the rest of the files in the Deployment directory and hen execute the script that was specified. In this case it would execute UnexaminedToTest.ps1. What the PowerShell script does is completely up to you, but one of the interesting things to note is that the BuildData (refer to Team Foundation Server SDK) object is actually exposed to the script and the script below shows how to exploit that so the script can grab the files from the drop location.

Once you have the script writen you are ready to test it out, hopefully it works as well for you as it does for us. If you have any issues using TFS Deployer please send me an e-mail or leave a comment against this post and I will help diagnose the issue.

Credits

TFS Deployer was the product of several people putting in hours above and beyond the call of duty. I’d like to thank Chris Burrows for building the core of TFS Deployer and extracting out a common TFS Listener component, and I’d like to thank Geoff Appleby for doing some of the early analysis around how to best integrate with PowerShell. Finally I’d like to thank Darren Neimke for letting us try out this deployment solution on one of his internal projects.

Bug 1894: TFS Integrator fails to kick off one or more builds when more than one are triggered.

This is a bug on TFS Integrator that I just logged on our internal TFS installation. I thought that I would post it here for those that have started using TFS. We will get a fix out for this soon - we may take the opportunity to use Windows Workflow Foundation (now that it is stable) so it could take a few weeks before someone has the time available. Bug follows.

I suspect that Team Build is the problem here in that it doesn’t support queuing on its side of the fence. I uncovered this with the CrmOnTime project. The CrmOnTime project consists of three seperate builds CrmOnTimeCommon, CrmOnTime and CrmOnTimeServices. When CrmOnTimeCommon builds dependencies are replicated to the other two projects. If these projects kick off their builds at the same time they seem to fail if they are using the same build server. If they each kick off on a different build server then there is no problem.

Unfortunately we still have to implement the fix in our code because we are a Continuous Integration engine our users should rightfully expect not to have to run multiple build servers just to get multiple dependent builds kicking off - we just need to queue the requests. I’m thinking that moving forward we should replace the internal state management with an event driven workflow or something like that.

Temporary Workarounds:

Structure the sleep periods such that you allow one build to complete before the other gets kicked off - or use multiple build servers.

Getting Started with TFS Integrator

ATTENTION: TFS Integrator is now obsolete. Please use TFS Continuous Integrator and TFS Dependency Replicator. For more information, read the post on why TFS Integrator is now obsolete.

---

This is a post that I’ve been meaning to write for a while. Earlier this year, when Chris Burrows joined our team we had the opportunity to spend a little bit of time building a TFS extension that Darren and I dreamed up whilst we were working on a client project together.

As you are probably no doubt aware, Team Foundation Server does not ship out of the box with a Continuous Integration capability. A lot of people, including myself consider this a glaring omission - but as a developer I have to appreciate the demands of tight delivery schedules.

What is Continuous Integration?

Continuous Integration (CI) is the process of continually scanning the source code repository for changes, and, once they have been detected automatically kicking off a build process to verify that the code compiles successfully. In many instances teams will tag on a round of build verification tests to ensure that not only does the code compile, but that it doesn’t catch on fire when you try to run it (smoke tests).

While not unique to agile methods, continuous integration is certainly one of the tools that agile development teams to keep a handle on their code base as it evolves throughout an iteration. It is no surprise then that when agile teams came to use TFS that there was a collective “what the” when it didn’t include a CI engine.

How did we build our TFS CI engine?

Fortunately for us, the team at Microsoft did build TFS with an eye on extensibility and they included in the system an “event service” which we can plug into to listen for events in the source code repository. The way it works is that whenever you do something significant in Team Foundation Server, such as checking in a file, that TFS sub-system notifies the TFS eventing system which then broadcasts that message to any subscribers. This is how you get an e-mail when another developer checks in some code.

This same eventing mechanism can actually be instructed to send a notification via the SOAP format to a web-service endpoint which might be hosted in something like ASP.NET, or event Windows Communication Foundation.

TFS Integrator uses this facility to kick off a build in the Team Build component of Team Foundation Server. When TFS Integrator initialises it reads a configuration file to identify which parts of the source tree it is interested in listening to then subscribes to the check-in event notifications that the eventing system sends out.

When it receives a notification it sleeps for a specified period of time then kicks off a build with the Team Build facility. Once the build is completed the eventing system sends how a notification to subscribers as to the outcome. The completed interaction between the system components looks like this:

 

All of this automation makes use of the tools available from the Visual Studio 2005 SDK which you can download from Microsoft.

How to configure TFS Integrator to support Continuous Integration?

Before we start you need to understand that the current build of TFS Integrator relies on the .NET 3.0 runtime (RC1) being installed. For this reason we don’t currently recommend installing it on the same computer as either your application or data tiers. Once you have downloaded and installed the .NET 3.0 runtime (RC1), the next thing that you are going to need to do is download the latest build of TFS Integrator.

The setup package itself does nothing more than drop a few files onto the file system, the rest of the configuration requires some manual intervention. The installation files are dropped into the “TFS Integrator” directory under program files.

The “TFS Integrator” directory contains three seperate configuration files:

• TfsIntegrator.exe.config
• ContinuousIntegration.xml
• DependencyReplication.xml

The first file, TfsIntegrator.exe.config contains the bootstrap information required to get TFS Integrator talking to your Team Foundation Server installation. In this configuration file there are three main settings that you should be interested in. These are BaseAddress, RegistrationUserName and TeamFoundationServerUrl.

BaseAddress is the address of the server itself. We would have liked to have allowed the program to determine this itself, but it can get tricky in multi-homed systems. The value that you need to provide here is the address that Team Foundation Server will be able to communicate to TFS Integrator on via the event notification service. Provided this address maps to the local host, the TFS Integrator service will self register the port as it starts up.

The next setting, RegistrationUserName is the account that you want TFS Integrator to register subscriptions under. Typically this is the same account that is used to run the service (especially in a domain configuration). Finally, the TeamFoundationServerUrl is the address of the Team Foundation Server that TFS Integrator will need to talk to.

Once the initial configuration is done the service can then be installed. So that it can run headless. You do this by running the “TfsIntegrator.exe” program with the “-i” switch.

 

After the installer registers the service you just need to go into the Services MMC snap-in (services.msc) and specify the logon credentials that the TFS Integrator service is going to use. This account should have access to TFS, be the same account as specified in the RegistrationUserName and have sufficient rights to write files to the “TFS Integrator” directory in program files.

Now that the basic configuration for TFS connectivity is out of the way its time to actually modify the ContinuousIntegration.xml file. The ContinuousIntegration.xml configuration file contains all the information that TFS Integrator needs to trigger the build of a “team build type” in Team Foundation Server.

The root element in the document is ContinuousIntegration, and under that you can have one or more TeamProject elements. The TeamProject element defines the name of the team project to which TFS Integrator will listen to events for. This element in turn contains multiple Build elements. The build element defines what path in the version control store to listen for changes in, what build type to kick off, and on what build machine. The sleep attribute defines the settle period that TFS Integrator will wait for further check-in notifications before spinning off the build.

Once you have made the modifications to the above configuration file you should be able to start up TFS Integrator. Congratulations! But what about DependencyReplication.xml?

What the heck is Dependency Replication?

Dependency Replication is a process that most development teams have to do as the scope of what they are trying to achieve grows. The idea is that within your organisation you might have a set of common frameworks that you use and maintain. Rather than linking in the source code into each new project that you undertake you treat the framework like an internal product which you build seperately.

The problem you then have is integrating the updated binaries into the other projects that depend on them. This can be quite a time consuming process, so much so that teams will often give up and end up compiling hundreds of different versions of the same code into their code base - which creates a maintenance nightmare.

TFS Integrator includes a dependency replication engine which extends the continuous integration feature. It does this by listening for the build completion even from TFS and using that to trigger the execution of some code which picks up the compiled build output and checking it back into Team Foundation Server in a specified location.

The effect is that with dependency replication the feedback loop is complete and dependant components can be linked together.

In the case of the framework example we could build up a scenario where TFS Integrator gradually rebuilds your entire code base.

1. Developer checks in some code in the framework code base.
2. Check-in notification sent to TFS Integrator.
3. TFS Integrator kicks off a build in Team Build.
4. Build completes.
5. Notification of successful build completion sent to TFS Integrator.
6. TFS interator checks in some of the drop files into version control.
7. Check-in notification sent to TFS Integrator.
8. TFS Integrator kicks off a build in Team Build.
9. Build completes.
10. Notification of successful build completion sent to TFS Integrator.
11. Rinse, repeat.

In the past I’ve spent significant amounts of time getting similar systems going on top of version control systems like Visual SourceSafe where sharing was used as the replication mechanism. I actually think that this approach is much cleaner and more obvious.

How to configure TFS Integrator to support Dependency Replication?

Like continuous integration, the configuration of dependency replication is controlled via a configuration file - in this case, DependencyReplication.xml.

The root element of the configuration file is DependencyReplication. Within it there are multiple TeamProject elements which define the TeamProject and what the root folder is (I’m not sure what the root folder configuration item is off the top of my head, but I suspect its the relative path to which check-ins will be made). Within each TeamProject element, one or more Dependency elements are defined which specify which build type the team  project depends on. Inside each Dependency element is a File element which specifies the drop location relative source file any dependencies that need to get checked back into source control. The destination is the fully specified path to put those files into version control.

In the configuration file above we are saying that SystemB depends on some files from SystemA (Output1.dll and Output2.dll) and that those files should be checked back into the $/SystemB/SystemB/Dependencies location in version control. The interesting thing to note is that the file name is also specified in the Destination path. This allows configuration managers to rename files as part of the replication process.

Once both continuous integration and dependency replication configuration is complete, you need to restart TFS Integrator to get it to pick up its latest settings.

Next Steps

If you are interested in trying out TFS Integrator in your environment, just go ahead and download it. We are very keen to see people use this tool because it is genuinely useful to us, and we think it will be useful for you too. We probaly won’t be giving away the source code at this stage but unless a future version of Team Foundation Server makes it redundant we will continue to make revisions to the code base and support it (as best we can).

If you want to get support for TFS Integrator then I recommend that you join the OzTFS mailing list and community which was recently setup by Grant Holliday, you can subscribe via e-mail. Once you are subscribed you can send an e-mail with your question. If you can’t post to that mailing list for any reason, just send an e-mail to me.

Thanks

Finally - before I sign off on this post, I want to thank a few people. TFS Integrator was not developed by me - in fact I can’t even claim sole responsibility for its design. Both Darren Neimke and Chris Burrows both helped get this software out the door. Chris in particular put in hours above and beyond the call of duty to get this out the door.

Event Monger Delay

I’m sorry to announce that development on Event Monger (my BTC-FTC project) has been delayed until after the Australian Tech.Ed event (beginning of September).

Unfortunately I’ve been pretty busy over the course of the last month or so on various projects and whilst I’ve been on leave this week I haven’t really had the focused time I’ve required to sit down and tackle this project.

Event Monger has a relatively large scope in terms of the scenarios it needs to support to be useful, as such it breaks the BTC-FTC mold a little bit (which is ironic given that it was supposed to be the first cab off the rank).

So – my sincere apologies for the delay, once I start work on the project again I’ll post up here with progress reports.

Who would like to test our Continuous Integration server for Team Foundation Server?

At Readify we (well, one guy mostly) have been building a continuous integration extension for Team Foundation Server. Obviously building a continuous integration extension for Team Foundation Server is pretty straight-forward but what we have done is try to simplify the process by removing the dependency on IIS (but we do rely on .NET 3.0 BETA 2 currently – for WCF).

The other significant feature is that you don’t need to use BisSubscribe.exe, just install the service onto a seperate build server (we don’t recommend putting it onto the same box running the application or data tiers just yet because of the BETA dependency). When the extension starts up (runs as a service) then it will self register with the TFS event notifications service.

We are already using this build internally and it is working great! We still have a few issues we want to address before putting it out there for wider distribution, but it is certainly a functional piece of software. If you want to try it out just leave a comment on this post or shoot me an e-mail directly at mitch.denny@readify.net.

Who is going to use Event Monger and how are they going to use it?

In my previous post about Event Monger I said that I would post up a list of specific end-user scenarios. A scenario is a dirt simple – plain english description of how someone will use the system. They aren’t really supposed to be a technical description, but more something that you can agree with the customer on – in this case the customer is me so it was easy to reach agreement.

Defining Personas

In Event Monger there are really three different types of users, Organisers, Speakers and Attendees. To help understand these users a little bit more those links go to copies of (very small) persona documents that I have written. If I find out more about the users that will be using the system I’ll update those documents.

The Workflow

The users of the system participate in an overall workflow which results in the running of a successful event. The workflow, combined with the use of Event Monger will have five stages that I’ve been able to distill, these are Install, Plan, Promote, Attend and Review.

Each of these steps in the workflow have scenarios that support them, and those scenarios can be mapped against the personas.

Installation Scenarios

The following scenarios are related to the initial installation of the Event Monger software. I want this to be as seemless as possible. The people doing the work should only need to have some ASP.NET 2.0 hosting space with a SQL Server account.

• Event Organiser
• The event organiser installs and configures Event Monger at a remote ASP.NET hosting provider.
• The event organiser logs into Event Monger and defines a list of other event organisers.
• The event organiser logs into Event Monger and adds event specific flair.

Planning Scenarios

This is really just the initial data entry required to get the event kicking over itself, doing things like setting the themes for the event by defining tracks for example.

• Event Organiser
• The event organiser defines a list of venues, rooms and dates, and provides links to useful information resources.
• The event organiser defines a list of event tracks with a brief overview of what they are about.

Promotion Scenarios

For a technical community event its a pretty good idea to put your ear to the ground and figure out what people want to hear about (within broad guidelines). So this is all about getting speakers to submit things they are happy to present on and then getting people who _might_ attend to vote on those submissions.

• Event Organiser
• The event organiser opens the Event Monger site up for submission of session abstracts.
• The event organiser accepts a session abstract.
• The event organiser rejects a session abstract.
• The event organiser opens the Event Monger site up for voting on accepted session abstracts.
• The event organiser creates the final schedule for the event based on prospective event attendee feedback.
• The event organiser sends a notification to prospective event speakers asking them to confirm their attendance.
• The event organiser sends a notification to prospective event attendees asking them to confirm their attendance.
• The event organiser sends a reminder to event speakers and attendees about the upcoming event.
• The event organiser changes the schedule based on a speaker cancellation.
• Event Speaker
• The prospective event speaker registers on the site and submits a session abstract.
• The prospective event speaker registers on the site and submits an additional session abstract.
• The prospective event speaker logs into the Event Monger site and confirms their attendance.
• Event Attendee
• The prospective event attendee registers on the site and votes on session abstracts.
• The prospective event attendee logs into Event Monger and votes on more session abstracts.
• The prospective event attendee logs into the Event Monger site and confirms their attendance at a particular venue.
• The prospective event attendee registers on the site for the first time and confirms their attendance.

Attendance Scenarios

Once you are at the event the site tends to be forgotten, but it can actually be a useful posting area for all sorts of information. For example, at Code Camp Oz last year a whole bunch of MVPs held a number of informal sessions on a much more diverse range of topics than the Code Camp was really intended to cover. A specific room wasn’t scheduled for them so they just put up on a whiteboard which room they were doing their stuff in – great for BOF sessions too.

Of course – this is also a good time to collect all the collateral that the presenters used during their session if they are happy to share it.

• Event Organiser
• The event organiser opens up the Event Monger site for the submission of community notices.
• Event Speaker
• The event speaker uploads slide decks and any other collateral to the Event Monger site and associates it with their session abstracts.
• Event Attendee
• The event attendee logs into the Event Monger site and reviews and adds to the community notices.

Review Scenarios

That’s a wrap! After the event there are a few things that need to be done like writing up a quick review of the event so that if someone comes to visit the site its clear that the event has passed. Its also a good time to send thank-yous so that hopefully the speakers and attendees come back again next time. I’m not quite sure how this part of the workflow works with multi-venue events that span different time periods.

• Event Organiser
• The event organiser closes down registration and writes a post event synopsis.
• The event organiser downloads a mailing list of all event speakers and attendees.
• The event organiser sends a thank-you message to event speakers.
• The event organiser sends a thank-you message to event attendees.
• Event Attendee
• The event attendee downloads collateral assocaited with their favourite sessions.

Your thoughts?

What are you thoughts about the scenarios that I have outlined above? I’m not sure if I am going to get to all of them before the 4th of August so I’m going to do a little bit of prioritisation. I’d love to hear from you on what is most important, what can be dropped, and have I missed something completely?

Introducing Event Monger

Well, the votes are in and it appears that my “By the Community, For the Community” project is to build an ASP.NET-based event management site – I have decided to call it Event Monger. My objective is to ship this application out on or before the 4th of August which means that I had better get cracking. I’m going to use my blog to document the process of designing and building the application.

What is Event Monger – really?

Event Monger is something that I have wanted to have for a long time as an co-organiser of a technical event like Code Camp Oz. The problem is that all the tools out there to date are really good at content management, but not that great at automating the workflow of an event, including things like:

• Accepting session submissions
• Ranking session submissions
• Scheduling accepted sessions
• Accepting attendee registrations
• Sending attendee and speaker notifications
• Presenting venue information
• Posting of during-event notices
• Accepting speaker uploads

In my next post on Event Monger I am going to list out the specific end-user scenarios that are going to be supported and where they sit in the overal workflow of managing an event. I’ll use the scenarios to drive the development forward and keep track of how things are progressing.

Shrinklet now hosted on SourceForge

Have you been wondering why I haven’t posted up anything about Shrinklet recently? Of course you were, you were just sitting there on the edge of your seat waiting for more Shrinklet goodness weren’t you.

The reason is that I have been waiting to get Shrinklet approved as a SourceForge project. My initial application was rejected because I didn’t provide enough detail and I didn’t notice the rejection notice so it sat there waiting for clarification.

I am now happy to report that the project page is up and running SourceForge right now and I spent an hour or so migrating the CVS repository over including the historical file versions (minus correct date information).

Feel free to go and check it out. Over the next few days I’ll set up the bug tracking and feature request systems the way I want. I will continue update my source code releases on Darren Neimke’s Project Distributor - actually - I hope over time I’ll be able to move everything over there once they start getting SourceForge like features.

Over the coming weeks I hope to start commiting sources for a 2.0 or 1.1 release (not sure depending on the features that get included), at which point I’ll need to branch the repository to do bug fixes on 1.0 versions.

Shrinklet 1.1.0.0 Released

Hi everyone. I’ve uploaded yet another version of Shrinklet, this time the version number 1.1.0.0 which means there has been a minor enhacement (versus just bug-fixes to date).

One of the issues with the previous version was that if you copied a Shrinkster.com URL into the clipboard it will still offer to shrink the URL which is kind of pointless.

So, based on the version 2.0 feedback that I received I have created this minor release to detect Shrinkster.com URLs and display an alternate message balloon.

The nice thing about this is that now when you click on the balloon it will take to you to the appropriate tracking page on Shrinkster.com.

You can download the 1.1.0.0 version of Shrinklet here.

Shrinklet 1.0.4.0 Released

I’ve just uploaded a new version of Shrinklet. No bug fixes in this release but instead a change to use a web-service instead of screen scraping to get the URL from Shrinkster.com (big thanks to Kyle for setting this up).

As a result I was able to eliminate about 50+ lines of code from the application and at the same time make the application immune to any changes on the pages at Shrinkster.com (nice!).

At the moment I expect this to be the final 1.0 release as I start looking at other features to add like automatic updates and integration with the tracking features.

In other news, I’ve uploaded the sources for releases to date up to Darren Neimke’s Project Distributor here. Darren’s site is a distribution point for .NET community projects that he and others are working on.