Archive for the ‘Version Control’ Category

Birth of a power tool: QualifiedItem and the Powershell pipeline

Saturday, February 14th, 2009

When I was pondering what the oft-requested Powershell interface should look like, one of the main goals was to offer the power of that raw version control API while being easier to use than tf.exe.  Both existing approaches were too clumsy for many of the tasks customers commonly request.  The canonical example seems to be “how do I add the items modified in changeset to a label.”  I once answered a forum post with a Powershell function targeting the API.  Fairly clean, but not exactly concise.  My coworker Mohamed answered a different customer with a tf-based solution.  Good display of his ingenuity, but when I’m using development tools I just want to be efficient, not clever.

for /f "usebackq tokens=2 delims=$" %i in (`tf changeset 1256 /i ^| find "$/"`) do tf label goodbuild $%i;C1256

Yuck (not to mention slow).

And so an idea was born.  Much progress came “free” from Powershell itself: it’s far, far easier to script than cmd.exe is, and when scripting fails, you can drop right into the TFS .Net API without missing a beat.  Some more came from conventions -- standardized verbs, standardized parameters, etc. – and examining how tf commands should be merged, split, or otherwise molded to fit them.  (I admit to being dragged kicking & screaming through this process!)  But ultimately, the biggest value would come from the ability to seamlessly string together multiple commands.

How to reconcile these goals?  To retain their power, cmdlets must output native API objects wherever possible.  The user receives the full data set in structured, strongly-typed form, including the ability to call properties and methods.  Yet for simplicity, the input parameters are rarely more than a couple strings.  In the pipeline I conceived, a form of weakly-typed items are the “glue” that allow rich outputs to feed limited inputs.

Recall that a QualifiedItem represents a simple tuple of {path, deletionID, versionspec}.  By qualifying individual items on the command line, tf.exe commands let you attach a deletion ID and/or versionspec directly to each.  This is the most general way to control the underlying API that tf offers.

The prototype I wrote in early 2007 was just a crude layer on top of tf.exe plus a “magic” extract-items.ps1 script that could turn any suitable object in the public API back into tf’s internal format – QualifiedItem[].  It worked like this:

tfps changeset 1256 | extract-items | tfps label goodbuild

The syntax was nowhere near idiomatic Powershell; the capabilities unlocked by inserting magic scripts in various places were not at all discoverable.  But I was hooked.  Henceforth, I used this little shell as my daily work environment for testing & debugging TFS.  (Direct programmatic access to the data returned by any tf.exe command sure sped things up.)  All the while I collected real-world use cases, tightened/tweaked/fixed, and shopped the demo around in search of a sympathetic ear.  As anyone who’s worked in a huge organization knows, nothing moves without a lot of pushing…and suffice to say that wasn’t my strong suit.  Eventually I did find myself in Brian’s office in spring 2008.  He gave me a green light, conditioned on fixing the fit & finish to Jeffrey’s satisfaction.

Redesigning the cmdlets based on PS conventions wasn’t a technical challenge per se (modulo the kicking :)).  But having to stick a magic cmdlet in between each pipe wasn’t going to fly, no matter its pretty new name Select-TfsItem and incorporation into the main (read: discoverable) C# snap-in.  The final innovation came from Lee Holmes, who suggested we use Powershell’s built-in type coercion to help the pipeline link up automatically: all you need is a constructor on the input type with a single parameter of the output type.  Hyung implemented it as QualifiedItemSpec:


A QualifiedItemSpec represents the collection of QIs extracted from a single input object.  Cmdlets then take an array of these collections as input.  Where appropriate, some cmdlets process each collection as a single entity and yield control; most, however, batch up the entire list of lists and execute a single server call in EndProcessing().  There are also subtleties in the way different object types are coerced into the much weaker QI type.  In the future we’ll look at the various ways such “chunks” of data stream along a TFS pipeline.

Select-TfsItem remains in the toolkit for completeness, but you can see in Reflector that its implementation has degenerated to nothing; the QualifiedItemSpec[] parameter does all the heavy lifting.  It’s still useful for quickly dumping the contents of a huge object like PendingSet into the console or a file.  Meanwhile, let’s see what the final version of our “label a changeset” script looks like:

Get-TfsChangeset 1256 | New-TfsLabel goodbuild

Easy as pie!  Except for the fact that *-TfsLabel commands haven’t actually shipped :)  I coded cmdlets replacing nearly all of tf.exe before leaving, and I know that power tools development remain in good hands, so I’ll leave you with the above example in the hopes it’ll work after the next release or two.

Advanced tf.exe syntax, or what on earth is a QualifiedItem?

Friday, February 13th, 2009

An item that’s qualified, duh :)  Real answer: it comes from tf.exe versionspec syntax.  Let’s demonstrate by example:

1) tf get foo.cs  # not qualified – falls back to default
2) tf get foo.cs /version:1234  # not qualified – falls back to /version parameter
3) tf get foo.cs;1234  # qualified
4) tf get foo.cs;X56  # qualified with deletion ID
5) tf get foo.cs;X56;LmyLabel  # qualified with both – betcha didn’t know this was possible
6) tf get foo.cs;1234 bar.cs;X56 baz.cs;789 *quux* /version:LmyLabel  # mix & match
7) tf history foo.cs;1234~5678  # qualified with two versions
8) tf history foo.cs;~5678  # default versionFrom, qualified versionTo
9) tf history foo.cs;5678  # same as above
10) tf history foo.cs;1234~  # qualified versionFrom, default versionTo

“Qualifying” an item means telling tf.exe exactly what version to operate on.  We do this by combining a name with a versionspec.  Most tf commands take qualified items as their main piece of input.  (Commands that don’t include things like Checkin, Shelve, Undo, or Status that need to use whatever’s already in the workspace; and of course ones like Workfold or Configure that don’t touch the repository at all).


If you examine tf.exe in Reflector, you’ll see that QualifiedItem is a simple class encapsulating a Path, a DeletionId, and a VersionSpec[] array.  The array almost always has just one member; only a few commands take two versionspecs, and none takes more.  Most items have a deletion ID of 0 to indicate “not deleted.”


To see how it’s used, start by drilling into the VersionControlCommand class to find the methods named ParseQualifiedItem().  I’ll leave this journey as an exercise for the reader.  Suffice to say here’s what you’d find:

Behavior - single versionspec

The goal is to give the user fine-grained control over tf operations.  Examples #1-4 are straightforward.  #5 is a quirk of syntax with no real import.  #6 shows the power of this approach: a single command pulls different versions of foo, bar, baz, while applying a fallback versionspec to all non-qualified items – in this case, the set intersection of all files in the current directory matching the *quux* pattern with the contents of the “myLabel” label.

If not specified by a /version parameter, the default versionspec is T.

Behavior – version range (2 versionspecs)

Example #7 is straightforward.  #8-10 show off some syntax sugar.  When you have a command that expects a version range, you need only type one end of the range if the default suffices for the other.  The default versionFrom spec is C1 and the default versionTo is T.

Note that there is no opportunity for fallback.  You can use the /version parameter or qualified items, but not both.  All of the commands that operate on a version range have additional limitations that make the general syntax seen in #6 impossible.

  • Diff – takes exactly one item with a version range (foo.cs;10~20), or two items with one versionspec each (foo.cs;10 bar.cs;20), but no other variations.
  • History – takes exactly one item with a version range.
  • Merge – takes exactly two items.  The first one has a version range and the second has no version.
  • Rollback – introduces some new parameters that are beyond the scope of this post, not to mention still subject to change before TFS 2010’s release.