Archive for the ‘TFS’ Category

Unlocking other people’s files. Yes, again.

Saturday, February 14th, 2009

You’re going about your business, trying to refactor some code, when suddenly things come to a crashing halt.  “The item foo.cs is locked for check-out by DOMAIN\user in workspace ws1.”  Since Microsoft shipped the first public betas of TFS in 2005, there have probably been more blog posts on this pitfall than any other -- I’ll pick on James for an example.  The frustration also spawned Status Sidekick (now part of TFS Sidekicks), the first 3rd-party TFS tool to my knowledge. 

The arrival of the Fall 2008 Powertools hinted at yet another possibility based on Powershell.  Script-friendly unlocking is particularly enticing for scenarios like the one I found myself in last week: transitioning a team from exclusive checkouts to an edit-merge-commit workflow.  Turning off the exclusive settings on the Team Project is the obvious first step, but what to do about the hundreds of existing pending changes?  I can’t ask everyone to suddenly checkin their work (which may be far from ready to share, or even build), but I also can’t reorganize the tree unless every lock is removed.  Thus:

del temp.hat 2> $null foreach ($pendingSet in get-tfspendingchange -user *) { $workspace = '"' + $pendingSet.Name + '";' + $pendingSet.OwnerName $pendingSet.PendingChanges | where {$_.IsLock} | foreach { $item = '"' + $_.ServerItem + '"' "lock /lock:none /workspace:$workspace $item" | out-file temp.hat -append } } tf "@temp.hat" del temp.hat

This isn’t the cleanest solution in the world.  You couldn’t just wire up the “find all locks in the system” example I provided for Brian’s post to a Remove-TfsLock cmdlet -- even if the Power Tools provided one (which they don’t yet).  By convention, a pipeline of TFS cmdlets transmits QualifiedItems, but to unlock an arbitrary item we need two additional pieces of information from the PendingSet object.  So we fall back on good old procedural loops around tf lock.

Could we improve the script?  Sure.  We could batch ~12 items at a time* for a better performance, though the gain is not nearly as big as using a @hatfile in the first place.  We could also be a little more robust.  Get-TfsPendingChange returns a full PendingSet in the general case, but sometimes the return type decays to a simple PendingChange[].  If our query returns exactly 1 workspace, then our logic for populating $workspace won’t work. 

Of course, we could also do a lot worse!  I originally tried to add the wrapper quotes in-place, but quickly got into syntax trouble.  After consulting the relevant Powershell blog post I eventually got it right, but it was an unreadable mess.  For example, line 4 read:

$workspace = "`"$($pendingSet.Name)`";$($pendingSet.OwnerName)"

Yuck.

*3200 character limit on each line of a tf.exe script file; TFS paths are 260 chars max

    Get-TfsPendingChange doesn’t always return pending changes

    Saturday, February 14th, 2009

    Depending on how extensively you’ve played with the new TFS powershell cmdlets, you may have noticed one oddity: Get-TfsPendingChange doesn’t always return a PendingChange[] array.

    PS C:\workspaces\ws1\MAIN> tfstatus

    Version CreationDa ChangeType      ServerItem
                    te
    ------- ---------- ----------      ----------
       7415  2/13/2009 Edit            $/Research Platform (R...rLife.v2 Web.vssscc
       8024  2/13/2009 Edit            $/Research Platform (R...ckerLife.v2 Web.sln

    PS C:\workspaces\ws1\MAIN> tfstatus -user rwilliams

    Name                Computer            OwnerName           Type
    ----                --------            ---------           ----
    CoatueResearch      CMXP123             COATUECAP\rwilliams Workspace
    BuildTypeEditor_... CMXP123             COATUECAP\rwilliams Workspace

    The second command is actually returning a PendingSet[] array.  Each PendingSet contains a PendingChange[] array that’s specific to a particular workspace or shelveset, plus some metadata about the workspace or shelveset itself.

    What gives?  Actually, the underlying APIs always return PendingSet[].  If Get-TfsPendingChange merely pushed the result of the API call onto the pipeline, as most other TFS cmdlets do, then every time you ran it you’d get something more like this:

    PS C:\workspaces\ws1\MAIN> $ws = get-tfsworkspace .
    PS C:\workspaces\ws1\MAIN> $ws.QueryPendingSets($null, $ws.Name, $ws.OwnerName, $false)

    Name                Computer            OwnerName           Type
    ----                --------            ---------           ----
    ws1                 RICHARD490          COATUECAP\rberg     Workspace

    …which is clearly not as useful as the output of the first “tfstatus” example above.  Furthermore, people only use Get-TfsPendingChange (and its tf.exe predecessor) to query across multiple workspaces 5% of the time, if that.  I decided to optimize for the 95% case.  That meant checking to see how many PendingSets were returned from the server, pushing its PendingChange[] array onto the pipeline instead if there was only 1.

    Note: the QueryShelvedChanges API we use takes a single shelveset.  Unfortunately that means the cmdlet does not support queries like “show me everyone who has shelved $/project/foo.cs” in a single server call.  (QueryShelvedChanges() does have the necessary overloads; we simply inherited this limitation from tf.exe and didn’t have time to rewrite it).  Thus, whenever you specify the –shelveset parameter to Get-TfsPendingChange, you are assured to get a PendingChange[].

    Birth of a power tool: QualifiedItem and the Powershell pipeline

    Saturday, February 14th, 2009

    When I was pondering what the oft-requested Powershell interface should look like, one of the main goals was to offer the power of that raw version control API while being easier to use than tf.exe.  Both existing approaches were too clumsy for many of the tasks customers commonly request.  The canonical example seems to be “how do I add the items modified in changeset to a label.”  I once answered a forum post with a Powershell function targeting the API.  Fairly clean, but not exactly concise.  My coworker Mohamed answered a different customer with a tf-based solution.  Good display of his ingenuity, but when I’m using development tools I just want to be efficient, not clever.

    for /f "usebackq tokens=2 delims=$" %i in (`tf changeset 1256 /i ^| find "$/"`) do tf label goodbuild $%i;C1256

    Yuck (not to mention slow).

    And so an idea was born.  Much progress came “free” from Powershell itself: it’s far, far easier to script than cmd.exe is, and when scripting fails, you can drop right into the TFS .Net API without missing a beat.  Some more came from conventions -- standardized verbs, standardized parameters, etc. – and examining how tf commands should be merged, split, or otherwise molded to fit them.  (I admit to being dragged kicking & screaming through this process!)  But ultimately, the biggest value would come from the ability to seamlessly string together multiple commands.

    How to reconcile these goals?  To retain their power, cmdlets must output native API objects wherever possible.  The user receives the full data set in structured, strongly-typed form, including the ability to call properties and methods.  Yet for simplicity, the input parameters are rarely more than a couple strings.  In the pipeline I conceived, a form of weakly-typed items are the “glue” that allow rich outputs to feed limited inputs.

    Recall that a QualifiedItem represents a simple tuple of {path, deletionID, versionspec}.  By qualifying individual items on the command line, tf.exe commands let you attach a deletion ID and/or versionspec directly to each.  This is the most general way to control the underlying API that tf offers.

    The prototype I wrote in early 2007 was just a crude layer on top of tf.exe plus a “magic” extract-items.ps1 script that could turn any suitable object in the public API back into tf’s internal format – QualifiedItem[].  It worked like this:

    tfps changeset 1256 | extract-items | tfps label goodbuild

    The syntax was nowhere near idiomatic Powershell; the capabilities unlocked by inserting magic scripts in various places were not at all discoverable.  But I was hooked.  Henceforth, I used this little shell as my daily work environment for testing & debugging TFS.  (Direct programmatic access to the data returned by any tf.exe command sure sped things up.)  All the while I collected real-world use cases, tightened/tweaked/fixed, and shopped the demo around in search of a sympathetic ear.  As anyone who’s worked in a huge organization knows, nothing moves without a lot of pushing…and suffice to say that wasn’t my strong suit.  Eventually I did find myself in Brian’s office in spring 2008.  He gave me a green light, conditioned on fixing the fit & finish to Jeffrey’s satisfaction.

    Redesigning the cmdlets based on PS conventions wasn’t a technical challenge per se (modulo the kicking :)).  But having to stick a magic cmdlet in between each pipe wasn’t going to fly, no matter its pretty new name Select-TfsItem and incorporation into the main (read: discoverable) C# snap-in.  The final innovation came from Lee Holmes, who suggested we use Powershell’s built-in type coercion to help the pipeline link up automatically: all you need is a constructor on the input type with a single parameter of the output type.  Hyung implemented it as QualifiedItemSpec:

    qualifieditemspec

    A QualifiedItemSpec represents the collection of QIs extracted from a single input object.  Cmdlets then take an array of these collections as input.  Where appropriate, some cmdlets process each collection as a single entity and yield control; most, however, batch up the entire list of lists and execute a single server call in EndProcessing().  There are also subtleties in the way different object types are coerced into the much weaker QI type.  In the future we’ll look at the various ways such “chunks” of data stream along a TFS pipeline.

    Select-TfsItem remains in the toolkit for completeness, but you can see in Reflector that its implementation has degenerated to nothing; the QualifiedItemSpec[] parameter does all the heavy lifting.  It’s still useful for quickly dumping the contents of a huge object like PendingSet into the console or a file.  Meanwhile, let’s see what the final version of our “label a changeset” script looks like:

    Get-TfsChangeset 1256 | New-TfsLabel goodbuild

    Easy as pie!  Except for the fact that *-TfsLabel commands haven’t actually shipped :)  I coded cmdlets replacing nearly all of tf.exe before leaving, and I know that power tools development remain in good hands, so I’ll leave you with the above example in the hopes it’ll work after the next release or two.

    Advanced tf.exe syntax, or what on earth is a QualifiedItem?

    Friday, February 13th, 2009

    An item that’s qualified, duh :)  Real answer: it comes from tf.exe versionspec syntax.  Let’s demonstrate by example:

    1) tf get foo.cs  # not qualified – falls back to default
    2) tf get foo.cs /version:1234  # not qualified – falls back to /version parameter
    3) tf get foo.cs;1234  # qualified
    4) tf get foo.cs;X56  # qualified with deletion ID
    5) tf get foo.cs;X56;LmyLabel  # qualified with both – betcha didn’t know this was possible
    6) tf get foo.cs;1234 bar.cs;X56 baz.cs;789 *quux* /version:LmyLabel  # mix & match
    7) tf history foo.cs;1234~5678  # qualified with two versions
    8) tf history foo.cs;~5678  # default versionFrom, qualified versionTo
    9) tf history foo.cs;5678  # same as above
    10) tf history foo.cs;1234~  # qualified versionFrom, default versionTo

    “Qualifying” an item means telling tf.exe exactly what version to operate on.  We do this by combining a name with a versionspec.  Most tf commands take qualified items as their main piece of input.  (Commands that don’t include things like Checkin, Shelve, Undo, or Status that need to use whatever’s already in the workspace; and of course ones like Workfold or Configure that don’t touch the repository at all).

    Implementation

    If you examine tf.exe in Reflector, you’ll see that QualifiedItem is a simple class encapsulating a Path, a DeletionId, and a VersionSpec[] array.  The array almost always has just one member; only a few commands take two versionspecs, and none takes more.  Most items have a deletion ID of 0 to indicate “not deleted.”

    qualifieditem

    To see how it’s used, start by drilling into the VersionControlCommand class to find the methods named ParseQualifiedItem().  I’ll leave this journey as an exercise for the reader.  Suffice to say here’s what you’d find:

    Behavior - single versionspec

    The goal is to give the user fine-grained control over tf operations.  Examples #1-4 are straightforward.  #5 is a quirk of syntax with no real import.  #6 shows the power of this approach: a single command pulls different versions of foo, bar, baz, while applying a fallback versionspec to all non-qualified items – in this case, the set intersection of all files in the current directory matching the *quux* pattern with the contents of the “myLabel” label.

    If not specified by a /version parameter, the default versionspec is T.

    Behavior – version range (2 versionspecs)

    Example #7 is straightforward.  #8-10 show off some syntax sugar.  When you have a command that expects a version range, you need only type one end of the range if the default suffices for the other.  The default versionFrom spec is C1 and the default versionTo is T.

    Note that there is no opportunity for fallback.  You can use the /version parameter or qualified items, but not both.  All of the commands that operate on a version range have additional limitations that make the general syntax seen in #6 impossible.

    • Diff – takes exactly one item with a version range (foo.cs;10~20), or two items with one versionspec each (foo.cs;10 bar.cs;20), but no other variations.
    • History – takes exactly one item with a version range.
    • Merge – takes exactly two items.  The first one has a version range and the second has no version.
    • Rollback – introduces some new parameters that are beyond the scope of this post, not to mention still subject to change before TFS 2010’s release.