Archive for the ‘Powershell’ Category

Why is the TFS Powershell snapin marked 32-bit only?

Friday, March 13th, 2009

Short answer: because the TFS core assemblies are marked 32-bit only.

Like everyone else, I wish the TFS team had supported x64 much earlier on.  You can see in Microsoft.TeamFoundation.Common.dll that while there are a decent number of routines under the NativeMethods static class that need to be ported, it’s not an enormous task.  Turns out that supporting setup and deployment on x64 servers (with all the requisite expansion of the test matrix, etc.) is by far the higher cost.  Good news is that x64 is supposed to be on the Rosario feature list.

In the meantime, I can’t have John Robbins’ impression of our power tool be negative!  I consider my trip to attend his “Native Windows Debugging” class one of the highlights of my MS career.  As a TFS power user I frequently live in x86 shells, so here’s a tidbit from my $profile that helps keep me sane:

$isX86Process = ( $env:Processor_Architecture -eq "x86" ) ... $Global:PscxHostTitlePreference = { $(if ($IsAdmin) { "Admin: " }) + $PscxWindowTitlePrefix + $(get-location).Path + $(if ($IsX86Process) {" (x86)"}) }

Naturally, I like Vivek’s auto-Fork example too – may need to incorporate somewhere into my library of stuff…

More on ConvertTo-FixedByte vs Powershell Community Extensions

Wednesday, February 25th, 2009

Last time we introduced two undocumented cmdlets that shipped with the TFS Fall ‘08 power tools.  The idea behind ConvertTo-FixedByte is probably familiar to anyone who’s used Format-Byte from PSCX.  Take a number and make it pretty.  That’s it -- even the dirt simple conversion from “0” to “<DIR>” is handled in the formatter XML, not the cmdlet.

However, there are some key differences:

  • ConvertTo-FixedByte lets you specify any width between 6 and 22, while Format-Byte is fixed at 10 characters.
  • ConvertTo-FixedByte allows the entire UInt64 range (~10^20), while Format-Byte only goes up to TB (~10^15).
  • ConvertTo-FixedByte does not allow pipeline input, while Format-Byte does.
  • ConvertTo-FixedByte uses the SI standard prefixes, as recommended by IEC, IEEE, EU, and NIST.  Format-Byte uses binary-based prefixes (although it does not display them as KiB, MiB, etc).
    PS> convertto-fixedbyte 2000 -width 10
          2 KB
    PS> format-byte 2000
      1.953 KB
  • ConvertTo-FixedByte uses the biggest possible unit so long as the value is at least one unit.  Format-Byte waits until you’ve exceeded 2^n to use the next biggest unit.
    PS> convertto-fixedbyte 1000 -width 10
          1 KB
    PS> format-byte 1024
       1024 B
  • Both cmdlets attempt to output a fixed width by rounding, but Format-Byte’s approach fails to handle some edge cases correctly:
    PS> convertto-fixedbyte (1000*999) -width 10
        999 KB
    PS> convertto-fixedbyte (1000*999+1) -width 10
    999.001 KB
    PS> convertto-fixedbyte (1000*999+499) -width 10
    999.499 KB
    PS> convertto-fixedbyte (1000*999+500) -width 10
          1 MB
    PS> convertto-fixedbyte (1000*999+501) -width 10
          1 MB
    PS> convertto-fixedbyte (1000*1000-1) -width 10
          1 MB
    PS> convertto-fixedbyte (1000*1000) -width 10
          1 MB
    PS> convertto-fixedbyte (1000*1000+1) -width 10
          1 MB
    PS> format-byte (1024*1023)
       1023 KB
    PS> format-byte (1024*1023+1)
    1023.001 KB
    PS> format-byte (1024*1023+511)
    1023.499 KB
    PS> format-byte (1024*1023+512)
    1023.5 KB
    PS> format-byte (1024*1023+513)
    1023.501 KB
    PS> format-byte (1024*1024-1)
    1023.999 KB
    PS> format-byte (1024*1024)
       1024 KB
    PS> format-byte (1024*1024+1)
          1 MB
  • Both cmdlets omit trailing 0’s, instead padding the front with extra spaces.  [thought there was a slight difference here, but now I can’t repro it!]
  • And of course: PSCX is open-source, while the Power Tools are not.  So I can’t read the PSCX source and Keith can’t read mine.  Even so, I hope some of the ideas can cross-pollinate :)

How does the TFS snap-in handle formatting?

Wednesday, February 25th, 2009

Powershell formatting is pretty flexible, as long as you know how it works.  So how do the version control cmdlets work?  Open up %programfiles(x86)%\Microsoft Team Foundation Server 2008 Power Tools\PowerShell\Microsoft.TeamFoundation.PowerTools.PowerShell.format.ps1xml and have a look for yourself.  Let’s consider the ubiquitous Item class for now:

<View> <Name>Item</Name> <ViewSelectedBy> <TypeName>Microsoft.TeamFoundation.VersionControl.Client.Item</TypeName> </ViewSelectedBy> <TableControl> <TableHeaders> <TableColumnHeader> <Width>7</Width> <Alignment>Right</Alignment> </TableColumnHeader> <TableColumnHeader> <Label>CheckinDate</Label> <Width>10</Width> <Alignment>Right</Alignment> </TableColumnHeader> <TableColumnHeader> <Label>ContentLength</Label> <Width>7</Width> <Alignment>Right</Alignment> </TableColumnHeader> <TableColumnHeader> <Label>ServerItem</Label> <Alignment>Left</Alignment> </TableColumnHeader> </TableHeaders> <TableRowEntries> <TableRowEntry> <TableColumnItems> <TableColumnItem> <PropertyName>ChangesetId</PropertyName> </TableColumnItem> <TableColumnItem> <ScriptBlock>$_.CheckinDate.ToString('d')</ScriptBlock> </TableColumnItem> <TableColumnItem> <ScriptBlock> if ($_.ItemType -eq [Microsoft.TeamFoundation.VersionControl.Client.ItemType]::Folder) { '&lt;DIR&gt;' } else { ConvertTo-FixedByte $_.ContentLength 7 } </ScriptBlock> </TableColumnItem> <TableColumnItem> <ScriptBlock> ConvertTo-FixedPath $_.ServerItem 7,10,6 </ScriptBlock> </TableColumnItem> </TableColumnItems> </TableRowEntry> </TableRowEntries> </TableControl> </View>

And a few examples of what it looks like, taken from an 80- and a 112-character-wide console:

PS> tfdir Changes CheckinDat Content ServerItem etId e Length ------- ---------- ------- ---------- 8224 2/25/2009 <DIR> $/Test-ConchangoV2/super/u...g/directorynamefromhell 8224 2/25/2009 11 KB $/Test-ConchangoV2/super/u...efromhell/devilsurf.jpg 8224 2/25/2009 12 B $/Test-ConchangoV2/super/u...orynamefromhell/foo.txt 8224 2/25/2009 5 MB $/Test-ConchangoV2/super/u.../PowerGUI.1.6.1.639.msi PS> tfdir Changes CheckinDat Content ServerItem etId e Length ------- ---------- ------- ---------- 8224 2/25/2009 <DIR> $/Test-ConchangoV2/super/ultra/very/very/mega/long/directorynamefromhell 8224 2/25/2009 11 KB $/Test-ConchangoV2/super/ultra/very/very/m...ong/directorynamefromhell/devilsurf.jpg 8224 2/25/2009 12 B $/Test-ConchangoV2/super/ultra/very/very/mega/long/directorynamefromhell/foo.txt 8224 2/25/2009 5 MB $/Test-ConchangoV2/super/ultra/very/very/m...torynamefromhell/PowerGUI.1.6.1.639.msi

You can see that the Powershell team’s “expand toward the right to minimize visual variance” concept is used to great effect.  However, we violated their “never provide a label” maxim in 3 out of 4 columns.  Oops!  Seriously though, I don’t think anyone would want to use a ScriptProperty with the exact parameters found in the formatter XML.  What we provided instead is far more useful: a couple of fully generic cmdlets named ConvertTo-FixedByte and ConvertTo-FixedPath.  I’ll cover the former in another post; there’s a fair amount of detail to dig into, but its use here is rather mundane. 

ConvertTo-FixedPath is what really makes this example shine.  If you’re like me, you hate managing long paths in shell environments.  Wrapping onto multiple lines, truncating the front, and (especially) truncating the end of the path are all unacceptable.  Cmdlets to the rescue!  This one has two modes (“parameter sets” in PS-speak).  The first is obvious enough:

PS> convertto-fixedpath 12345678901234567890 -width 16

12345678...67890

The 20-character input string is truncated to 16, but only by omitting things in the middle.  (Note that this mode has an off-by-one bug.  Hint: odd numbers.)  This allows you to answer the important questions: what is the file name? what branch/workspace are they located in?  Usually we don’t care as much about the intermediate directories.  Maybe the same is true for other kinds of strings too – let me know if you find it useful!

The other mode is designed explicitly for use in a ps1xml formatter.  Consider:

PS> convertto-fixedpath 12345678901234567890 -otherwidths 30, 20, 10

12345678...67890

Same output, but what the heck are the “other widths”?  Imagine this string were going into a variable-length field.  We want the string to expand dynamically, showing us as much of the filename as possible.  Yet if it does need to be cropped, we want it done intelligently, as if we’d known the width in advance and passed it as a –width parameter.

As it turns out, we can know the final width, given a little hackery.  The Powershell API lets you access the raw console host details, including its BufferSize.  All you need to do is subtract the widths of the other columns, plus room for the separator character PS places between each column.  So imagine the code above is used in a four-column formatter: 80 - 30 - 20 - 10 - (4*1) = 16.  Sure enough, if you scroll back up to the XML, you’ll see the values passed to –otherwidths are no coincidence.  We hand-tweaked every column width so it fit the data type as closely as possible without wrapping*, then plugged in those widths to the convertto-fixedpath cmdlet driving the final variable-length column.

This hack may not always work as Powershell migrates away from the ancient CSRSS hosting environment.  But in theory, anyone who implements PSHostRawUserInterface should be compatible.  It works in the Powershell v2 ISE environment that ships with Win7, for instance.  Custom-PS-tool-lovers, let me know how it goes.

*Except for column titles, obviously.  They were a necessary casualty of two higher-priority design considerations (a) keeping labels identical to the underlying property name (b) supporting the default 80-character console width without squishing average-sized paths.

Powershell: find SQL files orphaned from SSMS projects

Monday, February 23rd, 2009

We’re using SQL Management Studio to manage our database scripts.  (Yes, VS “Data Dude” is under evaluation, but it has lots of quirks we haven’t had time to get a grasp on yet.)  Since making DB development a collaborative, source-controlled process is new itself, we needed some extra checks & balances.  I remembered that James had a script for reconciling project files with what’s actually on disk.  Although SSMS doesn’t use msbuild for its project structure, it was still fairly easy to adapt.

function find-orphan {
  param([string]$proj = $(throw 'ssmssqlproj file is required'))
  
  $proj = resolve-path $proj
  $dir = split-path $proj
  
  $xml = [xml](cat $proj)
  $files_from_proj = ($xml.SqlWorkbenchSqlProject.Items.LogicalFolder | 
    where { $_.name -eq "Queries" }).items.filenode |
    foreach { join-path $dir $_.name } | 
    sort
    
  $files_from_dir = ls $dir -r -filter *.sql |
    foreach { $_.fullname } |
    sort
    
  compare $files_from_proj $files_from_dir
}

What does Get-TfsServer –All do, exactly?

Tuesday, February 17th, 2009

PS C:\workspaces\ws1> help tfserver –full

SYNTAX
    Get-TfsServer [-Name] <String> [-Credential <PSCredential>] [-All] [<Common
    Parameters>] 
  

    Get-TfsServer -Path <String> [-Credential <PSCredential>] [-All] [<CommonPa
    rameters>]

...

  -All
    Get Team Foundation Server object with the extended properties

  Required?                    false
    Position?                    named
    Default value
    Accept pipeline input?       false
    Accept wildcard characters?  false

... 

Not the greatest help text ever written, I know.  Oh well, there’s a lot more room on this blog anyway :)

First the basics: Get-TfsServer and Get-TfsWorkspace (aliases: tfserver, tfworkspace) are intended to be “official” versions of James Manning’s get-tfs.ps1 and get-workspace.ps1 cmdlets.  To incorporate them into the main snap-in, Hyung ported them to C# and standardized the parameters.  Both will now accept either –Path or –Name (but not both).  For tfserver, –Name is the default [position = 1] and all others are named, while the reverse is true for tfworkspace, reflecting what we felt was the most common use of each.  Bottom line, these will all work:

> $tfs = get-tfsserver njtfs
> $tfs = get-tfsserver –path .
> $ws = get-tfsworkspace .
> $ws = get-tfsworkspace –name njtfs
> $ws = get-tfsworkspace –servername .

There are also new parameters like –Credential on tfserver, and –Computer and –Owner on tfworkspace, which behave more or less how you’d expect.  In short, you can use tfserver to interface TFS with Powershell’s built-in credential management features, as well as tfworkspace to query for the Workspace object that represents remote workspaces you may not even own.

So back to the original topic.  The –All switch turns on the special script-friendly behavior that James pioneered and I later extended.

AddTypeExtensions

As you can see there are a few differences between the final shipping behavior and the various get-tfs.ps1 scripts floating on the web:

  • Once you choose –All, every TFS assembly is loaded immediately, rather than on-demand as you use the ScriptProperties.  The idea here is to “fail early” rather than introduce a ticking bomb into downstream scripts that may depend on these objects.
  • The types that we import as NoteProperties are prefixed with the same shorthand as the ScriptProperties, plus an underscore.  Even though we continue to weed out types that aren’t useful (anything private, an exception, or an event) there are still a ton to sort thru; this tweak helps segregate the huge list into “namespaces” that are still marginally navigable with Tab completion.
  • C# is a much uglier language to do this work than Powershell!

All in all, you can now do cool things like:

PS C:\workspaces\ws1> $tfs = get-tfsserver njtfs -all
PS C:\workspaces\ws1> $tfs.vcs.GetChangeset(1234) 
 

Changes Owner                  CreationDa Comment
   etId                                te
------- -----                  ---------- -------
   1234 COATUECAP\LDeGrazia    10/27/2007

PS C:\workspaces\ws1> new-object $tfs.VCS_ChangesetVersionSpec 1234

                            ChangesetId DisplayString
                            ----------- -------------
                                   1234 C1234

So why not have this option switched on permanently?  The vanilla TeamFoundationServer object isn’t very useful on its own; the entire point of James’ original wrapper was to quickly access the VersionControlServer and similar objects.  Well first of all, it’s noticeably slower – on my machine, testing a “cold” Powershell console, the cmdlet takes 3 seconds with the default parameters and 6 seconds with –All.  We’re loading more assemblies than ever, not to mention deep reflection on each one (in order to test an upcast and a string comparison, not exactly speedy operations in themselves).  More importantly, the extra properties add enormous clutter to the objects when not needed.  And I think you’ll be more likely to use those TFS objects in vanilla form in your future scripts.  For example, most of the version control cmdlets in the Fall 2008 suite take a TeamFoundationServer as either pipeline input or a named parameter.  I’m sure whatever Microsoft is cooking up for the next batch will include even more interactivity and flow between cmdlets.

Unlocking other people’s files. Yes, again.

Saturday, February 14th, 2009

You’re going about your business, trying to refactor some code, when suddenly things come to a crashing halt.  “The item foo.cs is locked for check-out by DOMAIN\user in workspace ws1.”  Since Microsoft shipped the first public betas of TFS in 2005, there have probably been more blog posts on this pitfall than any other -- I’ll pick on James for an example.  The frustration also spawned Status Sidekick (now part of TFS Sidekicks), the first 3rd-party TFS tool to my knowledge. 

The arrival of the Fall 2008 Powertools hinted at yet another possibility based on Powershell.  Script-friendly unlocking is particularly enticing for scenarios like the one I found myself in last week: transitioning a team from exclusive checkouts to an edit-merge-commit workflow.  Turning off the exclusive settings on the Team Project is the obvious first step, but what to do about the hundreds of existing pending changes?  I can’t ask everyone to suddenly checkin their work (which may be far from ready to share, or even build), but I also can’t reorganize the tree unless every lock is removed.  Thus:

del temp.hat 2> $null foreach ($pendingSet in get-tfspendingchange -user *) { $workspace = '"' + $pendingSet.Name + '";' + $pendingSet.OwnerName $pendingSet.PendingChanges | where {$_.IsLock} | foreach { $item = '"' + $_.ServerItem + '"' "lock /lock:none /workspace:$workspace $item" | out-file temp.hat -append } } tf "@temp.hat" del temp.hat

This isn’t the cleanest solution in the world.  You couldn’t just wire up the “find all locks in the system” example I provided for Brian’s post to a Remove-TfsLock cmdlet -- even if the Power Tools provided one (which they don’t yet).  By convention, a pipeline of TFS cmdlets transmits QualifiedItems, but to unlock an arbitrary item we need two additional pieces of information from the PendingSet object.  So we fall back on good old procedural loops around tf lock.

Could we improve the script?  Sure.  We could batch ~12 items at a time* for a better performance, though the gain is not nearly as big as using a @hatfile in the first place.  We could also be a little more robust.  Get-TfsPendingChange returns a full PendingSet in the general case, but sometimes the return type decays to a simple PendingChange[].  If our query returns exactly 1 workspace, then our logic for populating $workspace won’t work. 

Of course, we could also do a lot worse!  I originally tried to add the wrapper quotes in-place, but quickly got into syntax trouble.  After consulting the relevant Powershell blog post I eventually got it right, but it was an unreadable mess.  For example, line 4 read:

$workspace = "`"$($pendingSet.Name)`";$($pendingSet.OwnerName)"

Yuck.

*3200 character limit on each line of a tf.exe script file; TFS paths are 260 chars max

    Get-TfsPendingChange doesn’t always return pending changes

    Saturday, February 14th, 2009

    Depending on how extensively you’ve played with the new TFS powershell cmdlets, you may have noticed one oddity: Get-TfsPendingChange doesn’t always return a PendingChange[] array.

    PS C:\workspaces\ws1\MAIN> tfstatus

    Version CreationDa ChangeType      ServerItem
                    te
    ------- ---------- ----------      ----------
       7415  2/13/2009 Edit            $/Research Platform (R...rLife.v2 Web.vssscc
       8024  2/13/2009 Edit            $/Research Platform (R...ckerLife.v2 Web.sln

    PS C:\workspaces\ws1\MAIN> tfstatus -user rwilliams

    Name                Computer            OwnerName           Type
    ----                --------            ---------           ----
    CoatueResearch      CMXP123             COATUECAP\rwilliams Workspace
    BuildTypeEditor_... CMXP123             COATUECAP\rwilliams Workspace

    The second command is actually returning a PendingSet[] array.  Each PendingSet contains a PendingChange[] array that’s specific to a particular workspace or shelveset, plus some metadata about the workspace or shelveset itself.

    What gives?  Actually, the underlying APIs always return PendingSet[].  If Get-TfsPendingChange merely pushed the result of the API call onto the pipeline, as most other TFS cmdlets do, then every time you ran it you’d get something more like this:

    PS C:\workspaces\ws1\MAIN> $ws = get-tfsworkspace .
    PS C:\workspaces\ws1\MAIN> $ws.QueryPendingSets($null, $ws.Name, $ws.OwnerName, $false)

    Name                Computer            OwnerName           Type
    ----                --------            ---------           ----
    ws1                 RICHARD490          COATUECAP\rberg     Workspace

    …which is clearly not as useful as the output of the first “tfstatus” example above.  Furthermore, people only use Get-TfsPendingChange (and its tf.exe predecessor) to query across multiple workspaces 5% of the time, if that.  I decided to optimize for the 95% case.  That meant checking to see how many PendingSets were returned from the server, pushing its PendingChange[] array onto the pipeline instead if there was only 1.

    Note: the QueryShelvedChanges API we use takes a single shelveset.  Unfortunately that means the cmdlet does not support queries like “show me everyone who has shelved $/project/foo.cs” in a single server call.  (QueryShelvedChanges() does have the necessary overloads; we simply inherited this limitation from tf.exe and didn’t have time to rewrite it).  Thus, whenever you specify the –shelveset parameter to Get-TfsPendingChange, you are assured to get a PendingChange[].

    Birth of a power tool: QualifiedItem and the Powershell pipeline

    Saturday, February 14th, 2009

    When I was pondering what the oft-requested Powershell interface should look like, one of the main goals was to offer the power of that raw version control API while being easier to use than tf.exe.  Both existing approaches were too clumsy for many of the tasks customers commonly request.  The canonical example seems to be “how do I add the items modified in changeset to a label.”  I once answered a forum post with a Powershell function targeting the API.  Fairly clean, but not exactly concise.  My coworker Mohamed answered a different customer with a tf-based solution.  Good display of his ingenuity, but when I’m using development tools I just want to be efficient, not clever.

    for /f "usebackq tokens=2 delims=$" %i in (`tf changeset 1256 /i ^| find "$/"`) do tf label goodbuild $%i;C1256

    Yuck (not to mention slow).

    And so an idea was born.  Much progress came “free” from Powershell itself: it’s far, far easier to script than cmd.exe is, and when scripting fails, you can drop right into the TFS .Net API without missing a beat.  Some more came from conventions -- standardized verbs, standardized parameters, etc. – and examining how tf commands should be merged, split, or otherwise molded to fit them.  (I admit to being dragged kicking & screaming through this process!)  But ultimately, the biggest value would come from the ability to seamlessly string together multiple commands.

    How to reconcile these goals?  To retain their power, cmdlets must output native API objects wherever possible.  The user receives the full data set in structured, strongly-typed form, including the ability to call properties and methods.  Yet for simplicity, the input parameters are rarely more than a couple strings.  In the pipeline I conceived, a form of weakly-typed items are the “glue” that allow rich outputs to feed limited inputs.

    Recall that a QualifiedItem represents a simple tuple of {path, deletionID, versionspec}.  By qualifying individual items on the command line, tf.exe commands let you attach a deletion ID and/or versionspec directly to each.  This is the most general way to control the underlying API that tf offers.

    The prototype I wrote in early 2007 was just a crude layer on top of tf.exe plus a “magic” extract-items.ps1 script that could turn any suitable object in the public API back into tf’s internal format – QualifiedItem[].  It worked like this:

    tfps changeset 1256 | extract-items | tfps label goodbuild

    The syntax was nowhere near idiomatic Powershell; the capabilities unlocked by inserting magic scripts in various places were not at all discoverable.  But I was hooked.  Henceforth, I used this little shell as my daily work environment for testing & debugging TFS.  (Direct programmatic access to the data returned by any tf.exe command sure sped things up.)  All the while I collected real-world use cases, tightened/tweaked/fixed, and shopped the demo around in search of a sympathetic ear.  As anyone who’s worked in a huge organization knows, nothing moves without a lot of pushing…and suffice to say that wasn’t my strong suit.  Eventually I did find myself in Brian’s office in spring 2008.  He gave me a green light, conditioned on fixing the fit & finish to Jeffrey’s satisfaction.

    Redesigning the cmdlets based on PS conventions wasn’t a technical challenge per se (modulo the kicking :)).  But having to stick a magic cmdlet in between each pipe wasn’t going to fly, no matter its pretty new name Select-TfsItem and incorporation into the main (read: discoverable) C# snap-in.  The final innovation came from Lee Holmes, who suggested we use Powershell’s built-in type coercion to help the pipeline link up automatically: all you need is a constructor on the input type with a single parameter of the output type.  Hyung implemented it as QualifiedItemSpec:

    qualifieditemspec

    A QualifiedItemSpec represents the collection of QIs extracted from a single input object.  Cmdlets then take an array of these collections as input.  Where appropriate, some cmdlets process each collection as a single entity and yield control; most, however, batch up the entire list of lists and execute a single server call in EndProcessing().  There are also subtleties in the way different object types are coerced into the much weaker QI type.  In the future we’ll look at the various ways such “chunks” of data stream along a TFS pipeline.

    Select-TfsItem remains in the toolkit for completeness, but you can see in Reflector that its implementation has degenerated to nothing; the QualifiedItemSpec[] parameter does all the heavy lifting.  It’s still useful for quickly dumping the contents of a huge object like PendingSet into the console or a file.  Meanwhile, let’s see what the final version of our “label a changeset” script looks like:

    Get-TfsChangeset 1256 | New-TfsLabel goodbuild

    Easy as pie!  Except for the fact that *-TfsLabel commands haven’t actually shipped :)  I coded cmdlets replacing nearly all of tf.exe before leaving, and I know that power tools development remain in good hands, so I’ll leave you with the above example in the hopes it’ll work after the next release or two.