The Big Picture

OCS follows Exchange into 64-bit-only land

Devin L. Ganger — Fri, 29 Aug 2008 20:07:05 GMT

You may have missed this interesting blog post this morning amidst all the political kerfuffle, so let me sum up: the next version of OCS will only support x64 platforms.

This isn't the big deal it would have been for OCS 2007. A lot of the initial FUD around the 64-bit-only move in Exchange 2007 turned out to be mere steam. While there were some initial challenges involved in managing the new 64-bit Exchange deployment from 32-bit machines, Microsoft got a lot of the licensing figured out and released the appropriate sets of tools to allow management of Exchange 2007 from both 32-bit and 64-bit environments. I fully expect that the OCS group has been paying close attention to all of this and taken good notes.

There's no denying that Exchange 2007 benefits from the "64-bit only in production" stance -- and with the release of Windows Server 2008 and Hyper-V, not to mention Microsoft's updated support statement for virtualization environments, the need for 32-bit environments is going away. My biggest reason for wanting 32-bit Exchange environments was so I could run demos under Virtual Server; now that I have Hyper-V, I'm probably not in any rush to go back to Virtual Server and the 32-bit limitation. 64-bit hardware is the norm today, and the x64 Windows variants are solid and mainstream enough for my dedicated application servers. (Maybe not so for the desktop quite yet, but still getting there rapidly.)

The one thing I'm skeptical about, though, is whether the move to 64-bits is really going to reduce the total number of servers in the deployment. In Exchange 2007, I only saw the server reductions in very large environments; the mailbox-per-server gains we got from 64-bits was offset by the explicit breakout of roles and the business needs that drove redundant configurations like CCR (which meant no co-locating roles with the Mailbox role) and multiple HT/CAS servers. I'm wondering how this is going to play out with the next version of OCS, where it already has so many distinct roles in play.

What I *hope* to see is that the maximum capacity of each server role (such as the number of users per pool or the number of streams per mediation server) can be driven upwards; this makes the large datacenter configuration options much more attractive, because it does translate to a reduced number of servers. However, for organizations that still have relatively low bandwidth separating their various locations, 64-bits won't do much to help; OCS deployment planning is very dependent on bandwidth, and is often the top limit on scalability long before the limits of the 32-bit Windows environment.

These are not the solutions you're looking for

Devin L. Ganger — Fri, 27 Jun 2008 03:18:47 GMT

As IT professionals, we are more than often prone to fall to the perils of magical thinking. (I'm sure this is a side-effect of being human, which is a pesky and bothersome condition I will have to do something about one of these days.) Magical thinking in this context is when we have not internalized the intricacies of a problem and instead rely on formulas rather than true understanding to come up with solutions.

At one ISP I used to work at, we had a glorious reclaimed piece of technology, an Auspex NS-5500 file server. Every now and then on reboot, this old beast of a machine would fail to boot up; the cure was to open the cover over the drive cage and give it a good swift whack. We all assumed that this was because one of the drive connectors was a bit loose, but when our "magic" fix failed to work one night I discovered that it was in fact because one of the screws holding things in place was missing, allowing the drive bay to sag just a tiny bit. It was this tiny bit of sag that put just enough stress on the connector for drive 0. Had we actually opened the case up earlier, we'd have been able to solve the problem -- and prevent a year of whacking the server.

All too often, I see magical thinking in the field of security. Case in point: I recently heard about a gentleman who has a client that is requesting ETRN support be added back to Exchange 2007, either natively or through an add-on. They want to deploy the Edge role in their DMZ, have it queue up mail for the internal organization, and then have their Hub Transports (in the internal protected network) initiate a connection out to de-queue the messages using the ETRN SMTP extension. The reason they want this is that they've done due diligence and read some very thorough documents about computer network zones and have come to the conclusion that all network connections must be initiated from the most secure network. This, they say, removes the threat of malware taking over the Edge server in the DMZ and allowing an attacker to use it as a launching point to the protected network.

Now, the recommendation for connections to be initiated from a more secure network to a less secure network is a good general baseline to follow when it makes sense. However, it is not realistic in all cases (if we followed this to the letter, nobody would be able to receive e-mail from external senders except through random polling of Internet SMTP hosts, which is not at all scalable). This is doubly true if you don't understand how the underlying protocols work. Case in point: ETRN, defined by RFC 1985, "SMTP Service Extension for Remote Message Queue Starting". Quoting from section 3, "The Remote Queue Processing Declaration service extension" (emphasis added):

To save money, many small companies want to only maintain transient connections to their service providers. In addition, there are some situations where the client sites depend on their mail arriving quickly, so forcing the queues on the server belonging to their service provider may be more desirable than waiting for the retry timeout to occur.

Both of these situations could currently be fixed using the TURN command defined in [1], if it were not for a large security loophole in the TURN command. As it stands, the TURN command will reverse the direction of the SMTP connection and assume that the remote host is being honest about what its name is. The security loophole is that there is no documented stipulation for checking the authenticity of the remote host name, as given in the HELO or EHLO command. As such, most SMTP and ESMTP implementations do not implement the TURN command to avoid this security loophole.

This has been addressed in the design of the ETRN command. This extended turn command was written with the points in the first paragraph in mind, yet paying attention to the problems that currently exist with the TURN command. The security loophole is avoided by asking the server to start a new connection aimed at the specified client.

See the problem? ETRN was not designed to solve a security problem; it was designed to solve a financial problem back in days when always-on bandwidth was a lot more expensive and most ISPs metered traffic. It masquerades as solving a security problem only because it's designed to avoid a loophole in an insecure and exploitable feature. As a result, ETRN won't solve the problem these people want it to solve; all it does is tell the system in the DMZ to initiate a new connection to the Hub Transport servers. It doesn't reuse the existing connection initiated by the Hub Transport servers. They can't use a firewall rule to block outgoing access from the Edge to the Hub Transport and be safe, because they'll cut off all incoming traffic.

However, let us for a moment assume that it did work the way they wanted it to: my Hub Transport initiates an outbound SMTP session to the Edge. In this session, HT is the SMTP client, ET is the SMTP server. As soon as HT issues the ETRN command, they still have to swap roles -- HT is now using the SMTP server code paths, while the ET is using the SMTP client code paths. Any theoretical vulnerabilities that are in the HT SMTP implementation are still going to be there, still exposed to the message traffic about to be sent down the connection, still open to exploitation.

This is the magical thinking: firewalls and a DMZ will protect my traffic. This is not true; firewalls and networks zones are two components of a complete security plan. Neither firewalls nor network zones can protect legitimate traffic, nor are they designed to; they are designed to allow you to designate which traffic is legitimate. If you want to secure that traffic, you need to turn to other measures.

masteringdpm.com back online

Devin L. Ganger — Mon, 23 Jun 2008 15:26:49 GMT

Things got hairy enough last week that I forgot to post, but my hosting provider got the problem sorted out and the website is back online.

masteringdpm.com temporarily down

Devin L. Ganger — Wed, 18 Jun 2008 18:43:45 GMT

If you've tried to get to masteringdpm.com in the past couple of days, you may have gotten a cryptic error message instead of a site with DPM goodness. I'm working with my hosting provider to get it put back up ASAP and will post again once it's back up.

Tech-Talk: Making Backups Cool with DPM

Devin L. Ganger — Wed, 18 Jun 2008 17:55:50 GMT

While I was at the Tech-Ed NA IT Pro conference last week, Jason Buffington and I took the chance to invade the Tech-Ed Online fishbowl studio and record a quick Tech-Talk on using DPM. You can now view it online on the Tech-Ed IT Pro page and the Library page, or stream it directly. Now that Tech-Ed's over, maybe we'll both find the time to be on Xbox Live at the same time so we can continue our discussion in Call of Duty 4...

Updated Exchange Developer Roadmap

Devin L. Ganger — Tue, 17 Jun 2008 19:43:45 GMT

To reinforce yesterday's post about Exchange Web Services (EWS), I wanted to draw your attention to the Exchange Developer Roadmap posted on May 22 2008 on the Exchange API-spotting blog.

There shouldn't really be any surprises here, but there were a couple of items I wanted to highlight. First:

Given this commitment to Web services and our goal of making Exchange Web Services the richest developer interface for Exchange... (emphasis added)

Here's a preview of some of the functionality that we plan to add to the next release of Exchange Web Services:

Access to Folder Associated Items (FAI) and read/write access to user settings (Devin: this page in the MAPI reference indicates that FAIs are things like views and forms. I believe that this also fixes a known quirk of EWS that keeps you from creating Outlook-visible search folders that use certain property paths. I believe this also gives access to server-side rules, if they're not already accessible through a separate part of the API.)

Management of Personal Distribution Lists (Devin: very cool.)

Throttling capabilities that give Exchange administrators control over system resource consumption (Devin: this will be very nice for helping keep poorly written applications from taking down the Exchange servers.)

A powerful and easy-to-use server-to-server authentication model to enable building portals and enterprise mash-ups (Devin: let's hope this can ease some of the pain of building Exchange-aware SharePoint sites, at least those that don't require direct access to private mailbox content.)

An easy-to-use Microsoft .NET API that fully wraps the Web service calls, which makes Web service development even easier (Devin: I'll be interested in seeing how this stacks up against third-party offerings like the Independentsoft EWS client offering.)

Then they go on to list the APIs that will get removed (Exchange WebDAV, Store Events, CDO 3.0/CDOEx, and ExOLEDB) and moved to "extended support" (Exchange Server MAPI Client, CDO 1.2.1). Don't get too excited by the MAPI client -- it's not what you think:

Provides server applications a MAPI runtime for accessing Exchange.

Note: This is not the Outlook MAPI Client library that is included with Outlook.

and

Outlook's Exchange MAPI Store provider, available in the Outlook MAPI Client library can also be used to access an Exchange mailbox or public folder.

If you're going to start writing Exchange-aware applications, you should probably start looking at EWS first for future compatibility. If you're trying to support Exchange 2003 at the same time...good luck.

3Sharp, Podcasting, and You

Devin L. Ganger — Mon, 09 Jun 2008 18:25:11 GMT

The talented people at 3Sharp are one of the best reasons to work here. Our Platforms Group is just one piece of the pie here; we've got some top-tier development talent who can make SharePoint stand up and dance. Those guys down the hall have been working hard on a little surprise they like to call the Podcasting Kit for SharePoint, which Microsoft has just released on Codeplex as indicated in their press release. 3Sharpies John Peltonen, David Gerhardt, and Paul Robichaux are also blogging about it, so if you’re interested, check them out.

I've been hearing bits and pieces, but last week I got to sit down and take a good look at what they're doing. Wow. This is some cool stuff that is going to make sharing podcasts, video talks, and other knowledge sharing content a lot easier. I can't wait until I can start using it; I've already lined up some content that I can put up and I'm already thinking of some more I can do.

All purchases should be this easy

Devin L. Ganger — Mon, 09 Jun 2008 17:45:18 GMT

If you haven't seen me in person recently, you may not realize I'm a heretic. Yes, that's right -- I use an Apple 15" MacBook Pro with Vista as my laptop. It took some jiggling to get it all working -- an upgrade to Leopard (OS X 10.5) for the final release of BootCamp, an upgrade to Vista SP1, and finding a stable version of the Atheros wireless drivers -- but it's now reliable and fast.

There are some downsides to this particular laptop. It's only gives me 2GB of RAM, which means that I can't run a typical VM configuration (DC, DPM, Exchange) and still have enough power to run PowerPoint like I could under XP. The battery life is okay but not great; I run out on long flights.

I'm off to Tech-Ed this week, so I stopped by the Apple store in Bellevue Square Sunday to pick up a spare battery for the flight. I've had bad experiences at this store in the past; I don't give off the right vibe(or maybe I just look light a tightwad) and can't get seem to get the attention of the staff. I took a chance, though, and walked in the store.

This time, my customer service experience was great. I caught the eye of Associate 1; although he was busy with another customer, he called for help; I didn't even see him do it. A minute later, Associate 2 walks up to me. "I understand you're looking for a 15" MacBook Pro battery." Pleasantly shocked, I followed him over to the appropriate shelf and soon had the battery in hand. "Is there anything else I can help you with, or are you ready to check out?"

If you've not been into an Apple store recently, they're doing something absolutely sweet. Each customer service associate has a hip-mounted scanner/cardreader. They scan your merchandise on the spot, take and run your credit card, and ask you for an email address to send the receipt to. Boom -- it's all done, your card is charged, and you don't have to stand in line at the counter unless you're doing cash or check. This is a great concept I'd love to see other stores use. My receipt hit my Exchange account (and thus my Windows Mobile phone) as I was walking out of the store.

I love living in the future.

Doing UC in the Pacific Northwest

Devin L. Ganger — Tue, 06 May 2008 17:19:34 GMT

I've been sitting on a cool announcement for several days now, and I'm happy that it's now time to announce it.

I've been working with a group of people to get a new user group for Unified Communications (UC) put together here in the Pacific Northwest. While all of us are here in the Puget Sound area, our goal is to put in place a framework to empower a variety of events and meetings all throughout the region, not just based here in Seattle. Rather than be a typical boring user group with a jawbreaking acronym (PNWUCUG, which we do use), we're defining ourselves as people who do UC. This gives us a simpler name -- We do UC, hosted at ucdoers.org.

From our website:

We are the Pacific Northwest Unified Communications User Group (PNWUCUG) and we have a passion for UC. If you are one of the following, you could be one of us:

IT professionals in the Pacific Northwest who design, deploy, or manage Exchange Server, Live Communications Server, and Office Communications Server systems.
Developers who write or maintain solutions that integrate, extend, or provide UC capabilities to Exchange Server, Live Communications Server, and Office Communications Server and clients.
Industry experts with a recognized expertise in UC.
Hobbyists who are exploring Microsoft-based UC solutions.

One thing that's important for me to clarify -- my vision of this user group (which is echoed by the other folks who are getting it off the ground) is that it exists to support all Exchange, LCS, and OCS users, not just people running 2007 and doing the VoIP stuff. We may have a focus on UC, but that's mainly to align ourselves with the direction Microsoft is taking these products. If you're using Exchange, we want you to participate; we want to make sure we have content for you.

So, if this sounds like goodness to you, head on over to the blog for the announcement of our May 28th kick-off meeting at The Parlor Billiards & Spirits in Bellevue, WA. For those of you who can't be there in person, we're even going to have a Live Meeting feed for you -- how cool is that?

Exchange protocol documentation now available

Devin L. Ganger — Fri, 11 Apr 2008 02:21:31 GMT

Per the announcement on Tuesday (08 Apr), Microsoft has released a lot of new documentation for various Exchange and Outlook-Exchange protocols. This is some cool stuff -- just check out the list of what's available. However, as the web site warns, it's preliminary documentation. If you don't believe them, when you download the files (available in PDF format) the big fat "PRELIMINARY" watermark (in very bold font) will help remind you.[1]

I can already hear some of you out there: "So Microsoft released documentation on obscure or unimportant Exchange protocols. Big deal. I bet they've saved all the good stuff for licensing!" Well, I'm not going to deny that this is a complete set of documentation for every Exchange protocol you might ever want to know about -- after all, Microsoft is a company who believes in the value of intellectual property. They've kinda built a business plan around it, and it's both foolish and naive to somehow assume that they're just going to toss all of that overboard overnight. It's not even reasonable to expect them to completely abandon that position; it's an arguable proposition that Open Source principles work best in conjunction with an IP scheme that permits open licensing when the developers feel invested in doing so, alongside more restrictive licensing schemes. But that's a religious argument for another day.

This will be a long post. I'm going to split it into three sections: Appetizers, Main Course, and What's Missing.

Section 1: Appetizers

First, we have some housekeeping and overview documents and protocols:

Name	Description
MS-CAB	Cabinet File Format
MS-MCI	MCI Compression and Decompression
MS-OXDOCO	Outlook-Exchange Protocol Document Roadmap
MS-OXGLOS	Office Exchange Protocols Master Glossary
MS-OXPROTO	Office Exchange Protocols Overview
MS-OXREF	Office Exchange Protocols Master Reference
MS-PATCH	LZX DELTA Compression and Decompression

You may have noticed that these documents include a few things that aren't strictly Exchange or Outlook-specific, such as the CAB file format and various compression protocols. Just remember that the Exchange protocol documentation is part of a wider set of Interoperability Principles, and so it depends on technologies that are part of the more generic set of Windows technologies.

Section 2: Main Course

Okay, with roadmaps and preliminaries out of the way, let's take a look at the meat:

Name	Description
MS-NSPI	Name Service Provider Interface (NSPI) Protocol Specification
MS-OXABREF	Address Book Name Service Provider Interface (NSPI) Referral Protocol Specification
MS-OXBBODY	Best Body Retrieval Protocol Specification
MS-OXCDATA	Data Structures Protocol Specification
MS-OXCETF	Enriched Text Format (ETF) Message Body Conversion Protocol Specification
MS-OXCFOLD	Folder Object Protocol Specification
MS-OXCFXICS	Bulk Data Transfer Protocol Specification
MS-OXCICAL	iCalendar to Appointment Object Conversion Protocol Specification
MS-OXCMAIL	RFC2822 and MIME to E-mail Object Conversion Protocol Specification
MS-OXCMSG	Message and Attachment Object Protocol Specification
MS-OXCNOTIF	Core Notifications Protocol Specification
MS-OXCPERM	Exchange Access and Operation Permissions Specification
MS-OXCPRPT	Property and Stream Object Protocol Specification
MS-OXCROPS	Remote Operations (ROP) List and Encoding Protocol Specification
MS-OXCRPC	Wire Format Protocol Specification
MS-OXCSPAM	Spam Confidence Level, Allow and Block Lists Protocol Specification
MS-OXCSTOR	Store Object Protocol Specification
MS-OXCSYNC	Mailbox Synchronization Protocol Specification
MS-OXCTABL	Table Object Protocol Specification
MS-OXDISCO	Autodiscover HTTP Service Protocol Specification
MS-OXDSCLI	Autodiscover Publishing and Lookup Protocol Specification
MS-OXIMAP4	Internet Message Access Protocol Version 4 (IMAP4) Extensions Specification
MS-OXLDAP	Lightweight Directory Access Protocol (LDAP) Version 3 Extensions Specification
MS-OXMSG	.MSG File Format Specification
MS-OXMVMBX	Mailbox Migration Protocol Specification
MS-OXOAB	Offline Address Book (OAB) Format and Schema Protocol Specification
MS-OXOABK	Address Book Object Protocol Specification
MS-OXOABKT	Address Book User Interface Templates Protocol Specification
MS-OXOCAL	Appointment and Meeting Object Protocol Specification
MS-OXOCFG	Configuration Information Protocol Specification
MS-OXOCNTC	Contact Object Protocol Specification
MS-OXODLGT	Delegate Access Configuration Protocol Specification
MS-OXODOC	Document Object Protocol Specification
MS-OXOFLAG	Informational Flagging Protocol Specification
MS-OXOJRNL	Journal Object Protocol Specification
MS-OXOMSG	E-mail Object Protocol Specification
MS-OXONOTE	Note Object Protocol Specification
MS-OXOPFFB	Public Folder Based Free/Busy Protocol Specification
MS-OXOPOST	Post Object Protocol Specification
MS-OXORMDR	Reminder Settings Protocol Specification
MS-OXORMMS	Rights-Managed E-mail Object Protocol Specification
MS-OXORSS	RSS Object Protocol Specification
MS-OXORULE	E-mail Rules Protocol Specification
MS-OXOSFLD	Special Folders Protocol Specification
MS-OXOSMIME	S/MIME E-mail Object Protocol Specification
MS-OXOSMMS	SMS and MMS Object Protocol Specification
MS-OXOSRCH	Search Folder List Configuration Protocol Specification
MS-OXOTASK	Task-Related Objects Protocol Specification
MS-OXOUM	Voice Mail and Fax Objects Protocol Specification
MS-OXPFOAB	Offline Address Book (OAB) Public Folder Retrieval Protocol Specification
MS-OXPHISH	Phishing Warning Protocol Specification
MS-OXPOP3	Post Office Protocol Version 3 (POP3) Extensions Specification
MS-OXPROPS	Office Exchange Protocols Master Property List Specification
MS-OXPSVAL	E-mail Postmark Validation Protocol Specification
MS-OXRTFCP	Rich Text Format (RTF) Compression Protocol Specification
MS-OXRTFEX	Rich Text Format (RTF) Extensions Specification
MS-OXSHARE	Sharing Message Object Protocol Specification
MS-OXSMTP	Simple Mail Transfer Protocol (STMP) Mail Submission Extensions Specification
MS-OXTNEF	Transport Neutral Encapsulation Format (TNEF) Protocol Specification
MS-OXWAVLS	Availability Web Service Protocol Specification
MS-OXWOAB	Offline Address Book (OAB) Retrieval Protocol Specification
MS-OXWOOF	Out of Office (OOF) Web Service Protocol Specification
MS-OXWUMS	Voice Mail Settings Web Service Protocol Specification
MS-XJRNL	Journal Record Message Format Protocol Specification
MS-XLOGIN	SMTP Protocol AUTH LOGIN Extension Specification
MS-XWDVSEC	Web Distributed Authoring and Versioning (WebDAV) Protocol Security Descriptor Extensions Specification

On first glance, that's an impressive list. NSPIs, S/MIME, SMTP and POP3 extensions, RTF extensions, TNEF -- the list goes on. There's a lot of seriously crunchy material here. The question of the moment, though, is "just how detailed is all this documentation?"

Good question.

I haven't had time to look through it all in a lot of detail. To be honest, I suspect that a lot of it is in areas that I wouldn't be able to catch any glaring omissions or discrepancies (sorry, readers, I'm just not up on the latest specs for RTF). However, I did take a quick look through MS-XLOGIN, "SMTP Protocol AUTH LOGIN Extension Specification"[2], since I'm reasonably familiar with SMTP.

Let me skip to the chase: yup, this is preliminary work. On whole, it does a good job of documenting the flow of the LOGIN extension (which people have already mostly figured out how it works through years of careful protocol analysis). The most complicated part of it is that you're using Base64 to encode the credentials being passed -- not rocket science. However, there are some gaps in this straightforward documentation:

Nowhere did I find any guidance on what the user and passwords challenges are supposed to be computed on (only that they are to be Base64 encoded). This makes it more difficult to properly code a LOGIN implementation.
The samples they gave look like valid Base64, but according to my quick conversion tests in PowerShell, they aren't. I can't get any of the sample values to match what they should. Again, this means I can't work backwards to get the missing data.

I really hope this is the kind of stuff they're going to fix between this release and the final release, because without it, this documentation isn't nearly as useful as it could be. Some would even accuse it of being provided merely to give the appearance of interoperability while still keeping enough implementation details close to the chest to keep it from really happening. I, however, subscribe to the philosophy that one should never initially ascribe to malice what can be explained through other possibilities -- and I've done enough work on these sorts of projects to know that getting the right level of detail in a document like this is far from a no-brainer, especially if you're dealing with contractors or are having to generate the documentation after the fact. (I don't know that either of these possibilities are involved, but I'm guessing.)

Section 3: What's Missing

There are three obvious protocols missing in all of the above: MAPI, MAPI over RPC over HTTP, and Exchange ActiveSync. I can hear the screams now...but this is where I go back to the point that Microsoft still makes money from intellectual property. Microsoft's Web site offers a searchable IP Catalog that shows you exactly which protocols they offer for a licensing fee, and both MAPI (aka the Outlook Exchange Transport Protocol) and Exchange ActiveSync are on it, as well as several other important protocols for Unified Communications. Microsoft is under no obligation to make every single protocol available for free -- but the fact that they're finding value in doing it with the above protocols is pretty cool and interesting. [3]

[1] If the watermark bugs you, try seeing if your PDF client will allow you to view or print the document without annotations. Using Foxit Reader, I was able to make the watermark go away and actually read some of the text it obscured.

[2] SMTP Protocol? Seriously? Is that like PIN number or ATM machine? Attention, Microsoft technical writers: P stands for Protocol.

[3] You're free to speculate on what value they get, but not here, please. That's another religious discussion.