Windows

OCS follows Exchange into 64-bit-only land

Devin L. Ganger — Fri, 29 Aug 2008 20:07:05 GMT

You may have missed this interesting blog post this morning amidst all the political kerfuffle, so let me sum up: the next version of OCS will only support x64 platforms.

This isn't the big deal it would have been for OCS 2007. A lot of the initial FUD around the 64-bit-only move in Exchange 2007 turned out to be mere steam. While there were some initial challenges involved in managing the new 64-bit Exchange deployment from 32-bit machines, Microsoft got a lot of the licensing figured out and released the appropriate sets of tools to allow management of Exchange 2007 from both 32-bit and 64-bit environments. I fully expect that the OCS group has been paying close attention to all of this and taken good notes.

There's no denying that Exchange 2007 benefits from the "64-bit only in production" stance -- and with the release of Windows Server 2008 and Hyper-V, not to mention Microsoft's updated support statement for virtualization environments, the need for 32-bit environments is going away. My biggest reason for wanting 32-bit Exchange environments was so I could run demos under Virtual Server; now that I have Hyper-V, I'm probably not in any rush to go back to Virtual Server and the 32-bit limitation. 64-bit hardware is the norm today, and the x64 Windows variants are solid and mainstream enough for my dedicated application servers. (Maybe not so for the desktop quite yet, but still getting there rapidly.)

The one thing I'm skeptical about, though, is whether the move to 64-bits is really going to reduce the total number of servers in the deployment. In Exchange 2007, I only saw the server reductions in very large environments; the mailbox-per-server gains we got from 64-bits was offset by the explicit breakout of roles and the business needs that drove redundant configurations like CCR (which meant no co-locating roles with the Mailbox role) and multiple HT/CAS servers. I'm wondering how this is going to play out with the next version of OCS, where it already has so many distinct roles in play.

What I *hope* to see is that the maximum capacity of each server role (such as the number of users per pool or the number of streams per mediation server) can be driven upwards; this makes the large datacenter configuration options much more attractive, because it does translate to a reduced number of servers. However, for organizations that still have relatively low bandwidth separating their various locations, 64-bits won't do much to help; OCS deployment planning is very dependent on bandwidth, and is often the top limit on scalability long before the limits of the 32-bit Windows environment.

The OCS Edge Server: how many NICs do I need?

Devin L. Ganger — Thu, 14 Aug 2008 18:01:26 GMT

There are a lot of people out there who want to try to get around Microsoft's recommended configuration for the OCS Edge Server roles. For whatever reason, they don't like the thought of have two network interfaces, one on a publicly routable IP network, the other on the private network. I've talked in the past about some of the reasons why this configuration is not only recommended, but actually a good idea, but let's just say it took a lot of talking and thinking before I accepted that notion.

MVP Jeff Schertz has done a fantastic job of walking through the various permutations people have come up with, separating what will work from what won't, and explaining the pros and cons of each variant. I highly recommend this post.

I also want to amplify a point he makes: having multiple interfaces (whether physical or virtual) on the same subnet will cause interesting and otherwise inexplicable weirdness on a Windows machine. I'll write up the situation I'm seeing in a bit (not OCS!), but let me be clear: it's caused me all sorts of problems. Run, do not walk, away from any "solution" that requires this.

First Look at Microsoft Online Services: the Sign-In tool

Devin L. Ganger — Mon, 28 Jul 2008 18:30:18 GMT

Continuing from my previous post on MOS...

I didn't really mention this in the previous post, but MOS is designed to provide a hosted alternative to the server-side applications. One of the goals is to continue working with existing native clients and client access methods, so (for example) you can access your Exchange Online mailbox through OWA (running from MOS), through Outlook, or even through EAS/Windows Mobile. In order to do this, though, your client applications need to know how to talk to MOS and provide the proper credentials.

You can do this the hard way or the easy way. The hard way is running around and reconfiguring each application by hand and teaching your users how to use a separate set of credentials. The easy way is to use the MOS Sign-In tool, a little .NET 3.0 application that runs on the client desktop. It interacts with Outlook 2007 RTM/SP1, LiveMeeting 8, and IE7+.

When this application is run, it will invite the user to logon to MOS. The first time they do so, they're required to change their password. It then detects the apporpriate applications, offers to configure them to work with MOS, and then just sits quietly on the desktop, providing a seamless SSO experience.

To be continued...

First look at Microsoft Online Services: adding domains

Devin L. Ganger — Mon, 28 Jul 2008 18:21:46 GMT

I'm at an airlift here in Redmond for the new Microsoft Online Services (MOS), Microsoft's hosted services platform. Right now, MOS offers a combination of hosted Exchange (OWA, Outlook, and even EAS!), hosted SharePoint, and Live Meeting. We've just gone through an overview of the service, and it looks cool -- enough so that I'm now seriously considering switching my personal domains over to it (especially since they offer the ability to synchronize with your Active Directory deployment).

MOS is currently in beta and you can go sign up for a time-limited trial. There's only a certain number of trial accounts active at any given time, so your trial request may not be provisioned immediately; however, you can go to https://mocp.microsoftonline.com and sign up for one. You'll need a Windows Live account.

As you might imagine, MOS allows you to associate one or more DNS domains with your online account. When you register for your account, you're asked for a domain. This domain is not verified and, in fact, seems to be used simply as an internal administrative tag -- once your account and service is set up, you have to specifically add DNS domains. Adding them is a fairly simple process:

Register your domain name with a registrar.
Provision your domain with a DNS provider (often combined with step 1).
Add the domain name to your MOS Admin Center.
Run the verification wizard and add the auto-generated CNAME to your domain's DNS zone.
Validate the domain in the MOS Admin Center.
Start provisioning users with this domain, enable inbound e-mail on this domain, etc.

The verfication step is an important piece, because this helps MOS make sure that you're using a domain you're actually in control of. Otherwise, malicious people could sign in and hijack your domain, which would suck. The way Microsoft does this is actually simple and elegant: they generate a unique CNAME record (that looks very much like a GUID), and ask you to add this CNAME record, pointing back to a server under their control, to your zone. This has lots of advantages:

It's pragmatic. If you can add a CNAME record to a zone file, you effectively control the domain.
It avoids the nastiness that can result in WHOIS-based verification and allows people who register domains to continue using proxy companies, hiding their personal info from WHOIS spammers.
It's relatively easy. You simply have to add a simple record to your DNS; if you can't do this (or your DNS hoster can't do it for you), then you have much bigger problems managing your DNS and verifying your DNS domain under MOS is the least of your problems.
It's low-impact. The generated CNAME is highly unlikely to be queried during normal operations by your users; only MOS is likely to be looking for it. It doesn't require you to repoint your MX records or otherwise make major modifications to your infrastructure if all you want to do is start using online SharePoint and Live Meeting.

Note that just because you add a domain to MOS doesn't mean you have to use it for email! That's a separate operation, which is a two-step process of enabling inbound email for that domain and then updating your MX records appropriately.

More on other MOS functionality coming later...big thanks to the event staff for their kind permission for me to blog!

DPM 2007 Rollup packages now available

Devin L. Ganger — Thu, 10 Jul 2008 02:34:31 GMT

While I was away on vacation last week, Microsoft finally released the DPM 2007 Rollup packages to Microsoft Downloads. (I blame Jason Buffington; I'm sure he waited until I was out of office.) There are both x86 and x64 packages; both require you to download three separate files.

In addition to various bug fixes, this rollup (also known as a "feature pack") provides the following new functionality:

Official support for protecting Windows Server 2008 servers (and supported applications, such as Exchange Server 2007, running on Windows 2008), including protecting the system state.
You get support for backing up clustered Virtual Server 2005 R2 SP1 environments. Before, the cluster itself was not seen as a cluster by DPM, and depending on your configuration you may have needed to do some funky scripting.
Better tape handling. You can now share tape libraries between multiple DPM servers, reducing the cost of long-term tape retention and allowing better utilization of high-end tape libraries. You can also put multiple protection groups on a single tape; DPM 2007 RTM would start a new tape as it began writing each protection group, even if the previous tape was not fully used. This could get expensive.

I haven't yet been able to confirm whether the cleaning tape bug Tim noted has been fixed in this update, but I suspect not.

Applying this update is a four-step process:

Install the main DPM update (DataProtectionManager2007-KB949779.exe)on your DPM servers.
Install the SQL Server update (SqlPrep-KB949779.msp) on the machine hosting the SQL Server database for DPM. In a default install, this is the same machine that is your DPM server.
Update the agents on your protected servers to version 2.0.8107.0. You can push them out through the console or manually run the .msp update package on your protected machines (using any supported push mechanism). You will need to restart the protected machines for the new agent version to take effect.
Update the DPM Management Shell update (DPMManagementShell2007-KB949779.msp) on all of your DPM management stations (including the DPM servers themselves).

Although the official instructions give the update steps in the previous order, I have run all three udpates on my lab DPM servers before updating the agents on my protected servers, and as long as Microsoft doesn't say that's not supported, that's the way I'd recommend doing it -- that way, all of your PowerShell tasks are using the updates even if you don't have all the protection agents pushed out yet.

Hyper-V in the hizzouse!

Devin L. Ganger — Fri, 27 Jun 2008 03:28:20 GMT

Everyone's being so coy in the Windows blogosphere today. "As you may have heard..." Heck with that; this is wicked cool. Hyper-V has Released To Manufacturing ... and is already available for download. As the link explains, it'll start coming down the Windows Update pipe July 8th. If you don't want your Windows Server 2008 machine to be updated yet, don't be blindly accepting updates.

Why wouldn't you want to get it first thing?

You're running a previous version of Hyper-V. If so, be aware that upgrading your VMs is not automatic. It's not a horrible process, but it will take some time. You have to manually export each VM, remove the VMs from the server, upgrade the server, re-import the VMs, then update the Integration Services. The more VMs you have, the more time this will take.
You're running some software that is not yet compatible with Hyper-V RTM but works with an earlier build. In this case, you want to wait until that software has a patch available.

I fit into both categories. I think I'm going to wait until I'm back from vacation to do it.

Oh, yes, just because Hyper-V is now RTM doesn't mean that you can go run to install Exchange 2007 on it in production. See Scott Schnoll's post for more info.

These are not the solutions you're looking for

Devin L. Ganger — Fri, 27 Jun 2008 03:18:47 GMT

As IT professionals, we are more than often prone to fall to the perils of magical thinking. (I'm sure this is a side-effect of being human, which is a pesky and bothersome condition I will have to do something about one of these days.) Magical thinking in this context is when we have not internalized the intricacies of a problem and instead rely on formulas rather than true understanding to come up with solutions.

At one ISP I used to work at, we had a glorious reclaimed piece of technology, an Auspex NS-5500 file server. Every now and then on reboot, this old beast of a machine would fail to boot up; the cure was to open the cover over the drive cage and give it a good swift whack. We all assumed that this was because one of the drive connectors was a bit loose, but when our "magic" fix failed to work one night I discovered that it was in fact because one of the screws holding things in place was missing, allowing the drive bay to sag just a tiny bit. It was this tiny bit of sag that put just enough stress on the connector for drive 0. Had we actually opened the case up earlier, we'd have been able to solve the problem -- and prevent a year of whacking the server.

All too often, I see magical thinking in the field of security. Case in point: I recently heard about a gentleman who has a client that is requesting ETRN support be added back to Exchange 2007, either natively or through an add-on. They want to deploy the Edge role in their DMZ, have it queue up mail for the internal organization, and then have their Hub Transports (in the internal protected network) initiate a connection out to de-queue the messages using the ETRN SMTP extension. The reason they want this is that they've done due diligence and read some very thorough documents about computer network zones and have come to the conclusion that all network connections must be initiated from the most secure network. This, they say, removes the threat of malware taking over the Edge server in the DMZ and allowing an attacker to use it as a launching point to the protected network.

Now, the recommendation for connections to be initiated from a more secure network to a less secure network is a good general baseline to follow when it makes sense. However, it is not realistic in all cases (if we followed this to the letter, nobody would be able to receive e-mail from external senders except through random polling of Internet SMTP hosts, which is not at all scalable). This is doubly true if you don't understand how the underlying protocols work. Case in point: ETRN, defined by RFC 1985, "SMTP Service Extension for Remote Message Queue Starting". Quoting from section 3, "The Remote Queue Processing Declaration service extension" (emphasis added):

To save money, many small companies want to only maintain transient connections to their service providers. In addition, there are some situations where the client sites depend on their mail arriving quickly, so forcing the queues on the server belonging to their service provider may be more desirable than waiting for the retry timeout to occur.

Both of these situations could currently be fixed using the TURN command defined in [1], if it were not for a large security loophole in the TURN command. As it stands, the TURN command will reverse the direction of the SMTP connection and assume that the remote host is being honest about what its name is. The security loophole is that there is no documented stipulation for checking the authenticity of the remote host name, as given in the HELO or EHLO command. As such, most SMTP and ESMTP implementations do not implement the TURN command to avoid this security loophole.

This has been addressed in the design of the ETRN command. This extended turn command was written with the points in the first paragraph in mind, yet paying attention to the problems that currently exist with the TURN command. The security loophole is avoided by asking the server to start a new connection aimed at the specified client.

See the problem? ETRN was not designed to solve a security problem; it was designed to solve a financial problem back in days when always-on bandwidth was a lot more expensive and most ISPs metered traffic. It masquerades as solving a security problem only because it's designed to avoid a loophole in an insecure and exploitable feature. As a result, ETRN won't solve the problem these people want it to solve; all it does is tell the system in the DMZ to initiate a new connection to the Hub Transport servers. It doesn't reuse the existing connection initiated by the Hub Transport servers. They can't use a firewall rule to block outgoing access from the Edge to the Hub Transport and be safe, because they'll cut off all incoming traffic.

However, let us for a moment assume that it did work the way they wanted it to: my Hub Transport initiates an outbound SMTP session to the Edge. In this session, HT is the SMTP client, ET is the SMTP server. As soon as HT issues the ETRN command, they still have to swap roles -- HT is now using the SMTP server code paths, while the ET is using the SMTP client code paths. Any theoretical vulnerabilities that are in the HT SMTP implementation are still going to be there, still exposed to the message traffic about to be sent down the connection, still open to exploitation.

This is the magical thinking: firewalls and a DMZ will protect my traffic. This is not true; firewalls and networks zones are two components of a complete security plan. Neither firewalls nor network zones can protect legitimate traffic, nor are they designed to; they are designed to allow you to designate which traffic is legitimate. If you want to secure that traffic, you need to turn to other measures.

masteringdpm.com back online

Devin L. Ganger — Mon, 23 Jun 2008 15:26:49 GMT

Things got hairy enough last week that I forgot to post, but my hosting provider got the problem sorted out and the website is back online.

masteringdpm.com temporarily down

Devin L. Ganger — Wed, 18 Jun 2008 18:43:45 GMT

If you've tried to get to masteringdpm.com in the past couple of days, you may have gotten a cryptic error message instead of a site with DPM goodness. I'm working with my hosting provider to get it put back up ASAP and will post again once it's back up.

Tech-Talk: Making Backups Cool with DPM

Devin L. Ganger — Wed, 18 Jun 2008 17:55:50 GMT

While I was at the Tech-Ed NA IT Pro conference last week, Jason Buffington and I took the chance to invade the Tech-Ed Online fishbowl studio and record a quick Tech-Talk on using DPM. You can now view it online on the Tech-Ed IT Pro page and the Library page, or stream it directly. Now that Tech-Ed's over, maybe we'll both find the time to be on Xbox Live at the same time so we can continue our discussion in Call of Duty 4...