Well, the title says it all! You will have to excuse my lack of use of images, I'll have to get back to this post and add some later on, but at the moment you'll have to make do - I'll try to be as descriptive as I can!!

I'm assuming that you've bought some Office 365 Licenses, and also that you have an Exchange 2010 installation on-premises. I'm also assuming that you've got an Internet Domain Name set up internally (i.e. contoso.com, not contoso.local). Let's get started:

At this point you are faced with three options, as described in this MS article: http://community.office365.com/en-us/w/exchange/514.aspx?CompanyType=CompanyTenant&DapEnabled=0&HasLiteSKU=0&HasAdminPermissions=1

Hybrid routing implies the extra cost of buying FOPE (Forefront Online Protection for Exchange) licenses, so although it's the best solution out of the three it's a no-go for now. Of the other two, the one that makes more sense is a decentralized shared address space, where email flows to cloud recipients through the on-premises server and sent from cloud recipients to Internet recipients directly from the cloud. A centralized shared address space would only make sense if strict compliance was to be enforced by a security protocol like SOX or ISO.



Implementing a Decentralized Shared Address Space scenario


Setting up Domains and DNS Records 

1. Sign in to the Microsoft Online Services Portal with your Office 365 Administrator Credentials
2. On the Admin page, under Management, click Domains
3. Click Add a domain
4. Type the name of the domain you would like to add (e.g. itguy.gr)
5. Click Next
6. Click Verify domain
7. Add a TXT record on your public DNS system

Now comes the part where you choose your newly added domain's intent, i.e. what you'll use it for. For the moment let's choose just Exchange Online, for two reasons: SharePoint Online must be added in a separate process and for Lync Online you cannot use a domain that's already set up on-premises in a hybrid scenario.

On the second step of this wizard comes the truly confusing part; You'll be presented with a table with 3 types of DNS records: TXT, MX and CNAME. You can safely create the TXT one, but for the other two, if you create them you'll be effectively cancelling out the DNS records that have your on-premises system working!!! Think about it: you already have an autodiscover.yourdomain.com CNAME record, or else auto-configuration wouldn't work in your on-premise system. Similarly, if you overwrite your @.yourdomain.com MX record, say bye-bye to incoming mail to your on-premise Exchange system. So here's what you should do for the example domain we all love, contoso.com:

CNAME Alias: autodiscover.service.contoso.com
TTL:3600
Destination Host: autodiscover.outlook.com

MX Mail domain: service.contoso.com
TTL: 3600
Preference: 10
Mail server:  contoso-com.mail.eo.outlook.com

You're welcome!!


Preparing your on-premises organization for Office 365


Let's kick this off by saying that your on-premises Exchange 2010 Server(s) should all be on SP2. This means the EDGE server(s) as well.

First step will be to update your exchange schema for hybrid deployment, and mind you, the domain controller on which you update your schema must be a 64-bit machine and be included in the same AD site as your schema master.

So, open a command prompt, navigate to the directory location of your Exchange Server 2010 SP2 binaries and type setup /prepareAD

Now it's time to deploy your Federation Server Farm. I'd normally opt out of creating a server farm as most of my clients' environments use clustered Hyper-V servers so the individual VMs are safe enough, but in this instance ADFS 2.0 is an ultra-light application that you can set up on any domain-joined server. Effectively, the only pre-requisite is that these 2 servers must not utilize IIS for port 443 (HTTPS). That's all!

There's a great step-by-step article here on how to set up your ADFS server farm:

http://onlinehelp.microsoft.com/en-us/Office365-enterprises/ff652539.aspx

But we'll do a quick step-by-step nonetheless:



Deploying a Federation Server Farm


What you'll need:

• Two Windows 2008 or 2008 R2 hosts joined to your domain. One will server as the primary server (the one you will install ADFS first) and the other as backup.
• One IP address that will serve as your NLB cluster IP Address
• One DNS A record (in your internal DNS servers) in the form: sts.yourdomain.com
• One trusted third-party SSL certificate

Let's go:

1. Install the Network Load Balancing Feature on both servers and set up an NLB cluster. Make sure you define a port rule just for HTTPS (TCP port 443)
2. Go to the primary ADFS server (i.e. the one you will install ADFS first), go to IIS Manager and create a certificate request. Tho points to pay attention to:
1. As a certificate common name use: sts.yourdomain.com
2. As a bit length use: 2048
3. Now use the certificate request you created to obtain an SSL certificate from a third-party provider. (hint: if you're creating a lab or you just don't care too much about security give www.startssl.com a go... one year SSL certificate 100% free!!!)
4. Install the certificate in the primary ADFS server's IIS manager. Also go to bindings, and select the newly-imported certificate for HTTPS
5. Now extract the certificate with its private key (.pfx) and repeat step 4 on the secondary ADFS server's IIS
6. Create an ADFS service account
1. Create it as a user in Active Directory Users and Computers
2. Make sure "User cannot change password" and "Password never expires" is selected
3. Set the SPN of the service account as described here
7. Install ADFS 2.0 on both servers
1. Obtain installation package here
2. On the Server Role page of the ADFS 2.0 setup wizard, make sure you select "Federation Server"
3. Install the ADFS Update Rollup 1
4. Install Microsoft Online Services Sign-in Assistant
5. Install Microsoft Online Services Module for Windows PowerShell
8. Configure First Federation Server in Federation Server Farm
9. Add Secondary Federation Server to Federation Server Farm

Enabling single sign-on:

Careful! The first three steps are not included in any guide !!

• One Windows 2008 or 2008 R2 server
• Access to modify your external DNS records
• The certificate you used on your internal ADFS servers
• A working DMZ Topology
• Appropriate rules on your corporate firewall

Here we go:

1. First of all, configure your Windows server, position it on the DMZ and make sure there is connectivity both to the Internet and your internal network
2. Install the IIS Role with the default options, import the certificate you installed to the internal ADFS servers, and configure the bindings for port 443
3. Add the respective Host Name and IP address of the ADFS NLB Cluster you created earlier to the ADFS Proxy server's Hosts file
4. Create a new A record on your public DNS server with the name sts.yourdomain.com (exactly like the one you created on you internal DNS servers) and point it to the public IP address you will use for your ADFS Proxy
5. Configure your firewall so that HTTPS requests on this IP will be forwarded to your ADFS Proxy
6. Make sure you can:
• ping the load-balanced internal ADFS server farm (sts.yourdomain.com)
• browse to this link: https://sts.yourdomain.com/federationmetadata/2007-06/federationmetadata.xml
• On IE 9, hit the 'Compatibility View' button, then you should be able to browse the XML contents
7. Now run the ADFS 2.0 setup and select Federation Server Proxy
8. Once completed, make sure the "Start the ADFS 2.0 Federation Server Proxy Configuration Wizard when this wizard closes" check box is selected and click finish.

Once the AD FS 2.0 Federation Server Proxy Configuration Wizard starts, do the following:

1. On the "Specify Federation Service Name" type: sts.yourdomain.com, i.e. the exact same name of the internal load-balanced ADFS server farm we configured earlier
2. The "Use an HTTP proxy server when sending requests to this federation service" IS STUPID!!! Why on earth would we need yet another proxy, to proxy our proxy's requests!!!! Make sure this check box remains unselected and hit "Test Connection". If network connectivity is working you will get a "The specified Federation Service was contacted successfully" message.
3. Hit Next, and you're prompted for the logon credentials for the internal ADFS server farm. Remember the ADFS service account you created? That's the one. You're supposed to enter the credentials in the form
   
   yourdomain\adfsaccount
   password
   
   But first go to Server Manager's Main Page, click on "Configure IE ESC" and turn it off for both users and administrators. Don't, and you'll get "unable to establish a trust between the federation server proxy and the federation service" errors
4. On "Ready to Apply Settings, click on Next and watch the magic happen!

• Log on to a computer that sits outside your corporate network (so that it will be directed to your ADFS Proxy server
• In an Internet Explorer window, type https://portal.microsoftonline.com
• In the User ID field, type your corporate username (e.g. user@yourdomain.com)
• The web page should update with the "You are now required to sign in at yourdomain.com" message
• Click the sign-in link and in the sign-in page and enter your domain credentials
• If this takes you to the Microsoft Online Services Portal home screen, then you can pat yourself to the back; the ADFS system you just configured works
• If not, re-check what we've done so far and that you haven't skipped a step. You can also check Adam Conkle's great post on this TechNet thread: http://social.msdn.microsoft.com/Forums/en-US/Geneva/thread/bd815b4e-70be-4f1c-ac7f-a5e9ac5aea02?prof=required

Now to create a Federation Trust:

1. On EMC, go to Organization Configuration and on the Actions pane, click on New Federation Trust
2. Click on New and then on finish and you're done

Last but most definitely not least, we need to create a Hybrid Configuration:

1. Right click under Organization Configuration and choose New Hybrid Configuration
2. Go through the 2 steps of the wizard and click on finish
   This was too easy, now comes the tricky part
3. Right-click on the Hybrid Configuration and choose Manage. The Manage Hybrid Configuration opens
4. On the credentials screen, type in the appropriate usernames and passwords. What I suggest you do is (since we've already synchronized Active Directory) to enable an Active Directory user that is member of the on-premises Organization Management role group, as a Global Administrator in Office 365. This way, you can use the same account for both fields
5. Under Domains, choose your hybrid domain
6. On the next step you will be asked to create a TXT record on your public DNS servers with a Record Value that is a lengthy string of alphanumeric characters. Check the box to confirm that the TXT record has been created
7. Next up, choose your hybrid on-premises CAS and HUB servers
8. Under Mail Flow Settings type the Public IP address of your on-premises Exchange server (i.e. the one on the public MX record), then type the FQDN of the HUB server you chose in the previous step
9. Mail Flow Security is pretty much straightforward, the third-party certificate is already there, and for the Mail Flow Path choose the first option (as discussed in the beginning of this article).
10. Click next but don't go any further just yet! If you continue you will most likely get an error and the wizard will fail with something like:

Execution of the Get-FederationInformation cmdlet had thrown an exception. Federation information could not be received from the external organization.

OR

Exception has been thrown by the target of an invocation

What on earth is wrong? Well, either the MRSProxy is disabled, or WSSecurity is disabled, or the MRSProxy is unreachable from the Internet.


Regardless of where exactly the fault lies, do all of the following:

• On your Hybrid Server's Exchange PowerShell, enter this command:
  Set-AutodiscoverVirtualDirectory -Identity ‘autodiscover (Default Web Site)’ –WSSecurityAuthentication $true
  to enable WSSecurity authentication for autodiscover on your hybrid server, and...
• On your Hybrid Server's Exchange PowerShell, enter this command:
  Set-WebServicesVirtualDirectory -Identity "EWS (Default Web Site)" -MRSProxyEnabled $true -MRSProxyMaxConnections 100
  The rule of thumb is that the larger number you use for the MRSProxyMaxConnections parameter, the less chances there are that a timeout will occur for large simultaneous mailbox moves.
• Create a new publishing rule for Autodiscover in your TMG server.
  To get your hybrid scenario working, you will need to set up an additional ISA/TMG rule pointing to the hybrid server, using the proper public names (don’t forget autodiscover), bound to the OWA listener. You need to allow All Users (as opposed to Authenticated Users), set authentication to “No Authentication, but users can authenticate directly” and configure the following paths:

• /EWS/mrsproxy.svc/WSSecurity
• /EWS/Exchange.asmx/WSSecurity
• /Autodiscover/Autodiscover.svc
• /Autodiscover/Autodiscover.svc/WSSecurity

After configuring the rule, you need to put it above all the other Exchange rules, making it the first matching rule when federation traffic hits ISA/TMG.



Now you're ready to go through with the wizard and bask in the glory of your success!!!

• We've configured our SSO domain with Office 365
• set up public DNS records
• installed an internal ADFS server farm
• enabled Single Sign-On
• configured Active Directory Synchronization
• installed our ADFS proxy on the DMZ
• verified the ADFS bit is all interoperable
• configured our on-premises Exchange 2010 hybrid server for Office 365

Only thing left to do is to move our mailbox(es) up to the cloud!

1. On the EMC console, go to the on-premises node --> Recipient Configuration --> Mailbox
2. Right-click the mailbox you want to move and choose New Remote Move Request
3. On the Connection Configurations windows, under "FQDN of the Microsoft Exchange Mailbox Replication service proxy" type the Internet FQDN of your hybrid server. Definitely consider typing in a domain admin credentials for this test.
4. Under Move Settings, for the Target Delivery Domain browse for "yourdomain.mail.onmicrosoft.com"
5. If you followed this article to the letter, the move should go without a hitch.

You can follow the progress of the mailbox move from Recipient Configuration, Move Request in the EMC console (the Office 365 federated forest you added earlier)

1. On the EMC console, go to the Office 365 node --> Recipient Configuration --> Mailbox
2. Observe that the mailbox you just moved appears with a green arrow. This means that a move request exists for this mailbox (albeit a completed one). You will need to go to "Move Request" underneath select the move request and click on "Clear Move Request". This will enable you to move the mailbox back to the on-premises organization. 
3. Now right-click the test mailbox and click New Remote Move Request
4. On the Connection Configurations windows, under "FQDN of the Microsoft Exchange Mailbox Replication service proxy" type the Internet FQDN of your hybrid server. Definitely type in a domain admin credentials for this test, then consider using delegated permissions
5. Under Move Settings, for the Target Delivery Domain browse for "yourdomain.com". For Remote Target Database, type the on-premises database you want to move the mailbox to. Contrary to what other guides might say, just type in the database name without the server name.
6. This should be enough to get the test mailbox moving back to your on-premises exchange organization.

Finally, it's time to test email flow. Test for all possible cases:

• From Office 365 users to Internet (external) recipients
• From Office 365 users to on-premises recipients
• From on-premises users to Office 365 recipients
• From Internet (external) email users to Office 365 recipients

One potential issue here that has emerged on several of my clients; If you are using TMG with Exchange Edge collocated on the same server then you might have an issue with your Edge Receive Connector. By default, TMG controls (i.e. blocks) all changes made on the Edge Role. So if you have E-Mail Policy Integration mode enabled on your TMG server chances are that the creation of the Receive Connector for Office 365 on your Edge Server has been blocked by TMG.

Check for EventID 31506 from Microsoft Forefront TMG Control service in the Application Log. If you see this error and the Receive Connector is then this is what you should do:

• On the TMG Management Console, to to Troubleshooting and on the Tasks pane click on "Control E-mail configuration Integration"
• Set the status to "Disabled" and enforce the policy.
• Go to Forefront Online Protection for Exchange Administration center and make a note of the IP addresses listed under "IP addresses to configure on your firewall".
• Login to https://portal.microsoftonline.com/ with a Global Administrator account
• Click on Outlook
• On the upper right-hand side of the Outlook page click on Options and then "See All Options"
• On the upper left-hand side click on the drop-down list arrow right from "Manage Myself" and under "Select What to Manage" click on "My Organization"
• Go to Mail Control and on the right-hand side of the page, under Forefront Online Protection for Exchange click on "Configure IP safelisting, perimeter message tracing and email policies"
• This takes you to Forefront Online Protection for Exchange for your organization. Under Information, click on Configuration.
• Copy-paste the IP addresses and subnet ranges to a text document, then arrange them in a comma-separated way, like so:
  12.12.12.0/24, 14.14.14.0/26, 34.23.34.156
• Now open the Exchange PowerShell on the TMG/Exchange Edge server and type the following command:
  New-ReceiveConnector -Name "From Cloud" -Usage Internet -RemoteIPRanges <FOPE Outbound IP Addresses> -Bindings 0.0.0.0:25 -FQDN mailserverexternalFQDN.yourdomain.com -TlsDomainCapabilities mail.messaging.microsoft.com:AcceptOorgProtocol
  where <FOPE Outbound IP Addresses>, copy-paste the comma-separated list from the text document

This will create the receive connector and reinstate email flow from Office 365 to your on-premises organization




Fine tuning?

Now that everything's working you might want to do some fine tuning. This article has great info regarding the Mailbox replication Service. MRS is the service responsible for moving mailboxes, importing and exporting .pst files, and restoring disabled and soft-deleted mailboxes. So, depending on how you want to plan your mailbox moves and your migration to your desired hybrid model, you may want to tamper with the MRS configuration accordingly.

Got SCVMM 2012 RC on a Windows 2008 R2 Virtual Host. When I tried to add a new Library Share with Default Resources on an existing Library Server I got a weird error. Adding a new Library Server with another Share also produced the same error.



Here's the error:

Error (24374)
Addition of default resources to library share (\\server.domainname.local\foldername) failed. DetailedErrorMessage: VMM could not find the specified path \\server.domainname.local\foldername\ApplicationFrameworks\WebDeploy_x86_el-GR_2.0.1070.cr\WebDeploy.msi on the SCVMM.domainname.local server.
Ensure that you have specified a valid file name parameter, and then try the operation again.

Recommended Action
Fix the problem, then try operation again,


The server is otherwise running ace. The operation did create the ApplicationFrameworks folder inside said folder but it looked like this:












Now, the same folder inside the default MSSCVMMLibrary folder looked like this:













It is clearly a Locale issue.

Furthermore, adding an update server in the Fabric Workspace, produces Update Catalog categories in Greek in the Library Workspace. I might speak Greek but I work in English, and by then it was clear to me that this should be dealt with now, before I end up with a mixed English/Greek SCVMM Console that I won't be able to understand.

My system was set with a Greek locale, and for whatever reason, SCVMM was trying to put locale-specific files in the ApplicationFrameworks folder. That's perhaps all well for SAV (Server AppV), since the Greek version of the agent is available to SCVMM. The same is not true for the Web Deployment Tool. In fact the "WebDeploy_x86_el-GR_2.0.1070.cr" folder was empty. Why?

Having a look at the Official Microsoft IIS Site (http://www.iis.net/download/webdeploy), on the bottom right of the page you can see this:



















These, and only these, are the languages which the Web Deployment Tool v.2.0 is available for. So if your SCVMM server's system locale is set on one of the above languages you're good!

If not, then SCVMM tries to copy the Web Deployment Tool files in a language format that is not available (in my case Greek), hence the 24374 error. A bit disappointing but still, it's an RC version, hopefully such issues will be ironed out in the RTM version.

So, to the solution:

Clearly, changing the server Locale back to English should do the trick.

Go to Control Panel --> Region and Language, and change all tabs back to US-English. Then go to the Administrative tab and "Copy Settings" to "Welcome Screen and System Accounts" as well as "New User Accounts", like so:




























Normally, after a restart the issue should be fixed, but it isn't. Some of the locale settings have been saved in the user profiles, so these should be erased.

To do a proper job of it, do the following:

• log on the SCVMM as a local Administrator
• stop the System Center Virtual Machine Manager service
• go to: Control Panel --> System --> Advanced System Settings --> Advanced tab and under User Profiles click on Settings
• delete all User Profiles except the Default Profile.

Do a restart, and it all should be well... right? Well, if SCVMM uses a locally installed SQL instance, most likely, yes. If you're using SQL on a different server, read on.

You will notice that the System Center Virtual Machine Manager service doesn't start. In the SCVMM server's Application logs there's this error:





The signature "P6: System.FormatException" suggests there is something wrong with the System Format (which we just changed) and most importantly "P5: M.V.E.SqmRefresher.IsRefreshRequired" suggests there is something wrong with the SQM Engine refresh. SQM stands for System Quality Metrics and this most likely relates to some entry in the SCVMM database.

A look in the Application logs of the SQL server holding the SCVMM Database confirms that there's something wrong.





















Not much information here but there's definitely a correlation between the error in the SCVMM server and the one in SQL server.

We need to take a look in the VirtualManagerDB database, but what should we look for? All we have is the "SqmRefresher.IsRefreshRequired" signature from the SCVMM service error. Searching for the "Refresh" string could be a start.

So how do you search an entire database (i.e. all collumns within all tables) for a single keyword?

Luckily, Narayana (http://vyaskn.tripod.com/search_all_columns_in_all_tables.htm) has written a nifty stored procedure that does exactly that. Here it is:

 CREATE PROC SearchAllTables
(
@SearchStr nvarchar(100)
)
AS
BEGIN
-- Copyright 2002 Narayana Vyas Kondreddi. All rights reserved.
-- Purpose: To search all columns of all tables for a given search string
-- Written by: Narayana Vyas Kondreddi
-- Site: http://vyaskn.tripod.com
-- Tested on: SQL Server 7.0 and SQL Server 2000
-- Date modified: 28th July 2002 22:50 GMT

CREATE TABLE #Results (ColumnName nvarchar(370), ColumnValue nvarchar(3630))
SET NOCOUNT ON
DECLARE @TableName nvarchar(256), @ColumnName nvarchar(128), @SearchStr2 nvarchar(110)
SET @TableName = ''
SET @SearchStr2 = QUOTENAME('%' + @SearchStr + '%','''')
WHILE @TableName IS NOT NULL
BEGIN
SET @ColumnName = ''
SET @TableName =
(
SELECT MIN(QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME))
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_TYPE = 'BASE TABLE'
AND QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME) > @TableName
AND OBJECTPROPERTY(
OBJECT_ID(
QUOTENAME(TABLE_SCHEMA) + '.' + QUOTENAME(TABLE_NAME)
), 'IsMSShipped'
) = 0
)
WHILE (@TableName IS NOT NULL) AND (@ColumnName IS NOT NULL)
BEGIN
SET @ColumnName =
(
SELECT MIN(QUOTENAME(COLUMN_NAME))
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = PARSENAME(@TableName, 2)
AND TABLE_NAME = PARSENAME(@TableName, 1)
AND DATA_TYPE IN ('char', 'varchar', 'nchar', 'nvarchar')
AND QUOTENAME(COLUMN_NAME) > @ColumnName
)

IF @ColumnName IS NOT NULL
BEGIN
INSERT INTO #Results
EXEC
(
'SELECT ''' + @TableName + '.' + @ColumnName + ''', LEFT(' + @ColumnName + ', 3630)
FROM ' + @TableName + ' (NOLOCK) ' +
' WHERE ' + @ColumnName + ' LIKE ' + @SearchStr2
)
END
END
END
SELECT ColumnName, ColumnValue FROM #Results
END

 So now all we have to do to search the entire database for the string 'refresh' is this query:

EXEC SearchAllTables 'refresh'

Unfortunately, the word 'refresh' is referenced in 2586 columns. Browsing through all these would be tedious. So let's look for 'sqm' instead:

EXEC SearchAllTables 'sqm'

The result looks more promising - 'sqm' is referenced in only 5 columns.
















Entry #5 looks like it might be the winner. LastSQMRefreshTime might well be related to the IsRefreshRequired error.

The column is "PropertyName" and sits inside the "dbo.tbl_VMM_GlobalSetting" table, so let's have a look:

select * from dbo.tbl_VMM_GlobalSetting

Mind you, this is not the actual query I did during the troubleshooting process (forgot to take a screenshot) so the time is not right. The date/time I actually saw was a couple of days before I changed the system locale back to Eng-US.

So it was clear to me that when the System Center Virtual Machine Manager service is starting, it's checking for this specific value, it's not liking what it sees, so it stops and prints that error in the SCVMM server's Application log.

Maybe deleting this value will fix the issue. Or to be more exact, inserting a NULL where the date/time is:

update dbo.tbl_VMM_GlobalSetting set PropertyValue=NULL where propertyName='LastSQMEngineRefreshTime'

checking to see the value has actually switched to NULL:























It has, and guess what? The System Center Virtual Machine Manager service now starts!!!


Now a couple of words regarding this post:

Granted, it was an insignificant issue that I could well live with, and perhaps troubleshooting it took more time than it was worth. What I am interested in, and the reason I believe that such posts are worth sharing, is the process of troubleshooting.

Troubleshooting is a reward in itself, as it provides you with product insight that would otherwise be unattainable. You also come across valuable bits of information that you can re-use, e.g. that great SQL stored procedure that searches for keywords. It improves your confidence. Think about it; how many times do you come across an issue that the solution isn't a 10 minute Google search away??? It's a rare occurrence, and I welcome the challenge regardless of the issue's importance.

Now regarding SCVMM 2012. I'm a huge Microsoft enthusiast, in fact if you'd listen to me talk about IT, you'd think I was on Microsoft's payroll. But in the case of SCVMM, it just has these annoying little quirks that give you the feel it's an unfinished product. This is true for the 2012 as it's an RC version, but versions 2008 and 2008R2 were also quirky. Don't get me wrong, I'd pick the Hyper-V / SCVMM solution over VMWare any day and firmly believe that Microsoft will be the winner of the Hypervisor race. I just hope that the 2012 RTM version will rid itself of these minor issues and emerge as the flawless product it deserves to be.


Yours,


the ITGuy.

EDIT: This post is receiving a surprising amount of hits, and is being debated amongst colleagues in forums, events and gatherings, in my attempt to reach some sort of consensus. So far it appears the community is (unevenly) divided, but through the discussion process I have been offered one alternative other than my own that meets my standards. More on this towards the end of this post. I've also made edits in Italics throughout the post, to illustrate the debate process and my reaction to it.

*********************************************************************************



I've always wanted to do this and I am so happy I finally got down to it... turned out to be easier than I thought, too!

It all started with a need for a Lync Edge server for my Lab... I was already running a TMG server as a firewall/proxy with collocated Exchange 2010 Edge and thought I'd spare myself the expense of buying an extra server to use as a Hyper-V host and see if it was feasible to run Hyper-V on my existing TMG server and use it to host DMZ VMs. Is it something I would recommend for a production environment? It wouldn't be my first option, but I wouldn't discard the idea either.

(EDIT) Some people say that I should. It's a non-standard way of doing things, it's not an MS recommended topology (still haven't found any official indication that it's unsupported), and should be avoided at all costs. I say an engineer's job is to find solutions, and sometimes compromises need to be made. As long as the customer is aware of the dangers, and as long as I've exhausted all other options, if it's a working solution then I'll go for it.

If you look online there are tons of articles of how to use TMG 2010 with Hyper-V in a DMZ, but in a funky kind of way IMHO: You're supposed to have a Hyper-V host server with multiple DMZ VMs, TMG being one of them... which is fine if you're just using TMG just for Internet Proxying and Publishing internal sites.

What if you wanted to use your TMG as your firewall? You would have to weave an intricate web of VLANs on the Hyper-V host, assign them on the DMZ VMs, route traffic from these VLANs to virtual NICs on the TMG VM... if I was to graphically illustrate this, it would look something like:



It just looks wrong! And that's just for 4 VLANs, imagine what it would be like having more! There is apparently some disagreement with this (see http://social.technet.microsoft.com/Forums/en-AU/Forefrontedgesetup/thread/fad4ad74-b50f-4776-8ecf-fcaf257320ab) but I beg to differ.

Physical separation is the basis of any DMZ design worth its money, and VLANing is no substitute. In fact a quick Google search will reveal tons of info on how VLANs can been compromised in intrusion attempts. And for good reason: a VLAN is a means of segmenting networks, not a security counter-measure. Here's a great article by SANS: http://www.sans.org/security-resources/idfaq/vlan.php. More on this at the end of this article.

(EDIT) Overwhelming feedback on this matter attests to some Systems Engineers' lack of security awareness. This brings up a whole new issue, that of lack of role transcendence (something I've been meaning to blog about for a long time but never got down to it); IT Pros become overly entangled in their role and become dangerously close to perceive matters in a monolithic way. The distinction between Systems Engineers, Network Engineers, Security Engineers, etc, is OK as it serves the need for specialisation and expertise, but an awareness of all fields is paramount when designing an end-to-end solution. What is the point of having a DMZ if you're going to enforce network separation by using VLANs? It sounds like building a castle wall out of hay.


So here's my thought:

Couldn't we build a Windows 2008 R2 server with 4 physical NICs, install TMG 2010 and add the Hyper-V role on it, physically connect 2 NICs to the Internet and our LAN and assign the other 2 to be managed by Hyper-V Virtual Network Manager, then use TMG to route DMZ traffic to and from the Virtual NICs created by Hyper-V's VNM ??? It would look something like this:




The answer is we could, I have and it works beautifully!! Is it on a par with physical separation? No! Is it better than that other funky VLAN solution? A lot, because network separation and control happens at the hypervisor level which works at OSI layer 2 (just as VLANing is), BUT you also get the added benefit of OSI layer 3 to 7 vulnerability scanning BEFORE malicious attacks or malformed packets physically reach your internal network. Check the end of this article for a bit more info on this.

(EDIT) Apparently (and this comes from testing as I could not find any info online), the Forefront TMG Packet Filter takes precedence over the Virtual Network Switch Protocol. This ensures packets are grabbed and scanned by TMG before being handed over to Hyper-V.


So, let's begin!
I'm assuming you've already set up a Windows 2008 R2 box, installed TMG 2010 on it and set up basic routing and firewall rules. There are plenty of great articles on the subject out there. Make sure that routing and Internet connectivity works before you proceed.

Set up physical NICs on the TMG server as:

• INTERNAL, connected to the LAN
• EXTERNAL, connected to the Internet-facing router
• DMZ External, NOT physically connected anywhere
• DMZ Internal, NOT physically connected anywhere

Remember that only the EXTERNAL interface gets to have a gateway.

Your Network Connections should look like this:




Now add the Hyper-V role, open Hyper-V Manager and on the Actions column (right-hand side), click on Virtual Network Manager.

Click on "New Virtual Network" (left-hand side), choose "External" as type and then click on "Add"

Under "Name", type: "HYPER-V DMZ Internal"
Under "Connection Type" choose the NIC that corresponds with the "DMZ Internal" connection seen on the previous image. In this instance it's "Realtek PCIe GBE Family Controller #2"

Make sure the "Allow management operating system to share this network adapter" box is checked!

It should look like this:



Now do the same for the external DMZ:



Now click on OK.

After Hyper-V has done its thing, your Network Connections should look like this:





Note that while the physical DMZ adapters are disconnected, the Virtual NICs created by Hyper-V's Virtual Network Manager, appear connected. Had we used Internal or Private connection types in the Virtual Network Manager that do not require a physical adapter, routing outside the Hyper-V/TMG server would be impossible. Since we're not planning on using physical DMZ connections, this is how we achieve network separation on the hypervisor level (OSI Layer 2) without VLANs. Now we can use the Virtual NICs with both Hyper-V and TMG, and here's how:

First of all, create a VM with whatever attributes suits you (CPUs, RAM, disk type/size) but add only ONE virtual adapter to it at this time and assign it to the "HYPER-V DMZ External" Network.



Now start the VM and install Windows. Once Windows is installed, rename the Local Area Connection to "External", so you'll know which Virtual NIC it corresponds to. Now shut down the VM and open settings again.



Notice that although MAC ADDRESS is still on Dynamic, an actual MAC address has been assigned to the Network adapter. Now click on "Static" so the Settings window will look like so:



Click "Apply" and repeat for the "HYPER-V DMZ Internal" Network.

What have we achieved by this? Well, we've let Hyper-V choose an appropriate MAC address for our 2 VM adapters, so we are 100% sure there is absolutely no chance of a MAC address conflict. We also statically bound these MAC addresses to our VM so we're 100% sure they will never change. If they did it might trigger a spoof attack event on TMG and our VM would be blocked.

Now that we have our Hyper-V bit sorted, let's have a look at TMG.

TMG should already have seen your new NICs and set up Network Adapters like so:



Remember, we don't care about the disconnected Physical NICs, we're not using these at all.


Now, under "Networks" we need to declare both our DMZ NICs IP Ranges, and give them a name. That's how they'll appear in Firewall Policy rules.



Last but not least, we'll need to define the Network rules. Rule of thumb is: NAT anything that goes to the Internet, Route everything else:





And that's it, mission accomplished! Now it's just a matter of putting the appropriate Firewall Policy rules in place.



(EDIT) An increasing amount of people point out the fact that adding so many services to one Windows installation runs the inherent risk of a single point for multiple failures, i.e. if one service goes down the whole DMZ will likely go offline. I know this is true, but are we missing the point here? Our starting goal is not to build a rock-solid fail-proof solution, our starting goal is to make the best out of too little to begin with. Why go in the trouble of building a DMZ if your foremost goal is availability and not security? It truly beats the point!



*****************************************************************************
*****************************************************************************



There is a lot to be said regarding security in a DMZ which falls outside the scope of this article. For our purpose, I will stress the following security-related facts.
To understand security we need a basic knowledge of the OSI model. Here's a picture taken from Wikipedia.org (http://en.wikipedia.org/wiki/OSI_model)




















Both VLANs and the Hypervisor operate at Layer 2 (the Data Link layer), hence in respect to separation they can never match physical separation which occurs at Layer 1. However, network separation is not everything. As we deploy security measures, the more OSI Levels we cover/inspect the better our overall security will be. And TMG is aware of OSI levels 3 to 7.

I will present some examples of security vulnerabilities per OSI layer to better illustrate my point:

• Layer 3: IP Spoofing, RIP attacks, ICMP DoS attacks, Ping Flood, Ping of Death, Teardrop, Packet Sniffing
• Layer 4: TCP SYN Flooding, Man-in-the-Middle attacks, Land attack, UDP Flood attack, Port Scanning, Fingerprinting
• Layer 5: Session Hijacking, Brute-Force attacks on Session Credentials
• Layer 6: Fake Certificate attacks, Man-in-the-Middle attacks
• Layer 7: Application-Specific attacks

Now, having TMG operate as a firewall inside a Virtual Machine sitting on a Hyper-V host together with a number of other VMs with nothing separating it from internal servers but VLANs, means at the very least, that you allow potentially illegitimate traffic through your internal switches, disguised as tagged packets... Let's have a look at the diagram again:


Let's say that a request comes from the Internet for your Web Server; it travels through your router, gets tagged as VLAN4, travels through your LAN switches, reaches the Hyper-V server, gets sent to the TMG VM virtual NIC, gets processed and if all is right, gets flagged as VLAN2 or 3 and sent through Hyper-V's VNM to the Web Server's virtual NIC. What if that connection request originated from a spoofed IP address? You've got that packet travelling your entire LAN before TMG drops it. Best case scenario is you're adding unneeded complexity to your system, worst case is you're opening a can of worms in your internal network.

Isn't it much better if the first point of contact for incoming traffic is TMG? Now play that previous scenario on this diagram:



What you get is OSI level 3 to 7 security scanning BEFORE anything reaches your DMZs, let alone your LAN.

Given you have only ONE physical server for a firewall and DMZ VM host, this is clearly a winning solution!




*****************************************************************************
*****************************************************************************

(EDIT) Another solution was presented to me that meets almost all the standards I had in mind when I started this post. I'll give a short description here and promise to expand on it on a new post as soon as I get the chance.

You start off by having a Windows 2008 R2 CORE installation with the Hyper-V Role, then install TMG, and your DMZ hosts as VMs. Then, instead of using VLANs to route packets, you use Hyper-V's own Virtual Network Manager to create one Virtual Network for every physical NIC on your Hyper-V host. Then get this:

You DISABLE all NICs on the Hyper-V host, except the one you'll use for managing it!!! So in theory your CORE machine is impervious to attacks from the Internet AND the DMZ, because the host itself cannot physically connect to these networks since the corresponding NICs are disabled.

BUT, Hyper-V's VNM still routes traffic where it should and everything works beautifully, on the VM side. MS by-the-book goers are happy and I am kinda happy because I'm not relying on VLANs. Then again, I'm still having potentially malicious packets running around my Virtual Networks before TMG stops them, but regardless, it's a solid proposition and merit goes to Hyper-Vaggelis, Greece's resident Hyper-V MVP.

I promise I'll do a proper how-to on this as soon as I get the chance.

*****************************************************************************
*****************************************************************************



Yours,


the ITGuy.