Scripting Patch Management of Enterprise Web Clusters on Microsoft.com
One of the most common questions I am asked when meeting with customers is, How does MSCOM patch their Enterprise Web servers? I will cover exactly what our approach is, and give you a little background into some of the challenges we are faced with when patching our production Web servers. Also, I have included a sample script we currently use on the Microsoft.com Web team to patch our production Web servers.
To give you an overview of our environment, Microsoft.com is comprised of over 120 Web properties hosted on over 1000 Web servers. Our Web team supports sites that range from our corporate business presence of www.microsoft.com, to Developer/ITPro sites such as MSDN, and TechNet, including Download distribution sites such as Windows Update, and Download.microsoft.com. To maintain a high level of availability, each of our sites is comprised of multiple web clusters in multiple datacenters each running NLB. As I mentioned high availability is extremely important to our customers, and is invaluable in showcasing the Microsoft products and services we utilize here on Microsoft.com. Unfortunately, we are faced with the same challenges of having to apply hotfixes and service packs while making sure there are no service interruptions. This is a tough challenge. For example, on the Download.microsoft.com site it can take up to six hours to drain all the active connection from each server. We have architected each of these sites with added server capacity to ensure we can handle peak capacity and also handle any unplanned or planned outages such as a patch event.
Whether we are configuring new Web servers, maintaining configuration control or in this case patching our servers, we leverage admin scripting. Every engineer should have a working knowledge of the basics of a scripting language. This skill set is invaluable in managing enterprise Web servers. Scripting our deployments allows us the flexibility to perform controlled patching. The advantages of scripting your patch deployments are zero service interruption, and the ability to have coordinated customer deployments. For the advantages listed above we leverage the use of admin scripting during our deployments.
Listed below is a sample vbscript we have recently used to patch our Web servers. Copy the script below and save it as SrvPatch.vbs. You will need to create a text file named ServerList.txt containing all the servers you want patched. To execute the script from the command line type “SrvPatch.vbs ServerList.txt”. You will then be prompted to type in your password. The basic flow is of this script is to drain the live connections, run the appropriate patch and associated switches, place the server back into rotation, and loop thru the next server in your ServerList file.
‘ Script example: (Save the below script as SrvPatch.vbs)
‘ Script requires the SysInternals tool PSEXEC. http://www.sysinternals.com/Utilities/PsExec.html
‘ Syntax to execute script “SrvPatch.vbs ServerList.txt”
‘
Dim oFSO
Set oFSO = CreateObject("Scripting.FileSystemObject")
Dim oFile
Dim sServer
Dim sPass
dim sSystemRoot
Set WSHShell = WScript.CreateObject("WScript.Shell")
If Wscript.arguments.count > 1 Then
Wscript.Echo "Syntax: SrvPatch.vbs <Server List> [Password]"
WScript.Echo "Example: SrvPatch.vbs ServerList.txt Password"
WScript.Quit (0)
END If
set oFile = oFSO.OpenTextFile(WScript.arguments(0))
'sPass = WScript.arguments(1)
Wscript.Echo "Please enter password:"
sPass = wscript.stdin.readline
sSystemRoot = wshShell.ExpandEnvironmentStrings("%systemroot%")
Do while oFile.AtEndOfStream =false
sServer = oFile.ReadLine
GetInfo sServer, CountConnections
‘ Draining traffic from server
Set BeginDrain = CreateObject("wscript.Shell")
BeginDrain.Run "psexec \\" & sServer & " wlbs drain all",0,"true"
Do While CountConnections > 20
GetInfo sServer, CountConnections
wscript.echo sServer & ": " & CountConnections
wscript.echo "Sleeping..Draining.."
WScript.Sleep 10000
Loop
Set WLBSSuspend = CreateObject("wscript.Shell")
WLBSSuspend.run "psexec \\" & sServer & " wlbs suspend",0,"true"
wscript.echo sServer & ": Drained..WLBS Suspend..Patching to Begin"
‘ Launching your patch or executable file
Set LaunchPatch = CreateObject("wscript.Shell")
LaunchPatch.run sSystemRoot & "\system32\cmd.exe /c echo " & sPass & "| \\Server\share\patchfile.exe", 1,"true"
wscript.echo sServer & ": Patched......."
WScript.Sleep 10000
‘ Adding server back into rotation
Set WLBSResumeIntoRotation = CreateObject("wscript.Shell")
WLBSResumeIntoRotation.run "psexec \\" & sServer & " wlbs Resume",0,"true"
wscript.echo sServer & ": Resumed"
Set WLBSStartIntoRotation = CreateObject("wscript.Shell")
WLBSStartIntoRotation.run"psexec \\" & sServer & " wlbs Start",0,"true"
wscript.echo sServer & ": Started and taking traffic"
wscript.echo "------------------------------------"
Loop
oFile.Close
Function GetInfo(Computer, CountConnections)
REM On Error Resume Next
strComputer = Computer
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")
Set colItems = objWMIService.ExecQuery("Select * from Win32_PerfFormattedData_Tcpip_TCPV4",,48)
For Each objItem in colItems
CountConnections = objItem.ConnectionsEstablished
Next
rem wscript.echo "Connections at: " & CountConnections
End Function