An Uncommon Cause for IIS 503 Service Unavailable Errors

Recently, while migrating IIS websites to a new server, I encountered "Service Unavailable HTTP Error 503. The service is unavailable." errors, but only for HTTPS, while HTTP worked fine. Depending on the scenario, the problem could have just as easily impacted HTTP.

503error1a.jpg

The server was listening on port 443:

1Get-NetTCPConnection -LocalPort 443 -State Listen

503error2a.jpg

If you ever encounter a problem like this, stop the web publishing service:

1Stop-Service -Name w3svc -PassThru

503error3a.jpg

Then check to see if the server is still listening on port 443:

1Get-NetTCPConnection -LocalPort 443 -State Listen

503error2a.jpg

If it is, then something else is running that's listening on port 443. If this is a server that you've inherited, it may not be easy to determine what's listening on that particular port.

I'll install a PowerShell module named Carbon from the PowerShell Gallery to help determine what's going on.

503error4a.jpg

When a non-IIS web server is installed on Windows, it sets an ACL on the port it will listen on. You can see the ACL which references port 443 that's shown in the following image.

1Get-HttpUrlAcl

503error5b.jpg

Actually, there are two entries listed for port 443 in the previous image. The one with the GUID after it is fine since it's only for a particular application via a host header and not the entire namespace.

Although I wasn't sure, I was fairly certain the Dell EqualLogic SAN Headquarters application that was installed on this particular server was causing the problem so I decided to deinstall it since it was no longer needed.

503error6a.jpg

While the deinstall seemed to work fine and the application was removed from the server, it did not remove the ACL assigned to port 443.

1Get-HttpUrlAcl -Url https://+443/

503error7a.jpg

Since deinstalling the application did not resolve the problem, I decided to forcibly remove the ACL. Forcibly deleting ACL's is NOT something I recommend, but I was to the point where I had nothing else to lose because the alternative was to simply spin up another server.

1Revoke-HttpUrlPermission -Url https://+443/ -Principal 'NT AUTHORITY\SYSTEM'

503error8a.jpg

Once that entry was removed and the web service was restarted, the problems with IIS listening on port 443 were resolved.

503error9a.jpg

In addition to the steps previously listed, I went through all of the normal items you would check in this scenario. Most of them were ruled out because the website was working on port 80.

I decided to turn my checklist of items for 503 service unavailable errors into a Pester test to make it easier to know where to start troubleshooting.

 1Describe 'HTTP Error 503' {
 2    $SiteName = 'Default Web Site'
 3
 4    It 'Should have a W3SVC service that is running' {
 5        (Get-Service -Name W3SVC).Status |
 6        Should -Be 'Running'
 7    }
 8    It "Should have an IIS website named '$SiteName' that is started" {
 9        (Get-Website -Name $SiteName).State |
10        Should -Be 'Started'
11    }
12    It 'Should have an IIS application pool that is started' {
13        (Get-WebAppPoolState -Name ((Get-Website -Name $SiteName).applicationPool)).value |
14        Should -Be 'Started'
15    }
16    It 'Should have a process named w3wp.exe that is running' {
17        Get-Process -Name w3wp -ErrorAction SilentlyContinue |
18        Should -Not -BeNullOrEmpty
19    }
20    It 'Should not have any WAS warnings or errors in the system event log' {
21        2,3 | ForEach-Object {
22            Get-WinEvent -FilterHashtable @{LogName='System';Data='WAS';Level=$_} -ErrorAction SilentlyContinue |
23            Should -BeNullOrEmpty
24        }
25    }
26    It 'Should not have any WAS warnings or errors in the application event log' {
27        2,3 | ForEach-Object {
28            Get-WinEvent -FilterHashtable @{LogName='Application';Data='WAS';Level=$_} -ErrorAction SilentlyContinue |
29            Should -BeNullOrEmpty
30        }
31    }
32    It 'Should not have any W3SVC warnings or errors in the system event log' {
33        2,3 | ForEach-Object {
34            Get-WinEvent -FilterHashtable @{LogName='System';Data='W3SVC';Level=$_} -ErrorAction SilentlyContinue |
35            Should -BeNullOrEmpty
36        }
37    }
38    It 'Should not have any W3SVC warnings or errors in the application event log' {
39        2,3 | ForEach-Object {
40            Get-WinEvent -FilterHashtable @{LogName='Application';Data='W3SVC';Level=$_} -ErrorAction SilentlyContinue |
41            Should -BeNullOrEmpty
42        }
43    }
44    It 'Should not have any 503 errors in the IIS log for the website' {
45        Get-ChildItem -Path 'C:\inetpub\logs\LogFiles' -Include *.log -Recurse |
46        Where-Object LastWriteTime -gt (Get-Date).AddDays(-1) |
47        Select-String -SimpleMatch '503' |
48        Should -BeNullOrEmpty
49    }
50    It 'Should not have any HTTP entries in the system event log' {
51        (Get-WinEvent -FilterHashtable @{LogName='System';ProviderName='Microsoft-Windows-HttpEvent'}).Count |
52        Should -Be 0
53    }
54}

503error10b.jpg

The way I've written this test, the last entry is almost always going to fail because there will most likely be at least some informational entries in that particular event log. Starting from the top, check the first item that fails.

One informational entry in the event log stood out that appears to have occurred when the ACL on port 443 was set, although it still didn't specify which application had set the ACL.

1Get-WinEvent -FilterHashtable @{LogName='System';ProviderName='Microsoft-Windows-HttpEvent';Level=4} -MaxEvents 1 |
2Select-Object -Property Message

503error11a.jpg

Information about this particular problem seems to be non-existent, although I did come across a couple of really old articles that had some information about it.

µ