Blog » Via Scott Mattie
Troubleshooting a SCVMM 2012 transport error
Last week, I had someone reach out to me asking for help with a new SCVMM 2012 server. It was a new installation and for the most part it worked well for them during the past month. However, recently they started seeing the error below, which prevent them access to the Virtual Machine Manager:
At first they ignored this error, since rebooting the SQL server seemed to fix the issue. As time went on, the error become more common and they found themselves rebooting more frequently. Sadly, they did not share that information with me that they had to reboot so often. With the information they had shared with me listed above, I recommended that they investigate the following areas:
- Check for any auto services that are not started
- Check to ensure SQL, VMM and WS-Management services (etc.) are started
- Check Application and System logs for any service complainants
- Check Application and System logs for anything wrong with SCVMM and/or SQL Server
For the most part, all services were started and not having any issues. They did not see any credential errors with any service in the logs. At first they suspected the SCVMM application had an issue based on the error logs below:
Application fault again:
At this point, I asked them if they had any experience with SCVMM and were aware of the relationship it has with SQL server. They stated that they had some, but were not sure of the SQL details, so at that point I explained that all of the information they see in the SCVMM console is stored in a database on the SQL server and the application depends on it. At that point they understood why I was asking them about SQL server.
Then they shared with me this gem of information that I was looking for to help them:
I also asked them to log into SQL and check the error logs to confirm what we were seeing from this error message on the SCVMM server. (Above)
They first looked at the installation logs and saw an underlying error in their initial setup logs were coming from SQLServer :
InnerException.Type: System.Data.SqlClient.SqlException, InnerException.Message: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 – The semaphore timeout period has expired.)
When I was able to finally persuade them to look at the SQL server, there are no other databases on that server. However, we did see this error message was all over the place:
A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 – The semaphore timeout period has expired.)
At this point, I let them know that “The semaphore timeout period has expired” is a Winsock error that usually indicates the TCP peer did not ACK some data. This typically means that some part of the network layer croaked. (i.e. a NIC or switch is malfunctioning) It could also probably happen if the server machine was unplugged or hard powered off, but SQL Server itself or the client process wouldn’t cause this type of error. Even if you forcibly terminated the SQL Server process on the server, you wouldn’t see this behavior. It would at least send a TCP Reset. (RST)
I asked them if they knew how TCP works and then explained the 3 way handshake that occurs with these diagrams:
Basic overview:
Then I went further and explained some more details with this one:
I recommended that they either use Netmon or Wireshark to perform a network captures to look at where the failure is actually occurring. However, they saw this error message below and just asked the data center people to look at the SQL server NIC and switches. (It appears it’s either a NIC issue or a networking issue on the SQL server side. I found these 2 logs after digging around for a while and they seem to be consistent throughout)
Personally, I would have gone a step further and pin-pointed it to the actual device. I would have done a quick TCP based trace route, like tcptrace. (It will require you to install wincap) A lot of people are not aware that ICMP traffic, like ping and tracert, is often either dropped or deprioritized a lot in busy networks and it can lead to false alarms. If your network traffic is leaving a building, then I would recommend using a TCP connection to test, so that way I am guaranteed to get a response back. After that, then I would have confirmed my findings with the packet capture by looking for a TCP 3 way handshake.
Moving forward, I hope that people were able to see the key here is to look at basic networking and event viewer (plus other application) logs to get a true sense of what is going on. Sometimes, your perception of things can lead you down the wrong path as you allow it to become a reality. In this case, most of the early troubleshooting was focused on SCVMM and not a lot of attention was placed on the SQL server.
http://www.discountasicsshoes.org/All-Products/ Cheap Asics Shoes
http://www.discountasicsshoes.org/Onitsuka-Tiger-Mini-Clubman-c15 Onitsuka Tiger Mini Clubman
http://www.discountasicsshoes.org/Asics-Gel-Kayano-17-c3/ Asics Gel Kayano 17
http://www.discountasicsshoes.org/Asics-Gel-Lyte-III-c4/ Asics Gel Lyte III
http://www.discountasicsshoes.org/Asics-Gel-Stratus-2.1-c5/ Asics Gel Stratus 2.1
http://www.discountasicsshoes.org/Asics-Suroingu-Japan-Ar-c6/ Asics Suroingu Japan Ar
http://www.discountasicsshoes.org/Asics-Top-Seven-c7/ Asics Top Seven
http://www.discountasicsshoes.org/Asics-Whizzer-Lo-c8/ Asics Whizzer Lo
http://www.discountasicsshoes.org/Onitsuka-Tiger-Monete-Pokhara-c9/ Onitsuka Tiger Monete Pokhara
http://www.discountasicsshoes.org/Onitsuka-Tiger-California-78-c10/ Onitsuka Tiger California 78
http://www.discountasicsshoes.org/Onitsuka-Tiger-Kanuchi-c11/ Onitsuka Tiger Kanuchi
http://www.discountasicsshoes.org/Onitsuka-Tiger-Mexico-66-c12/ Onitsuka Tiger Mexico 66
http://www.discountasicsshoes.org/Asics-Alton--c2/ Asics Alton
http://www.discountasicsshoes.org/Onitsuka-Tiger-Mid-Runner-c14/ Onitsuka Tiger Mid Runner
http://www.discountasicsshoes.org/Asics-Gel-Kinsei-2-c27/ Asics Gel Kinsei 2
http://www.discountasicsshoes.org/-Onitsuka-Tiger-Olympos-c16 Onitsuka Tiger Olympos
http://www.discountasicsshoes.org/Onitsuka-Tiger-Rotation-77-c17/ Onitsuka Tiger Rotation 77
http://www.discountasicsshoes.org/Onitsuka-Tiger-Tokidoki-Mex-Lo-c18/ Onitsuka Tiger Tokidoki Mex Lo
http://www.discountasicsshoes.org/Onitsuka-Tiger-Ultimate-81-c19/ Onitsuka Tiger Ultimate 81
http://www.discountasicsshoes.org/Onitsuka-Tiger-Mexico-66-Deluxe-c20/ Onitsuka Tiger Mexico 66 Deluxe
http://www.discountasicsshoes.org/Onitsuka-Tiger-Mexico-66-Lauta-c21 Onitsuka Tiger Mexico 66 Lauta
http://www.discountasicsshoes.org/Onitsuka-Tiger-X-Tokidoki-Fabre-c22/ Onitsuka Tiger X Tokidoki Fabre
http://www.discountasicsshoes.org/Asics-Gel-Virage-4-c23/ Asics Gel Virage 4
http://www.discountasicsshoes.org/Asics-Stormer-c24/ Asics Stormer
http://www.timberland-discountedshoes.com/Timberland-men-shoes-f2-4.html Timberland men shoes
http://www.timberland-discountedshoes.com/Timberland-men-Hiking-f2-5.html Timberland men Hiking
http://www.timberland-discountedshoes.com/Timberland-men-Sandals-f2-15.html Timberland men Sandals
http://www.timberland-discountedshoes.com/Timberland-women-Boots-f2-6.html Timberland women Boots
http://www.timberland-discountedshoes.com/Timberland-women-Shoes-f2-7.html Timberland women Shoes
http://www.timberland-discountedshoes.com/Timberland-women-Hiking-f2-8.html Timberland women Hiking
http://www.timberland-discountedshoes.com/Timberland-women-sandals-f2-13.html Timberland women sandals
http://www.timberland-discountedshoes.com/Timberland-kids-Boots-f2-9.html Timberland kids Boots
http://www.louboutinshoes-discount.org/Christian-Louboutin-Pumps-c19.html Christian Louboutin Pumps
http://www.louboutinshoes-discount.org/Christian-Louboutin-Peep-Toe-Pumps-c22.html Christian Louboutin Peep-Toe Pumps
http://www.louboutinshoes-discount.org/Christian-Louboutin-Pointed-Toe-Pumps-c27.html Christian Louboutin Pointed Toe Pumps
http://www.louboutinshoes-discount.org/Christian-Louboutin-Mary-Jane-Pumps-c26.html Christian Louboutin Mary Jane Pumps
http://www.louboutinshoes-discount.org/Christian-Louboutin-Sandals-c38.html Christian Louboutin Sandals
http://www.louboutinshoes-discount.org/Christian-Louboutin-Wedges-c28.html Christian Louboutin Wedges
http://www.louboutinshoes-discount.org/Christian-Louboutin-Evening-c30.html Christian Louboutin Evening
http://www.louboutinshoes-discount.org/Christian-Louboutin-Slingbacks-c29.html Christian Louboutin Slingbacks
http://www.louboutinshoes-discount.org/Christian-Louboutin-Sandal-Booties-c31.html Christian Louboutin Sandal Booties
http://www.louboutinshoes-discount.org/Christian-Louboutin-Boots-c33.html Christian Louboutin Boots
http://www.louboutinshoes-discount.org/Christian-Louboutin-Ankle-Boots-c32.html Christian Louboutin Ankle Boots
http://www.louboutinshoes-discount.org/Christian-Louboutin-Over-The-Knee-Boots-c34.html Christian Louboutin Over The Knee Boots
http://www.louboutinshoes-discount.org/Christian-Louboutin-Flats-c35.html Christian Louboutin Flats
http://www.louboutinshoes-discount.org/Womens-Christian-Louboutin-Sneakers-c36.html Womens Christian Louboutin Sneakers
http://www.louboutinshoes-discount.org/Mens-Christian-Louboutin-Sneakers-c37.html Mens Christian Louboutin Sneakers
http://www.winterjackets-monclercoats.com/moncler-coats-women-c95/ moncler coats women
http://www.winterjackets-monclercoats.com/moncler-coats-men-c96/ moncler coats men
http://www.winterjackets-monclercoats.com/Moncler-Men-T-shirts------c78/ Moncler Men T-shirts
http://www.winterjackets-monclercoats.com/Moncler-Women-T-shirts--c79/ Moncler Women T-shirts
http://www.winterjackets-monclercoats.com/Moncler-Men-Vests-c80/ Moncler Men Vests
http://www.winterjackets-monclercoats.com/Moncler-Women-Vests-c81/ Moncler Women Vests
http://www.winterjackets-monclercoats.com/Moncler-Men-Jackets-c82/ Moncler Men Jackets
http://www.winterjackets-monclercoats.com/Moncler-Women-Jackets----c83/ Moncler Women Jackets
http://www.winterjackets-monclercoats.com/Moncler-Men-Sweaters-c86/ Moncler Men Sweaters
http://www.winterjackets-monclercoats.com/Moncler-Women-Sweaters-c87/ Moncler Women Sweaters
http://www.winterjackets-monclercoats.com/Moncler-Couple-Jackets-c88 Moncler Couple Jackets
http://www.winterjackets-monclercoats.com/Moncler-Kids-c89 Moncler Kids
http://www.winterjackets-monclercoats.com/Moncler-Boots-----c90/ Moncler Boots
http://www.winterjackets-monclercoats.com/Moncler-Scarf-&-Cap-c91/ Moncler Scarf & Cap
http://www.winterjackets-monclercoats.com/Moncler-Men-Shoes--c94/ Moncler Men Shoes
http://www.winterjackets-monclercoats.com/Moncler-Lovers-Clothes-c93/ Moncler Lovers Clothes
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Mary-Jane-Pumps-t1-30.html Christian Louboutin Mary Jane Pumps
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Daffodile-Pumps-t1-32.html Christian Louboutin Daffodile Pumps
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Peep-Toe-Pumps-t1-31.html Christian Louboutin Peep Toe Pumps
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Platform-Pumps-t1-33.html Christian Louboutin Platform Pumps
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Pointed-Toe-Pumps-t1-34.html Christian Louboutin Pointed Toe Pumps
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Lady-Peep-t1-45.html Christian Louboutin Lady Peep
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Bianca-t1-46.html Christian Louboutin Bianca
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Pumps-t1-44.html Christian Louboutin Pumps
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Sandals-t1-35.html Christian Louboutin Sandals
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Wedges-t1-43.html Christian Louboutin Wedges
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Slingbacks-t1-36.html Christian Louboutin Slingbacks
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Evening-t1-42.html Christian Louboutin Evening
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Ankle-Boots-t1-37.html Christian Louboutin Ankle Boots
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Boots-t1-38.html Christian Louboutin Boots
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Over-Knee-Boots-t1-39.html Christian Louboutin Over Knee Boot
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Sneakers-t1-40.html Christian Louboutin Sneakers
http://www.cheaplouboutins-onlineoutlet.org/Christian-Louboutin-Flats-t1-41.html Christian Louboutin Flats
http://www.wintercoats-moncler.com/Moncler-T-shirts-Men/ Moncler T-shirts Men
http://www.wintercoats-moncler.com/Moncler-T-shirts-Women/ Moncler T-shirts Women
http://www.wintercoats-moncler.com/Moncler-Jackets-Men/ Moncler Jackets Men
http://www.wintercoats-moncler.com/Moncler-Jackets-Women/ Moncler Jackets Women
http://www.wintercoats-moncler.com/Moncler-Jackets-Kids Moncler Jackets Kids
http://www.wintercoats-moncler.com/Moncler-Vests-Men/ Moncler Vests Men
http://www.wintercoats-moncler.com/Moncler-Vests-Women Moncler Vests Women
http://www.wintercoats-moncler.com/Moncler-Sweaters-Men/ Moncler Sweaters Men
http://www.wintercoats-moncler.com/Moncler-Hoody-Sweaters-Women/ Moncler Hoody/Sweaters Women
http://www.wintercoats-moncler.com/Moncler-Boots/ Moncler Boots
http://www.wintercoats-moncler.com/Moncler-Couple-Jackets/ Moncler Couple Jackets
http://www.wintercoats-moncler.com/Moncler-Scarf---Cap/ Moncler Scarf & Cap
http://www.wintercoats-moncler.com/Moncler-Lovers-Clothes/ Moncler Lovers Clothes
http://www.wintercoats-moncler.com/Moncler-Men-Shoes/ Moncler Men Shoes