Possible network latency

[Albert Jurado]

I'm in the mist of troubleshooting possible high network
re-transmissions. I'm basically attempting to capture enough data to
prove that the network is not the bottle neck. I have complaints from
user that their systems are slow but it seems that the application they
are using is the bottleneck. We have several in-house developed
applications that the end users uses that communicates with a SQL
server. They also browse the internet frequently. I've been looking
for articles that describe what the average re-transmission rate is for
a standard TCP/IP networked workstation but I could not find any. I've
attempted a simple test like running the trouble application and then
performing a simple copy & paste (of a 1gb file) from a file server to
the workstation's desktop while pinging the SQL server at the same time
and I did not see the time change from <1ms. The application ran slow.
Plus the file copied over without any issue.



A brief description of the network is as follows. We have 5 floors with
each floor having a wiring closet. In each closet we have a Cisco 3750
cluster of switches. Each floor has fiber running down to the core
switch on the 2nd floor.



The reason we suspect re-transmission is because some workstations show
a high "segments retransmitted" when you run netstat -s. If I run
Wireshark on the suspect workstations what should I be looking for in
the capture? Will I capture re-transmission that corresponds to the
netstat -s output?



Thx.





Albert

[Visser, Martin]

You won't find a "average re-transmission" statement because there really isn't one. On a well provisioned local area network you could expect it to be zero. On a heavily congested wireless or WAN network it might be 20% before it becomes unworkable.

If netstat -s shows retransmission by the client than either the workstation or the LAN should show retransmissions. (The client will retransmit either if it receives if receives duplicate ACKs for segments it has said or if the retransmission timeout (RTO) the TCP stack calculates has exceeded before it receives an ACK.

On Wireshark if you seclect Analyze:Expert Info Composite and Notes tab you will see a TCP retransmission count if you are getting any. You can also just apply the Display Filter "tcp.analysis.retransmission" to see all relevant packets.

If you are getting a lot of retransmissions where client and server are on a local switched LAN you may want to look at the physical error counts on your switches - and look for physical layer issues.

Regards, Martin

Martin Visser

Technology Consultant
Technology Solutions Group

Hewlett-Packard
410 Concord Road
Rhodes NSW 2138
Australia

Mobile: [phone]
Fax: [phone]
E-mail: martin.visserAThp.com

This email (including any attachments) is intended only for the use of the individual or entity named above and may contain information that is confidential, proprietary or privileged. If you are not the intended recipient, please notify HP immediately by return email and then delete the email, destroy any printed copy and do not disclose or use the information in it.





________________________________
From: wireshark-users-bounces-IZ8446WsY0/***@public.gmane.org [mailto:wireshark-users-***@wireshark.org] On Behalf Of Albert Jurado
Sent: Thursday, 3 July 2008 12:40 AM
To: wireshark-users-IZ8446WsY0/***@public.gmane.org
Subject: [Wireshark-users] Possible network latency

I'm in the mist of troubleshooting possible high network re-transmissions. I'm basically attempting to capture enough data to prove that the network is not the bottle neck. I have complaints from user that their systems are slow but it seems that the application they are using is the bottleneck. We have several in-house developed applications that the end users uses that communicates with a SQL server. They also browse the internet frequently. I've been looking for articles that describe what the average re-transmission rate is for a standard TCP/IP networked workstation but I could not find any. I've attempted a simple test like running the trouble application and then performing a simple copy & paste (of a 1gb file) from a file server to the workstation's desktop while pinging the SQL server at the same time and I did not see the time change from <1ms. The application ran slow. Plus the file copied over without any issue.

A brief description of the network is as follows. We have 5 floors with each floor having a wiring closet. In each closet we have a Cisco 3750 cluster of switches. Each floor has fiber running down to the core switch on the 2nd floor.

The reason we suspect re-transmission is because some workstations show a high "segments retransmitted" when you run netstat -s. If I run Wireshark on the suspect workstations what should I be looking for in the capture? Will I capture re-transmission that corresponds to the netstat -s output?

Thx.


Albert

[Albert Jurado]

Thx.



One particular capture shows a count of 1534 suspected retransmissions
(that number seems high). When I look at the details I see [TCP
Retransmission] Close Response. The protocol in question is SMB. The
communication is between a workstation and a SQL server. Not sure what
the close response means. Is this a valid retransmission? Or is there
a preference setting I'm missing?



Albert Jurado

Network Manager

First Commercial Insurance Company

2300 W 84 St.

Hialeah, FL 33016

Phone: [phone] ex. 1206

Mobile: [phone]

Email: ajurado-***@public.gmane.org



From: wireshark-users-bounces-IZ8446WsY0/***@public.gmane.org
[mailto:wireshark-users-bounces-IZ8446WsY0/***@public.gmane.org] On Behalf Of Visser,
Martin
Sent: Thursday, July 03, 2008 12:18 AM
To: Community support list for Wireshark
Subject: Re: [Wireshark-users] Possible network latency



You won't find a "average re-transmission" statement because there
really isn't one. On a well provisioned local area network you could
expect it to be zero. On a heavily congested wireless or WAN network it
might be 20% before it becomes unworkable.



If netstat -s shows retransmission by the client than either the
workstation or the LAN should show retransmissions. (The client will
retransmit either if it receives if receives duplicate ACKs for segments
it has said or if the retransmission timeout (RTO) the TCP stack
calculates has exceeded before it receives an ACK.



On Wireshark if you seclect Analyze:Expert Info Composite and Notes tab
you will see a TCP retransmission count if you are getting any. You can
also just apply the Display Filter "tcp.analysis.retransmission" to see
all relevant packets.



If you are getting a lot of retransmissions where client and server are
on a local switched LAN you may want to look at the physical error
counts on your switches - and look for physical layer issues.



Regards, Martin

Martin Visser

Technology Consultant
Technology Solutions Group


Hewlett-Packard
410 Concord Road
Rhodes NSW 2138
Australia

Mobile: [phone]
Fax: [phone]
E-mail: martin.visserAThp.com

This email (including any attachments) is intended only for the use of
the individual or entity named above and may contain information that is
confidential, proprietary or privileged. If you are not the intended
recipient, please notify HP immediately by return email and then delete
the email, destroy any printed copy and do not disclose or use the
information in it.







________________________________

From: wireshark-users-bounces-IZ8446WsY0/***@public.gmane.org
[mailto:wireshark-users-bounces-IZ8446WsY0/***@public.gmane.org] On Behalf Of Albert
Jurado
Sent: Thursday, 3 July 2008 12:40 AM
To: wireshark-users-IZ8446WsY0/***@public.gmane.org
Subject: [Wireshark-users] Possible network latency

I'm in the mist of troubleshooting possible high network
re-transmissions. I'm basically attempting to capture enough data to
prove that the network is not the bottle neck. I have complaints from
user that their systems are slow but it seems that the application they
are using is the bottleneck. We have several in-house developed
applications that the end users uses that communicates with a SQL
server. They also browse the internet frequently. I've been looking
for articles that describe what the average re-transmission rate is for
a standard TCP/IP networked workstation but I could not find any. I've
attempted a simple test like running the trouble application and then
performing a simple copy & paste (of a 1gb file) from a file server to
the workstation's desktop while pinging the SQL server at the same time
and I did not see the time change from <1ms. The application ran slow.
Plus the file copied over without any issue.



A brief description of the network is as follows. We have 5 floors with
each floor having a wiring closet. In each closet we have a Cisco 3750
cluster of switches. Each floor has fiber running down to the core
switch on the 2nd floor.



The reason we suspect re-transmission is because some workstations show
a high "segments retransmitted" when you run netstat -s. If I run
Wireshark on the suspect workstations what should I be looking for in
the capture? Will I capture re-transmission that corresponds to the
netstat -s output?



Thx.





Albert

[Visser, Martin]

Albert,

The information field value "[TCP Retransmission] Close Response" is populated by information from two different sources. "Close Response" is simply the acknowledgement from your SMB server to a corresponding "Close Request" - thus is is really at the application layer. If you open up the Details for "Close Response" and SMB:SMB Header you should see a highlighted entry "Response to: nnnnn" where nnnn is the frame number for the Request. The client has closed the connection because it no longer requires it, so this is all normal.

The "TCP Retransmission" is how Wireshark interprets your packet trace. Because of TCP segments arriving out of order (probably because of network errors or congestion) the server has had to send this particular more than once.

Thus the two parts of the information field are not related. If TCP Retransmissions occur on basically random parts of the TCP stream, then you will need to look at what lower layer issues (congestion/packet loss) that you might have. The best way to do this methodically is to "stress" the network, generating traffic that isn't application dependant. You want to look at tools such as NLANR Iperf or the like to this. You will want to see what "goodput" you are getting and while you are testing watch the network switch/router stats for where packets are being lost.

Hope this helps, Martin


Martin Visser

Technology Consultant
Technology Solutions Group
HP Australia

410 Concord Road
Rhodes NSW 2138
Australia

Mobile: [phone]
Fax: [phone]
E-mail: martin.visserAThp.com

This email (including any attachments) is intended only for the use of the individual or entity named above and may contain information that is confidential, proprietary or privileged. If you are not the intended recipient, please notify HP immediately by return email and then delete the email, destroy any printed copy and do not disclose or use the information in it.





________________________________
From: wireshark-users-bounces-IZ8446WsY0/***@public.gmane.org [mailto:wireshark-users-***@wireshark.org] On Behalf Of Albert Jurado
Sent: Thursday, 3 July 2008 11:54 PM
To: Community support list for Wireshark
Subject: Re: [Wireshark-users] Possible network latency

Thx.

One particular capture shows a count of 1534 suspected retransmissions (that number seems high). When I look at the details I see [TCP Retransmission] Close Response. The protocol in question is SMB. The communication is between a workstation and a SQL server. Not sure what the close response means. Is this a valid retransmission? Or is there a preference setting I'm missing?

Albert Jurado
Network Manager
First Commercial Insurance Company
2300 W 84 St.
Hialeah, FL 33016
Phone: [phone] ex. 1206
Mobile: [phone]
Email: ajurado-***@public.gmane.org

From: wireshark-users-bounces-IZ8446WsY0/***@public.gmane.org [mailto:wireshark-users-***@wireshark.org] On Behalf Of Visser, Martin
Sent: Thursday, July 03, 2008 12:18 AM
To: Community support list for Wireshark
Subject: Re: [Wireshark-users] Possible network latency

You won't find a "average re-transmission" statement because there really isn't one. On a well provisioned local area network you could expect it to be zero. On a heavily congested wireless or WAN network it might be 20% before it becomes unworkable.

If netstat -s shows retransmission by the client than either the workstation or the LAN should show retransmissions. (The client will retransmit either if it receives if receives duplicate ACKs for segments it has said or if the retransmission timeout (RTO) the TCP stack calculates has exceeded before it receives an ACK.

On Wireshark if you seclect Analyze:Expert Info Composite and Notes tab you will see a TCP retransmission count if you are getting any. You can also just apply the Display Filter "tcp.analysis.retransmission" to see all relevant packets.

If you are getting a lot of retransmissions where client and server are on a local switched LAN you may want to look at the physical error counts on your switches - and look for physical layer issues.

Regards, Martin

Martin Visser

Technology Consultant
Technology Solutions Group

Hewlett-Packard
410 Concord Road
Rhodes NSW 2138
Australia

Mobile: [phone]
Fax: [phone]
E-mail: martin.visserAThp.com

This email (including any attachments) is intended only for the use of the individual or entity named above and may contain information that is confidential, proprietary or privileged. If you are not the intended recipient, please notify HP immediately by return email and then delete the email, destroy any printed copy and do not disclose or use the information in it.




________________________________
From: wireshark-users-bounces-IZ8446WsY0/***@public.gmane.org [mailto:wireshark-users-***@wireshark.org] On Behalf Of Albert Jurado
Sent: Thursday, 3 July 2008 12:40 AM
To: wireshark-users-IZ8446WsY0/***@public.gmane.org
Subject: [Wireshark-users] Possible network latency
I'm in the mist of troubleshooting possible high network re-transmissions. I'm basically attempting to capture enough data to prove that the network is not the bottle neck. I have complaints from user that their systems are slow but it seems that the application they are using is the bottleneck. We have several in-house developed applications that the end users uses that communicates with a SQL server. They also browse the internet frequently. I've been looking for articles that describe what the average re-transmission rate is for a standard TCP/IP networked workstation but I could not find any. I've attempted a simple test like running the trouble application and then performing a simple copy & paste (of a 1gb file) from a file server to the workstation's desktop while pinging the SQL server at the same time and I did not see the time change from <1ms. The application ran slow. Plus the file copied over without any issue.

A brief description of the network is as follows. We have 5 floors with each floor having a wiring closet. In each closet we have a Cisco 3750 cluster of switches. Each floor has fiber running down to the core switch on the 2nd floor.

The reason we suspect re-transmission is because some workstations show a high "segments retransmitted" when you run netstat -s. If I run Wireshark on the suspect workstations what should I be looking for in the capture? Will I capture re-transmission that corresponds to the netstat -s output?

Thx.


Albert