This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Why Do I Have Video Stutter Through Netgear MCAB1001 MoCA Adapters–Possibly Related to TCP Window

0

I use a pair of Netgear MCAB1001 MoCA (multimedia over coax cable) adapters for my home theater setup. The connections are: PC-->Cat6-->WNDR3700 router-->Cat6-->MoCA-->coax cable-->MoCA-->Cat6-->bluray player. LAN Speed Test (www.totusoft.com) indicates I have a sustained read/write of 84Mbps over the adapters on large files. Ping latency from my computer to the bluray player is less than 1ms. However, I cannot stream video that is over 16Mbps (with regular surges to 25-30Mbps in high-detail scenes) without intolerable stutter. Even though the MoCA adapters are only supposed to operate on Layer 2 (I think), I believe I have narrowed the problem down to some type of TCP Window problem or packet alteration. Here are the troubleshooting steps I've taken so far:

  1. Verified there are no other equipment problems or incompatibilities. Streamed the 16Mbps video directly to my LG BD670 and LG BD570 bluray players using 3 different sources (2 Win7 x64 PCs and a Netgear NAS)--no errors. Streamed using CIFS and DLNA--no errors. Played videos from a USB drive attached to the bluray player--no errors. Swapped out all Cat6 ethernet cables--no errors. Streamed other high bitrate videos to confirm that I don't have CODEC incompatibilities. Swapped the MoCA adapters with two separate adapters. Connected the MoCA adapters directly together to eliminate house coax cable problems. Changed the communication frequencies on the MoCA adapters. Essentially, all high bitrate videos play flawlessly in all configurations except when they are streamed through the MoCA adapters.
  2. Changed the setup to: PC-->WNDR3700 router-->MoCA-->coax-->MoCA-->PC. Then used Jperf to check throughput. When the TCP Window is left at the default 8K, I only obtain about 20Mbps throughput, which--I think--is why the video stutters when it surges to 30Mbps during high-detail portions. However, if I start increasing the TCP window size with Jperf, throughput starts to climb. If I set the TCP Window at 128K, I obtain 90Mbps throughput.
  3. Using TCP Optimizer, I've altered all settings--one at a time--in an attempt to discover the offending setting. I performed the same tests on two separate computers (both Win7 x64) in an attempt to identify one of the PCs as the problem. Stuttering still occurred.
  4. I performed two captures. I started the captures with the bluray player off in order to capture the initial handshake.
  5. Capture 1 streams the video directly from my computer to the bluray player via Cat6 and static IPs. It plays perfectly. I cut out a significant chunk of the middle of the capture in order to meet the upload limits of my Windows SkyDrive. The 140MB capture file is here: Capture 1
  6. Capture 2 streams the video from my computer through the MoCA adapters and then to the bluray player via Cat6 and static IPs. Stuttering begins about 40 seconds into the capture and doesn't quit until it hits a lower bitrate portion of the video (at the very end of the capture). Like Captue 1, I cut a significant portion of the middle out in order to meet my SkyDrive limits. The 115MB capture file is here: Capture 2

(If the files are too large, just tell me what I should capture and I'll make new files).

After testing with Jperf, I'm inclined to believe that I have a TCP Window scaling problem. When I stream the video(s) directly from my computer to the bluray player, TCP Window scaling works correctly and allows the video to play stutter-free. However, when I stream the video through the MoCA adapters, something seems to prohibit the TCP Window from growing to the size it needs. Either that or the MoCA adapters are somehow altering packets.

This is my first attempt at performing a detailed study of ethernet protocol. I'm hungry for knowledge but I just don't know what I'm looking at in this case. Are the MoCA adapters interfering with TCP window scaling? Or, are they somehow altering packets?

Kurt--The captures you requested are linked below.
Capture 3
192.168.1.1--Static IP on laptop (Y530--Fast Ethernet).
192.168.1.2--Static IP on BD670 bluray player (Fast Ethernet)
Config: Y530-->Cat5e-->MoCA-->short coax-->MoCA-->Cat6-->BD670 bluray player
Capture started before BD670 turned on in order to capture handshake.
Start of stutter marked with ping -l 444 192.168.1.2 (packet #115785)
End of stutter marked with ping -l 444 192.168.1.2 (packet #279526)

Capture 4
192.168.1.15--Dynamic IP on laptop (Y530--Fast Ethernet).
192.168.1.9--Dynamic IP on desktop computer (INTEL64--GigE)
Config: Y530-->Cat5e-->MoCA-->house coax-->MoCA-->Cat6-->WNDR3700 router-->Cat6-->INTEL64
Transfer rate indicated by Win7 dialog: 10.6MBps
The video file is 6GB and produced a huge capture file. I captured the first 13 seconds of the transfer (including the laptop handshake with the router). There appeared to be no change in the remaining capture packets.

Kurt, Thanks for your initial analysis. I don't yet have the capability to do multipoint captures. In the meantime, I ran some captures under additional circumstances in the hopes that it might reveal something.

INTEL64: Win7 x64 PC, IP=192.168.1.9
Y530: Win7 x64 PC, IP=192.168.1.15
Handshake captured whenever possible

Capture 5: Y530 to INTEL64 No MoCA No IPV6 JPerf default settings
Config: Y530-->WNDR3700-->INTEL64.
IPV6 disabled in Y530 NIC setup for all remaining captures to remove the possibility that IPV6 is interfering. JPerf default settings, including the TCP Window. I wanted to check to see how JPerf and the TCP Window behaved with no MoCA in the topology. Wireshark statistics show about 90Mbps, nearly maxing out the fast ethernet NIC in the Y530. The TCP Window in the capture expands rapidly and stops at 66560. It is, however, the first time I've seen "Len=xxx" entries other than zero.

Capture 6: Y530 to INTEL64 No MoCA No IPV6 Jperf TCP Window 8K
Config: Y530-->WNDR3700-->INTEL64. Same config as Capture 5 but with a forced starting TCP Window of 8K.
The capture results appear almost identical to Capture 5. Throughput reaches 90Mbps with a TCP Window of 66560. The "Len=932" entries are also present again. This result contradicts with 20Mbps throughput I obtained in this same configuration with the MoCAs in the topology.

Capture 7: Y530 to INTEL64 No MoCA No IPV6 Jperf TCP Window 128K
Config: Y530-->WNDR3700-->INTEL64. Same config as Capture 5 but with a forced starting TCP Window of 128K.
Throughput is erratic but generally about 70Mbps. The TCP Window expands to 66560, and there are multiple "TCP Window Full" entries. Len=1452.

More JPerf captures to come...

I'm hoping you might be able to identify why JPerf behaves one way when the MOCAs are in the topology and a different way when they are not.

The next test I performed was using VLC Media Player on my laptop (Y530) to stream the video from my desktop PC (INTEL64). I'm hoping that by using a player different from the bluray player, we might be able to correlate something.

Capture 8: Y530 to INTEL64 No MoCA No IPV6 VLC Media Player
Config: INTEL64-->WNDR3700-->Y530.
Video file on INTEL64 and played using VLC Media Player on Y530. Statistics graph shows variable bitrate maxing out around 35Mbps. TCP Window=65340. No stutter.

Capture 9: Y530 Through MoCA to INTEL64 No IPV6 VLC Media Player Stutter Range Marked with Pings (Abbreviated)
Config: INTEL64-->WNDR3700-->MoCA-->house coax-->MoCA-->Y530
I played the same video from Capture 8 but with the MoCAs added to the topology. Stutter begin and end is marked with ping -l 444 192.168.1.9. Stutter starts at packet #117542 and ends at #137810. Throughput on the statistics chart takes a noticeable nosedive at the same time. The capture also indicates dropped packets.

Capture 10: Y530 to BD670 No MoCA No IPV6 No Stutter Ping RTT Check
Config: Y530-->Cat6-->BD670 bluray
Per your request, I performed pings while playing the video with no MoCA in the topology. The pings are at #172494, #174473, #178838 and a couple other places. What do you think the RTT says about possible delays?

Capture 11 and Capture 12 are dual sniff captures at each end of the topology that you requested.
INTEL64: 192.168.1.9
Y530: 192.168.1.15
Config: Y530-->MoCA-->house coax-->MoCA-->WNDR3700-->INTEL64. No IPV6
I streamed the video from INTEL64 to VLC Media Player on my laptop (Y530). I captured at both ends and marked the beginning and end of stutter with pings to 192.168.1.9.

asked 27 May '12, 16:30

Bob%20K's gravatar image

Bob K
1●1●2●2
accept rate: 0%

edited 04 Jun '12, 21:22

can you please create a new capture file and

  1. set a marker in the capture file as soon as stutter begins. You can do this by pinging one machine from the other with a certain payload length, like this: ping -l 444 192.168.1.1. Prepare the command at the CLI and then just press enter as soon as it happens.

  2. set another marker, as soon as the stutter stops. Don't cut out that part. Rather limit the capture size at the end. ping -l 445 192.168.1.1

  3. Copy the video file between two Winodws machines (from your share as you did in capture 2) and check the max. throughput (capture that traffic as well)

(28 May '12, 05:02) Kurt Knochner ♦

2 Answers:

0

Take a look specifically at all packets with the SYN flag set, as these will be the ones doing window size and window scaling negotiations. Look for a situation in which the PC or the blueray player tries to enable scaling and the option is rejected. If the rejection comes from the MoCA you have part of the problem. If at the other end in the same time period you see the MoCA trying to negotiate a corresponding scaling and size with its device and being rejected by the PC or blueray, then you have the underlying problem. If not, then it is likely to be a problem with the MoCA implementation. On the other hand, if neither the PC nor the blueray try to set up scaling, then that is the problem.

Even if the ping latency is under 1ms, it could cause the time-bandwidth product to go too large without window scaling. For comparison, when doing the transfer without the MoCAs in the path, confirm that window scaling is being negotiated.

answered 27 May '12, 17:08

inetdog's gravatar image

inetdog
167●1●7
accept rate: 14%

Thanks for the suggestion. I've spent about 6 hours analyzing both captures linked above but see no discrepancies between the two. As far as I can tell, the SYN and ACK entries match correctly. If there are critical differences between the files, I am not currently able to detect them. Any other suggestions?

(28 May '12, 01:05) Bob K

More. On both captures, the TCP Window grows as expected. The maximum TCP Window size on the 'stutter' capture is 121856 and the non-stutter is 143168. There are no indicated errors.

(28 May '12, 01:21) Bob K

0

Here's my analysis.

relevant data:

Capture #1: tcp.stream eq 13
Capture #2: tcp.stream eq 12
Capture #3: tcp.stream eq 12
Capture #4: tcp.stream eq 87

Some Results:

  1. Capture 4 is IPv6 and SMBv2, so it's not really comparable. The reported throughput in Wireshark is ~ 94 Mbit/s (Filter: tcp.stream eq 87, then Statistics -> Summary). I assume the transfer rate shown in the windows 7 (10.6MBps) must be MByte/s instead of Mbit/s. This throughput is consistent with your speed measurements (jperf and totusoft). So, the MoCA are not (necessarily) the bottleneck (HOWEVER see also below - Conclusion).
  2. There is no window size (or window scaling problem) in Capture #1 and #3. If you look at the "TCP Stream Graph -> Window Scaling Graph" (for the related TCP streams), you will see a pretty "nice" picture with a "constant" window size (select the SYN packet, before you draw the graph).
  3. All other TCP Stream Graphs look O.K., except the "Throughput Graph", which shows a pretty variable throughput for the LG player (not the windows copy operation - capture #4), with a "base" around 20 Mbit/s and a peak around 40 Mbit/s (Capture #3). There is a remarkable difference among the TCP Throughput Graph" of capture #1 (no MoCA) and Capture #2 and #3 (MoCA in place and stutter). The "base" of the throughput graph in capture #1 is remarkably higher than in the other two captures. The graph of capture #4 (file copy via MoCA), is very "smooth" at ~ 94 Mbit/s. There must be a reason for the difference among captures #1 (NO MoCA) and #2/#3 (MoCA).
  4. Capture #3: Your LG player requests the data with a throughput of ~ 14.5 MBit/s (just wireshark summary - just the arithmetic average), although the windows size would permit a much higher throughput. I can only guess, but I believe, the player requests the data just "on demand", without much buffering. This is also true, if there is no MoCA involved (Capture #1). That would be O.K. for "streaming".

Facts:

  1. The problem is not related to TCP (windows size/scaling, etc.)
  2. The problem is not related to packet loss (almost no re-transmits)
  3. There is no problem if you download the file from Windows to Windows via the MoCA (94 Mbit/s)
  4. There is no problem if you do speed measurements via the MoCA (jperf, etc. - 90 Mbit/s)
  5. The LG player seems to request data only with the required bit rate, probably without much buffering in advance. O.K. for "streaming".
  6. There is a slightly higher RTT with the MoCA in place. Look at the TCP 3-way handshake in capture #1, #2 and #3. If I assume that the capture was done on the Windows PC hosting the share, the RTT is ~ 1,5-2 ms higher than without the MoCA (Delta SYN-ACK -> ACK). This matches with the RTT of the ping marker in capture #3 (~ 3 ms). The later (icmp) could also be a load problem on the LG player, while it's playing the movie (see TODO below).

Conclusion:

The problem is probably related to the MoCA, because it might add delay while trans-coding the data. Maybe it adds delay only in one direction. You can't tell with only one capture point. Unfortunately I have (yet) no idea why there is no problem in capture #4 (file copy via the MoCA). The suspected delay should have an effect there as well.

TODO:

To verify my delay assumption, you could generate some more captures:

  1. repeat capture #1 and ping the LG player while the movie is beeing played. Then check the RTT of the ping. If it's the same RTT as in capture #3, it's a load problem on the player. If there is no higher RTT, it supports my delay assumption.
  2. repeat capture #2/#3 with two data capture points.
    • One in front of the client (or on the client if there is no other option)
    • one in front of the LG player (use a small switch with port mirroring capabilities, or a PC with 2 interfaces - bridge mode: http://wiki.wireshark.org/CaptureSetup/Ethernet). Important: Synchronize the clocks of the capturing PCs with NTP before capturing.

Possible Solution:

The LG player could compensate by downloading the file with a higher "bit rate" and by using a larger internal buffer, to prevent stutter, if there is not enough data. However, that's apparently not the case, even in capture #1 it requests the data with a much lower bit-rate than it would be possible according to the announced window size. Apparently it downloads just what is needed, which is O.K. for "streaming".

Maybe a firmware update of the LG player could help.

Any other thoughts on this?

BTW: If you think my analysis is plain wrong, please feel free to correct me. I'm willing to extend my knowledge by learning from "weird" real world problems :-)

Regards
Kurt

answered 29 May '12, 04:13

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k●10●39●237
accept rate: 15%

edited 29 May '12, 05:34

Wow. At lot of new captures :-)

I don't yet have the capability to do multipoint captures

At least for Capture #8 and #9 you can sniff on both your laptop and your PC. I'm not sure if that will help to track down the problem, but it's the best thing you can do for now.

(31 May '12, 20:25) Kurt Knochner ♦

Yes, I'm determined to figure this out! Or, at least I will try to provide meaningful data to people who can figure it out!

(31 May '12, 20:31) Bob K

Kurt--A double-sniff capture is now uploaded. I'll be darned if I can detect any problems.

Notably, I get significant packet loss when streaming through the MoCAs to VLC Media Player, which should have enough buffer to make up for any delay problems.

(08 Jun '12, 22:14) Bob K