Hi Folks! This is my first post to the forum, and I couldn't find a detailed analysis for my issue. I've worked in the network forensics analysis field for awhile at a major financial company using enterprise tools, but never really dug into all the intricacies of Wireshark - until I changed jobs and my new employer didn't have any tools! Now I'm working on my Wireshark cert, and taking opportunities to look at any/all traffic of interest. As part of this effort, I bought a cheap aggregated 100 Mb TAP for use at home - and didn't install it. I knew that I should be watching my own traffic, but just didn't have time or motivation at the time to get started. I recently changed ISP's from AT&T to Charter for various good reasons - when suddenly the need for the TAP presented itself. I hadn't anticipated probs and at first the 100 Mb BW was awesome. However, weirdness began after about 1 month of service. I had sent AT&T's 2WIRE router back to them, and moved my personal Buffalo AirStation into position connecting to Charter's Cisco cable modem with the rest of my home network behind it. I didn't optimally configure the internet link on the router, so I wasn't very surprised when I first experienced lost connectivity. I figured that Charter had changed DNS or DHCP servers, the default gateway, or some combination thereof. I'd heard of that before. I properly reconfigured my router for DHCP on the internet link, release/renewed and obtained a new Charter IP. Service was restored, problem solved (for the moment). A week later I had another disconnect. When I removed the router and directly connected my old XP workstation used to originally test the link, service was again restored. I reconnected the router and rebooted it, and was online again, for another few days. Then the disconnect started happening every day or two. At this point, my wife was flaming me to fix it (must be my fault) - and I wanted some background info before calling Charter. I knew that they were going to blame my router, suggest I should pay for theirs instead, and it was going to be a long and painful support call. So I inserted the TAP between the cable modem and router, installed a USB Ethernet adapter on my XP PC for TAP data collection, turned off the TCP stack, and configured dumpcap to collect a large ring buffer of files with snaplen set to 92. The XP's primary interface was still connected inside the router, and I downloaded a ping plotter and set it to ping Charter's gateway for my current subnet. I was ready for the next outage. Within a couple of days, I had another incident. This time I download the router's system log before rebooting it, checked the ping plotter to see exactly when the disconnect happened, and grabbed the corresponding PCAP to keep it from being overwritten. The ISP link continues to drop, so now I have several instances recorded. It usually happens between 4:30-5:30 AM. There is no indication of a problem in the router's logs. While the connection is OK and when initiated by the router, a DHCP request resolves successfully. The XP workstation connected directly to the cable modem never has a problem reconnecting via DHCP that needs intervention. The ping plotter provides a log for attempts to ping the gateway, every 10-15 sec. These average 10 ms almost all night until the failure occurs. The traces shown lots of Charter network activity, but nothing that would indicate a specific type of initiated conversation behavior between any Charter system and my router/workstation. The only thing you see just before the pings stop is lots of ARP (but NO USER) activity. After the gateway ping fails, the router starts broadcasting ARP to see where the gateway is, and it continues without ever reconnecting on it's own. There appears to be some other attempted connections/browser activity from the router IP that show as retrans during this time. DHCP requests are directed to the wrong server or don't answer. I haven't ruled out possible malware on my network, but even if present - I don't think it's contributing to this problem. Charter's not telling me that they're booting me based on my outbound traffic. You have to physically power cycle both the modem and router to re-establish link. Just releasing/renewing the IP on the router won't work. I'm patched to the latest software version on my router. Right now the only solution I have is to give my wife a wireless light switch that has the cable modem and router plugged in, and when the internet doesn't work turn it off and then back on. I had to google a phone support number, but after that opened a support case with Charter, got an English speaking tech on the phone within 5 mins, and started hoping that maybe it wouldn't be so bad. In fact, he determined in less than 5 minutes that if the directly connected PC worked fine, then it HAD TO BE my router and he was done. Sigh. I have anonymized my capture with TraceWrangler (cool tool!), and can provide it for analysis. I'm hoping that I have enough info to prove or at least accurately infer what Charter is doing at 5 AM that causes this. I'll probably never get them to admit any fault and unless Buffalo has a suggestion, I most likely will have to continue with the remote power cycling. However, I'm hoping that we can discuss enough detail here to at least shame them publicly and confirm what everybody already suspects - it's not always the user's router. Maybe we can even diagnose the activity to determine if it's something they can do to avoid causing the condition (a change in their procedures), and make the world better for Charter customers. Unless they're TRYING to make you use their router... Is this the place to out an ISP? Looking forward to your replies! asked 09 Jun '15, 21:20  Lucidcryotank edited 13 Jun '15, 18:45 |
3 Answers:
Kurt, WARNING (I guess that's fair, but unnecessary if you have good people skills and kinda irritating don't you think?) - you may not like what you read here. I'm going to call this my answer to my own question and close it out. You must get a lot of exercise jumping to conclusions. I didn't say I was going to sue Charter, I'm not that naive. I've been insulted by smarter guys than you before about lots of different network issues, and they were wrong too. Everybody's entitled to their opinion. I think I did prove something here - these problems are NOT the fault of my router or network behind it, and they DO originate from Charter's network. As Laura Chappell or Gerald Coombs would tell you - packets don't lie. HOWEVER, they don't always tell ALL of the truth, depending on where and how they were collected, and when taken out of context. From the packet data I was able to collect from my TAP, I don't think that any of my statements can be ruled out. I never really expected to fix this problem, I just wanted more eyes on the traces to make sure I wasn't missing anything, get used to the forum, play with anonymizing data, use CloudShark, and leave some food for thought about Charter along with solid packet details (my trace) - mission accomplished! So even though I can't prove any of my accusations against Charter, I'm also not the first person to make these same claims. In this case if it's NOT MY ROUTER, then how many other Charter customers are experiencing the same problems - and it's not their routers either? What Charter is doing is WRONG whether by stupid negligence or intent, and more people should know. I won't be taking your recommendation, because that's also what Charter wants and it goes against my moral standards. I'll just keep power cycling for now, wait until AT&T catches up to and matches the price and bandwidth that Charter currently offers, and switch back when they do. Notice I used some all caps words, but stopped short of bolding things and including little winks. Enjoy! answered 17 Jun '15, 21:00 Lucidcryotank edited 17 Jun '15, 21:02 1
I guess that's the best solution for you to be able to keep up your moral standards and it's also a pragmatical solution, which I like. As there is nothing I can contribute to solve your problem in a way that's acceptable for you, I wish you all the best with your further endeavor. BTW: I'll accept your answer as the correct one, as you suggested in your 2nd sentence. (17 Jun '15, 23:18) Kurt Knochner ♦ Read the comment thread immediately below this answer to see additional information concerning Charter's efforts to resolve the issue after I filed a complaint with the FCC! (24 Jun '15, 16:22) Lucidcryotank A Charter engineering manager called me tonight, I sent him a couple of PCaps, they're going to review with their team. (25 Jun '15, 22:20) Lucidcryotank I'm going to build on this a little with some new info and call it closed. Charter wanted me to remove my router and just have a single workstation connected to see when it would happen again. Unsatisfactory solution for a house with 3 connected adults on phones, laptops, the TV, game consoles, and used to streaming Netflix at night. They didn't like my answer, so they notified the FCC that I was refusing to help them tshoot further. HOWEVER... just before that happened, I asked the engineer working on the problem if he thought it might be weather related. I asked him twice and he never answered. So now about 100 yards down my street Charter has their big orange coax looking just like a donkey's tail. The junction boxes are open, and multiple long runs of cable now loop in a daisy chain between the junction boxes. Cabling runs out into the street and lays by the curb until the next junction box, spanning about 8 houses. They are bypassing the existing infrastructure, patching around it with temporary cabling. It looks like a layer one problem to me. Regardless, since they did that a week or so ago, I don't think my cable modem and router have locked one time. So I may not absolutely prove that Charter was at fault, or that my router is at fault, but the probs have practically disappeared with the new cabling. I guess we just have to draw our own conclusions here. (24 Jul '15, 18:20) Lucidcryotank and yeah, putting the cable modem and router both on a wireless remote switch certainly made the frequently recurring problem easier and quicker to resolve by anybody in the household without my intervention. (24 Jul '15, 18:23) Lucidcryotank |
A few comments: ALWAYS take a systematic approach to troubleshooting packet captures, and start with what the user expects to be able to do. Here the troubleshooting seems a bit all over the place. Not only is it wrong technically to look at DNS as a cause for a failed ping to an IP address (for example), but when you go down that rabbit hole you make a simple problem more complicated needlessly. Trust me, you can spend hours chasing down red herrings if you just start by opening a capture file and looking around at things like this. From the trace file, assuming your topology info is correct and your router not being in the same network as your IP gateway is just a result of packet sanitization, we can say for sure that during the service interruption your router could not ping its IP gateway. THAT is something you can potentially take to your ISP without speculating on a cause, and it's a symptom their first-line support should understand. Further, if you see it from the tap leaving your router, it puts all your thoughts on 'network malware' to bed where you can focus on this single L3 hop without speculating on (uncommon) things you suspect they may be doing beyond their CMTS. As has been said, there isn't enough information in that trace file to conclude anything. A ping request not responded to after it had been successful in packets 1-2 means that this level of connectivity was lost, but from this there's no way to say why, and no 'tail to be pinned', so to speak, from this stage. Having said that, for the comments about ISPs not caring based on the math of one customer's monthly rate versus the support cost of a call, I can't speak for all ISPs but at least for myself I'd never look at it that way. In my experience, if a problem is enough of an outlier to not fit within what initial support lines can troubleshoot, it WILL get to engineering-type support people who would understand what a network tap is fairly quickly (provided it really is an outlier, and not a common problem being over-complicated). Even now where I'm in a design/planning role, and mainly in the the cellular side now, I'll still regularly read through escalated 'one-off' tickets for residential internet service and put an A-game effort in to them because 1) they represent the squeaky wheel of a greater customer base that warrants caring of such wheels and 2) one-off cases reported by individuals can often represent greater problems that few people have reported yet, and I really want to know about those. Many/most network engineering-types feel the same way, and the HFC/Docsis network admins that I know care about every single cable modem. answered 18 Jun '15, 19:05  Quadratic
I was not my intention to generally blame ISPs. It's just my experience, that it often takes very long to find the right people who are able and willing to understand the problem at a technical level. My experience with different ISPs (mostly in Germany) is, that the motivation to look at a problem is directly connected to the service type. If its a highly standardized thing like a Cable link (for home users), that's being sold to millions of customers, your chances to get the right person on the phone is virtually zero. If it's a business type service, it's much more easy, because you'll have the right people to talk to anyway (often dedicated engineers) and they are able to directly look at the routers while you are talking to them. The same is true for small, local ISPs. Apparently, every case (and ISP) is different, but my general perception is certainly right for the german market, and I don't think it's totally different in other countries: If the service is highly standardized, with low margins, the motivation to invest too much time for support cases is pretty low. If they would investigate every case where an end user BELIEVES to have found a problem with the service, they would virtually drown in support tickets. So my whole point is: If a home user of a large ISP, with one of those highly standardized services, believes he can prove that there is something wrong with the service, then I'm absolutely confident that this will lead to nowhere! You simply won't find the right person who is willing to listen to you, especially if you don't follow their recommendation (please use our own router) and thus you've left the "known paths" of their highly standardized service. (19 Jun '15, 06:55) Kurt Knochner ♦ Quadratic, Thank you so much for contributing some helpful comments from an insider's perspective! Note that I never performed any DNS analysis for this event, just made a speculative comment - because there is no DNS traffic to analyze. You are correct about the only thing we can say for sure, during the service interruption my router could not ping its IP gateway OR get an ARP back from it. However, when you take that to Charter, all you get it "It's your router, fix it or try ours" - case closed. First line support did/does NOT understand or appear to care. As for my possible consideration for malware, I speculated that if possible malware exfiltrating from my network were detected by Charter, they may knock me off on purpose to mitigate what they may consider to be suspicious behavior. I don't think that any of the traffic seen leaving my network could be seen as such, but there were some SSL sessions that couldn't really be analyzed without a key, and I didn't capture full packets in order to get more history. This happens intermittently, on a frequency of every 1-4 days. Also, Charter didn't notify me of any such action, but not sure how they'd do that anyway. As far as having evidence and being able to prove that Charter is at fault for anything (accidental or not), I agree that the data from this capture at this location is inconclusive. However, if you read the news yesterday about FCC's lawsuit against AT&T for bandwidth throttling for their unlimited data 4G service, I think it's possible that the FCC could take an interest in this behavior too. I'm curious about what the FCC did to make their case against AT&T, how/what data was obtained, methods of analysis used, etc. All I know for certain is that there were numerous customer complaints. I believe there are engineers at ISP's that do care that would analyze these types of probs if they were aware and not prohibited by some internal policy, but there's no escalation or followup when contacting tier 1 support - just the lame mantra "it's your router, try ours". Therefore, it makes Charter's response seem suspiciously to be by design, especially in light of the data. I think I will attempt to contact the FCC to see if they have an interest in this. Nothing ventured, nothing gained. In this case, I'd just be happy to have 3 tier take this issue seriously, do some tshooting, and fix what appears to be Charter's prob. At this point, Charter has shown no interest in doing this. (19 Jun '15, 11:32) Lucidcryotank For any other ISP customer having similar probs and reading this, if you've exhausted all reasonable efforts and end up like I did, here's a URL of interest: (19 Jun '15, 11:37) Lucidcryotank Not that it will help, but Charters Consumer satisfaction rating is poor even for a cable company, see here and there seems to be a lot of like minded other customers here. (19 Jun '15, 14:41) grahamb ♦ I've submitted a complaint to the FCC, we'll have to see if/how this process works. It should be interesting. Who knows, I may be the only person to offer up a tracefile to them. (19 Jun '15, 19:53) Lucidcryotank Hold on to your hats, folks! After submitting a complaint to the FCC on the 19th, somebody from Charter Corporate called me today. They said they'd received the notice from the FCC, they were terribly sorry for their initial support response, the issue is being escalated in order for me to receive the kind of internet service they promised, and one of their managers is going to follow up with me to resolve the issue. How about that? I'll post updates until the deal is done, and hopefully they'll tell me what they found and fixed - assuming that they do actually find and fix something. Looks like I got their attention after all. (24 Jun '15, 16:21) Lucidcryotank showing 5 of 6 show 1 more comments |
seriously? WARNING:: You may not like the facts in the following text! Continue to read at your onwn risk :-) I don't think that ANY (large) ISP out there is interested in a conversation like this with (small) customers (home users) and you won't get the right people on the phone, unless you are a really big customer. So, what are your options? First of all, they are very, very limited ;-)) Yes, you can try to proof that they are doing something wrong, but this is going to take a looooong time and the chances that they will listen to you are extremely low, because they get hundreds of similar tickets every day, all claiming that the ISP is doing something wrong, and in the end it's 99% of the users doing something wrong. So, you will automatically fall into the 99% for them. I've done such investigations in professional environments with business contracts of several thousand $ per month, and even there the ISPs claimed it's not their fault, regardless the fact that the proof was right there in front of them. So, good luck with that approach as a "small home user" ;-)) However, I don't want to discourage you. So, if you think you can achieve something, go ahead and post the capture file somewhere, so we can have a look. But, don't invest too much time, as your chances that something will change are not that good. And, after all: it could be your router ;-) What you could try:
Then, after the next "disconnect", take a look at the ping results.
As you see, it's not that easy to draw meaningful conclusions from ping tests, if you are capturing only on your side. You would need to convince the ISP to do a packet capture in parallel on their side, which is 100% not going to happen for a home user! Regards answered 10 Jun '15, 03:38  Kurt Knochner ♦ edited 10 Jun '15, 03:40
(10 Jun '15, 04:13) Anders ♦ Kurt, Hey, I'm a dreamer - what can I say? I look at packets and perform analysis at work, then come home and focus my skills and attention on what's become an aggravation with my ISP. I've also used enterprise tools on ISP links to large companies to show that their SLA wasn't being met, that they had issues on their circuits, or that QoS wasn't being marked properly. However, I never paid that much attention on my home service, or tried using Wireshark to see the details. I didn't have a TAP to get between the modem and router, but now I do. How/where do I post a PCAP of my data for collaboration? How do I determine the cable modem's IP? I don't think it has one, as it seems like my Charter assigned IP's next hop is directly to the subnet gateway. A traceroute to the default gateway for my Charter subnet from the inside workstation shows that it's the very next hop after my LAN gateway, with nothing in between. I've been pinging the default gateway for the DHCP subnet sourced from my workstation on the LAN side of my router, but in the capture the ping appears sourced from the Charter IP that my router obtains via DHCP on the internet interface. I also see traffic to several other Charter systems, inlcuding the DHCP servers. When my ping to Charter's gateway fails, I see the ICMP request with no reply. If you can't ping the gateway, you can't get out to go anywhere on the internet. Anders, What you suggested is what I suspect, but I don't understand why a PC would renegotiate successfully after the test and connect, wherease my router won't - as it's fully configured for DHCP to receive DNS and default gateway from Charter, and does so successfully AFTER I hard boot both the cable modem and router. While it's down before the reboot, I can see my router arping for the gateway that it last knew, but an arp response is never received. This happens whether the subnet changes or stays the same after a power cycle and reconnect. It's like whatever they do breaks the arp process and/or DHCP negotiation from my router through the cable modem to Charter's edge. I don't know why Charter isn't responding to my router's arp to find the previously working (or any other) gateway, or sending out DHCP offers after whatever it is they do. I'v cloned the MAC from my originally connected workstation, and I don't understand why Charter's systems would discriminate against me until both a cable modem and router reboot. I'm trying to see if any of the captured data analyzed in Wireshark can provide any clues, but I'm not sure what else to look at that may be significant - and I understand that the actual problem may not leave any evidence in the PCAP data. I'm really just checking to see. (10 Jun '15, 22:29) Lucidcryotank There may be a Charter DNS component for the failure to renegotiate as well, but that isn't apparent to me from the packet data. (10 Jun '15, 22:34) Lucidcryotank
google drive, dropbox, cloudshark.org. Then post the link here. Please limit the capture file to a reasonable size. I'm not going to look at the 100 Mbyte file ;-)
Isn't it doing DHCP? If no, can you please post the type of the modem (Cisco ????) and explain how the router/PC gets an IP address?
Where exactly do you see that? In the capture file between router and modem or anywhere else?
Not necessarily. Think about firewalls not answering pings, but still forwarding frames. But in your case, that assumption is probably correct. Hard to tell, because we don't have enough information about your infrastructure yet. A simple drawing, with IP addresses, MAC addresses, device types, etc. would certainly help.
Maybe because the DHCP request of the PC is different than the one of your router? BTW: Can you please add more details what happens when the PC is connected?
Unfortunately, that's missing in your problem description so far! You may know all this, because you are working on the problem for days/weeks. For us this is all new and unknown. So a much more detailed problem description, probably with a simple drawing, would certainly help to understand what's going on. (11 Jun '15, 02:08) Kurt Knochner ♦
Why would a DNS server problem have any influence on a failed ping to the gateway IP? (11 Jun '15, 02:09) Kurt Knochner ♦ How do you realize the disconnect - the running ping stops, seen in the pingplotter output on the PC. what do you see at that moment in the capture file (ECHO REQ but no answer) - right what do you see at the moment when the connection is re-established (DHCP REQ, what else) - typical DHCP negotiation, which was continuously failing until I rebooted both cable mode and router. Before that, the router just keeps broadcasting ARP to find the gateway, and DHCP requests by the router go unanswered. In the trace, the reboot happens around packet 11391. I think it's possible that there's a DNS issue after getting knocked off (see packet 94 - 05:15:11.277 CST) where the Charter systems can't resolve their own IP's on my original subnet after the event (seems like some sort of resolution problem, at least temporarily), and that's why the expected systems can't respond to an ARP or DHCP request from my router - at least as long as my router still has the old IP, gateway, and DNS configured. That's just a guess, I could be wrong, not sure you can prove that from the packet details. It doesn't make sense that a router internet link release/renew doesn't work without rebooting the cable modem after the event, unless the modem got latched in some sort of confused state - which I think happened on their end instead of mine. I don't know what Charter does whenever they move you to another DHCP server and gateway, or why the PC would recover without rebooting the modem and the router wouldn't. I say the PC recovers. The pings don't seem to go down at all when it's connected alone. I couldn't find a time when the pings failed to respond. I haven't looked very closely at that, I'd need to swap router/PC and live without my router long enough for it for the event to hopefully occur, and then find any evidence of a ping drop - which didn't make an obvious appearance when looking at the pingplotter responses for PC only connection traffic. (13 Jun '15, 14:39) Lucidcryotank I want to know for sure if my router is the problem (what Charter says), or not. Prove them right or wrong based on evidence. Just like you do in a business network. We should be able to do that. I don't want to rent Charter's router, and why should I buy another one (I would if it would solve the issue) if it's just going to keep happening - if it turns out to be due to Charter's activity? I have support through Buffalo for my router, so if it's my router I can maybe get them to create a bugfix - IF it's their problem. (13 Jun '15, 14:50) Lucidcryotank If I were you. I would try to explain the ARP Requests myself. Becsause they are looking at the first few a little bit unexspected to me. (14 Jun '15, 02:28) Christian_R The constant arp was the first thing I noticed as well, but it turns out that this is normal for a DSL network. This is a Cadant system (Charter edge) that's performing some sort of proxy arp function, always trying to identify the endpoints on each gateway. This link provides some background on this behavior. http://www.dslreports.com/forum/r25953464-TWC-Cadant-CMTS-wtf-Hudson-Valley-NY (14 Jun '15, 14:55) Lucidcryotank When I have an outage, the router still thinks it's connected, communicating, and has an outside (Charter) IP address - even though the gateway can't be reached and there's no outbound traffic to the internet. If I release and renew on the router, an IP is never obtained. Rebooting just the router still won't obtain a Charter IP. You must reboot both the router and the modem to restore communication. (14 Jun '15, 14:58) Lucidcryotank So I think you should have a closer look here: So the sender of these ARP is working well. I think to follow these ARPs could bring you closer to the root cause. (14 Jun '15, 16:17) Christian_R And that would be somewhere inside Charter's network. Buffalo isn't going to be much help: Dear Phillip, Thank you for contacting Buffalo Support. This particular issue isn't one that support normally handles under your support plan, which primarily covers setup of the unit in a standard environment. We do have quite a few resources on our website, and you can also take a look at your support plan and see what it covers there. Our community forum is also a great place to ask for help and talk to people that may have experience with advanced setups in a range of environments. Buffalo Support Email is intended for customers in the United States and Canada. Any replies to this response will be handled in the order it was received. Thanks, Sergio Technical Support Associate Buffalo Americas My setup isn't advanced, it's actually pretty typical. I told them about my TAP so they took that as an excuse to quit. (14 Jun '15, 17:46) Lucidcryotank This is part of my question. Is there any way to look at the packet data and infer what problem may exist on Charter's network where the arps don't get answered? (14 Jun '15, 17:50) Lucidcryotank
the missing ping response and the unanswered DHCP REQUEST/DISCOVER messages are an indicator that something is wrong with the Charter network or with the modem. However, it's impossible to tell what could have been failed and where! What I don't understand: why are there ARP requests to 192.168.3.216 from your router (Frame #100 ++)? Did you get an IP address in the range of 192.168.x.x? If not, I would not fully understand those ARP requests. It's continuously sending ARP requests for 192.168.3.216, while its also happily forwarding TCP frames to the MAC address of the gateway (Frame #136 ++). Do you (or Buffalo) have any explanation for that? The other ARP broadcasts you are seeing are "normal" on a cable modem network, meaning you will see them on other sites as well, so nothing to worry about. The "blast" of ARP requests directly after your last successful ping response (Frame #3) could be coincidental or an indicator for a problem. I can't tell.
No way, unless you have PSI skills ;-) Based on the information you have provided, I can tell you, that there is NO clear sign what is going wrong. There is "evidence", that something is wrong, but I'm pretty sure it's by far not enough for Charter to take this seriously enough. BTW: What happens if you reboot only the modem? Does that solve the problem as well?
Seriously, what do you expect from both of them? If Charter is spending more than 5 minutes with support per month with a home user, they will earn 0 $$ in that month with that customer, as the support personnel will cost more that you are paying. The same holds true for Buffalo. You are paying maybe 50 $ for their router, which means their will to invest hours of compley troubleshooting is to near to zero. It might be frustrating for you, but it's a fact. My advice: Ask Charter to send you their router. If the problem still persists, you can blame them until they have fixed the problem and then they have no excuse to refuse support ;-) Regards (14 Jun '15, 23:27) Kurt Knochner ♦ why are there ARP requests to 192.168.3.216 from your router (Frame #100 ++)? These are ALL broadcasts from my router trying to find the subnet gateway that it used to talk to. Before I anonymized the data, and before the event when the gateway pings failed (earlier in the original data, trimmed out), there was a specific host IP on a different subnet from anything else seen. Charter DNS servers couldn't resolve the IP to a name. It is the source of the ARP requests to many different subnets. In another trace this resolved at layer 2 to a Cadant DSL system, Charter's edge, arp proxy that appears to be tasked with checking to see who all of the live systems are in whatever broadcast domain this is for multiple, non-contiguous IP subnets. In every case it was ARP'ing to get specific network IP's to talk back to their gateways. The first 3 packets from 100-102 are my router arping to that specific IP/Cadant system. The rest are simply to the subnet as the first 3 tries to the Cadant system failed. I'm not sure why my router tried the Cadant first, unless that's the last system that told my router to report it's IP to the subnet gateway. Did you get an IP address in the range of 192.168.x.x? no. If not, I would not fully understand those ARP requests. It's continuously sending ARP requests for 192.168.3.216, while its also happily forwarding TCP frames to the MAC address of the gateway (Frame #136 ++). Do you (or Buffalo) have any explanation for that? It's not forwarding frames to the last known gateway MAC. The first TCP frames are retrans from my router to yv-in-f188.1e100.net, identitifed as hpvroom traffic. They are PSH ACK's, so that means they're supposed to be expedited. I have no idea what traffic this traffic is - sourced from either inside my network or from my FW. Maybe my cel phone, checking something in the background. It appears to part of or similar to some earlier communication. The original packet is #134. It seems that these retrans never reach their destination. This is true for all other traffic sourced from my router's IP in the trace - you see some first packets of traffic destined for Amazon or Google, then retrans. (16 Jun '15, 19:46) Lucidcryotank I'll try just rebooting the cable modem next time, but I don't think my router will release and renew until the lease is up - so I'd be down until then unless I managed the router to do that. If I have to grab a device and logon to the router to do that, it's just become easier to put both the router and cable modem on the same wireless remote light switch and use that to power cycle them both at the same time from upstairs. At least that's what I have setup for my wife when she's home and I'm not, and the internet is down. It's also easier and cheaper to do this than pay Charter extra each month to insult my router, network, and me for the privilege of stopping outages that are simply caused because I'm not running their router. I'm not connecting a business or hosting a webserver, etc. - so while outages are aggravating, fixing it isn't worth renting Charter's router to me. I do believe that Charter may be intentionally using some methodology to determine that I have a router, it's not theirs, and they're trying to coerce their customers into renting Charter gear instead of using their own router by configuring and managing their systems WITH FULL KNOWLEDGE/DESIGN FOR THE EFFECT, to knock folks off in this fashion. If you're connected using a single PC and nothing appears to be routed from behind it based on some analysis they perform on your traffic, they leave you alone. I also disagree that there's nothing that can be determined from these traces. I think it's obvious that the problem ISN'T on my router, that it's activity on Charter's network, and therefore something Charter is either accidentally or intentionally doing (the latter is my opinion) that causes it. (16 Jun '15, 20:03) Lucidcryotank I think that Charter has done this in such a fashion to also purposely mask what they're doing - as if it could be proved, I think it could result in a class action lawsuit against them for upselling you to rent their router in order to consistently access the 100 Mb link for all the devices in your home that you already paid for. (16 Jun '15, 20:08) Lucidcryotank 1
I'm sorry, but I believe you are heading into the totally wrong direction. Are you serious? You are thinking about to start a lawsuit, which will cost you (at least) several thousand $, for a 50 $ cable link? How are you going to prove any of your accusations/assumptions in court? I'm sorry, but I stay with my last recommendation: Ask Charter to send you their router. If the problem still persists, you can blame them until they have fixed the problem and then they have no excuse to refuse support ;-) (17 Jun '15, 04:57) Kurt Knochner ♦ showing 5 of 18 show 13 more comments |
First time I've used Cloudshark, it's very cool. My data's been anonymized, which means I'll have to do some translation so you know what's what if needed. I already had the info you've requested before I obscurred it. I'll provide that soon. I wanted to use the anonymizer for experience, and think it's not smart to post all of your particulars on the internet. Look for the ICMP source near the beginning, that's from the IP that Charter assigned to my router - and the destination is the gateway that was specified with it. The actual PING source is my PC on the router LAN side.
I wouldn't ask for you to look at a 100 Mb file, and you don't have to look at all if you don't want to. Sorry if this info was clear in the first post, I thought it was.
My data: https://www.cloudshark.org/captures/383ad1bef646
I don't have a console or webgui link to the cable modem, so I really don't know what it's doing. I thought is was simply a layer two conversion for the RG6 coax media, and that all of these endpoints were located directly on the subnet with Charter's gateways (they change intermittently, along with my outside IP). I believe my router negotiates through the L2 cable modem link over Charter's subnet with their DHCP server, who then provides the default gateway. It's the same for the PC when it's directly connected to the cable modem and I release/renew my Charter IP.
The only capture location is my TAP connected inline between the cable modem RJ45 and my router's Internet interface. The PC has a monitor link to the TAP, and data link to the router's inside LAN. When I connect the PC directly, I simply swap it's data network cable for the router's connection to the TAP. I don't know if my connection goes through a Charter firewall before it hits the DHCP server or gateway first, but I doubt it. The cable modem is a Cisco DPC3008, Docsis 3.0, HW V1.0.
Evidently there's something about the DHCP negotiation using the PC's NIC that is different from the router's, but I believe I have the router configured correctly or an Internet interface release/renew on the router would fail when the link is up and pings to the gateway are working. I'm going to have to start another comment.
Here's a drawing.
In the anonymized data, 199.77.184.213 is my assigned outside IP via DHCP from Charter. The default gateway for the subnet is 192.168.3.216. Anything sourced from .213 is either my router's internet IP, or traffic from my LAN that appears to be sourced from that IP. Everything else not sourced from .213 is Charter intra-system traffic, or is network traffic from the internet or Charter hosts. The MAC f2:18:92:af:8c:69 is Charter's edge device, some sort of Cadant system. MAC f2:03:36:b2:fd:85 belongs to my PC's LAN interface, but is also the MAC that I cloned for use on the router's internet interface.
The timestamps in the CloudShark trace are offset 5 hours later than my actual time and I don't know how to correct in CloudShark, so 10:15:01 is actually 05:15:01.
I have no idea what hpvroom traffic or some of the other traffic is - maybe some background traffic from my phone or PC.