Introduction

Packet loss on an intermediary hop during network troubleshooting can indicate potential issues like congestion, rate limiting, or hardware problems. Using the MTR (My Traceroute) tool, you can identify and analyze packet loss to determine whether it’s a genuine issue affecting connectivity or a benign rate-limiting mechanism that doesn’t impact the final destination. This article explains how to read MTR output, troubleshoot packet loss on intermediary hops, and emphasizes the importance of bidirectional testing and numeric output to fully understand the network path.

Understanding MTR Output

MTR combines ping and traceroute to display real-time network performance data for each hop along the path to a destination. The output includes:

• Host: IP address or hostname of each hop.

• Loss%: Percentage of packets lost at each hop.

• Snt: Number of packets sent.

• Last: Latency of the last packet (in milliseconds).

• Avg: Average latency across all packets.

• Best: Lowest latency recorded.

• Wrst: Highest latency recorded.

• StDev: Standard deviation of latency, indicating consistency.

When troubleshooting packet loss, focus on the Loss% column to identify where packets are being dropped and whether the loss persists to the destination.

Importance of Numeric Output

Using numeric output (IP addresses) instead of DNS-resolved hostnames is critical for network engineers. MTR’s default behavior attempts to resolve each hop’s IP address to a hostname using PTR (reverse DNS) records. However, PTR records may be missing, outdated, or misleading, which can obscure the network path. Numeric output, enabled with the -n or --no-dns option, ensures the following:

• Accurate Path Visibility: Displays raw IP addresses, revealing the exact routers involved, even if PTR records don’t resolve or provide vague names.

• Troubleshooting DNS Issues: If DNS resolution fails or returns incorrect names, numeric output avoids confusion and ensures engineers see the true network path.

• Consistency: IP addresses are less likely to change mid-troubleshooting compared to hostnames, which may depend on DNS server reliability.

• Clarity for Escalation: When sharing MTR results with ISPs or network administrators, numeric output provides unambiguous data, making it easier to pinpoint problematic hops.

Example command for numeric output:

mtr -n -r -c 100 [destination]

This ensures all hops are shown as IP addresses, aiding precise analysis.

Identifying Packet Loss on Intermediary Hops

Packet loss on an intermediary hop (a router between your system and the destination) may not always indicate a problem. Here’s how to interpret it:

1. Check Loss% at Each Hop:

   • Run MTR with sufficient cycles and numeric output for reliable data:

     mtr -n -r -c 100 [destination]

   • Look for hops with non-zero packet loss (e.g., 5–20% or higher).

2. Does Loss Persist to the Destination?:

   • If packet loss occurs at an intermediary hop (e.g., hop 3) but not at subsequent hops or the destination, it’s likely due to rate limiting. Many routers prioritize traffic and limit ICMP (ping) responses, causing apparent packet loss in MTR without affecting actual data transfer.

   • If packet loss continues through to the destination (e.g., loss at hop 3 and all subsequent hops, including the target), it indicates a genuine issue, such as network congestion, a faulty router, or a misconfigured firewall.

3. Example Output (Numeric):

   Host                    Loss%   Snt   Last   Avg   Best  Wrst  StDev
   1. 192.168.1.1         0.0%   100    1.2   1.3   1.0   2.0   0.3
   2. 10.0.0.1            0.0%   100    2.5   2.7   2.0   3.5   0.4
   3. 172.16.0.1         10.0%   100   10.1  10.5   9.8  12.0   0.6
   4. 203.0.113.1         0.0%   100   15.3  15.7  14.5  17.2   0.8
   5. 142.250.190.14      0.0%   100   20.1  20.4  19.5  22.0   0.7

   In this example, hop 3 shows 10% packet loss, but subsequent hops and the destination (142.250.190.14) show 0% loss. This suggests rate limiting at hop 3, not a connectivity issue. The numeric output ensures the IP addresses are clear, even if PTR records are unavailable.

Is It Rate Limiting or a Problem?

To determine if packet loss on an intermediary hop is due to rate limiting:

• Verify Downstream Hops: If packet loss stops after the affected hop and the destination shows 0% loss, the issue is likely rate limiting. Routers often deprioritize ICMP traffic to focus on actual data, causing MTR to report loss.

• Test with Different Protocols: ICMP-based MTR may show loss due to firewall restrictions. Try TCP or UDP-based MTR to test the same path:

  mtr -n --tcp -r -c 100 [destination]
  mtr -n --udp -r -c 100 [destination]

  If TCP/UDP shows no loss, ICMP rate limiting is likely the cause.

• Check Latency Trends: If latency (Avg, Last) increases significantly at the hop with packet loss and persists, it may indicate congestion or a failing router rather than rate limiting.

Importance of Bidirectional Testing

Packet loss on an intermediary hop in one direction doesn’t tell the full story, as network paths are often asymmetric (the return path may differ). To ensure a complete understanding:

1. Run MTR from Both Ends:

   • From your system to the destination, using numeric output:

     mtr -n -r -c 100 [destination]

   • From the destination back to your system (if possible). This requires access to the destination server or a cooperative remote host running MTR toward your public IP.

2. Why Bidirectional?:

   • Packet loss in one direction may not affect the return path. For example, a router may rate-limit ICMP in the outbound direction but not inbound.

   • Bidirectional results help identify if the issue is specific to one path or affects both directions, which is critical for diagnosing problems like routing loops or asymmetric congestion.

   • Numeric output ensures both directions show clear IP addresses, avoiding confusion from unresolved or misleading PTR names.

3. How to Perform Bidirectional Testing:

   • If you control the destination server, run MTR from that server back to your public IP with -n:

     mtr -n -r -c 100 [your-public-ip]

   • If you don’t have access, ask the destination’s network administrator to run MTR toward your IP or use a public looking-glass server (e.g., lg.he.net) to simulate the return path.

   • Compare both MTR reports to identify where packet loss occurs and whether it persists to the endpoint.

Troubleshooting Steps

1. Confirm Packet Loss:

   • Run MTR with numeric output and sufficient cycles:

     mtr -n -r -c 100 [destination] > mtr-outbound.txt

   • Check if loss is isolated to an intermediary hop or persists to the destination.

2. Test for Rate Limiting:

   • Run MTR with TCP or UDP to bypass ICMP restrictions, using numeric output:

     mtr -n --tcp -r -c 100 [destination]

   • If loss disappears, the issue is likely ICMP rate limiting.

3. Perform Bidirectional Testing:

   • Run MTR from the destination to your IP (if possible) with -n and compare results:

     mtr -n -r -c 100 [your-public-ip] > mtr-inbound.txt

   • Numeric output ensures both reports show clear IP addresses for accurate comparison.

4. Analyze Latency and Jitter:

   • High StDev or Wrst values at the lossy hop may indicate congestion, even if it’s rate-limited.

   • Compare latency trends in both directions to pinpoint bottlenecks.

5. Escalate if Necessary:

   • If packet loss persists to the destination in either direction, contact your ISP or the network administrator of the affected hop.

   • Share both MTR reports (outbound and inbound) with numeric output to provide a complete and unambiguous picture.

Best Practices

• Use Numeric Output: Always use -n to ensure IP addresses are displayed, avoiding issues with unresolved or misleading PTR records.

• Use Sufficient Cycles: Run MTR with at least 30–100 cycles (-c 30 or -c 100) to ensure statistically significant results.

• Test Multiple Protocols: Use --tcp or --udp to confirm if loss is ICMP-specific.

• Check During Different Times: Network conditions vary, so test during peak and off-peak hours to identify intermittent issues.

• Save and Share Results: Export MTR reports for documentation:

  mtr -n -r -c 100 [destination] > mtr-report.txt

• Combine with Other Tools: Use ping, traceroute, or nslookup to complement MTR findings.

Common Scenarios and Solutions

• Loss Only on Intermediary Hop:

  • Likely rate limiting. Verify with --tcp or --udp and -n. If the destination is unaffected, no action is needed unless you’re troubleshooting ICMP-specific services.

• Loss Persists to Destination:

  • Indicates a real issue. Check bidirectional MTR results with numeric output to identify the problematic hop and escalate to the relevant ISP or network admin.

• High Latency with Loss:

  • Suggests congestion or hardware issues. Test bidirectionally with -n and report to the network provider with MTR data.

• No Response (???) at a Hop:

  • Could be a firewall blocking ICMP. Use --tcp or --udp with -n to test further.

Conclusion

Packet loss on an intermediary hop in MTR output is often due to rate limiting, especially if it doesn’t affect the destination. By using numeric output with -n, testing with different protocols, and performing bidirectional testing, you can accurately distinguish between benign rate limiting and genuine network issues. Numeric output ensures network engineers see the full, unambiguous path, even if PTR records are missing or misleading. Always gather bidirectional MTR results with numeric output when possible to understand the complete network path and ensure effective troubleshooting. If issues persist, share detailed MTR reports with your ISP or network administrator for further investigation.