Read a TCP Three-Way Handshake in Wireshark

The TCP three-way handshake is the foundational process for establishing a reliable connection between a client and a server. Understanding its nuances in…

The TCP three-way handshake is the foundational process for establishing a reliable connection between a client and a server. Understanding its nuances in a packet capture is crucial for diagnosing connectivity issues, application performance bottlenecks, and security posture. This article details how to effectively interpret TCP handshakes using Wireshark, focusing on the packets exchanged and common deviations that indicate problems.

Understanding the Three-Way Handshake Fundamentals

The Transmission Control Protocol (TCP) uses a three-step process to establish a connection before data transfer can begin. This ensures both parties are ready, acknowledge each other's initial sequence numbers, and agree on connection parameters. The steps are:

  1. SYN (Synchronize Sequence Numbers): The client initiates the connection by sending a TCP segment with the SYN flag set to 1. This segment also includes an initial sequence number (ISN) for the client's data stream.
  2. SYN-ACK (Synchronize-Acknowledge): The server receives the SYN, acknowledges it, and responds with a segment where both the SYN and ACK flags are set to 1. This segment contains the server's ISN and an acknowledgment number (client's ISN + 1).
  3. ACK (Acknowledge): The client receives the SYN-ACK from the server and completes the handshake by sending an ACK segment. This segment has the ACK flag set to 1 and an acknowledgment number (server's ISN + 1).

Upon successful completion, a full-duplex connection is established, and application data can be exchanged.

Capturing and Filtering for Handshakes in Wireshark

To analyze handshakes, you first need a packet capture. Launch Wireshark and select the appropriate network interface. Start the capture before attempting the connection you wish to analyze.

Initial Filter for SYN Packets

The most straightforward way to identify connection attempts is to filter for the initial SYN segment. In Wireshark's display filter bar, use:

tcp.flags.syn == 1 and tcp.flags.ack == 0

This filter specifically looks for packets where the SYN flag is set and the ACK flag is not set. This is critical because SYN-ACK packets also have the SYN flag set, but they also have the ACK flag set.

Alternatively, you can use:

tcp.flags == 0x002

This filters for packets where only the SYN flag (0x002 in TCP flag hexadecimal representation) is set. This is often more precise.

Refining the View: Following TCP Stream

Once you identify a SYN packet of interest:

  1. Right-click on the SYN packet in the packet list pane.
  2. Select "Follow" -> "TCP Stream".

Wireshark will then apply a display filter (e.g., tcp.stream eq X) that shows all packets belonging to that specific TCP connection, making it much easier to observe the entire handshake and subsequent data exchange.

Analyzing Handshake Success and Failure Scenarios

Successful Handshake

A successful handshake will show a clear sequence of SYN, SYN-ACK, and ACK packets. Observe the sequence numbers (Seq) and acknowledgment numbers (Ack) in the TCP header for each step:

  • Client (SYN): Seq=X Ack=0 Win=[Window Size] Len=0
  • Server (SYN-ACK): Seq=Y Ack=X+1 Win=[Window Size] Len=0
  • Client (ACK): Seq=X+1 Ack=Y+1 Win=[Window Size] Len=0

The "Len=0" indicates that these control packets do not carry application data. Window size (Win) advertises the receive buffer size, an important aspect for flow control.

Common Handshake Failure Patterns

1. SYN, No SYN-ACK

If you see a SYN packet sent from the client, but no corresponding SYN-ACK from the server, this indicates that the server either did not receive the SYN, or received it but did not respond. This is a common indicator of:

  • Firewall blocking: An intermediate firewall (or the server's host-based firewall) is dropping the SYN packet or blocking the SYN-ACK response.
  • Incorrect IP address or port: The client is trying to connect to a non-existent IP address or a service not listening on the specified port.
  • Network routing issue: The SYN packet is not reaching the server due to routing problems.
  • Server down or service not running: The server is unreachable, or the target service (e.g., HTTP on port 80, SSH on port 22) is not listening.

In Wireshark, you might see retransmissions of the initial SYN packet after a timeout, as the client attempts to establish the connection multiple times. This usually appears as [TCP Retransmission] in the Info column.

2. SYN, then RST

If the client sends a SYN, and the server immediately responds with a Reset (RST) flag set, it means the server received the SYN but refused the connection. This typically happens when:

  • Port is closed: The service the client is trying to reach is not listening on that specific port on the server. The server explicitly rejects the connection.
  • Host-based firewall rejection: A firewall on the server is configured to send RSTs for connection attempts to certain ports, rather than silently dropping them.

A RST packet will have tcp.flags.reset == 1. You might see RST, ACK or simply RST depending on the context.

3. SYN, SYN-ACK, No ACK

This scenario is less common but indicates that the server successfully responded to the client's SYN, but the client either did not receive the SYN-ACK or failed to send the final ACK. Potential causes:

  • Return path asymmetry or blocking: The SYN-ACK from the server is not reaching the client due to routing issues or an intermediate firewall blocking the return traffic.
  • Client-side resource exhaustion: The client might be overwhelmed and unable to process the SYN-ACK or send the final ACK.

In this case, the server will often retransmit the SYN-ACK several times, eventually giving up and resetting the connection itself, or the connection will simply time out on the server side.

4. Repeated SYNs with No Progress

Observing multiple SYN packets sent from the client without any response from the server strongly suggests an upstream issue where the server isn't seeing the packets or its responses are being dropped. As mentioned in the introduction, this often points to a firewall silently dropping the return traffic (the SYN-ACK). It's crucial to check both directions of traffic flow if possible.

Example Wireshark Output for a Blocked Connection

Here's a simplified example of what you might see for a SYN, no SYN-ACK scenario where a firewall blocks the connection to a target server (192.168.1.100 on port 80):

No.     Time        Source          Destination     Protocol Length Info
1       0.000000    192.168.1.10    192.168.1.100   TCP      74     50000 > 80 [SYN] Seq=0 Win=65535 Len=0
2       1.000000    192.168.1.10    192.168.1.100   TCP      74     [TCP Retransmission] 50000 > 80 [SYN] Seq=0 Win=65535 Len=0
3       3.000000    192.168.1.10    192.168.1.100   TCP      74     [TCP Retransmission] 50000 > 80 [SYN] Seq=0 Win=65535 Len=0
... (eventually client gives up)

If the server sent a RST because the port was closed, it would look like this instead:

No.     Time        Source          Destination     Protocol Length Info
1       0.000000    192.168.1.10    192.168.1.100   TCP      74     50000 > 80 [SYN] Seq=0 Win=65535 Len=0
2       0.000030    192.168.1.100   192.168.1.10    TCP      60     80 > 50000 [RST, ACK] Seq=0 Ack=1 Win=0 Len=0

Troubleshooting and Next Steps

When a handshake fails, the Wireshark capture provides critical clues for subsequent troubleshooting. Based on the patterns observed:

  • No SYN-ACK:
    • Check server status: Is the server online? Is the service listening on the correct port (e.g., netstat -tulnp on Linux, netstat -an | find "LISTENING" on Windows)?
    • Firewall rules: Inspect all firewalls between client and server (network, host-based) for rules blocking the specific port or IP addresses.
    • Routing: Perform a traceroute or tracert from the client to the server to identify any routing failures.
  • RST Response:
    • Service configuration: Confirm the service is configured to listen on the intended port. This is a clear indication that the server received the SYN but actively refused the connection.
    • Application logs: Check server application logs for error messages related to connection attempts.
  • SYN-ACK, No final ACK:
    • Reverse path routing: Verify routing from server back to client.
    • Client-side firewall: Ensure the client's firewall isn't dropping the incoming SYN-ACK.
    • Client resource limits: Rarely, client-side resource exhaustion might prevent the final ACK.

Common Pitfalls

  • One-sided Capture: Only capturing traffic on the client or server side can lead to misdiagnosis. Always try to capture at the point of failure or both endpoints if possible. If you only see SYNs and no replies on the client, the issue could be upstream (server not responding) or downstream (server responding, but packets not reaching client).
  • Misinterpreting RST: An RST can signify a closed port, but it can also be sent by a firewall or intrusion prevention system as an active rejection for various reasons. Don't assume a closed port without further investigation.
  • Ignoring Retransmissions: Frequent retransmissions of SYN or SYN-ACK packets usually indicate packet loss or network congestion, exacerbating handshake failures and increasing connection setup times.
  • Not checking port numbers: Always verify that the client is attempting to connect to the correct destination port and that the server is listening on that port.

Back to the knowledge base · Ask the AI assistant