Recently, I had to troubleshoot an issue where there was some improper API use and it was being blamed on the application. The traffic is SNAT’d, so essentially backend servers are being proxied. What I mean by this, is in the “backend servers” perspective, the requests appear to be coming from the F5, not the original source IP. Since the SSL is offloaded on the F5, we can only trace the unencrypted traffic on the backend and due to how noisy it is and all request appear to be coming from the F5 self IP, this becomes very difficult to troubleshoot. So we need to troubleshoot on the frontend, where the public source IP is preserved. This traffic is encrypted, so we need means of viewing the traffic unencrypted. This can be done with tcpdump and ssldump, which both are installed on the F5 by default. So now we are ready to start capturing the traffic. You can do so with tcpdump, but you should be very strict with your parameters, as you could cause performance issues sending to much garbage to stdout. You will want to save to file, so we can later decrypt with ssldump. Here is an example, listening on the frontend VLAN: Now you should have the PCAP file in your present directory and you can view it via tcpdump, wireshark, or any other packet analysis tool that you have available, as PCAP is the industry standard packet capture format. If you were to open the file up in Wireshark, you would notice that the SSL/TLS payload displays ***‘* Record Layer: Handshake Protocol: Encrypted Handshake Message’***, so you aren’t able to view the unencrypted data natively, which is expected. This is where ssldump comes in, which can utilized your F5 private keys to decrypt the trace. You need to identify what SSL cert/key pair are used on the VIP you are troubleshooting. You can find this out by looking over the VIP configuration, which will use a specific SSL profile. In looking into that profile, it will mention what cert/key pair it uses. You will be able to find these certs/keys in /config/ssl/ssl.crt & ssl.key on 9.x/10.x or it could be stored in /config/filestore/files\_d/Common\_d/certificate\_d & certificate\_key_d in 11.x. Once you find the key, you can decrypt the PCAP, using the following example: So there you have it, you can decrypt SSL traffic if you have the private key with only tcpdump and ssldump. You can perform the same task in using tcpdump to output to PCAP and then using the private key in Wireshark to decrypt the traffic, although I find it easier to troubleshoot using tools on the F5 if I can. Let me know if you have any questions on the subject or any suggestions to improve this method. Feel free to comment!
There are many situations where you need to be able to capture traffic at ANY point in the network. Whether it testing from the client side, server side, or any transit component, such as your ASA. Since the traditional tools such as tcpdump or Wireshark aren’t available, you will have to use the proprietary tools on the ASA itself. You will start off by creating an ACL to match the traffic you are wanting to capture. This should be a strict, such as a source and dest that are experiencing the issues you are troubleshooting: Now that you have configured the access list to match your traffic, you can start the capture. I am capturing traffic before and after it goes over the VPN, so I will monitor the interface closest to the source, which is the ‘inside’ interface. There aren’t any “required” options besides the name but if you don’t specify an ACL and interface, you won’t match any traffic. There are a few options you may want to use other than those two: * type (default is raw-data but there are other options such as ISAKMP) * headers-only (no data, just headers; similar to tcpdump -q) * buffer (default is 512KB. Increase with caution, you don’t want to use up all your memory) * real-time (have output display in your terminal) * \*more\* — use ‘capture
?’ to see all options
We will go with the basics; access-list and interface:
After this, you can check your console to see if the traffic is matching your ACL and being captured with ‘show capture'; if the bytes are a non-zero value and increasing, it’s working!
If you want to see the content, you have to specify the capture name:
So if you were troubleshooting connectivity and you see this successful ICMP traffic, the loss you are experiencing must be somewhere else. Now you can assure that packets are hitting the firewall and this should help you verify your high level packet flow, to hopefully pinpoint where the issue is. If you want to see more information, there are a few more useful options when displaying the capture:
* access-list (yes, you can use an additional access-list to filter the output of ‘show capture’)
* count (# of packets to display)
* detail (display more information such as TTL, DF bit, etc)
* dump (display the hex dump for each packet)
* \*more\* — use ‘show capture ?’ to view all options
So if you would like to view the hex dump of 1 packet, you can with:
Often, you may have a large capture and you don’t want to sift through it on the ASA. You might want to look at the capture file in wireshark or tcpdump. You can do so by copying the file via tftp to a workstation or server. But how will the output be formatted… with just headers, hex dumps or .. ? Well, when you transfer it via copy, you specify to use pcap format, which you will be able to read in just about any packet analysis program as it is the most commonly used. You can do this via:
Awesome! So although the ASA capture utility isn’t as versatile as tcpdump or wireshark for example, it is still great to start with and verify packet flow, etc. And if you want to save a larger capture for future analysis, it can be copied and opened with another packet analysis tool. From there you can use your advanced filters, etc.
I hope you enjoyed the basic tutorial on how to use the ASA capture utility… perhaps soon I will write one up on how to effectively use packet-tracer, which is yet another useful ASA command line tool for verifying packet flow and it will also give you a better understanding of how the ASA processes packets. (order of NAT/ACL processing, route lookup, etc.)