Blue Coat Troubleshooting Basics

A few useful 'set piece' procedures for gathering diagnostic information from a Blue Coat Proxy SG appliance.

Full policy trace

Enable this under Configuration > Policy > Policy options > enable full policy execution. Then, go to https://x.x.x.x:8082/policy (where x.x.x.x is the IP of your proxy) and delete the default trace. Also make sure there are no tracing rules in CPL or the VPM otherwise the trace will not work. Then, replicate the problem by going to the problematic website a few times (making a note of the IP address the client is using) and then turn off the policy trace.

Filtered policy trace

Add the following to your local policy file (replacing z.z.z.z with the IP address of the client you are testing from).

<proxy>
client.address=z.z.z.z trace.rules(all) trace.request(yes) trace.destination(trace.html)

Then, replicate the issue.
Then go to https://x.x.x.x:8082/policy (where x.x.x.x is the IP of your proxy) and download the file trace.html

Sysinfo

Go to https://x.x.x.x:8082/sysinfo (replacing x.x.x.x with the IP address of your proxy) and save the page as a text file. Then please zip and email it to me.


Sysinfo stats

Go to https://x.x.x.x:8082/Diagnostics/Snapshot/sysinfo_stats/download/all to download an archive of all the available snapshots on the SG.

HTTP Debug:

The HTTP debug can be done using the advanced URL https://xx.xx.xx.xx:8082/HTTP/Debug. Set the mask enabling all options and clear the log just before browsing to the site both times. Saving the output from each.

Packet capture

Login to https://x.x.x.x:8082/PCAP/statistics and make sure filtering is off. Then, start the packet capture, try to access the problematic site through the proxy and then stop the capture. Then please download and compress the packet capture and send it to me with a note of the client IP address, proxy IP address and the IP address and URL of the website you tried to view.

Blue Coat - SSL reverse proxy for Exchange Active Sync (with client certificates)

I've recently had the pleasure of setting up a reverse SSL proxy for Exchange Active Sync (EAS) with the requirement of SSL on both sides of the connection, with client certificates for mobile devices.
After some coffee, some 'what have I done to deserve this' moments and a bit of reading - it's actually not quite as painful as you might imagine. There's a good amount of documentation on how to achieve this with Microsoft ISA / TMG but not much for other vendors. Here's how I set it up with a Blue Coat  Proxy SG running SGOS 6.4.2.1.

1. To make things a bit easier, you'll need all of your certificates in PEM format (Base 64 encoded - the ones if you open in a text edit start with ===BEGIN CERTIFICATE/KEY===). If you generate them from your Microsoft enterprise CA you'll most likely be given them in PFX or P12 unless you want to generate CSRs from the Blue Coat device. You'll need to use openssl to convert them into a PEM file containing the unencrypted private key and the certificate.
You will need the certificates for your enterprise CA, exchange server if not issued by your CA and one for the Blue Coat device itself  to identify to the clients (issued by your enterprise CA).

2. Add the certificate for the Blue Coat device to your proxy as a keyring (configuration -> SSL -> Keyrings). This should be the only certificate with a private key contained in the PEM file.

3. Add the other CA certificates to the proxy in  CA certificates section (configuration -> SSL -> CA certificates). This will ensure the proxy trusts any certificates issued by your enterprise CA (clients and the Exchange server). It's quite common for Exchange to use a self-signed certificate - if this is the case you'll need to export this certificate and add this to the Blue Coat as well (otherwise you'll need to add policy later on to ignore the fact it's issuer isn't trusted.

4. Create a new CA Certificate list and add in the CA certificates you created above. We'll use this shortly on our proxy service listener.

5. Under Configuration -> Services -> Proxy Services, create a new service group and named it 'Reverse proxy'. (this is optional but keeps things more organised). Create a new and fill in the details as below.
Name: Something meaningful to you
Service group: Group you created above
Proxy: HTTPS Reverse Proxy
Keyring: This is the keyring for the Blue Coat appliance you created previously (the one with the private key).
CCL: This is the CCL you created in step 4. This will make sure the proxy only accepts certificates which were issued by your enterprise CA.
SSL versions: Enable whatever you require here - but make sure you tick Verify Client and don't enable Forward Client Cert (it's impossible to forward the actual client certificate so the Blue Coat forwards some information from the certificate in the HTTP headers. This doesn't work for our goal which is why we're using KCD).
ADN: unticked - it's not relevant here.
Listeners: This is the address that you will NAT from your firewall (or a public address if your proxy is deployed in that way). This will be where clients connect. You can use an IP that isn't in use by the proxy so long as you add it as a Virtual IP.

6. Join the proxy to your domain (you'll need a domain administrator account for this). Under Authentication -> Windows Domain create a new domain and then join it. Make a note of the name you join to the domain with as you'll need this for the KCD settings. If you have problems joining - make sure the DNS on your proxy is pointing toward your Active Directory server. Once joined, go to Authentication -> IWA -> IWA servers and make sure under Client Authentication Modes - 'Kerberos credentials' is ticked.

7. Create a certificate authentication realm under Authentication -> Certificate. Then under Certificate Main, configure the extractors so the proxy is able to obtain the username from the clients certificate.
Your certificates might vary so you might need to try a few options to find what works for you. Our AD / CA setup is mostly standard so far as I know so these settings should work. You'll get errors in the event and access logs saying 'unable to extract username' if these settings aren't relevant to your environment. The options on the other tabs can be left at default.



8. Over to your AD setup for a moment now to enable your Blue Coat device to perform KCD for authentication to your Exchange server. In the AD 'Users And Computers' view - find the new device representing the proxy (the name you chose in step 6)and edit the properties. Under the delgation tab, configure your options as in the screenshot.

For the user / computer options which are blanked out in the screenshot - this needs to be the object for your Exchange server. Click apply / OK and close this window.














9. Back to the proxy now. Go to configuration -> Forwarding -> Forwarding Hosts and create a new forwarding host. This needs to have the below settings.
Host: IP address of your Exchange server
Type: Server
Ports: 443
Others left unticked as per the default.

10. Open up the VPM via Configuration -> Policy -> Visual Policy Manager.
For the purpose of this guide - I'll be working with a totally blank policy to keep things simple. The policy layers should be setup as below.

Layer 1 - Web Authentication Layer.

Destination: Request URL object. Simple match - value should be the DNS name clients will be accessing for your reverse proxy. Eg, activesync.mydomain.co.uk

Action: Combination object containing an Authenticate Object (Realm should be your certificate authentication realm from step 7. Mode: Auto) and a Kerberos Constrained Delegation object. (Authentication type should be origin, IWA Realm should be the IWA direct realm from step 6. Service Principal Name should be 'host/' then the exact name of your Exchange server without the domain).

Layer 2 - Web Access Layer.

Source: Authenticated user (no configuration for this object)

Destination: Request URL object (Advanced match - scheme any, Path "/Microsoft-Server-ActiveSync"). This will ensure only EAS connections are allowed to this reverse proxy connection. 

Layer 3 - SSL Access Layer

NOTE: This layer isn't strictly necessary but it's useful to troubleshoot SSL problems whilst you're setting up. The destination is the IP of your Exchange server and the Action is an SSL certificate validation action to ignore untrusted issuer.

Layer 4 - Forwarding Layer

Source: Authenticated User (no options again)

Destination: Server URL object (simple match again with the DNS name of your reverse proxy connection).

Action: Select Forwarding Object. Move the object name you created in step 9 into the right hand column and then ok.

11. We're done! Install the VPM policy and then you're ready to test. Make sure the client certificates you deploy contain the full certificate chain or you'll get SSL errors when your devices try connecting to the proxy. If you use Active Directory to issue user certificates you'll probably find they're in PFX format. In this case, they should contain the full chain by default.

Check Point dynamic object for DNS

Although Check Point use domain objects for resolving DNS entries to IP addresses in the rulebase - the documentation recommends not to use them for various good reasons. I've written a simple script to make use of Check Points rarely used dynamic objects. The script pings a host by DNS, takes the IP address then updates a dynamic object you can then use in your firewall policy. Add this to a cronjob and then you can have a dynamic dynamic object(!?).
Download here

#!/bin/bash -f
#Dynamic Object update script - dyno.sh
#Stuart Green - 08/02/2013
#stuart@root9.net
#
#PURPOSE: 
# This script will create and subsequently update a Check Point dynamic
# object which can be used in the rulebase in place of a domain object which is
# known to be unreliable and resource intensive. The script should be placed in
# a crontab and run at the interval you choose.
#
#REQUIREMENTS:
# DNS needs to be configured on your gateway.
# ICMP and DNS need to be allowed outbound from your gateway / cluster object.
# Tested on R75.40 SecurePlatform and Gaia - should be fine with other versions.
# 
#USAGE: 
# Copy this script somewhere on your firewall.
# Edit the value of CDIR in this script to be the value of $CPDIR from your 
# shell. Eg, the value you get when you run 'echo $CPDIR'.
# Edit the value of DNSNAME to be the domain you want to resolve to an IP.
# Edit the value of CPOBJECT to represent a UNIQUE Check Point dynamic object.
# Add a cronjob to execute the script at your desired time interval (~10min).
#
#EXAMPLE: 
# run 'crontab -e' to edit your crontab
# paste in the following line (minus quotes) to run the script every 10 minutes: 
# '10 * * * * /home/admin/dyno.sh'
#
# Create a dynamic object in your policy with the EXACT same name as CPOBJECT.
# Use the object where required.
#
# If you need multiple dynamic objects, you can copy the script with a 
# different name, modify the values of DNSNAME and CPOBJECT then add the new
# script to a new cronjob.
#
# LIMITATIONS:
# Domains that return multiple IPs will only update the dynamic object with the
# FIRST IP address!
CDIR="/opt/CPshrd-R75.40"
source $CDIR/tmp/.CPprofile.sh
DNSNAME="www.mywebsite.com"
CPOBJECT="DO_www.mywebsite.com"
IP=$(/bin/ping -c1 $DNSNAME | /bin/egrep -oh -m1 "([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})")
$FWDIR/bin/dynamic_objects -do $CPOBJECT > /dev/null 2>&1
$FWDIR/bin/dynamic_objects -n $CPOBJECT > /dev/null 2>&1
$FWDIR/bin/dynamic_objects -o $CPOBJECT -r $IP $IP -a > /dev/null 2>&1
EXIT_STATUS=$?
if [[ $EXIT_STATUS -eq 0 ]]; then
  echo "Dynamic object $CPOBJECT updated. $DNSNAME resolves to $IP" >> /var/log/messages
fi
if [[ $EXIT_STATUS -ne 0 ]]; then
  echo "Dynamic object $CPOBJECT not updated successfully" >> /var/log/messages
fi

Check Point - upgrade_export and the migrate command

What's the difference between the two you may ask. In true Check Point style the documentation implies they're different commands with different functions.

 Let's compare the binaries...



md5sum migrate
d9889562e3093d70669ff511b8fbc9d4  migrate

md5sum upgrade_export
d9889562e3093d70669ff511b8fbc9d4  upgrade_export

So the difference - nothing. They're exactly the same binary. The only apparent difference is you get a deprecation warning if you use upgrade_export but not migrate. Thanks for the misleading documentation Check Point...

OpenELEC Raspberry Pi - Build R12577

Time for another new image. The OpenELEC guys have added something pretty cool to this release. If you've got a new Pi with 512MB of RAM it can run OpenELEC entirely from RAM! I can't test this myself because I've got a first revision, but Matt @ RaspPiBeginners has tested it out and it works well.

OpenELEC R12577 - Raspberry Pi 1GB image

F5 and Websockets

If you're interested in getting some shiny new HTML5 websockets working through an F5 load balancer with HTTP profiles enabled, you're in for some disappointment.
Despite websockets being documented in the RFC as proxy compatible - they don't work through an F5 when you are using full-proxy mode (i.e. you have an HTTP profile assigned). This looks to be down to the websockets connection sending an 'upgrade' notification to the client which turns the HTTP connection into a standard TCP connection (still on port 80/443 however but not strictly HTTP).

Anyway, unless you know of some hardcore iRule magic - I had to have a bit of an architectural rethink. Our websockets application on the backend works off of a virtual host which has the websockets proxy enabled on it. It runs on the same server as the main web application so you can see where disabling the HTTP profile becomes a problem if I want to run any iRules relating to HTTP. The way we've got around it is to create a new public virtual server then redirect any connections going to the /websocket  to the new virtual server using an iRule. On the new virtual server - you still assign an HTTP profile and then create an iRule so that only connections to the /websocket path go to the virtual server and anything else is redirected back to the main VS. It's a bit of a crude solution, but it's simple enough to do the job. For what it's worth, websockets support is in the pipeline apparently.
So for the iRules...

iRule for VS_main_web_app

when HTTP::REQUEST {
if {[HTTP::URI] starts_with '/websocket'}
  {
  HTTP::redirect "https://websocket.domain.com"
  }
}

iRule for VS_websocket_only

when HTTP::REQUEST {
if {![HTTP::URI] starts_with "/websocket"}
  {
  HTTP::redirect "https://mainapp.domain.com"
  }
else
  {
  HTTP::disable
  }
}

SSL Decryption with Wireshark (Private key and Pre-Master secret)

Troubleshooting communication problems with Wireshark can be difficult at the best of times, yet alone when the connection is encrypted with SSL/TLS.
There are a couple of ways you can approach decrypting the SSL/TLS traffic. One assumes you have root access to the server you are having problems with and you're able to obtain a copy of the public and private key. If you only have access to the server as a client with no special privileges, there are still a couple of ways to take look at the encrypted data.

Before you look at either method, it would be useful to do a little background reading on SSL/TLS as the encryption methods might not work quite as you think.
In a nutshell, when a client (in most cases, a web browser) makes a connection to a web server requiring SSL/TLS encryption - the encrypted channel is setup using a symmetric session key. This key is a random string generated by the client and then encrypted and transmitted using the servers public key, known as the Pre-master Secret. Once shared, the client and server use this shared key to encrypt and decrypt traffic. There's a more detailed version of this here, but knowing this you be able to see how you can decrypt the traffic using the SSL session key or the servers private key.

Method 1 : Decrypting the traffic with the server private key.

1. You'll need the private key from the server first. This can be hidden in a variety of locations so you'll have to refer to your server documentation for where to find it. Usually it will be a single file with an extension of .CRT, .PEM, .DER or in some cases .PFX or .P12. The latter two are certificate containers which contain the private key, public key and the certificate of the issuing CA.
2. Once you've found the private key, it will make things easier if they're in PEM format with an unencrypted key. The best way to do the conversion is to use OpenSSL. It can be a bit confusing at first but you'll soon get used to how it works. There is a good selection of commands here (they also offer an upload / convert tool. Not to say they're not trustworthy, but I wouldn't recommend giving your private key to anyone. Ever.)
3. Now you have the private key, you'll need to load this into Wireshark and configure it to use this key for any traffic to or from your web server. Go to Edit > Preferences. Then Protocols > SSL. Next to the RSA keys list text, click the edit button. From here select the servers private key and enter the IP address of the web server that will be present in the capture. Select port 443 (or whichever port your application runs on) and the protocol which is inside the encrypted tunnel. In most cases, this will be HTTP but could be IMAP, SMTP etc. The password field can be left blank if you've saved the key in an unencrypted form, otherwise provide the password here. Click OK until you're back to the Wireshark main screen.
4. Now start the capture and generate some encrypted traffic to your web server. Make sure you are using a totally new session. Clearing the cache then closing and reopening your browser would be ideal. Stop the capture when you've generated a few connections.
5. Back in the main window, you should now find that the SSL wrapping has been removed and you're able to view the protocol details within.

Method 2: Decrypting with Pre-Master Secrets

This method is relatively new to Wireshark and allows you decrypt the encrypted traffic using the Pre-Master Secret which is generated by the client. By default, this key isn't logged anywhere for obvious reasons but with Chrome it's possible to set an environment variable and have these written to disk.

1. (Windows 7) Right click on 'My Computer' and then go to properties.Then click Advanced System Settings > Environment Variables. Then under system variables - create a new variable named SSLKEYLOGFILE with the value being a text file. In this case I went with C:\premaster.txt. Click OK through all open dialogs. I've found this didn't take effect immediately and needed a reboot before it started logging.
2. Back in Wireshark, head to Edit > Preferences > Protocols > SSL. Under the option for '(Pre)Master-Secret log file name' - select your log file you created above (so C:\premaster.txt).
3. Start your capture in Wireshark and then generate a few SSL connections in Chrome. Stop the capture when you're done.
4. In Wireshark, you should again see that your encrypted traffic is now unwrapped ready for some troubleshooting. If you don't, check the pre-master file you've created exists and has some contents. If not you should double check your environment variable.

These is of course a much easier method for looking into encrypted traffic if you're only looking at HTTP traffic (or traffic that supports a proxy setting) and that's to use Fiddler. Fiddler is an excellent local proxy application which can perform SSL termination and re-encryption (otherwise known as 'man-in-the-middle' / MITM). If you don't need a full packet capture and can make do with the HTTP requests and responses this might save you a lot of pain.