It took all day, but I tracked down a strange bug:
BUG DESCRIPTION:
Symantec Encryption (PGP) interferes with SSL/TLS under specific conditions, causing encrypted connections to fail. The failure occurs under conditions where the browser would normally open a dialog to inform the user that there was a certificate authentication error, and normally gives the user an opportunity to abort the connection or establish one with the invalid certificate. PGP makes it impossible for the user to override the certificate authentication errror, even when the user knows that he/she can safely establish the connection. Different browsers report the failure in different ways, but in all cases, the user is unable to establish the connection.
STEPS TO REPLICATE...
FIRST, ESTABLISH MAC OS X NORMAL BEHAVIOR:
1. On a Mac OS X system where Symantec Encryption / PGP is NOT installed, open a browser.
2. Attempt to establish a secure connection to a web server where the following is true:
2a. The web server has a certificate that cannot be authenticated by Mac OS X. For example, it may be a self-signed certificate.
2b. The connection is established on a non-standard port.
Example: In my test case, I was attempting to connect to a Sophos UMT (firewall) using "https://<FQDN_or_IP_address>:4444. The Sophos UMT devices have a secure web server for configuration at port 4444.
3. Note that the browser opens a dialog box, indicating that the website certificate cannot be authenticated. The browser prompts you to abort the connection, but also provides a way for you to ignore the warning and connect despite the authentication failure.
NEXT, REPLICATE THE BUG, WHERE THE CONNECTION FAILS COMPLETELY:
4. Install PGP WDE. I have replicated this bug using Symantec Encryption 10.3 MP1 under Mac OS 10.8.3, and PGP 10.1.2 under Mac OS 10.6.8. Reboot after installation, but there is no need to open the Desktop application or enter a license code, etc.
5. Open a browser. Attempt the same connection as before (in step 2, above). Note the following errors:
5a. SAFARI: Safari can't open the page "https://<FQDN_or_IP_address>:4444/" because Safari can't establish a secure connection to the server "FQDN_or_IP_address".
5b. CAMINO: Data Transfer Interrupted. The connection to <FQDN_or_IP_address>:4444 has terminated unexpectedly. Some data may have been transferred. [...]
5c. CHROME: SSL connection error. Unable to make a secure connection to the server. This may be a problem with the server, or it may be requiring a client authentication certificate that you don't have. Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.
5d. Firefox: Not tested.
6. This bug has been replicated with a Sophos UMT firewall running as a VMware instance, and also on a Sophos UMT firewall with a publicly facing interface on the Internet.
7. This bug does not appear when testing on various browser running in VMware guest OSs (NATed) on the same Mac OS X computer with Symantec Encryption installed - Windows, Linux, and also Tor (Vidalia Control Panel) with the TorBrowser.
The easiest way to replicate this bug is to do the following:
6. Install VMware Fusion 5.0.3 on your Mac
7. Download the demo UMT from Sophos - I have replicated the bug using both the .iso version, the VMware instance, and the ESX instance converted to a VMware instance using VMware's OVF tool.
8. The Sophos examples are configured with bridged network interfaces. The easiest way to connect is to set your Mac OS to a fixed IP address on the same subnet as the UTM default IP address as displayed in its console. Once you connect to the UTM with your browser (PGP not installed), change its IP address to one that works on your LAN, then restore your Mac OS IP address and connect to the UTM on its new address.
ADDITIONAL NOTES:
9. I do not know exactly which conditions are required to replicate this bug. Is an alternate port (e.g., 4444) truly required? Exactly what types of certificate failures are necessary? Must the web server run a specific version of SSL/TLS to see the bug?
10. All I know is that I found this issue while trying to test a Sophos UTM. Obviously Sophos knew nothing of this bug, and it took a very long time and a lot of effort to isolate this issue to the presence (or lack thereof) of PGP.
At this point, I am tired of configuring test scenarios, rebooting, isolating, replicating, verifying, etc. I didn't make much progress on my primary task, which was to test the Sophos UTM firewall.