Scenario

It has been that situation again when I have spent a whole day trying to research and figure out the solution to a problem that has bugged me. Hopefully this will save researching time for people having the same issue.

If you, like me, are trying to get cURL, or rather, plugin such as WordPress OpenID (which uses PHP libcurl) to work with HTTPS sites that use self-signed certificates but do not wish to compromise the security by disabling CURLOPT_SSL_VERIFYPEER and/or CURLOPT_SSL_VERIFYHOST option in the code, the following might be the answer for you.

Some background

You can skip this part actually. But if you are interested in some technical details, you can continue reading. This post is written on the assumption that Ubuntu Linux is used. It should apply to other distros as well with minor differences.

When we are using cURL to retrieve a HTTPS site that is not using a CA-signed certificate, the following problem occurs. Of course, this can simply be overcome by using the -k option.

root@ubuntu:/etc# curl https://example.selfip.com
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). The default
 bundle is named curl-ca-bundle.crt; you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

However, in the case of applications such as WordPress OpenID plugin, we have to amend the code to disable both CURLOPT_SSL_VERIFYPEER and/or CURLOPT_SSL_VERIFYHOST. Not a very good idea as this will disable verification for authentic sites as well. If we know example.selfip.com, in this case is one of our own trusted test servers, we can add its certificate to the trusted list.

Now that we know what to do, we have to find out how to do it. It took me quite a while to look for the solution as I was looking in the wrong places. WordPress OpenID and cURL documentations/forums did not point me in the right direction. I decided to take a look at the source code in cURL and discover that ultimately, the responsibility of the verification lies with OpenSSL.

Arm with this and some right keyword searching, it has led me to this HOWTO, which provided some insight on how OpenSSL recognizes certificate authorities, and an important program. Right now, we are all set to add the new certificate so that our OpenSSL can recognize the HTTPS server.

The solution

Identify which directory your OpenSSL installation uses.

root@ubuntu:~# openssl version -d
OPENSSLDIR: "/usr/lib/ssl"

Change to that directory.

root@ubuntu:~# cd /usr/lib/ssl

List the directory contents. You should see a directory called “certs”.

root@ubuntu:/usr/lib/ssl# ls -la
total 24
drwxr-xr-x  4 root root  4096 2009-04-24 14:37 .
drwxr-xr-x 50 root root 12288 2009-04-24 17:56 ..
lrwxrwxrwx  1 root root    14 2009-04-24 14:37 certs -> /etc/ssl/certs
drwxr-xr-x  2 root root  4096 2009-04-24 14:37 engines
drwxr-xr-x  2 root root  4096 2009-04-24 14:37 misc
lrwxrwxrwx  1 root root    20 2009-04-24 14:37 openssl.cnf -> /etc/ssl/openssl.cnf
lrwxrwxrwx  1 root root    16 2009-04-24 14:37 private -> /etc/ssl/private

Change to that directory.

root@ubuntu:/usr/lib/ssl# cd certs

List the directory contents. You should see from the symlinks that the certificates are actually stored in “/usr/share/ca-certificates”.

root@ubuntu:/usr/lib/ssl/certs# ls -la
total 648
drwxr-xr-x 2 root root  16384 2009-06-24 12:32 .
drwxr-xr-x 4 root root   4096 2009-04-24 14:39 ..
lrwxrwxrwx 1 root root     26 2009-06-24 12:32 00673b5b.0 -> thawte_Primary_Root_CA.pem
lrwxrwxrwx 1 root root     59 2009-06-24 12:32 2edf7016.0 -> Verisign_Class_1_Public_Primary_Certification_Authority.pem
lrwxrwxrwx 1 root root     61 2009-06-24 12:32 thawte_Primary_Root_CA.pem -> /usr/share/ca-certificates/mozilla/thawte_Primary_Root_CA.crt
lrwxrwxrwx 1 root root     94 2009-06-24 12:32 Verisign_Class_1_Public_Primary_Certification_Authority.pem -> /usr/share/ca-certificates/mozilla/Verisign_Class_1_Public_Primary_Certification_Authority.crt
.
.
.

Change to that directory.

root@ubuntu:/usr/lib/ssl/certs# cd /usr/share/ca-certificates

List the directory contents. As you can see, I have added “example.selfip.com.crt” here.

root@ubuntu:/usr/share/ca-certificates# ls -la
total 56
drwxr-xr-x 11 root root  4096 2009-06-24 15:02 .
drwxr-xr-x 98 root root  4096 2009-04-24 17:51 ..
drwxr-xr-x  2 root root  4096 2009-04-24 14:35 brasil.gov.br
drwxr-xr-x  2 root root  4096 2009-04-24 14:35 cacert.org
drwxr-xr-x  2 root root  4096 2009-04-24 14:35 debconf.org
-rw-r--r--  1 root root  1220 2009-06-24 11:57 example.selfip.com.crt
drwxr-xr-x  2 root root  4096 2009-04-24 14:35 gouv.fr
drwxr-xr-x  2 root root 12288 2009-04-24 14:35 mozilla
drwxr-xr-x  2 root root  4096 2009-04-24 14:35 quovadis.bm
drwxr-xr-x  2 root root  4096 2009-04-24 14:35 signet.pl
drwxr-xr-x  2 root root  4096 2009-04-24 14:35 spi-inc.org
drwxr-xr-x  2 root root  4096 2009-04-24 14:35 telesec.de

Change to “/etc” directory and edit the file “ca-certificates.conf”.

root@ubuntu:/usr/share/ca-certificates# cd /etc
root@ubuntu:/etc# nano ca-certificates.conf

Add “example.selfip.com.crt” to the file and save it.

-------------------------------------------------------------------------------
  GNU nano 2.0.9                         File: ca-certificates.conf

# This file lists certificates that you wish to use or to ignore to be
# installed in /etc/ssl/certs.
# update-ca-certificates(8) will update /etc/ssl/certs by reading this file.
#
# This is autogenerated by dpkg-reconfigure ca-certificates.
# Certificates should be installed under /usr/share/ca-certificates
# and files with extension '.crt' is recognized as available certs.
#
# line begins with # is comment.
# line begins with ! is certificate filename to be deselected.
#
example.selfip.com.crt
brasil.gov.br/brasil.gov.br.crt
cacert.org/cacert.org.crt
cacert.org/class3.crt
.
.
.
-------------------------------------------------------------------------------

Execute the program “update-ca-certificates –fresh”.
Note: You might like to backup /etc/ssl/certs before executing the command.

root@ubuntu:/etc# update-ca-certificates --fresh
Clearing symlinks in /etc/ssl/certs...done.
Updating certificates in /etc/ssl/certs....done.
Running hooks in /etc/ca-certificates/update.d....done.

Test with curl on your target HTTPS site and it should work now.

root@ubuntu:/etc# curl https://example.selfip.com
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
.
.
.

Excellent References:
http://www.madboa.com/geek/openssl/#verify-system
http://manpages.ubuntu.com/manpages/karmic/man8/update-ca-certificates.8.html