Scenario
It has been that situation again when I have spent a whole day trying to research and figure out the solution to a problem that has bugged me. Hopefully this will save researching time for people having the same issue.
If you, like me, are trying to get cURL, or rather, plugin such as WordPress OpenID (which uses PHP libcurl) to work with HTTPS sites that use self-signed certificates but do not wish to compromise the security by disabling CURLOPT_SSL_VERIFYPEER and/or CURLOPT_SSL_VERIFYHOST option in the code, the following might be the answer for you.
Some background
You can skip this part actually. But if you are interested in some technical details, you can continue reading. This post is written on the assumption that Ubuntu Linux is used. It should apply to other distros as well with minor differences.
When we are using cURL to retrieve a HTTPS site that is not using a CA-signed certificate, the following problem occurs. Of course, this can simply be overcome by using the -k option.
root@ubuntu:/etc# curl https://example.selfip.com curl: (60) SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). The default bundle is named curl-ca-bundle.crt; you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option.
However, in the case of applications such as WordPress OpenID plugin, we have to amend the code to disable both CURLOPT_SSL_VERIFYPEER and/or CURLOPT_SSL_VERIFYHOST. Not a very good idea as this will disable verification for authentic sites as well. If we know example.selfip.com, in this case is one of our own trusted test servers, we can add its certificate to the trusted list.
Now that we know what to do, we have to find out how to do it. It took me quite a while to look for the solution as I was looking in the wrong places. WordPress OpenID and cURL documentations/forums did not point me in the right direction. I decided to take a look at the source code in cURL and discover that ultimately, the responsibility of the verification lies with OpenSSL.
Arm with this and some right keyword searching, it has led me to this HOWTO, which provided some insight on how OpenSSL recognizes certificate authorities, and an important program. Right now, we are all set to add the new certificate so that our OpenSSL can recognize the HTTPS server.
The solution
Identify which directory your OpenSSL installation uses.
root@ubuntu:~# openssl version -d OPENSSLDIR: "/usr/lib/ssl"
Change to that directory.
root@ubuntu:~# cd /usr/lib/ssl
List the directory contents. You should see a directory called “certs”.
root@ubuntu:/usr/lib/ssl# ls -la total 24 drwxr-xr-x 4 root root 4096 2009-04-24 14:37 . drwxr-xr-x 50 root root 12288 2009-04-24 17:56 .. lrwxrwxrwx 1 root root 14 2009-04-24 14:37 certs -> /etc/ssl/certs drwxr-xr-x 2 root root 4096 2009-04-24 14:37 engines drwxr-xr-x 2 root root 4096 2009-04-24 14:37 misc lrwxrwxrwx 1 root root 20 2009-04-24 14:37 openssl.cnf -> /etc/ssl/openssl.cnf lrwxrwxrwx 1 root root 16 2009-04-24 14:37 private -> /etc/ssl/private
Change to that directory.
root@ubuntu:/usr/lib/ssl# cd certs
List the directory contents. You should see from the symlinks that the certificates are actually stored in “/usr/share/ca-certificates”.
root@ubuntu:/usr/lib/ssl/certs# ls -la total 648 drwxr-xr-x 2 root root 16384 2009-06-24 12:32 . drwxr-xr-x 4 root root 4096 2009-04-24 14:39 .. lrwxrwxrwx 1 root root 26 2009-06-24 12:32 00673b5b.0 -> thawte_Primary_Root_CA.pem lrwxrwxrwx 1 root root 59 2009-06-24 12:32 2edf7016.0 -> Verisign_Class_1_Public_Primary_Certification_Authority.pem lrwxrwxrwx 1 root root 61 2009-06-24 12:32 thawte_Primary_Root_CA.pem -> /usr/share/ca-certificates/mozilla/thawte_Primary_Root_CA.crt lrwxrwxrwx 1 root root 94 2009-06-24 12:32 Verisign_Class_1_Public_Primary_Certification_Authority.pem -> /usr/share/ca-certificates/mozilla/Verisign_Class_1_Public_Primary_Certification_Authority.crt . . .
Change to that directory.
root@ubuntu:/usr/lib/ssl/certs# cd /usr/share/ca-certificates
List the directory contents. As you can see, I have added “example.selfip.com.crt” here.
root@ubuntu:/usr/share/ca-certificates# ls -la total 56 drwxr-xr-x 11 root root 4096 2009-06-24 15:02 . drwxr-xr-x 98 root root 4096 2009-04-24 17:51 .. drwxr-xr-x 2 root root 4096 2009-04-24 14:35 brasil.gov.br drwxr-xr-x 2 root root 4096 2009-04-24 14:35 cacert.org drwxr-xr-x 2 root root 4096 2009-04-24 14:35 debconf.org -rw-r--r-- 1 root root 1220 2009-06-24 11:57 example.selfip.com.crt drwxr-xr-x 2 root root 4096 2009-04-24 14:35 gouv.fr drwxr-xr-x 2 root root 12288 2009-04-24 14:35 mozilla drwxr-xr-x 2 root root 4096 2009-04-24 14:35 quovadis.bm drwxr-xr-x 2 root root 4096 2009-04-24 14:35 signet.pl drwxr-xr-x 2 root root 4096 2009-04-24 14:35 spi-inc.org drwxr-xr-x 2 root root 4096 2009-04-24 14:35 telesec.de
Change to “/etc” directory and edit the file “ca-certificates.conf”.
root@ubuntu:/usr/share/ca-certificates# cd /etc root@ubuntu:/etc# nano ca-certificates.conf
Add “example.selfip.com.crt” to the file and save it.
------------------------------------------------------------------------------- GNU nano 2.0.9 File: ca-certificates.conf # This file lists certificates that you wish to use or to ignore to be # installed in /etc/ssl/certs. # update-ca-certificates(8) will update /etc/ssl/certs by reading this file. # # This is autogenerated by dpkg-reconfigure ca-certificates. # Certificates should be installed under /usr/share/ca-certificates # and files with extension '.crt' is recognized as available certs. # # line begins with # is comment. # line begins with ! is certificate filename to be deselected. # example.selfip.com.crt brasil.gov.br/brasil.gov.br.crt cacert.org/cacert.org.crt cacert.org/class3.crt . . . -------------------------------------------------------------------------------
Execute the program “update-ca-certificates –fresh”.
Note: You might like to backup /etc/ssl/certs before executing the command.
root@ubuntu:/etc# update-ca-certificates --fresh Clearing symlinks in /etc/ssl/certs...done. Updating certificates in /etc/ssl/certs....done. Running hooks in /etc/ca-certificates/update.d....done.
Test with curl on your target HTTPS site and it should work now.
root@ubuntu:/etc# curl https://example.selfip.com <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> . . .
Excellent References:
http://www.madboa.com/geek/openssl/#verify-system
http://manpages.ubuntu.com/manpages/karmic/man8/update-ca-certificates.8.html
Your post on cURL(http://www.onlinesmartketer.com/category/php/curl/) really helped me…Thanks(it saved me lot of time)
Thanks, this saves some time for me 🙂
Fab!!! Thx!
hi, on redhat where the ca-certificates.conf file is? and wich command shall i use to update-ca-certificates -refresh?
I don’t have access to a Redhat server and hopefully this should help you. 🙂
http://nl.globalsign.com/en/support/ssl+certificates/redhat/red+hat+enterprise+linux/install+certificate/
Man Lifesaver for me..!!!
Thanks boss.
Actually when someone doesn’t understand then its up to other viewers that they will help, so here it takes place.
how to add the support for fedora 9?
Unfortunately, I do not have Fedora installed to test it out. But the concept shouldn’t be too far apart between Ubuntu and Fedora. Probably someone in the community could help you out.
Thank you!
Got the same problem. Now I’m just using a different certificate instead.
Good post. Got same error while accessing the https url with curl. I used the domains certificate and it again gave same error … At last I downloaded the intermediate CA certificate and followed your instructions and it fixed the issue.
Great ! Helps a lot !
Excellent! This is not just for cURL, but if you use a self-signed certificate for any site that uploads files with flash (aaargh!!), then you need this too! I needed it for Magento stores to upload file in the administration interface.
thank you!
This works for imap connection with php and a self signed certificate.
“This post is written on the assumption that Ubuntu Linux is used. It should apply to other distros as well with minor differences.”
In fact it will not, as distros all use different systems for maintaining their trust stores. The system you describe is Debian’s system, which Ubuntu uses. Other Debian derivatives mostly use the same system, but non-Debian derived distros mostly do not.
On OpenSUSE – https://github.com/openSUSE/ca-certificates – I believe (going by the source) you will want to add the certificate to /etc/pki/trust/anchors and run update-ca-certificates (which, despite the identical name, is not the same as Debian’s update-ca-certificates).
On Fedora, RHEL, Arch, and derivatives / compatible distros, you will want to add the certificate to /etc/pki/ca-trust/source/anchors and run update-ca-trust (see ‘man update-ca-trust’).
Thanks for sharing. It will benefit people using other distros.
This tutorial is great, thanks.
I tried but It’s not working in my situation because the server use a certificate chain with extended validation and only send the first cert of the chain after doing your suggested steps the curl certificate errors still shows up because doesn’t have the root cert of the issuer. Just the same case that it’s described here:
http://www.apsis.ch/pound/pound_list/archive/2013/2013-01/1357296249000/index_html?fullMode=1
For references in this case you could do an online test of the conection to the https server using https://www.ssllabs.com/ssltest/
when it finish (are forward a little) you will see the it shows an extra download result, example of my case:
DigiCert SHA2 Extended Validation Server CA
Fingerprint: 7e2f3a4f8fe8fa8a5730aeca029696637e986f3f
RSA 2048 bits (e 65537) / SHA256withRSA
Go to the cert issuer site to find and download the necessary cert, digicert in my case, if the new cert is not in PEM format you could import it in mozilla and the export to the PEM format and then you could do the steps of this great tutorial to solve the issue and get curl or any other based openssl cert validation of connections
Thanks again for the tips
do we have to add my target url instead of example.selfip.com ? i tried but getting this error curl: (7) Failed to connect to example.selfip.com port 443: Connection timed out
I am really in need for support.
You should be adding the domain, and its relevant cert, you want to connect to, not example.selfip.com.
I am trying to connect https://api.royalmail.com/tracking — Should i have to use full url or simple https://api.royalmail.com ?
I believe i have to put this in my ca-certificates.conf instead of example.selfip.com.crt right?
will it become like api.royalmail.com.crt ? Please suggest.
“example.selfip.com” is just an example domain. You just need to replace it with your relevant domain, in this case “api.royalmail.com”. “api.royalmail.com.crt” file should contain api.royalmail.com certificate. After that, you can just use curl to test https://api.royalmail.com/tracking
Thank you!I’ll try later