int X509_verify_cert(ctx)
X509_STORE_CTX *ctx;
The default verification routine internal_verify is defined in x509_vfy.c. This is the routine invoked by X509_verify_cert unless you have changed the default verify function.
Let's take the case first where you are verifying a single certificate, not a certificate chain. In that case, X509_verify_cert creates a stack of certificates, puts the certificate to be verified on it, and then checks the depth of the stack to be sure it's ok. Default maximum depth is 10, so a stack of 1 is ok :) If we had a chain here we would do some other processing but we don't, so instead we move on to the check against trusted certs.
First, get the last cert on the stack, see if it is self-signed and if it is, and the number of certs on the stack is 1, set an error, call the default verify callback (null_callback which does nothing, unless you have set another default callback), and if the error is unchanged return this.
In our case the stack is just the one certificate. So if it is self-signed we will get the error, and depending on the callbak either return with it, or proceed on to the next step.
If the cert was not self-signed we will also move on to the next step.
Now we enter a loop: break if number of certs on stack is greater than depth, break if cert is self-signed, and if neither of these, try to get the issuer cert from our trusted list. If we can't find it, break. Otherwise, put it on the stack and go around again.
After exiting the loop we have a chain of certs that either ends early because depth exceeded, ends early because we couldn't find an issuer, or ends with a self-signed cert.
In the first two cases we will flag an error that we couldn't find the issuer; in the third case we call
internal_verify
(or some other function if you have used X509_STORE_set_verify_func to make it call something else.)
internal_verify does the following for each cert in the stack, starting at the end and working back towards the beginning:
If an error is encountered anywhere along the way, the default callback (null_callback, unless you have used X509_STORE_set_verify_cb_func to modify this) is called to handle the error. (null_callback, not surprisingly, does nothing and returns the error unchanged.) If the callback returns 0 then the error is cleared and processing continues; otherwise processing stops and the error is returned.
Now suppose you pass a certificate chain in there along with the initial certificate? In this case you have called X509_STORE_CTX_init() with a stack of X509 as the last argument instead of NULL.
In this case, X509_verify_cert creates a stack of certificates, puts the certificate to be verified on it, makes a temp copy of the cert chain which we use for all work for the rest of this routine.
If the stack has more certs than the depth (default depth is 10), skip the next step; likewise, if the cert is self-signed, skip the next step.
Otherwise, we have a cert chain so we look in it for the issuer cert. If we find one, put it on the stack and remove it from our temp chain. Make that the current working certificate. We go back and repeat until the chain is empty or we don't find one of the issuer certs.
In theory the cert chain should have been a chain of issuers, so the whole chain should have been pushed onto the stack.
Now we get the last cert on stack and see if it's self=signed; if so and it's the only one in the chain we flag an error, and call the verify callback, If the callback doesn't like it either we then return the error.
In our case there is a chain, presumably containing more than one cert, and we expect that chain to have made it on the stack, so we don't have the error.
Instead we pop the self-signed cert off the stack and save it elsewhere for later processing if needed.
Now we enter a loop:
Now we enter a loop: break if number of certs on stack is greater than depth, break if cert is self-signed, and if neither of these, try to get the issuer cert from our trusted list. If we can't find it, break. Otherwise, put it on the stack and go around again.
After exiting the loop we have a chain of certs that either ends early because depth exceeded, ends early because we couldn't find an issuer, or ends with a self-signed cert.
In the first two cases we will flag an error that we couldn't find the issuer; in the third case we call
internal_verify
and we are done, just as before.