RSA Key Exchange with Windows Crypto API and OpenSSL Part 1

Introduction

Microsoft Crypto API (CAPI) was first released with the Windows NT4 operating system in 1996. The OpenSSL project, that was originally a fork of SSLeay by Eric Young and Tim Hudson, was initiated in 1998 and has since become one of the most widely distributed cryptographic libraries available.

I recently required a Windows application using CAPI that can sign and verify files using the RSA digital signature algorithm, but it needed to read RSA keys and signatures generated by OpenSSL. The keys are stored using Abstract Syntax Notation One (ASN.1) and Privacy Enhanced Mail (PEM) format. The signatures are stored in binary using big-endian convention.

In this post, we’ll focus specifically on RSA key generation, the importation/exportation of RSA keys and the key management standards used to exchange keys in a platform independent manner. I understand RSA is being phased out in favor of Elliptic Curve Cryptography (ECC) that may be discussed in a future post. I’m also aware ECC will eventually be phased out in favor of quantum resistant cryptography which is still under a lot of research and development, but that could be 10+ years away and RSA still offers good security margin, albeit with less efficiency.

Part 2 will focus specifically on generation and exchange of session keys over TCP for symmetric encryption, but the bulk of work needed to reach that stage is really within this post.

Key Management Standards

The 2 main issues developers appear to complain about for interoperability between OpenSSL and Microsoft Crypto API are:

1. Signatures exported by OpenSSL functions use big-endian convention, Microsoft Crypto API uses little-endian
2. Public and Private Keys exported by OpenSSL functions use ASN.1 structures, Microsoft CryptoAPI use their own structures or what are referred to as “Blobs”

However, both API support the following standards for public key management:

• PKCS #8: Private-Key Information Syntax Standard
• ASN.1 encoding rules: Specification of Basic Encoding Rules (BER), Canonical Encoding Rules (CER) and Distinguished Encoding Rules (DER)

We just need to import and export keys using these standard formats to successfully exchange keys between the 2 API. Then there’s the issue of textual encoding using Privacy Enhanced Mail (PEM) format which is defined in RFC1421, RFC1422 and RFC1423. Essentially, this encoding uses the base64 algorithm.

RSA Digital Signatures and RSA Key Exchange

When signing a file, we derive a cryptographic hash from its data. This hash is then encrypted using an RSA private key and modular exponentiation. The resulting ciphertext is called a signature. Verification of the signature involves decryption using an RSA public key and Modular Exponentiation. When exchanging session keys, the client side will generate a value derived from a cryptographic pseudo-random number generator (CSPRNG). This value will be used as the symmetric encryption key. It’s then encrypted using an RSA public key and modular exponentiatation before being sent to a remote server. The server will perform RSA decryption using the private key to recover the same session key.

Byte Order

OpenSSL exports signatures using the Big-Endian convention whereas Microsoft Crypto API uses Little-Endian. High end servers and mainframes in the 80s and 90s used Big-Endian architectures like SPARC, MIPS and POWER. The legacy of this are many cryptography libraries using Big-Endian convention to store data on disk. To accomodate this on Windows which predominantly runs on X86 architecture, we use the following piece of code to swap the order of bytes after signing and before verification. Then we have no problem verifying signatures generated by OpenSSL.

// used to convert digital signature from big-endian to little-endian
void byte_swap(void *buf, int len) {
    int     i;
    uint8_t t, *p=(uint8_t*)buf;
    
    for(i=0; i<len/2; i++) {
      t = p[i]; 
      p[i] = p[len - 1 - i];
      p[len - 1 - i] = t;
    }
}

RSA key context

The following structure is defined to hold RSA keys.

typedef struct _RSA_CTX_t {
  #ifdef CAPI
    HCRYPTPROV prov;
    HCRYPTKEY  privkey, pubkey;
    HCRYPTHASH hash;
    DWORD      error;
  #else
    EVP_PKEY   *pkey;
  #endif    
} RSA_CTX, PRSA_CTX;

RSA Key Generation

CAPI uses 65537 as the public exponent in key generation so we need to use the same for OpenSSL.

int RSA_genkey(RSA_CTX* ctx, int keyLen) {
    #ifndef CAPI
      BIGNUM *e=NULL;
      RSA    *rsa;
    #endif  
    int ok=0;  
    if (ctx==NULL) return 0;
        
    #ifdef CAPI
      // 1. release public if already allocated
      if (ctx->pubkey != 0) {
        CryptDestroyKey(ctx->pubkey);
        ctx->pubkey = 0;
      }

      // 2. release private if already allocated
      if (ctx->privkey != 0) {
        CryptDestroyKey(ctx->privkey);
        ctx->privkey = 0;
      }

      // 3. generate key pair for digital signatures
      ok = CryptGenKey(ctx->prov, AT_SIGNATURE,
        (keyLen << 16) | CRYPT_EXPORTABLE,
        &ctx->privkey);
    #else
      // 1. initialize public exponent
      BN_dec2bn(&e, "65537");
      
      // 2. create new RSA context
      rsa = RSA_new(); 
      
      // 3. generate key pair for digital signatures
      if (RSA_generate_key_ex(rsa, keyLen, e, NULL)) {
        // 4. create new EVP key context
        if ((ctx->pkey = EVP_PKEY_new()) != NULL) {
          // 5. assign RSA context to EVP key context        
          ok = EVP_PKEY_assign_RSA(ctx->pkey, rsa);
        }
      }
      BN_free(e);    
    #endif
    return ok;  
}

Reading and writing PEM files

Before using the CAPI functions, we need to decode the PEM files into ASN.1 encoded structures. The CryptStringToBinary and CryptBinaryToString APIs can convert to and from PEM, however these were only made available since Windows XP and Windows 2003. See PEM_read_file and PEM_write_file functions for more details.

Importing public and private keys

• Crypto API

For the Public key, decode the ASN.1 structure into a Public Key Info structure before importing to CAPI key object using CryptImportPublicKeyInfo API.

        // 1. convert DER to RSA public key info
        if (CryptDecodeObjectEx(
            X509_ASN_ENCODING | PKCS_7_ASN_ENCODING,
            X509_PUBLIC_KEY_INFO, derData, derLen,
            CRYPT_DECODE_ALLOC_FLAG, NULL,
            &keyData, &keyLen))
        {
          // 2. import public key blob
          ok = CryptImportPublicKeyInfo(ctx->prov, 
             X509_ASN_ENCODING,
            (PCERT_PUBLIC_KEY_INFO)keyData, &ctx->pubkey);

          // 3. release allocated memory
          LocalFree(keyData);
        }

For the Private Key, decode the ASN.1 structure into a Private Key Info structure. Convert the PrivateKey value into a CAPI Private Key Blob before importing into a CAPI key object. Unfortunately, there’s no CryptImportPrivateKeyInfo API, hence the extra call to CryptDecodeObjectEx.

        // 1. convert PKCS8 data to private key info
        if(CryptDecodeObjectEx(
          X509_ASN_ENCODING,
          PKCS_PRIVATE_KEY_INFO,
          derData, derLen,
          CRYPT_DECODE_ALLOC_FLAG,
          NULL, &pki, &pkiLen))
        {
          // 2. convert private key info to RSA private key blob
          if(CryptDecodeObjectEx(
            X509_ASN_ENCODING | PKCS_7_ASN_ENCODING,
            PKCS_RSA_PRIVATE_KEY,
            pki->PrivateKey.pbData, 
            pki->PrivateKey.cbData,
            CRYPT_DECODE_ALLOC_FLAG,
            NULL, &keyData, &keyLen))
          {
            // 3. import private key blob
            ok = CryptImportKey(
              ctx->prov, 
              keyData, keyLen, 0,
              CRYPT_EXPORTABLE, 
              &ctx->privkey);
              
            LocalFree(keyData);
          }
          LocalFree(pki);
        }

• OpenSSL

OpenSSL offers a much simpler solution with a single API call for both private and public keys. We also don’t have to decode the PEM format before hand. Nice, eh?

    FILE *fd = fopen(ifile, "rb");
  
    if (fd != NULL) {
      // private key for signing?
      if (pemType == RSA_PRIVATE_KEY) {
        ctx->pkey = PEM_read_PrivateKey(fd, NULL, NULL, NULL);
      // public key for verifying?  
      } else if (pemType == RSA_PUBLIC_KEY) {
        ctx->pkey = PEM_read_PUBKEY(fd, NULL, NULL, NULL);
      }
      ok = (ctx->pkey != NULL);
      fclose(fd);
    }

Exporting keys

• Crypto API

Since 2000, we can use CryptExportPKCS8 to export the private key.

      // 1. calculate size of PKCS#8 structure
      if(CryptExportPKCS8(
        ctx->prov, AT_SIGNATURE, szOID_RSA_RSA, 
        0, NULL, NULL, &pkiLen))
      {
        pki = malloc(pkiLen);
        
        if(pki != NULL) {
          // 2. export PKCS#8 structure to memory
          ok = CryptExportPKCS8(
                ctx->prov, AT_SIGNATURE, szOID_RSA_RSA, 
                0, NULL, pki, &pkiLen);
          if(ok) {
            // 3. write memory to file in PEM format
            PEM_write_file(RSA_PRIVATE_KEY, 
                ofile, pki, pkiLen);
          }
          free(pki);
        }
      }

Exporting the public key using CryptExportPublicKeyInfo before encoding with ASN.1.

      // 1. get size of public key info
      if (CryptExportPublicKeyInfo(ctx->prov, 
          AT_SIGNATURE,
          X509_ASN_ENCODING | PKCS_7_ASN_ENCODING,
          NULL, &pkiLen))
      {
        // 2. allocate memory
        pki = malloc(pkiLen);

        // 3. export public key info
        if (CryptExportPublicKeyInfo(ctx->prov, 
            AT_SIGNATURE,
            X509_ASN_ENCODING | PKCS_7_ASN_ENCODING,
            pki, &pkiLen))
        {
          // 4. get size of DER encoding
          if (CryptEncodeObjectEx(
            X509_ASN_ENCODING | PKCS_7_ASN_ENCODING,
            X509_PUBLIC_KEY_INFO, pki, 0,
            NULL, NULL, &derLen))
          {
            derData = malloc(derLen);
            if (derData) {
              // 5. convert to DER format
              ok = CryptEncodeObjectEx(
                X509_ASN_ENCODING | PKCS_7_ASN_ENCODING,
                X509_PUBLIC_KEY_INFO, pki, 0,
                NULL, derData, &derLen);

              // 6. write to PEM file
              if (ok) {
                PEM_write_file(RSA_PUBLIC_KEY, 
                    ofile, derData, derLen);
              }
            }
            free(derData);            
          }
        }
      }

• OpenSSL

      FILE *fd = fopen(ofile, "wb");
      if (fd != NULL) {
        if (pemType == RSA_PUBLIC_KEY) {
          ok = PEM_write_PUBKEY(fd, ctx->pkey);
        } else if (pemType == RSA_PRIVATE_KEY) {        
          ok = PEM_write_PKCS8PrivateKey(fd, 
              ctx->pkey, NULL, NULL, 0, NULL, NULL);   
        }
        fclose(fd);
      }  

Signing a file

• Crypto API

• OpenSSL

Verifying signature

• Crypto API

• OpenSSL

RSA Tool Usage

• Key Generation

 

• Signing a file

• Verifying signature

Summary

The purpose of this post was to cover the main problems of key exchange between OpenSSL and Microsoft Crypto API. For symmetric key exchange, so long as we use ASN.1 encoding for the exchange of public and private keys and remember that OpenSSL uses Big-Endian convention instead of Little-Endian by CAPI, there isn’t a significant problem. In part 2, we’ll examine how to perform End-To-End encryption of network traffic between a windows machine using Crypto API and Linux using OpenSSL. Source code for the RSA tool can be found here