RSA Key Exchange with Windows Crypto API and OpenSSL Part 1


Microsoft Crypto API (CAPI) was first released with the Windows NT4 operating system in 1996. The OpenSSL project, which was originally a fork of SSLeay by Eric Young and Tim Hudson, was initiated in 1998 and has since become one of the most widely distributed cryptographic libraries available.

I recently required a Windows application using CAPI that can sign and verify files using the RSA digital signature algorithm, but It needed to read RSA keys and signatures generated by OpenSSL. The keys are stored using Abstract Syntax Notation One (ASN.1) and Privacy Enhanced Mail (PEM) format. The signatures are stored in binary using big-endian convention.

In this post, we’ll focus specifically on RSA key generation, the importation/exportation of RSA keys and the key management standards used to exchange keys in a platform independent manner.

I understand RSA is being phased out in favor of Elliptic Curve Cryptography (ECC) which may be discussed in a future post. I’m also aware ECC will eventually be phased out in favor of quantum resistant cryptography which is still under a lot of research and development, but that could be 10+ years away and RSA still offers good security margin, albeit with less efficiency.

Part 2 will focus specifically on generation and exchange of session keys over TCP for symmetric encryption, but the bulk of work needed to reach that stage is really within this post.

Key Management Standards

The 2 main issues developers appear to complain about for interoperability between OpenSSL and Microsoft Crypto API are:

  1. Signatures exported by OpenSSL functions use big-endian convention, Microsoft Crypto API uses little-endian
  2. Public and Private Keys exported by OpenSSL functions use ASN.1 structures, Microsoft CryptoAPI use their own structures or what are referred to as “Blobs”

However, both API support the following standards for public key management:

We just need to import and export keys using these standard formats to successfully exchange keys between the 2 API.

Then there’s the issue of textual encoding using Privacy Enhanced Mail (PEM) format which is defined in RFC1421, RFC1422 and RFC1423.

Essentially, this encoding uses the base64 algorithm.

RSA Digital Signatures and RSA Key Exchange

When signing a file, we derive a cryptographic hash from its data. This hash is then encrypted using an RSA private key and modular exponentiation. The resulting ciphertext is called a signature. Verification of the signature involves decryption using an RSA public key and Modular Exponentiation.

When exchanging session keys, the client side will generate a value derived from a cryptographic pseudo-random number generator (CSPRNG). This value will be used as the symmetric encryption key. It’s then encrypted using an RSA public key and modular exponentiatation before being sent to a remote server. The server will perform RSA decryption using the private key to recover the same session key.

Byte Order

As stated already, OpenSSL exports signatures using the Big-Endian convention whereas Microsoft Crypto API uses Little-Endian.

High end servers and mainframes in the 80s and 90s used Big-Endian architectures like SPARC, MIPS and POWER. The legacy of this are many cryptography libraries using Big-Endian convention to store data on disk.

To accomodate this on Windows which predominantly runs on X86 architecture, we use the following piece of code to swap the order of bytes after signing and before verification.

Then we have no problem verifying signatures generated by OpenSSL.

RSA key context

The following structure is defined to hold RSA keys.

RSA Key Generation

CAPI uses 65537 as the public exponent in key generation so we need to use the same for OpenSSL.

Reading and writing PEM files

Before using the CAPI functions, we need to decode the PEM files into ASN.1 encoded structures. The CryptStringToBinary and CryptBinaryToString APIs can convert to and from PEM, however these were only made available since Windows XP and Windows 2003.

See PEM_read_file and PEM_write_file functions for more details.

Importing public and private keys

  • Crypto API

For the Public key, decode the ASN.1 structure into a Public Key Info structure before importing to CAPI key object using CryptImportPublicKeyInfo API.

Private Key, decode the ASN.1 structure into a Private Key Info structure. Convert the PrivateKey value into a CAPI Private Key Blob before importing into a CAPI key object.

Unfortunately, there’s no CryptImportPrivateKeyInfo API, hence the extra call to CryptDecodeObjectEx.

  • OpenSSL

OpenSSL offers a much simpler solution with a single API call for both private and public keys. We also don’t have to decode the PEM format before hand. Nice, eh?

Exporting keys

  • Crypto API

Since 2000, we can use CryptExportPKCS8 to export the private key.

Exporting the public key using CryptExportPublicKeyInfo before encoding with ASN.1.

  • OpenSSL

Signing a file

  • Crypto API

  • OpenSSL

Verifying signature

  • Crypto API

  • OpenSSL

RSA Tool Usage

  • Key Generation


  • Signing a file

  • Verifying signature


The purpose of this post was to cover the main problems of key exchange between OpenSSL and Microsoft Crypto API.

For symmetric key exchange, so long as we use ASN.1 encoding for the exchange of public and private keys and remember that OpenSSL uses Big-Endian convention instead of Little-Endian by CAPI, there isn’t a significant problem.

In part 2, we’ll examine how to perform End-To-End encryption of network traffic between a windows machine using Crypto API and Linux using OpenSSL.

Source code for the RSA tool can be found here

This entry was posted in crypto api, cryptography, openssl, programming, security, windows and tagged , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s