PEM: Privacy Enhanced Mail

Privacy Enhanced Mail (Privacy-enhanced Electronic-Mail) is a proposed IETF standard for securing electronic mail using public-key cryptography.  The .pem file extension is used to signify a certificate file that is base64 encoded.  This means that the files are encoded in ASCII and are readable in a text editor.  This is in contrast to the .crt file extension; this is used to signify binary certificate files.  PEM files wrap base64 encoded certificate in BEGIN/END markers that specify the type of certificate encoded between the makers.

PEM BEGIN/END Markers

BEGIN/END markers in PEM files start with the string “—–“ followed by “BEGIN” or “END” and then a marker name representing the type of certificate between the markers.  These markers are not standardized, but OpenSSL v1.68 defined the following markers in crypto/pem/pem.h:

"X509 CERTIFICATE", "CERTIFICATE", "CERTIFICATE PAIR", 
"TRUSTED CERTIFICATE","NEW CERTIFICATE REQUEST", "CERTIFICATE REQUEST", 
"X509 CRL", "ANY PRIVATE KEY", "PUBLIC KEY", "RSA PRIVATE KEY", 
"RSA PUBLIC KEY", "DSA PRIVATE KEY", "DSA PUBLIC KEY", "PKCS7", 
"PKCS #7 SIGNED DATA", "ENCRYPTED PRIVATE KEY", "PRIVATE KEY", 
"DH PARAMETERS", "SSL SESSION PARAMETERS", "DSA PARAMETERS", 
"ECDSA PUBLIC KEY", "EC PARAMETERS", "EC PRIVATE KEY”, "CMS"

PEM Certificate

A PEM Certificate is a base64 encoded certificate, in the format described by its marker tags.  The cryptographic definition of a public key certificate is an electronic document that uses a digital signature to bind a public key with an identity.  This identity could contain the name of a person or organization, an address, etc.  Certificates are used to verify that a public key belongs to a specific individual or organization.

X.509 Certificate

X.509 is an ITU Telecommunication Standardization Sector (ITU-T) standard for a public key infrastructure (PKI) and Privilege Management Infrastructure (PMI).  X.509’s specifications include, but are not limited to, standard formats for public key certificates, certificate revocation lists, attribute certificates, and a certification path validation algorithm.  The Internet Engineering Task Force (IETF) has a working group Public Key Infrastructure (X.509), more commonly known as PKIX, and the terminology X.509 Certificate usually refers to and IETF PKIX certificate, as well as the CRL Profile of the X.509 v3 certificate standard.  This standard is specified in RFC 5280.

5.509 Certificate Syntax

The X.509 v3 digital certificate contains the following fields, described using Abstract Syntax Notation One (ASN.1):

  • Certificate
    • Version
    • Serial Number
    • Algorithm ID
    • Issuer
    • Validity
      • Not Before
      • Not After
    • Subject
    • Subject Public Key Info
      • Public Key Algorithm
      • Subject Public Key
    • Issuer Unique Identifier (optional)
    • Subject Unique Identifier (optional)
    • Extensions (optional)
  • Certificate Signature Algorithm
  • Certificate Signature

Common X509 Filename Extensions

  • .pemPrivacy Enhanced Mail:  This is a base64 encoded DER certificate
  • .cer, .crt, .der– These are usually in binary DER form, but are sometimes also base64 encoded.
  • .p7b, .p7c – These are a SignedData structure with no data except one or more certificates or one or more certificate revocation lists.  These are signed with PKCS#7.
  • .p12 – This is a password protected certificate and exchanged public and private objects in a single file using PKCS#12.
  • .pfx – PFX is a predecessor of PKCS#12.  It usually is generated by IIS and contains data in the PKCS#12 format.

Relevant Algorithms

  • PKCS#7 – Cryptographic Message Syntax Standard: This is based on the PEM standard.  The Cryptographic Message Standard (CMS) is usually used to sign or encrypt messages under a PKI.    This is also used for certificate dissemination, and also commonly used for single sign-on operations.
  • PKCS#12 – Personal Information Exchange Syntax Standard:  PKCS#12 is an algorithm published by RSA Laboratories.  It is used to store X.509 private keys with accompanying public key certificates.  It protects the contents of the file with a password-based symmetric key.  As of now, it is the only standard way to store private keys and certificates in a single encrypted file.

The XML Signature: Syntax and Usage

The W3C XML Signature defines XML syntax for demonstrating the authenticity of a digital message or document.  If a digital signature is deemed to be a valid signature, it is highly likely that the message was created by a known sender and that the message has not been altered in transit. Therefore, XML Signatures provide a good method to detect a forgery or tampering of important XML data.

XML Signature Definition

XML Signatures are defined in the W3C recommendation XML Signature Syntax and Processing.  The W3C recommendation document defines different resources that an XML signature may be applied to; this article only discusses the XML Signature in context of an enveloped signature in XML data.

An Enveloped XML signature is an XML signature that is computed over XML content that contains the signature as an element.  The XML content provides a root XML document element, which contains the signature element somewhere along the XML tree.  According to the W3C recommendation, the XML signature should not take itself into account when computing the signature value.  Note that there is a difference between an enveloped signature as described above, and an enveloping signature.  In an enveloping signature, the signature is the parent object.

XML Signature Requirements

The requirements for XML signatures can be found in the W3C working draft XML-Signature Requirements.  Some important aspects of the requirements are as follows:

  • XML Signatures are generated from a hash over the canonical form of a signature manifest.
  • The signature manifest is a collection of references to the objects being signed.
  • An XML Signature must be an XML Element.
  • The XML Signature must preserve the XML Tree, except for the addition of the XML Signature Element.

Aside from enveloped XML signatures, W3C requires that detached XML signatures are possible (the signature is not part of the XML tree it is signing), but those are outside of the scope of this document.

XML Signature Syntax

XML Signatures are represented by the Signature XML element.  The Signature element contains exactly one SignedInfo element, and has zero or one ID attributes.  The SignedInfo element contains exactly one CanonicalizationMethod element and exactly one SignatureMethod element.  The SignatureMethod element contains one or more Reference elements.  Each Reference element has zero or one URI attributes.  Each Reference element can contain zero or one Transforms element, and must contain exactly one DigestMethod element, and exactly one DigestValue element.  The Signature element must contain exactly one SignatureValue element.  The Signature element can contain zero or one KeyInfo elements, and zero or more Object elements.  Each Object element has zero or one ID attributes.  Note that the Object element is typically used for enveloping signatures, not enveloped signatures.

A breakdown of the XML Signature XML Tree is described by the following diagram.   The following description of the above is taken from the W3C recommendation for XML Signatures.  In the description “?” denotes zero or one occurrence; “+” denotes one or more occurrences; and “*” denotes zero or more occurrence.

  <Signature ID?> 
     <SignedInfo>
       <CanonicalizationMethod/>
       <SignatureMethod/>
       (<Reference URI? >
         (<Transforms>)?
         <DigestMethod>
         <DigestValue>
       </Reference>)+
     </SignedInfo>
     <SignatureValue> 
    (<KeyInfo>)?
    (<Object ID?>)*
   </Signature>

XML Signature Verification

When an XML Signature is created, the SignedInfo element is the element that is actually signed.  XML Signature validation requires two steps, which are as follows:

  1. Signature Validation – The value of the element SignatureValue must match the result of processing SignedInfo with the CanonicalizationMethod and SignatureMethod.
  2. Reference Validation – All content identified by a Reference object must be transformed if applicable, then processed with the specified DigestMethod.  The result must match the DigestValue for that reference.

These steps can be performed in either order, but for the sake of understanding and convenience, we will delve into Reference Validation first.

Reference Syntax

Each Reference element contains zero or one URI attributes.  The Schema definition for Reference elements also contains optional attributes Id and Type, but we will not consider those in this document.  The URI attribute identifies a data object using a URI-Reference.  The value of this attribute can be any URI, where URI syntax is dictated by RFC 3986.  The URI can also be in the form of an XPointer, which is both a common and useful tool when dealing with enveloped XML Signatures.

Defining a URI with an XPointer

The XPointer is a fragment identifier for a URI described by W3C in XML Pointer Language (XPointer) Version 1.0.  A fragment identifier is a short string of characters that refers to a resource that is subordinate to another resource.  According to the XPointer specification version 1.0, an XPointer can act as a fragment identifier for resources with the following Internet media types:

  • text/xml
  • application/xml
  • text/xml-external-parsed-entity
  • application/xml-external-parsed-entity

XPointer is based on the XML Language Path (XPath) and can be used to address internal structures of XML documents.  XPath is described by W3C in XML Path Language (XPath) Version 1.0.

The character “/” is used to represent the root of an XML document.  Since an enveloped XML Signature is usually a Signature over the entire XML document containing the signature, it is common to see a Reference element definition such as:

<ds:Reference URI=”#xpointer(/)”>… </ds:Reference>

You’ll notice that in the previous example the Reference element has a prefix of “ds:”.  This means that the XML Namespace ds contains the element Reference.  XML Namespaces are described in Namespaces in XML 1.0 (Third Edition).  XML Namespaces provide a way to qualify element and attribute names in XML by associating them with namespaces identified by URI resources.  The use of the namespaces ds  or dsig are common, but optional.

Reference Transforms

The Transforms element is optional, and contains an ordered list of Transform elements.  The Transform elements describe how the data object was created before being digested.  The output of each Transform serves as the input to the next Transform.  The input to the first transform is the dereferenced URI attribute of the Reference element.  Transform elements consist of an Algorithm attribute that specifies a URI for the algorithm.  Depending on the algorithm, the content of the Transform element can provide additional data to determine how the algorithm processes the data.

DigestMethod

A Reference element must contain a DigestMethod element.  The required attribute Algorithm specifies a URI to an algorithm that is used to compute the DigestValue over the data object that is the result of the last transform.  If no transforms were performed, the DigestValue is computed over the dereferenced URI of the Reference element.  DigestValue is also a required element.  The DigestValue element contains the encoded value of the digest.  The digest is always encoded using base64.

Reference Validation

Reference validation is the first step of core validation in XML Signature verification.  W3C breaks reference validation into the following ordered steps:

  1. Canonicalize the SignedInfo element based on the Canonicalization Method in SignedInfo.
  2. For each Reference in SignedInfo:
    1. Obtain the data object to be digested.  This involves dereferencing the Reference object’s URI and applying transforms as necessary.
    2. Digest the resulting data object using the DigestMethod specified in the Reference section.
    3. Compare the generated digest value against DigestValue in the Reference.  If the generated digest does not match, validation fails.

Signature Syntax

The Signature element must contain a SignatureValue element.  The SignatureValue element contains the base64 encoded signature computed over the SignedInfo element.  The SignedInfo element must contain a single CanonicalizationMethod element and a single SignatureMethod element.  Both the CanonicalizationMethod element and the SignatureMethod element have the required attribute Algorithm which represents a URI to the algorithm used for canonicalization and signing, respectively.

Signature Validation

The W3C breaks signature validation into the following steps:

  1. Obtain the key information, either from a KeyInfo element or from an external source.  Documentation for the KeyInfo element is available from W3C.
  2. Obtain the canonical form of the SignatureMethod using the CanonicalizationMethod and use the result (and previously obtained KeyInfo) to confirm the SignatureValue over the SignedInfo element.

Verification

After Reference validation and signature validation are completed, the XML Signature has been verified.  If either of these two steps fails, the signature is not verified.  Verification implies that you can be sure beyond a reasonable doubt that the xml originated from a valid source.  Hopefully now you know a little more about XML Signatures.  Happy Security!

3 Line RSA in perl

The following 3 line implementation of RSA in perl is pretty cool:

#!/bin/perl -sp0777i<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<j]dsj
$/=unpack('H*',$_);$_=`echo 16dio\U$k"SK$/SM$n\EsN0p[lN*1
lK[d2%Sa2/d0$^Ixp"|dc`;s/\W//g;$_=pack('H*',/((..)*)$/)

You can get a up-til-now comprehensive overview of the perl hacker community’s work on decreasing the size of this implementation at http://www.cypherspace.org/rsa/story.html.

Rest in peace Peter Junger!