Calling openssl from x64 application

[KeesVer]

Hello,

I'm calling openssl from Delphi (10.3.1) using the indy libraries. This works perfectly when using a 32 bit application. However when the application is compiled to 64 bits, I get an access violation on this line: rc := EVP_PKEY_set1_RSA(@key, FRSA);

I use dll's copied from https://indy.fulgan.com/SSL/

Any idea what is causing this?

I have attached a test application but you'll need a private key to use it (I can send a temporary key when needed)

Thanks,

Kees

Code:

function TOAuth1SignatureMethod_RSA_SHA1.Hash_HMAC_SHA1(const AData, AKey: string): string;
var
 buffer: TBytes;
 clean_key: string;
 mdLength: TIdC_UInt;
 mdctx: EVP_MD_CTX;
 outbuf: TBytes;
 KeyBuffer: pBIO;
 FRSA: PRSA;
 md: PEVP_MD;
 rc: Integer;
 key: EVP_PKEY;
 p: Integer;

begin
 if not IdSSLOpenSSL.LoadOpenSSLLibrary then
   raise Exception.Create('LoadOpenSSLLibrary failed');

 // Load private key (key must include header and footer)
 //-----BEGIN PRIVATE KEY-----
 // ....
 // ....
 //-----END PRIVATE KEY-----

 p := AKey.IndexOf('-----END PRIVATE KEY-----');
 if p < 0 then
   raise Exception.Create('Private key error');

 clean_key := AKey.Substring(0, p + Length('-----END PRIVATE KEY-----'));
 buffer := TEncoding.ANSI.GetBytes(clean_key);
 KeyBuffer := BIO_new_mem_buf(buffer, Length(buffer));

 if KeyBuffer = nil then
   raise Exception.Create('RSA out of memory');

 try
   FRSA := PEM_read_bio_RSAPrivateKey(KeyBuffer, nil, nil, nil);
   if not Assigned(FRSA) then
     raise Exception.Create('Private key error');
 finally
   BIO_free(KeyBuffer);
 end;

 md := EVP_get_digestbyname('SHA1');

 rc := EVP_DigestInit(@mdctx, md);
 if rc <> 1 then
   raise Exception.Create('EVP_DigestInit failed: ' + rc.ToString);

 rc := EVP_PKEY_set1_RSA(@key, FRSA); <== Exception occurs here
 if rc <> 1 then
   raise Exception.Create('EVP_PKEY_set1_RSA failed: ' + rc.ToString);

 rc := EVP_DigestSignInit(@mdctx, nil, md, nil, @key);
 if rc <> 1 then
   raise Exception.Create('EVP_DigestSignInit failed: ' + rc.ToString);

 buffer := TEncoding.ANSI.GetBytes(AData);
 rc := EVP_DigestSignUpdate(@mdctx, buffer, Length(buffer));
 if rc <> 1 then
   raise Exception.Create('EVP_DigestSignUpdate failed: ' + rc.ToString);

 rc := EVP_DigestSignFinal(@mdctx, nil, @mdLength);
 if rc <> 1 then
   raise Exception.Create('EVP_DigestFinal failed: ' + rc.ToString);

 SetLength(outbuf, mdLength);
 rc := EVP_DigestSignFinal(@mdctx, PIdAnsiChar(@outbuf[0]), @mdLength);
 if rc <> 1 then
   raise Exception.Create('EVP_DigestFinal failed: ' + rc.ToString);

 //EVP_Cleanup();
 Result := TNetEncoding.Base64.EncodeBytesToString(outbuf);
 Result := Result.Replace(#13#10, '', [rfReplaceAll]);
end;

[rlebeau]

Code:

 rc := EVP_PKEY_set1_RSA(@key, FRSA); <== Exception occurs here

Check to make sure that the EVP_PKEY_set1_RSA function pointer is not nil after calling LoadOpenSSLLibrary():

Code:

if not IdSSLOpenSSL.LoadOpenSSLLibrary then
   raise Exception.Create('LoadOpenSSLLibrary failed');

if not Assigned(EVP_PKEY_set1_RSA) then // <-- here
   raise Exception.Create('EVP_PKEY_set1_RSA is not available');

...

EVP_PKEY_set1_RSA() is not marked as a critical function being used by Indy's own purposes, so if it fails to load from the DLLs, Indy will simply ignore it.

if the EVP_PKEY_set1_RSA pointer is not nil, but you are still getting the AV, then something else is going on.

[KeesVer]

Hello Remy,

Thanks for your reply. 


I added a check on the function pointer and it is not nil. I also checked this line in IdSSLOpenSSLHeaders: @EVP_PKEY_set1_RSA := LoadFunctionCLib(fn_EVP_PKEY_set1_RSA,False); and this proves the function is loaded.

Anything else we can check?

Kees

[rlebeau]

Anything else we can check?

Do you have the same error if you use EVP_PKEY_new() to allocate the EVP_PKEY dynamically instead of declaring it statically on the stack? Eventually, Indy will be updated to support OpenSSL 1.1.x and you will have to do this anyway, since OpenSSL 1.1.0+ now uses opaque structures that must be allocated dynamically, so you may as well prepare for this sooner rather than later.

Try something more like this:

Code:

function TOAuth1SignatureMethod_RSA_SHA1.Hash_HMAC_SHA1(const AData, AKey: string): string;
const
  cPrivateKeyBegin = '-----BEGIN PRIVATE KEY-----';
  cPrivateKeyEnd = '-----END PRIVATE KEY-----';
var
  buffer: TBytes;
  clean_key: string;
  mdLength: TIdC_UInt;
  mdctx: PEVP_MD_CTX;
  KeyBuffer: PBIO;
  FRSA: PRSA;
  md: PEVP_MD;
  key: PEVP_PKEY;
  rc: Integer;
  iBegin, iEnd: Integer;
begin
  if not IdSSLOpenSSL.LoadOpenSSLLibrary then
    raise Exception.Create('LoadOpenSSLLibrary failed');

  // Load private key (key must include header and footer)
  //-----BEGIN PRIVATE KEY-----
  // ....
  // ....
  //-----END PRIVATE KEY-----

  iBegin := AKey.IndexOf(cPrivateKeyBegin);
  if iBegin < 0 then
    raise Exception.Create('Private key error');

  iEnd := AKey.IndexOf(cPrivateKeyEnd, iBegin + Length(cPrivateKeyBegin));
  if iEnd < 0 then
    raise Exception.Create('Private key error');
  Inc(iEnd, Length(cPrivateKeyEnd));

  clean_key := AKey.Substring(iBegin, iEnd - iBegin);
  buffer := TEncoding.ANSI.GetBytes(clean_key);

  md := EVP_get_digestbyname('SHA1');
  if md = nil then
    raise Exception.Create('SHA1 lookup error');

  mdctx := EVP_MD_CTX_create();
  if mdctx = nil then
    raise Exception.Create('MD_CTX out of memory');

  try
    rc := EVP_DigestInit(mdctx, md);
    if rc <> 1 then
      raise Exception.Create('EVP_DigestInit failed: ' + rc.ToString);

    KeyBuffer := BIO_new_mem_buf(PByte(buffer), Length(buffer));
    if KeyBuffer = nil then
      raise Exception.Create('RSA out of memory');

    try
      FRSA := PEM_read_bio_RSAPrivateKey(KeyBuffer, nil, nil, nil);
      if FRSA = nil then
        raise Exception.Create('Private key error');
    finally
      BIO_free(KeyBuffer);
    end;

    try
      key := EVP_PKEY_new();
      if key = nil then
        raise Exception.Create('PKEY out of memory');

      try
        rc := EVP_PKEY_set1_RSA(key, FRSA);
        if rc <> 1 then
          raise Exception.Create('EVP_PKEY_set1_RSA failed: ' + rc.ToString);

        rc := EVP_DigestSignInit(mdctx, nil, md, nil, key);
        if rc <> 1 then
          raise Exception.Create('EVP_DigestSignInit failed: ' + rc.ToString);

        buffer := TEncoding.ANSI.GetBytes(AData);

        rc := EVP_DigestSignUpdate(mdctx, PByte(buffer), Length(buffer));
        if rc <> 1 then
          raise Exception.Create('EVP_DigestSignUpdate failed: ' + rc.ToString);

        rc := EVP_DigestSignFinal(mdctx, nil, @mdLength);
        if rc <> 1 then
          raise Exception.Create('EVP_DigestFinal failed: ' + rc.ToString);

        SetLength(buffer, mdLength);

        rc := EVP_DigestSignFinal(mdctx, PIdAnsiChar(PByte(buffer)), @mdLength);
        if rc <> 1 then
          raise Exception.Create('EVP_DigestFinal failed: ' + rc.ToString);
      finally
        EVP_PKEY_free(key);
      end;
    finally
      RSA_free(FRSA);
    end;
  finally
    EVP_MD_CTX_destroy(mdctx);
  end;

  Result := TNetEncoding.Base64.EncodeBytesToString(PByte(buffer), mdLength);
  Result := Result.Replace(#13#10, '', [rfReplaceAll]);
end;

BTW, why are you implementing an HMAC SHA1 hash manually? Indy does have a TIdHMACSHA1 class available for that (as well as a TIdEncoderMIME class for encoding data to base64, without line breaks). OpenSSL also has its own HMAC functions, too.

[KeesVer]

Hello again,

Thanks, that did it, after allocating the key through EVP_PKEY_new() the call to EVP_PKEY_set1_RSA succeeded. 

Why I choose this path?
I have little knowledge on this subject and when I needed this hash function I could not find any samples that pointed me in the right direction. I tried several Delphi alternatives but they did not give the right results. I did find a working C# example a decided to convert it to Delphi.

Maybe you can help out here?

If I do this:

  ih := TIdHMACSHA1.Create;
  ih.Key := IndyTextEncoding_8Bit.GetBytes(clean_key);
  indy_buf := ih.HashValue(IndyTextEncoding_8Bit.GetBytes(AData));
  indy_hash := TNetEncoding.Base64.EncodeBytesToString(indy_buf);

this hash value is calculated: 'SFYyolkv4nUPbzKoFOmHuPaDQE4='

The hash value should be: 
'jSXIuYXqNb23f0RHybCpYWlzNAfABN1ie2hGPA7Cj0svoXX2sLaTZdTt11nb8DieULXivftOzxKDPkWjGiJJQsxUaPpltsfYOCg/jSeNEo+clfRpIP3K4UGXDjCQNwV+WNCkggA9S87oEylUBPOdShibOAC1oMocAO/B9SOZm/o='

What is causing this difference? Do I need to setup some additional parameters?

Thanks Kees

[rlebeau]

  ih.Key := IndyTextEncoding_8Bit.GetBytes(clean_key);

That will not work. Your clean_key contains a base64-encoded string, but the TIdHMAC.Key property expects actual bytes instead. You will have to decode the base64 string first.

  indy_buf := ih.HashValue(IndyTextEncoding_8Bit.GetBytes(AData));

What does your AData actually look like? If it contains non-ASCII characters, 8Bit is likely to lose them. Typically, UTF8 is the preferred encoding when hashing string data.

  indy_hash := TNetEncoding.Base64.EncodeBytesToString(indy_buf);

Like I said earlier, Indy has a base64 encoder:

Code:

indy_hash := TIdEncoderMIME.EncodeBytes(indy_buf);

this hash value is calculated: 'SFYyolkv4nUPbzKoFOmHuPaDQE4='

The hash value should be: 
'jSXIuYXqNb23f0RHybCpYWlzNAfABN1ie2hGPA7Cj0svoXX2sLaTZdTt11nb8DieULXivftOzxKDPkWjGiJJQsxUaPpltsfYOCg/jSeNEo+clfRpIP3K4UGXDjCQNwV+WNCkggA9S87oEylUBPOdShibOAC1oMocAO/B9SOZm/o='

That is WAY TOO LONG for a valid HMAC-SHA1 hash, even when encoded in base64. SHA-1 produces a 20-byte hash, and HMAC-SHA1 simply invokes SHA-1 multiple times but the result is still 20 bytes. The "expected" base64 you have shown above decodes to 128 bytes. The "Indy" base64 you have shown above decodes to 20 bytes. Where are you getting this "expected" hash from exactly?

[KeesVer]

Hello,

I can't say why the hash calculated using OpenSSL is different. All I know is that it works. 

I updated my test application so that I have an OpenSLL and Indy version of the hashing algorithm (see file OAuth1SignatureMethod_RSA_SHA1). The hash being calculated is very different and the Indy hash is not working (Jira returns an HTTP 500).

I built this application based on this manual: https://developer.atlassian.com/cloud/ji...ntication/

If you want you can check it out.

If you do, please compile this application (I use Delphi 10.3.1), then press the button labeled '#1: Get Request-Token and Auth-Code'. This should open the login window for Jira.

You can also check 'Use indy' and then try again. This will result in a Http 500 exception.

Thanks Kees

[rlebeau]

I can't say why the hash calculated using OpenSSL is different. All I know is that it works. 

Like I said, HMAC-SHA1 by itself cannot produce such a long hash. Are you perhaps actually showing the COMPLETE OAuth signature after EVERYTHING is hashed and concatenated together? That is a very different thing than just hashing individual values by themselves. Start with hashing 1 value and make sure it hashes correctly, then another value, and another as needed until you read the final result. Find a decent OAuth example that shows you what the intermediate hashes should be at each step, not just what the input and final result are.

I updated my test application so that I have an OpenSLL and Indy version of the hashing algorithm (see file OAuth1SignatureMethod_RSA_SHA1). The hash being calculated is very different and the Indy hash is not working (Jira returns an HTTP 500).

Well, considering that Indy uses OpenSSL internally by default, the hashes should be identical. Which likely means either you are not actually hashing the same input data, or you are not using the hash algorithms correctly.

I built this application based on this manual: https://developer.atlassian.com/cloud/ji...ntication/

If you want you can check it out.

If you do, please compile this application (I use Delphi 10.3.1), then press the button labeled '#1: Get Request-Token and Auth-Code'. This should open the login window for Jira.

You can also check 'Use indy' and then try again. This will result in a Http 500 exception.

Internally, TIdHMACSHA1 uses OpenSSL's HMAC functions (specifically, HMAC_CTX_init(), HMAC_Init_ex(EVP_sha1), HMAC_Update(), and HMAC_Final()), rather than the Digest Signing functions that your OpenSSL code is using. If you use the HMAC functions in your code, are you able to get a result that Jira accepts?