TIdMessage.LoadFromStream: Missing fields

[rlebeau]

// BART
// Result := IOHandler.ReadLnRFC(LMsgEnd);
result := IOHandler.ReadLn; // Bart: I called ReadLn direct for testing, but not the problem.

TIdIOHandler.ReadLnRFC() simply calls TIdIOHandler.ReadLn(), and then sets LMsgEnd=True if the line is '.', otherwise it sets LMsgEnd=False and strips off the 1st '.' if the line begins with '..'.  This is the decoding process for handling dot-transparency in emails.

And now: Where is the FInputBuffer data loaded? I may be blind but I do not see it.
The Stream is loaded into the IoHandler; But how is the FInputbuffer loaded??

Whenever a TIdIOHandler reading method, such as ReadLn(), needs bytes that are not already buffered in the InputBuffer, TIdIOHandler.ReadFromSource() is called to read whatever bytes are available from its data source and put them into the InputBuffer, repeating as needed until the read operation can finish its work.

TIdIOHandler.ReadFromSource() calls TIdIOHandler.ReadDataFromSource(), which TIdIOHandlerStreamMsg overrides to read from its FReceiveStream:

Code:

function TIdIOHandlerStreamMsg.ReadDataFromSource(var VBuffer: TIdBytes): Integer;
var
  LTerminator: String;
begin
  if not FTerminatorWasRead then
  begin
    Result := inherited ReadDataFromSource(VBuffer);

Code:

function TIdIOHandlerStream.ReadDataFromSource(var VBuffer: TIdBytes): Integer;
begin
  // We dont want to read the whole stream in at a time. If its a big
  // file will consume way too much memory by loading it all at once.
  // So lets read it in chunks.
  if Assigned(FReceiveStream) then begin
    Result := IndyMin(32 * 1024, Length(VBuffer));
    if Result > 0 then begin
      Result := TIdStreamHelper.ReadBytes(FReceiveStream, VBuffer, Result);
    end;
  end else begin
    Result := 0;
  end;
end;

Code:

class function TIdStreamHelperVCL.ReadBytes(const AStream: TStream; var VBytes: TIdBytes;
  const ACount, AOffset: Integer): Integer;
var
  LActual: Integer;
begin
  Assert(AStream<>nil);
  Result := 0;

  if VBytes = nil then begin
    SetLength(VBytes, 0);
  end;
  //check that offset<length(buffer)? offset+count?
  //is there a need for this to be called with an offset into a nil buffer?

  LActual := ACount;
  if LActual < 0 then begin
    LActual := AStream.Size - AStream.Position;
  end;

  //this prevents eg reading 0 bytes at Offset=10 from allocating memory
  if LActual = 0 then begin
    Exit;
  end;

  if Length(VBytes) < (AOffset+LActual) then begin
    SetLength(VBytes, AOffset+LActual);
  end;

  Assert(VBytes<>nil);
  Result := AStream.Read(VBytes[AOffset], LActual);
end;

So, in this case, TIdMessage.LoadFromStream() creates a TIdMessageClient with a TIdIOHandlerStreamMsg as its IOHandler, which reads data from the input TStream. Thus, when TIdMessageClient.ReceiveHeader() calls IOHandler.ReadLn(), it will keep reading raw bytes from the TStream until a LF line break is read and buffered in the InputBuffer, then it will extract the whole line from the InputBuffer.

The FInputBuffer is only 5 bytes big.

If TIdIOHandlerStreamMsg.ReadDataFromSource() fails to read from the TStream, ie end-of-stream is reached, it will return a final email terminator (<CRLF>.<CRLF>), just in case, which will force TIdIOHandler.ReadLnRFC() to return LMsgEnd=True if it hasn't already:

Code:

function TIdIOHandlerStreamMsg.ReadDataFromSource(var VBuffer: TIdBytes): Integer;
var
  LTerminator: String;
begin
  if not FTerminatorWasRead then
  begin
    Result := inherited ReadDataFromSource(VBuffer);
    if Result > 0 then begin
      FLastByteRecv := VBuffer[Result-1];
      Exit;
    end;
    // determine whether the stream ended with a line
    // break, adding an extra CR and/or LF if needed...
    if (FLastByteRecv = Ord(LF)) then begin
      // don't add an extra line break
      LTerminator := '.' + EOL;
    end else if (FLastByteRecv = Ord(CR)) then begin
      // add extra LF
      LTerminator := LF + '.' + EOL;
    end else begin
      // add extra CRLF
      LTerminator := EOL + '.' + EOL;
    end;
    FTerminatorWasRead := True;
    // in theory, CopyTIdString() will write the string
    // into the byte array using 1-byte characters even
    // under DotNet where strings are usually Unicode
    // instead of ASCII...
    CopyTIdString(LTerminator, VBuffer, 0);
    Result := Length(LTerminator);
  end else begin
    Result := 0;
  end;
end;

[12:18:34/555] D3: IdMessageClient.ProcessMessage...
[12:18:34/571] D3: IdIOHandler.ReadLn: FInputBuffer.Size=5
[12:18:34/571] D3: IdIOHandler.ReadLn: length(LResult)=2
[12:18:34/586] D3: IdIOHandler.ReadLn: Line okay
[12:18:34/586] D3: IdIOHandler.ReadLn: Result=

This implies that the very 1st call to IOHandler.ReadLn() in TIdMessageClient.ReceiveHeader() is failing to read any bytes at all from the TStream, causing TIdIOHanderStreamMsg to return its generated terminator (the 5 bytes you are logging), which ReadLn() will then extract from the InputBuffer up to the 1st CRLF (the 2 bytes you are logging) and then strip off the CRLF from the Result (the blank string you are logging).  TIdMessageClient.ReceiveHeader() will then interpret that as end-of-headers, causing TIdMessageClient.ReceiveHeaders() to exit so TIdMessageClient.ProcessMessage() will then call TIdMessageClient.ReceiveBody() next.

However, it is concerning that the input TStream would fail to read any bytes at all, when it clearly has bytes available, according to your logging of its Size.  Any such failure would be happening outside of Indy's control.

Which goes right back to my original question to you 2 weeks ago - when you save a TIdMessage to a TStream and then load it back, are you remembering to reset the TStream.Position back to 0 in between?  If you are loading from a file instead, then that is a non-issue, since file streams always start at Position 0.

You will have to debug the TStream itself to find out what is going on with it.

[12:18:34/624] D3: IdIOHandler.ReadLn: FInputBuffer.Size=3
[12:18:34/624] D3: IdIOHandler.ReadLn: length(LResult)=3
[12:18:34/640] D3: IdIOHandler.ReadLn: Line okay
[12:18:34/640] D3: IdIOHandler.ReadLn: Result=.

This is from TIdMessage.ReadBody() calling IOHandler.ReadLn(), extracting the 2nd half of TIdIOHandlerStreamMsg's generated terminator from the InputBuffer, which TIdMessageClient.ReceiveBody() will then interpret as end-of-message.