TIdSMTPServer - Questions (differences between v9 and v10)

[Justin Case]

Hi,

Moons ago I was writing a smtp server in v9. V9 rocked, the event handlers (via the object inspector) made it nice and easy to use but there was some architectural bug in it involving the dots - from memory it needed two dots on some lines for some reason.. Anyway eventually i setup another machine with v10, had a newborn son, forgot all about it...

9 odd years later.. I'm back on this project..

I'm using 10.2.3

So with the online documentation being down on ww2.indyproject.org, I'm kind of flying blind. To get started, I double clicked all the events in object inspector and put in a ShowMessage() so that i knew what order they fire in (yes I had exceptions regarding drawing - but it got me what i needed after multiple program resets).

So I have these in this order (I'm manually typing these in - can't copy from the VM - sorry for any typo's but you'll get the idea):

procedure TDataModule1.IdSMTPServer1.Connect()
procedure TDataModule1.IdSMTPServer1BeforeCommandHandler()
TDataModule1.IdSMTPServer1Reset()
IDSMTPServer1SPFCheck()
IdSMTPServer1AfterCommandHandler()
IdSMTPServer1UserLogin()
IdSMTPServer1MailFrom()
IdSMTPServerRcptTo()
IdSMTPServer1Received()
IdSMTPServer1MsgReceive()
IdSMTPServer1Disconnect()

That all seems relatively straight forward however I'm having problems with SPFCheck(). My client (Outlook express - yes it's old but ok for testing) sends my machine name (Asuras) which obviously isn't a domain and as it's a SMTP client, not another SMTP server, that obviously won't resolve. I suspect a connecting server will send a domain?

So how do I proceed with this? - obviously if another server connects then the SPFCheck is a great idea but if it's a client then I'm in a fix and it will fail (as a bodge I'm using "If Pos('.', ADomain) then.." to test for a domain). Will IdSMTPServer cause a disconnect / client abort if I use spfNone / spfNeutral or will it just continue?

EG I'm with yahoo. If my email client were to connect to them, it would send "HELO/EHLO Asuras". Based on the indy logic, Asuras means nothing to Yahoo and so it would probably perform a spfCheck against my IP. My IP is part of my ISPs network and thus Yahoo has nothing to pass or fail me with until I login - but in the sequence above, that comes two events later.

Also if a SMTP client connects, it has to send a username and password. How do SMTP servers communicate mail to each other without this? - As the sequence above suggests it seems like a inbound connection has got to login first before it can specify the MailFrom details and trigger that event. How can I know in the UserLogin() event that the connecting user will be sending to a local address in order that I can set VAuthenticated := True ? - those details don't come through until RcptTo()!!

AThread has vanished.. now we have AContext. Whats the difference here? Presumably they still run like threads?

Oh one other thing, whats IdSMTPRelay all about?

Thanks for your time,

JC

[rlebeau]

but there was some architectural bug in it involving the dots - from memory it needed two dots on some lines for some reason..

That is simply how the SMTP protocol works.  Inside the DATA command that the client sends to the server to deliver the email content, any line of text that begins with a dot must be escaped with an extra dot, because the command is terminated by a single dot on a line by itself.  But TIdSMTPServer handles those dots for you, you shouldn't need to deal with them manually on the server side.

I'm using 10.2.3

Why?  That is a very old version.  The current version is 10.6.3.3, which you can download from Indy's GitHub repo.

So with the online documentation being down on ww2.indyproject.org, I'm kind of flying blind.

This blog post has some links to archived copies of the documentation.  However, even so, the documentation was never updated with all of the latest developments in later Indy 10 releases, but the basics should be there.

To get started, I double clicked all the events in object inspector and put in a ShowMessage() so that i knew what order they fire in (yes I had exceptions regarding drawing - but it got me what i needed after multiple program resets).

Like all of Indy's servers, TIdSMTPServer is multi-threaded.  Its events are fired in worker threads, not in the main UI thread.  But ShowMessage() is not thread-safe, so don't use it outside of the main UI thread.  You should instead use the Win32 MessageBox() (which is thread-safe), or better use OutputDebugString() and then view the messages in the IDE's debug log, or in SysInternals DebugView when running your server outside of the IDE.

So I have these in this order (I'm manually typing these in - can't copy from the VM - sorry for any typo's but you'll get the idea):

You are missing a few events in that list (perils of typing manually).

You should be getting OnBeforeCommandHandler and OnAfterCommandHandler events for every command the client sends.

And you should be getting a 2nd OnSPFCheck event before the OnMailFrom event.

But that is besides the point...

My client (Outlook express - yes it's old but ok for testing) sends my machine name (Asuras) which obviously isn't a domain and as it's a SMTP client, not another SMTP server, that obviously won't resolve.

If it were a proper hostname, it should resolve to an actual IP on their ISP.

I suspect a connecting server will send a domain?

A relaying SMTP server should send its own DNS name, just like any other client.

So how do I proceed with this?

You could simply ignore it.  You don't have to implement SPF if you don't want to.  The default action is spfNeutral if you don't assign an OnSPFCheck handler.  Also, Indy does not implement the actual SPF validation itself, so you would have to perform the necessary DNS validation yourself in the OnSPFCheck event.  If you are not prepared to do that, then don't use that event.  Otherwise, you could just validate what you can and reject what you can't, or be neutral about it.

obviously if another server connects then the SPFCheck is a great idea but if it's a client then I'm in a fix and it will fail (as a bodge I'm using "If Pos('.', ADomain) then.." to test for a domain).

Indy 10 has IsDomain() and IsFQDN() functions in the IdGlobalProtocols unit which may help you.

In any case, you could always just reverse-DNS the client's IP address to get its hostname.  You can use Indy's  GStack.HostByAddress() method for that purpose.

Will IdSMTPServer cause a disconnect / client abort if I use spfNone / spfNeutral or will it just continue?

It will continue, as those two values are treated as success cases, as are spfPass and spfSoftFail too.  Only spfFail, spfTempError, and spfPermError will fail the current SMTP command being validated.  The connection will not be closed either way.  The client can send further commands if it wants to, though it will likely disconnect if the validating command fails.

EG I'm with yahoo. If my email client were to connect to them, it would send "HELO/EHLO Asuras". Based on the indy logic, Asuras means nothing to Yahoo and so it would probably perform a spfCheck against my IP. My IP is part of my ISPs network and thus Yahoo has nothing to pass or fail me with until I login - but in the sequence above, that comes two events later.

The first OnSPFCheck event comes during the HELO/EHLO command.  But there is another OnSPFCheck event that comes during the  MAIL FROM command.  So, you could pass the 1st check if you don't have enough information to perform the validation with, and then pass/fail the 2nd check as needed.

Also if a SMTP client connects, it has to send a username and password. How do SMTP servers communicate mail to each other without this?

Relaying between servers does not use authentication.  They rely on SPF/DKIM/DMARC, whitelists/blacklists, secure tunnels, etc to verify each other and block out unwanted senders.  SMTP servers that receive the initial email from users will handle authentication before accepting emails and beginning the relay process.

As the sequence above suggests it seems like a inbound connection has got to login first before it can specify the MailFrom details and trigger that event.

That is not correct.  The only requirement TIdSMTPServer has for accepting a MAIL FROM command is that a previous HELO/EHLO command must have been accepted first.  There is no authentication requirement because of relaying from other servers.

In your case, you are getting the OnUserLogin event triggered because your Outlook client is authenticating itself when sending an email.

Although you can't force authentication, you can have your OnMailFrom event reject the sender when AContext.LoggedIn is False.  Or, you can have your OnSPFCheck event reject the sender when AContext.HELO or AContext.EHLO are True and AContext.LoggedIn is False.

How can I know in the UserLogin() event that the connecting user will be sending to a local address in order that I can set VAuthenticated := True ?

You can't, since that information is not known yet at that time.  The OnUserLogin event only tells you who is authenticating with your server, nothing more.  And the OnMailFrom event only tells you who is sending the email, nothing more.

those details don't come through until RcptTo()!!

Exactly true.  They are stored in the AContext.RCTPList property, which you can then use during email processing in the OnBeforeMsg, OnReceived, and OnMsgReceive events.

AThread has vanished.. now we have AContext. Whats the difference here? Presumably they still run like threads?

Yes, Indy still uses threads.  The data model was decoupled from the threading model in Indy 10 so client data could more easily be passed around between threads, as a client could be serviced by multiple threads, and a thread could service multiple clients. At least, that was the plan, but that proved to be a failed experiment and was later abandoned. In practice, each client is still serviced by 1 thread, and there is 1 TIdContext active per thread (however, threads can now be pooled and reused between connections, at least. Indy 9 did not support that).

Oh one other thing, whats IdSMTPRelay all about?

TIdSMTPRelay is an SMTP client similar to TIdSMTP, except that instead of connecting to a single SMTP server, it can connect to several servers.

TIdSMTP connects to 1 specified server and sends 1 email at a time, specifying all of the recipients of the email to that server.  That server will receive the email and deliver it or relay it to other servers as needed.

TIdSMTPRelay, on the other hand, groups an email's recipients together by their domains, then does a DNS lookup on each domain to discover its registered SMTP server, and then connects to each server and delivers the email to only the recipients of that server.

Basically, TIdSMTP is what an end client would use, and TIdSMTPRelay is what a relaying server would use to forward emails to another server.  Of course, nothing stops an end user from trying to use TIdSMTPRelay directly to bypss their own ISP, but they will likely be blocked by the receiving servers since they are not themselves a validated relay server.

[Justin Case]

Hi Remy,

Thank you (as always) for your brilliant expertise and explanations - you are a gem to the world of programming.

Can I also ask what this pipelining feature / property is all about in IdSMTPServer?

[rlebeau]

Can I also ask what this pipelining feature / property is all about in IdSMTPServer?

Pipelining is a feature of the SMTP protocol that allows a client to send multiple commands at a time, and the server to send multiple responses at a time, instead of the client sending each command individually and waiting for the server to reply before sending the next command. The server still has to reply to each command in order, but the commands and responses can be grouped inside fewer TCP packets. You can simply opt to enable or disable this feature as you want.

[Justin Case]

Ah ok, so not knowing if my incoming connections might use it, should I then enable it just in case? - Is there anything extra I need to do at my end to handle them or does IdSMTPServer break them apart and handle them seperately? (I'm going to assume it probably does)

One other thing, going back to the SPF check, I'm ok with resolving stuff using IdDNSResolver but I noticed last night that the IdServerContext that is passed as a parameter has a SMTPState which is set after each succesful stage.

I've put those states into a case statement and so that I can perform SPF checks in there with each stage. Is that a appropriate way forward or is there a better way?

[rlebeau]

Ah ok, so not knowing if my incoming connections might use it, should I then enable it just in case?

That is up to you to decide for your particular use case. I suggest you read RFC 2920 for how Pipelining actually works and rationales behind it.

Is there anything extra I need to do at my end to handle them or does IdSMTPServer break them apart and handle them seperately? (I'm going to assume it probably does)

Pipelining does not change the format of SMTP traffic, it affects only the transmission at the TCP level. Indy doesn't care if the client pipelines its commands or not, since all incoming data is buffered no matter what. Regardless of whether pipelining is used or not, TIdSMTPServer will read and reply to each command individually, though the responses will be buffered if pipelining is active. The use of pipelining simply reduces how many TCP packets are used back and forth, nothing more.

I noticed last night that the IdServerContext that is passed as a parameter has a SMTPState which is set after each succesful stage.

I've put those states into a case statement and so that I can perform SPF checks in there with each stage. Is that a appropriate way forward or is there a better way?

I thought about that before replying earlier. I suppose you could if you wanted to, since the OnSPFCheck event doesn't tell you which command is being validated (even though the triggerer knows that information). When the OnSPFCheck event is triggered for HELO/EHLO then the current SMTPState should be idSMTPNone, and when triggered for MAIL FROM then the current SMTPState should be idSMTPHelo. Those are the only commands that SPF is used for, since they are validating sender identification.

[Justin Case]

Hi again,

Just wondering.. in v10, there is a RcptTo event:
IdSMTPServer1RcptTo(ASender: TIdSMTPServerContext; const AAddress: String; AParams: TStrings; var VAction: TIdRCPToReply; var VForward: String);

I've got that one coded up, if the AAddress is local, it's accepted, if it's external it's only accepted if the user is logged in (pluis a few other checks). Oh and I have SPF fully implemented checking TXT (parsing a, mx, include: ), A records and MX..

However, then I hit a snag...

IdSMTPServer1MsgReceive(ASender: TIdSMTPServerContext; AMsg: TStream; var VAction: TIdDataReply);

EEeeeek!

So, I've done:
Msg := TIdMessage.Create;
Msg.LoadFromStream(AMsg);

But now I am stuck with a new problem... who is it TO?

ASender.RCPTList.EMailAddresses (and similar under ASender.RCPTList) !!!

But there could be many there.. how do I know which one I approved earlier in RcptTo? - I could use a variable somewhere but I kinda thought it would be made available in the parameters!

It looks like I need to loop through the items and process every single address all over again to determine if it's ok or a fail. Is that correct or am I missing something? Should I just call the RcptTo event on each address?

Obviously if it's an outgoing mail I guess that's straight forward but if it's incoming for a local user and there's lots of addresses in there that is gonna require more cpu cycles right?

[rlebeau]

IdSMTPServer1MsgReceive(ASender: TIdSMTPServerContext; AMsg: TStream; var VAction: TIdDataReply);

EEeeeek!

Yeah, Indy 9 gave you a choice of 3 different events to receive emails: OnReceiveRaw, OnReceiveMessage, and OnReceiveMessageParsed.  In Indy 10, there is only a single event, OnMsgReceive (effectively the same as OnReceiveRaw).  TIdSMTPServer no longer makes any assumptions about what the data represents or what to do with the data, it lets you handle all of that in your own code.

So, I've done:
Msg := TIdMessage.Create;
Msg.LoadFromStream(AMsg);

But now I am stuck with a new problem... who is it TO?

ASender.RCPTList.EMailAddresses (and similar under ASender.RCPTList) !!!

Yes, ASender.RCPTList contains the actual recipients that the client specified in its RCTO TO commands.  These are the actual recipients you must deliver to.  The TStream in the OnMsgReceive event is the raw data that the client wants delivered to each recipient.  It can be any arbitrary data, but it is usually an RFC 822/2822 formatted email.  Its To: and Cc: headers (if present) are not required to match the ASender.RCPTList and should not be considered for delivery purposes.  The ASender.RCPTList is the source of truth.  In fact, this is how BCC (Blind Carbon Copy) is implemented - BCC recipients are only available in the ASender.RCPTList and not in the TStream data. This is how clients can send emails to hidden recipients without letting all recipients know who the email is being sent to.

But there could be many there.. how do I know which one I approved earlier in RcptTo? - I could use a variable somewhere but I kinda thought it would be made available in the parameters!

It is, actually. If you reject a recipient in the OnRcptTo event, it will not be added to the ASender.RCPTList.  Only approved recipients are stored in that list. This was the same in Indy 9, too.

In Indy 10, the email data will not be accepted unless at least 1 recipient is approved (Indy 9 would accept an email with no approved recipients). The client will be told which recipients are rejected. If you reach the OnMsgReceive event, it means the client is ok with the rejected recipients and has decided to continue on with delivery to the remaining recipients who were approved.

It looks like I need to loop through the items and process every single address all over again to determine if it's ok or a fail.

You need to process the ASender.RCPTList to know who has been accepted to deliver the data to, yes.  But you should have already validated each recipient in the OnRcptTo event.  You don't need to do it again. By the time you reach the OnMsgReceive event, the client has been told you are accepting the email for delivery and thus you are now required to deliver the provided data to approved recipients.

Is that correct or am I missing something? Should I just call the RcptTo event on each address?

No.

Obviously if it's an outgoing mail I guess that's straight forward but if it's incoming for a local user and there's lots of addresses in there that is gonna require more cpu cycles right?

Each entry in the ASender.RCPTList is an individual recipient.  If you read the ASender.RCPTList.EMailAddresses property, it will consolidate the entries into a single string.  You don't need to do that (unless you want to store that string somewhere, of course).

[Justin Case]

Remy, thanks for that advice. I didn't realise that the rejected recipients would not be in RCPTList - that all makes sense now (and of course now you've mentioned it, I remember seeing it in the indy source).

Thanks for your sterling efforts in helping me!

[Justin Case]

Well after mulling this over, despite your advice to me it makes sense to actually call RCPTTo event again and this is why..

That event filters out the ok addresses.. but then in the MsgReceive event I still need to know if incoming mail is for a local account or external. The LAction variable parameter of the call to the RCPTTo is rather useful here and despite essentially duplicating the previous behaviour that variable then makes it easier to determine what to do with the incoming mail:

Code:

procedure TDataModule1.IdSMTPServer1RcptTo(ASender: TIdSMTPServerContext;
  const AAddress: String; AParams: TStrings; var VAction: TIdRCPToReply;
  var VForward: String);
var
Local: Boolean;
begin
Local := False;

//Check if address is local
Query1.Close;
Query1.SQL.Text := SQL('CheckAddressDomainIsLocal');
Query1.Prepare;
Query1.ParamByName('address').AsString := Utils.Parse('@', AAddress, 1);
Query1.ParamByName('domain').AsString := Utils.Parse('@', AAddress, 2);
Query1.Open;

with Query1 do
  begin
  First;

  while not EOF do
      begin
      //Record found in database so sending to local address
      Local := True;
      Break;
      end;
  end;

if ASender.LoggedIn then
  begin
  if not Local then
      begin
      VAction := rWillForward;
      end;

  if Local then
      begin
      VAction := rAddressOk;
      end;
  end
else
  begin
  //VAction := rNoForward; - If user is/was known and has forwarding address
  // or not local and no forwarding address

  if not Local then
      begin
      VAction := rRelayDenied;
      end;

  if Local then
      begin
      VAction := rAddressOk;
      end;
  end;
end;

Code:

procedure TDataModule1.IdSMTPServer1MsgReceive(
  ASender: TIdSMTPServerContext; AMsg: TStream; var VAction: TIdDataReply);
var
I: Integer;
LAction : TIdRCPToReply;
LForward: String;
Local: Boolean;
Msg: TIdMessage;
SMTP: TIdSMTP;
begin
LAction :=  rRelayDenied;

SMTP := TIdSMTP.Create;

Msg := TIdMessage.Create;
Msg.LoadFromStream(AMsg);

for I := 0 to ASender.RCPTList.Count -1 do
  begin
  IdSMTPServer1RcptTo(ASender, ASender.RCPTList.Items[I].Address, nil, LAction, LForward);

  case LAction of //Useful
      rAddressOk:
        begin
        {TODO : Save to database}
        Msg.SaveToFile(ExtractFilePath(ParamStr(0)) + 'email.eml');
        end;
      rWillForward:
        begin
        {TODO : Ssnd onwards elsewhere}
        //SMTP.Send(Msg); 
        end;
      end;
  end;

VAction := dOk;
Msg.Free;
end;

If you have any better ideas.. otherwise this works for me!