Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
A bug in new version of Indy in TIdSMTP
#1
I found a problem with it. Older version worked fine.


Code:
IdSMTP1->Host = "smtp.charter.net";
IdSMTP1->Port = 25;
IdSMTP1->Connect();

The server issues:
571 impout002 charter.net OTQuMjUzLjI0My4zNQ== You must connect from Charter IP space.  E1110

But IdSMTP raises an exception:

Project Project1.exe raised exception class EIdSMTPReplyError with message 'impout003 charter.net OTQuMjUzLjI0My4zNQ== You must connect from Charter IP space.  E1110'.

There is also problem connecting to:

Code:
IdSMTP1->Host = "mobile.charter.net";
IdSMTP1->Port = 587;
IdSMTP1->UseTLS = utUseImplicitTLS;
IdSMTP1->Connect();

Project Project1.exe raised exception class EIdSocketError with message 'Socket Error # 10060 Connection timed out.'.

But it is NOT socket error nor firewall. Older version of Indy connects fine using exact same settings above (587 + implicit TLS).

Older version of Indy (5373) works fine for both of the above settings... So not sure what changed in the meantime in IdSMTP.
maybe this line:
FExplicitTLSProtPort := 587; // TODO: define a constant for this!

In above connection 587 is implicit port (it is non-standardl but that's how Charter have it set up and until recently it connected well using older TIdSMTP).
Reply
#2
(05-07-2018, 08:40 PM)johnmay Wrote:
Code:
IdSMTP1->Host = "smtp.charter.net";
IdSMTP1->Port = 25;
IdSMTP1->Connect();

The server issues:
571 impout002 charter.net OTQuMjUzLjI0My4zNQ== You must connect from Charter IP space.  E1110

That error message means your TIdSMTP is connecting to Charter's SMTP server from a machine that is not connected to Charter's network and thus does not have an IP address that is assigned by Charter.  That has nothing to do with Indy.

(05-07-2018, 08:40 PM)johnmay Wrote: But IdSMTP raises an exception:

Project Project1.exe raised exception class EIdSMTPReplyError with message 'impout003 charter.net OTQuMjUzLjI0My4zNQ== You must connect from Charter IP space.  E1110'.

As it should be.  When you connect to an SMTP server, the server sends an initial SMTP greeting before the client can send any SMTP commands.  In the above case, that greeting has a reply code of 571 (which is technically an invalid reply code, as any reply code >= 560 is undefined by the SMTP protocol specs).  All 5xx reply codes are fatal errors.  The only reply code that is allowed in the greeting to indicate a successful SMTP session is 220, per the SMTP protocol specs.  TIdSMTP detects that non-220 reply code and raises the exception, and closes the socket connection.

This is normal and expected behavior.

(05-07-2018, 08:40 PM)johnmay Wrote: There is also problem connecting to:

Code:
IdSMTP1->Host = "mobile.charter.net";
IdSMTP1->Port = 587;
IdSMTP1->UseTLS = utUseImplicitTLS;
IdSMTP1->Connect();

Project Project1.exe raised exception class EIdSocketError with message 'Socket Error # 10060 Connection timed out.'.

SMTP port 587 is an EXPLICIT TLS port.  You should be using utUseExplicitTLS on it.

SMTP port 465 is an IMPLICIT SSL port.  You should be using utUseImplicitTLS on it.

SMTP port 25 doesn't use SSL/TLS.  You should be using either utNoTLSSupport or utUseExplicitTLS on it.

That being said, do note that setting the UseTLS property CAN CHANGE the Port property!  In your case, you are setting the Port property to the standard EXPLICIT TLS port 587 before setting the UseTLS property to utUseImplicitTLS.  The UseTLS property setter knows that 587 is a standard SMTP port, and will consequently change the Port property to the SMTP IMPLICIT SSL port 465.  So that is the port you are actually connecting to, not to port 587 like you specified.  And that would explain the timeout error, because Charter doesn't use port 465.

If you want to use a specific port with a non-standard UseTLS value, you need to set the Port property after setting the UseTLS property:

Code:
IdSMTP1->Host = "mobile.charter.net";
IdSMTP1->UseTLS = utUseImplicitTLS;
IdSMTP1->Port = 587; // <-- moved down here
IdSMTP1->Connect();

(05-07-2018, 08:40 PM)johnmay Wrote: Older version of Indy connects fine using exact same settings above (587 + implicit TLS).

Older version of Indy (5373) works fine for both of the above settings...

The UseTLS property setter in 5373 was not aware of explicit TLS ports, that logic was added in 5382.  In which case, in 5373, setting the UseTLS property to utUseImpliciTLS would change the Port to 465 only if the Port was 25, which is not the case in your situation.  Now, it does change the Port from 587 to 465, because port 587 is expected to use EXPLICIT TLS, not IMPLICIT SSL.

(05-07-2018, 08:40 PM)johnmay Wrote: In above connection 587 is implicit port

It is SUPPOSED to be an EXPLICIT TLS connection.

(05-07-2018, 08:40 PM)johnmay Wrote: it is non-standardl but that's how Charter have it set up and until recently it connected well using older TIdSMTP

Then you should complain to Charter, because this is a major flaw on their part. Everyone else in the world uses EXPLICIT TLS on SMTP port 587, and IMPLICIT SSL on SMTP port 465.

Reply
#3
Actually the problem is deeper than that.

This code also causes Socket Error # 10060 Connection timed out


Code:
IdSMTP1->Host = "smtp.charter.net";
IdSMTP1->Port = 25;
IdSMTP1->UseTLS = utUseExplicitTLS;
IdSMTP1->Connect();

And you wrote above that it is OK to use explicit TLS on port 25. Apparently, using code above, it is not. I thought it was exception issue but the exception is not the problem here but now when I understand what is going on here the UseTLS parameter logic is the problem - there is more to this but I'll get to that later.

This means that any provider which uses port 25 will now get Connection timed out if SMTP is setup with perfectly valid setup shown above. If the order of setup is reversed it works as expected:


Code:
IdSMTP1->Host = "smtp.charter.net";
IdSMTP1->UseTLS = utUseExplicitTLS;
IdSMTP1->Port = 25;
IdSMTP1->Connect();

So in other words, now TIdSMTP expects the programmer to issue commands in certain order because it modifies the internal port when UseTLS property is changed. I expect that you will receive many more questions about this behavior because the above code which uses both port 25 and explicit TLS is not unusual setup and can be used by a number of providers.

I made a change by setting port later and now my code works back as expected so thank you for pointing that out. But the problem here is having some assumptions about server setup and I'll explain some of the cases below why these assumptions should not be made:

- With regard to Charter.net - I know that they should be using explicit TLS on port 587. I agree with you on that. Problem is they don't. Now, I cannot just go and complain to them because I am not even their user - but they have probably hundreds of thousands of users. Maybe if I complain to them they would change that. Maybe that initial setup was done by incompetent administrator by mistake. But now after years have passed, they have a problem - they have to explain to hundreds of thousands of (technically challenged) users what the correct setup of ports should be and what is implicit and explicit TLS. The users will have no idea what explicit TLS is, they don't even know what TLS is. That is why they probably won't do it in the first place and the non-standard setup of port + implicit TLS will remain there. This is a real world scenario as opposed to idealistic scenario. Thus, any Indy programmer will have similar issue if they have to deal with Charter.net SMTP server.

- Some providers don't use standard setup and I've seen the cases of port 80 and 2525 being used for outgoing port. Sometimes this is used to avoid antiviruses blocking port 25. So any assumptions that certain ports should be used with certain setups here too falls apart.

- Take an example of a user having a local antivirus which requires the user to modify the port and then uses implicit TLS (locally) to scan for viruses before sending to servers. This is not a random example, some antiviruses really work like that. In such case it is not safe to assume that certain ports or explicit/implicit connection would be used. It can be any combination and SMTP component should be able to accept any combinations of parameters (within the valid port ranges that is).

From my perspective, the TIdSMTP component should accept parameters in any order before you actually connect. Only then parameters can be fixed. But by having this change to modify port when setting UseTLS you mislead the programmer into thinking he/she has set the port to their specification when in reality Indy has modified the port after changing UseTLS. In fact, in my own code I logged the connection to port 587, when in reality Indy was using port 465. Or in case of port 25, it would use port 587 instead using first piece of code shown above. That code logic is a bit flawed (and I don't mean this as criticism, just a constructive suggestion). If you don't want programmers to set parameters in any order then use Connect() with parameters. Then they can set it only in 1 order and no combinations or possible coding errors will occur.
Reply
#4
(05-08-2018, 09:56 AM)johnmay Wrote: This code also causes Socket Error # 10060 Connection timed out

Code:
IdSMTP1->Host = "smtp.charter.net";
IdSMTP1->Port = 25;
IdSMTP1->UseTLS = utUseExplicitTLS;
IdSMTP1->Connect();

And you wrote above that it is OK to use explicit TLS on port 25. Apparently, using code above, it is not.

Yes, it is OK. But you are not actually connecting to port 25, you are really connecting to port 587 instead, and apparently that Charter server doesn't use port 587, it uses port 25 instead, hence the tiimeout error on connect. So again, because you are setting the Port property to a standardized port 25 before setting the UseTLS property, the UseTLS property setter will change the Port property from 25 to 587 because of utUseExplicitTLS. You need to swap the order of the assignments:

Code:
IdSMTP1->Host = "smtp.charter.net";
IdSMTP1->UseTLS = utUseExplicitTLS; // <-- do this first
IdSMTP1->Port = 25; // <-- do this second
IdSMTP1->Connect();

Or, use utNoTLSSupport instead (which is the default):

Code:
IdSMTP1->Host = "smtp.charter.net";
IdSMTP1->Port = 25;
IdSMTP1->UseTLS = utNoTLSSupport;
IdSMTP1->Connect();

(05-08-2018, 09:56 AM)johnmay Wrote: This means that any provider which uses port 25 will now get Connection timed out if SMTP is setup with perfectly valid setup shown above.

No, that is not what it means. The only reason this is failing is because you use are using the UseTLS property but Charter is not using the standard Explicit TLS port. This is not a problem for most providers, who use standard ports the right way.

(05-08-2018, 09:56 AM)johnmay Wrote: If the order of setup is reversed it works as expected:

Code:
IdSMTP1->Host = "smtp.charter.net";
IdSMTP1->UseTLS = utUseExplicitTLS;
IdSMTP1->Port = 25;
IdSMTP1->Connect();

Only because you are using the UseTLS property, and Charter is using a non-standard setup on their end. Letting UseTLS manage the Port property is OK for MOST providers, who use standard port setups the right way.

(05-08-2018, 09:56 AM)johnmay Wrote: So in other words, now TIdSMTP expects the programmer to issue commands in certain order because it modifies the internal port when UseTLS property is changed.

Yes, and that logic has been in place for a very long time, this is not new functionality. The swapping of implicit and non-implicit ports was implemented well over a decade ago. The swapping of explicit and non-explicit ports was added almost 2 years.

Besides, this only affects protocols that implement standardized ports (and for good reason), it does not affect non-standardized ports.

(05-08-2018, 09:56 AM)johnmay Wrote: I expect that you will receive many more questions about this behavior because the above code which uses both port 25 and explicit TLS is not unusual setup and can be used by a number of providers.

Actually, no. This behavior has been in place for a long time, and I've had very few questions on it, because it "just works" for most people. It is the people that run into special situations that end up asking questions, and that has been few. Also, port 25 is deprecated anyway, most providers that support explicit TLS use port 587 instead (as they should be), and that is what most 3rd party email software expect. So again, swapping port 25 with port 587 when enabling explicit TLS "just works" for most people.

Why Charter is not following standard rules, I have no idea. But as I have explained, there is a workaround available to handle their non-standard behavior. You are not the first person to need to use it, and you probably won't be the last, but the situations that need it are few and far apart.

(05-08-2018, 09:56 AM)johnmay Wrote: I made a change by setting port later and now my code works back as expected so thank you for pointing that out. But the problem here is having some assumptions about server setup and I'll explain some of the cases below why these assumptions should not be made:

Thank you. But I'm not changing Indy's behavior at this point. The current behavior was added for a reason, and it serves its purpose well under normal conditions. Charter doesn't want to play by the same rules that everyone else follows, then they should expect to break someone's code along the way.

(05-08-2018, 09:56 AM)johnmay Wrote: - Some providers don't use standard setup and I've seen the cases of port 80 and 2525 being used for outgoing port.

And those are perfectly fine, and the UseTLS property setter will not change the Port property in those cases. It is only STANDARD PORTS that get changed around (for SMTP, ports 25, 465, and 587 only).

(05-08-2018, 09:56 AM)johnmay Wrote: From my perspective, the TIdSMTP component should accept parameters in any order before you actually connect.

And for the most part, it does. But Ports are a specific use case, and they need to follow certain rules. If you don't like the outcome, just overwrite the result.

If you don't like what Indy does, feel free to change it. The UseTLS property setter is virtual, you can override it to bypass the Port swapping logic if you want. There is a reason why all of Indy's internal data members are declared as protected instead of private, and why a good deal of its class methods are virtual - user customization.

Reply
#5
Well, as long as I can set any combination I need, I don't have a problem in changing the order, just it seems illogical to me.
Will this logic be applied to POP and IMAP so I prepare in advance for it and set UseTLS before Port like in SMTP case?
Reply
#6
(05-08-2018, 07:31 PM)johnmay Wrote: Will this logic be applied to POP and IMAP so I prepare in advance for it and set UseTLS before Port like in SMTP case?

Yes, similar UseTLS logic applies to POP3 and IMAP as well (and NNTP and FTP, too).  However, those protocols do not have a dedicated Explicit TLS port, like SMTP does.  Their Explicit TLS usage is typically on their normal non-SSL/TLS ports.  So, you will probably see the Port swap occur only when using utUseImplicitTLS rather than utUseExplicitTLS. For historical reasons, SMTP is different in regard to its use of a dedicated Explicit TLS port.

Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)