TIdHTTP Lightstreamer Client

[rsiw2000]

Hello,

I´m trying to write a Lightstreamer client using C++ Builder Tokyo and TIdHTTP component.
I wrote this code based on the python client demo. 
TIdHTTP is receiving data using a simple TStream descendant and I get these data on the Write method.

This is part of the connection thread, it creates the session and stay connected to receive data.

Code:

pHTTP = new TIdHTTP(0);
pHTTP->Request->CustomHeaders->Add("Accept: */*");
pHTTP->Request->CustomHeaders->Add("Content-Type: application/x-www-form-urlencoded");
pHTTP->Request->CustomHeaders->Add("Connection: Keep-Alive");
UnicodeString params, url = L"http://push.lightstreamer.com/lightstreamer/create_session.txt";
params.sprintf(L"LS_op2=create&LS_cid=mgQkwtwdysogQz2BJ4Ji+kOj2Bg&LS_adapter_set=DEMO&LS_protocol=TLCP-2.1.0";
// parametros
params = TIdURI::ParamsEncode(params, enUTF8);
// conexão
pSSParams = new TStringStream(params);
pHTTP->Post(url, pSSParams, pClienteLS->RcvdStream);

The connection returns the following headers:

HTTP/1.0 200 OK
Server: Lightstreamer-Server/7.1.0 b4 build 1915 (Lightstreamer Server - www.lightstreamer.com) ENTERPRISE edition
Content-Type: text/enriched; charset=ISO-8859-1
Cache-Control: no-store
Cache-Control: no-transform
Cache-Control: no-cache
Pragma: no-cache
Expires: Thu, 1 Jan 1970 00:00:00 GMT
Date: Fri, 3 May 2019 12:03:13 GMT
Content-Length: 500000
Connection: Keep-Alive
x-accel-buffering: no

When SessionId arrives, I subscribe some stocks using another TIdHTTP and stock data becomes to arrive.
Server is sending data all the time, but TIdHTTP connects and receives some data, then stops for a few minutes, then another block of data arrives, then stops again, then receive another block, a lot of times. 
It seems like to be filling a buffer, precisely a 32k (32768 bytes) buffer.
When I force TIdHTTP to disconnect, remaining data arrives in the Write method.

So:
What i'm doing wrong ?
Why received data does not reaches the stream Write method immediately ?

At the python example there's no different header (no header, in fact.). 
The Lightstreamer docs also does not have any reference to the need of some specific header.

[rlebeau]

Code:

pHTTP->Request->CustomHeaders->Add("Accept: */*");
pHTTP->Request->CustomHeaders->Add("Content-Type: application/x-www-form-urlencoded");
pHTTP->Request->CustomHeaders->Add("Connection: Keep-Alive");

You should not be using the CustomHeaders property for those particular headers.  The Request object has individual properties for them instead:

Code:

pHTTP->Request->Accept = "*/*";
pHTTP->Request->ContentType = "application/x-www-form-urlencoded";
pHTTP->Request->Connection = "Keep-Alive";

Code:

UnicodeString params, url = L"http://push.lightstreamer.com/lightstreamer/create_session.txt";
params.sprintf(L"LS_op2=create&LS_cid=mgQkwtwdysogQz2BJ4Ji+kOj2Bg&LS_adapter_set=DEMO&LS_protocol=TLCP-2.1.0";
// parametros
params = TIdURI::ParamsEncode(params, enUTF8);
// conexão
pSSParams = new TStringStream(params);
pHTTP->Post(url, pSSParams, pClienteLS->RcvdStream);

When sending an "application/x-www-form-urlencoded" post, you should be using a TStringList for the post fields. Let TIdHTTP handle encoding them for you (especially since ParamsEncode() is not the correct thing to use in this context anyway):

Code:

TStringList *params = new TStringList;
params->Add(L"LS_op2=create");
params->Add(L"LS_cid=mgQkwtwdysogQz2BJ4Ji+kOj2Bg");
params->Add(L"LS_adapter_set=DEMO");
params->Add(L"LS_protocol=TLCP-2.1.0");
pHTTP->Post(url, params, pClienteLS->RcvdStream);
delete params;

When SessionId arrives, I subscribe some stocks using another TIdHTTP and stock data becomes to arrive.

Session IDs are (usually) exchanged back and forth between client and server using HTTP cookies.  Are the two TIdHTTP objects sharing a single TIdCookieManager object?  If not, then the second TIdHTTP won't be able to send back cookies received by the first TIdHTTP.  TIdHTTP instances do not share cookies with each other by default, you have to set that up manually.

Server is sending data all the time, but TIdHTTP connects and receives some data, then stops for a few minutes, then another block of data arrives, then stops again, then receive another block, a lot of times. 

Sounds like the server is using server-side pushes to deliver streaming live data.  TIdHTTP is not designed to handle streaming data, though there are some workarounds for that (like using the TIdHTTP.OnChunkReceived event or hoNoReadChunked flag if the server is sending the pushes using HTTP chunking, or by using a custom TStream class with overwritten Write() method).  Otherwise, you may have to resort to using TIdTCPClient instead, manually implementing enough of the HTTP protocol to get the streaming started, and then you can read the data however you want.

It seems like to be filling a buffer, precisely a 32k (32768 bytes) buffer.

Yes, that is possible.  TIdCustomHTTP.ReadResult() calls TIdIOHandler.ReadStream() to read data, which uses a local buffer (which is 32K by default, but can be customized via the TIdIOHandler.RecvBufferSize property) to receive data from the socket before writing it to the TStream.  ReadStream() calls TIdIOHandler.ReadBytes() to fill that local buffer, and ReadBytes() waits for the requested number of bytes to fill the buffer before exiting.

Which, under normal conditions, should be perfectly fine.  HTTP has a structure to it.  Unless the HTTP response body uses an arbitrary data type that is terminated only by closing the socket, or the response body uses a MIME type (like 'multipart/x-mixed-replace') without being wrapped in HTTP chunking, then the response data has known sizes up front, which buffering takes into account.

What i'm doing wrong ?

Nothing.  This is just the way TIdHTTP works.

If the response has a 'Transfer-Encoding: chunked' header, the data is read in complete chunks (each chunk has its own size, and each chunk is read in blocks of RecvBufferSize up to that size) until the last chunk is received.  Only complete chucks are passed to the TIdHTTP.OnChunkReceived event and written to the target TStream.

Otherwise, if the response has a 'Content-Length' header, the data is read in blocks of RecvBufferSize and written to the TStream until the specified number of bytes have been received.

Otherwise, if the data has a 'Content-Type: multipart/...' header, the response is read in arbitrary blocks based on available data from the socket (and some of those blocks may themselves be read in blocks of RecvBufferSize) and written to the target TStream until the terminating MIME boundary has been received.

Otherwise, the data is read in blocks of RecvBufferSize and written to the TStream until the socket is disconnected.

Why received data does not reaches the stream Write method immediately ?

Because TIdIOHandler.ReadBytes() is being told by TIdHTTP to wait for X number of bytes to arrive (up to 32K at a time), so that is what it is doing.  TIdHTTP is not asking the socket "how many bytes do you have right now? X? OK, give me X bytes" (except MAYBE in the MIME w/o chunking scenario). It is (usuallly) telling the socket "OK, I want X bytes, so wait until X bytes are available and then give them to me".

[rsiw2000]

Thanks Remy, for your extensive answer, I'll digest this and give a feedback.

[rsiw2000]

Yes, I've understood this very clearly.
I'd like to thank you again for this explanation.

---

Another question:

The remote server sends a very large content-size and starts sending data (realtime quotes requested in subsequent posts).

My app receives these data in the Write method of a TStream descendant.

Each data content is terminated with a CR LF.

When content-length is reached, the component is reconnected as explained by the vendor docs.

I can't do any king of pooling or buffering as data consumption must occur ASAP.

To minimize the problem with TIdHTTP receive buffer, I've wrote the following code:

Code:

void __fastcall TLSConexaoHttp::HTTPConnected(TObject *Sender)
{
TIdHTTP * pHTTP = static_cast<TIdHTTP*>(Sender);
pHTTP->IOHandler->WriteBufferFlush();
pHTTP->IOHandler->RecvBufferSize = 8;
}

I can say this resolved my issue, It's working fine, the test program works for many consecutive hours receiving data.

Of course this is a PoC and this code later will be introduced in a very large system that already uses a TIdHTTP to receive data from another vendor server.

My feeling is that I'm writing a very ugly code doing things this way.

I'd like to know if there's any better approach to do this or even if it's possible to write a class to replace the receive buffer (of course, the goal is no buffering).

[rlebeau]

I'd like to know if there's any better approach to do this or even if it's possible to write a class to replace the receive buffer (of course, the goal is no buffering).

I already told you one way:

Otherwise, you may have to resort to using TIdTCPClient instead, manually implementing enough of the HTTP protocol to get the streaming started, and then you can read the data however you want.

As I said earlier, TIdHTTP simply is not designed for live streaming, and it seems the server data you are receiving does not follow any of the formats that TIdHTTP does provide workarounds for.

I suggest manually implementing a minimal HTTP client using TIdTCPClient directly so you have full control over the reading, then you can use the TIdIOHandler.ReadLn() method in a loop, since you say the server data is CRLF delimited. Then you don't need a custom TStream descendant at all.

[rsiw2000]

Yes, you're right, this connection is http but it's implementation via TIdHTTP requires a lot of subversion on HTTP concepts.

Perhaps, but you likely don't need the whole HTTP protocol just to get a stream going. A simple GET/POST, maybe some authentication, those can be accomplished fairly easily by hand.

PS: please stop over-quoting. This is a forum, not email. People can see the past replies, so there is no need to quote the entire discussion in each reply you post. Notice how I've been editing your replies to remove redundant quoting. Be more selective in what you quote. Thanks.

[rsiw2000]

I will take the developing this from TCP component in consideration.

I will take care about over-quoting in the future.

Thanks.