Receive http response messages using OpenSSL in C

Question

I have some problems when trying to receive http response message of a website.
This is my function:

void Reveive_response(char *resp, SSL *ssl) {

    const int BUFFER_SIZE = 1024;
    char response[1048576];
    char *buffer = NULL;            // to read from ssl
    char *check = (char *) malloc(BUFFER_SIZE*sizeof(char));
    int bytes;                      // number of bytes actually read
    int received = 0;               // number of bytes received

    buffer = (char *) malloc(BUFFER_SIZE*sizeof(char));     // malloc
    memset(response, '\0', sizeof(response));               // response
    assign = '\0'
    do{
        memset(buffer, '\0', BUFFER_SIZE);          // empty buffer
        bytes = SSL_read(ssl, buffer, BUFFER_SIZE);
        if (bytes < 0) {
            printf("Error: Receive response\n");
            exit(0);
        }
        if (bytes == 0) break;
        received += bytes;
        printf("Received...%d bytes\n", received);
        strncat(response, buffer, bytes);   // concat buffer to response
    } while (SSL_pending(ssl));             // while pending
    response[received] = '\0';
    printf("Receive DONE\n");
    printf("Response: \n%s\n", response);
    free(buffer);
    strcpy(resp, response);                 // return via resp

}

When I call the function, it seems like the response message is not complete. Like this:

Received...1014 bytes
Received...1071 bytes
Receive DONE
Response: 
HTTP/1.1 200 OK
<... something else....>
Vary: Accept-Encoding
Content-Type: text/html
Conne

Then if i call the function again, it returns:

Received...39 bytes
Receive DONE
Response:
ction: keep-alive
Content-Length: 0

The field Connection was split. Why my function didn't receive all the response message? I used do while loop inside. Please tell me where did i go wrong? Thank you.

Answer 1

There is nothing wrong. This is simply how TCP works. It is a streaming transport, it has no concept of message boundaries. There is no 1-to-1 relationship between the number of bytes sent and the number of bytes read. Your reading receives arbitrary bytes, which you are then responsible for processing as needed. Keep reading, buffering and parsing the HTTP data as you go, until you discover the end of the response (see RFC 2616 Section 4.4 Message Length for details). Looping on SSL_pending() is not sufficient (or correct).

In this case, you have to read CRLF-delimited lines one at a time until you reach a CRLF/CRLF pair indicating the end of the response headers, then you need to analyze the headers you have received to know whether a response body is present and how to read it, as it may be in one of several different encoded formats. If present, you can then read the body (decoding it as you go along) until you reach the end of the body as specified by the headers.

See the pseudo-code I posted in my answer to the following question:

Receiving Chunked HTTP Data With Winsock

That said, you really should not be implementing HTTP (let alone HTTPS) manually to begin with. HTTP is not trivial to implement from scratch, and neither is SSL/TLS for that matter. You have dived head-first into a deep well without understand some important basics of network programming and OpenSSL programming. You should use an existing HTTP/S library instead, such as libcurl, and let it handle the details for you so you can focus on your code's business logic and not its communications logic.