Java 17: Missing content-length header on a multipartfile request

sttp client
1

Hello,
I am using STTP 3.8.15 with ZIO backend, Java 17 ZULU.

So, given this code

  override def uploadFile(
    connection: Connection,
    path: String,
    file: File,
    overwrite: Boolean
  ): IO[BusinessError, Unit] =
    sendRequest(s"File upload to $path", AcceptedContent.any)( // custom function
      basicDBRequest
        .post(uri"https://something/put")
        .auth.bearer(connection.token.value)
        .multipartBody(
          multipart("path", path),
          multipart("overwrite", overwrite),
          multipartFile("contents", file)
        ),
      noOpCallHandler[String, String]
    ).mapErrorCause(fromError)
      .ignore

The server returns 400 with Content-Length header must be provided. This is required to avoid uploading partial data..

Using requestbin, I see that indeed there is no content-length header sent:

transfer-encoding: chunked
accept: */*
accept-encoding: gzip, deflate
content-type: multipart/form-data; boundary=8f9cab0d-2f53-4d2b-b8b1-1414d7b7dbb2

I’ve tried to set one, but frankly, however I try to calculate and set it, I always get another error: Too many bytes in request body. Expected: 37131145, got: 37131695

  1. Shouldn’t the content-length header be sent automatically in this case? The length of the body is known as it’s an existing file.
  2. If not, how can I have calculate the body of such multipart request or ask STTP to define Content-Length?

I noticed, that with Java 11, the content-length header is sent instead of Transfer-Encoding: chunked. Am I able to configure STTP (and underlaying classes) to use content-length header?

Thank you!

BTW the Slf4jLoggingBackend does not log POST requests, I see only GET ones. I have configured it this way for my tests

HttpClientZioBackend.layer().map(env =>
  ZEnvironment(Slf4jLoggingBackend(
    delegate                  = env.get[SttpClient],
    beforeCurlInsteadOfShow   = true,
    logRequestBody            = false,
    logRequestHeaders         = true,
    logResponseBody           = false,
    logResponseHeaders        = true,
    beforeRequestSendLogLevel = LogLevel.Warn,
    responseLogLevel          = _ => LogLevel.Warn,
    responseExceptionLogLevel = LogLevel.Warn
  ))
)
2

@leszek The change is caused by Java HTTP Client, which is the underlying sender for your backend. I suspect that with Java version update it started to send multipart bodies as chunked, and, according to RFC, the Content-Length header is illegal for this content type (http - Chunked encoding and content-length header - Stack Overflow).
There’s also an interesting rationale on Wikipedia:

Chunked transfer encoding allows a server to maintain an HTTP persistent connection for dynamically generated content. In this case, the HTTP Content-Length header cannot be used to delimit the content and the next HTTP request/response, as the content size is not yet known. Chunked encoding has the benefit that it is not necessary to generate the full content before writing the header, as it allows streaming of content as chunks and explicitly signaling the end of the content, making the connection available for the next HTTP request/response.

Your receiving server seems to be incorrectly handling the request, if it requires the header to be present.
I don’t think there’s a reliable method that you could use to calculate content length of the chunks and add the header by yourself in this case.

Regarding Slf4jLoggingBackend, I’ll check this issue separately.

3

Thank you @kciesielski .

Yes, I found similar information too. I also read that Transfer-Encoding: chunked is used only if the size of the body is unknown, but that is not the case here. I have the file and the size is known.

Would it be possible to check this case? Could it be some misconfiguration from the STTP itself?

4

It’s up to the Java Http client to decide to chunk, and then it assumes that the size is unknown. It’s the internal logic of this client on which sttp doesn’t have any influence.

However, there may be a workaround. Sttp allows to instance the Java client manually

import java.net.http.HttpClient
val httpClient: HttpClient = ???
val backend = HttpClientZioBackend.usingClient(httpClient)

maybe there’s a possibility to configure it to disable this chunking behavior.

5

Thank you, I will explore this idea.

6

For the record. I ended up using HttpURLConnectionBackend and wrap it around the ZIO effect.
This backend produces the Content-Length header.

7

Sounds good, fortunately there is some flexibility when it comes to backends :slight_smile:
If this backend uses content-length then I suspect it doesn’t use transfer-encoding: chunked?

8

Exactly. These headers are exclusive.

9

Regarding logging of GET requests, but no POST requests via Slf4j, I could not reproduce the issue.
Here’s a small program I wrote that can be run with scala-cli: SttpZioLog.scala · GitHub
I tried with different kinds of responses and status codes (with requestbin) and with an entirely failing the effect with an unknown host.
Maybe your issue is caused by some specific configuration of sttp or your logging backend, but I would need to know more about this specific case.

10

Thank you @kciesielski for your time and checking this. I’ve checked our other POST sttp calls and the logging works, but the POST upload call mentioned above does not and I cannot find any reasonable difference.

I tried to replicate our settings in your code example, but the POST log was always working.

If I manage to find it, I will start a new discussion or report an issue.

Thank you

1 Like