Sending large files through multipart IS #1780

Hi,

I am trying to send large files thourgh multipart but it doesn’t seem to get the right output buffer from Http.
I have tried to send in contentLength to PUT Header, MultipartBody and to multipart(as header(string, string)). None of them registers with contentLength in class HttpURLConnectionBackend. This section seems to be the problem:

val contentLength = partsWithHeaders
      .map { case (headers, p) =>
        val bodyLen: Option[Long] = p.body match {
          case StringBody(b, encoding, _) =>
            Some(b.getBytes(encoding).length.toLong)
          case ByteArrayBody(b, _)   => Some(b.length.toLong)
          case ByteBufferBody(_, _)  => None
          case InputStreamBody(_, _) => None
          case FileBody(b, _)        => Some(b.toFile.length())
          case NoBody                => None
          case StreamBody(_)         => None
          case MultipartBody(_)      => None
        }

        val headersLen = headers.getBytes(Iso88591).length

        bodyLen.map(bl => dashesLen + boundaryLen + crLfLen + headersLen + crLfLen + crLfLen + bl + crLfLen)
      }
      .foldLeft(Option(finalBoundaryLen)) {
        case (Some(acc), Some(l)) => Some(acc + l)
        case _                    => None
      }

I get an heap out of memory exception here:

private[client3] def transfer(is: InputStream, os: OutputStream): Unit = {
    var read = 0
    val buf = new Array[Byte](IOBufferSize)

    @tailrec
    def transfer(): Unit = {
      read = is.read(buf, 0, buf.length)
      if (read != -1) {
        os.write(buf, 0, read)
        transfer()
      }
    }

    transfer()
  }

Then why am I doing it this way, the stream I try to send is a large file that i find inside of an even larger zip. So i utilize the zipEntry data to get inputstream, filename and size to the file I need. We used to send this as a file before but discspace has limited our possibilities. Is there any way of forcing it to use an stroutputbuffer, as the one used in multipartFile.

But shouldn’t this switch case at least look for a ContentLength in the multipart?

val contentLength = partsWithHeaders
      .map { case (headers, p) =>
        val bodyLen: Option[Long] = p.body match {
          case StringBody(b, encoding, _) =>
            Some(b.getBytes(encoding).length.toLong)
          case ByteArrayBody(b, _)   => Some(b.length.toLong)
          case ByteBufferBody(_, _)  => None
          case InputStreamBody(_, _) => Some(r.headers.find(_.name == HeaderNames.ContentLength).toString.split(":").last)
          case FileBody(b, _)        => Some(b.toFile.length())
          case NoBody                => None
          case StreamBody(_)         => None
          case MultipartBody(_)      => None
        }

        val headersLen = headers.getBytes(Iso88591).length

        bodyLen.map(bl => dashesLen + boundaryLen + crLfLen + headersLen + crLfLen + crLfLen + bl + crLfLen)
      }
      .foldLeft(Option(finalBoundaryLen)) {
        case (Some(acc), Some(l)) => Some(acc + l)
        case _                    => None
      }

Something like the solution above. r might not be the correct header here. But if content size is delivered in multipartBody it should count for it in the bodyLen.map above.

Could you maybe describe the problem in more detail - I think you’re using the HttpURLConnectionBackend? Is the problem that the headers are incorrect, or that there’s an out-of-memory exception? Are you trying to set the content-length on a part? Can you post some runnable code to reproduce the problem?

Did you try using e.g. HttpClientSyncBackend? (which is now the default one)

As for the snippet you posted, I think what might be missing is checking if the content-length isn’t explicitly set, and using this if the bodyLen is a None. Though this seems to be a separate problem from the OOM, which I think is the core of the issue?