Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .codegen.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{ "engineHash": "6772894", "specHash": "7f8494a", "version": "10.21.0" }
{ "engineHash": "dabe8ed", "specHash": "7f8494a", "version": "10.21.0" }
67 changes: 67 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@
- [Example Delays (with default settings)](#example-delays-with-default-settings)
- [Retry-After Header](#retry-after-header)
- [Network Exception Handling](#network-exception-handling)
- [Retrying Uploads](#retrying-uploads)
- [Streams That Can Be Retried](#streams-that-can-be-retried)
- [Handling Errors for Streams That Cannot Be Retried](#handling-errors-for-streams-that-cannot-be-retried)
- [Customizing Retry Parameters](#customizing-retry-parameters)
- [Custom Retry Strategy](#custom-retry-strategy)
- [Timeouts](#timeouts)
Expand Down Expand Up @@ -129,6 +132,70 @@ This means:
- Network exception retries are tracked independently from HTTP error retries, each with their own counter and backoff progression.
- A request can fail up to `maxRetriesOnException` times due to network exceptions, but each exception retry also increments the overall attempt counter, so the total number of retries across both exception and HTTP error types is bounded by `maxAttempts`.

### Retrying Uploads

To retry a request that uploads content, the SDK has to send the content again. The SDK does not copy the content into memory to do that. Instead, it rewinds the `InputStream` you passed to the position it had when the upload started. Whether the request can be retried therefore depends on the type of the stream.

| Stream type | Retried automatically | How the content is sent again |
| ------------------------------------------------------------------------------------------------------- | --------------------- | -------------------------------------------------------------------------- |
| `FileInputStream` | Yes | The file position is moved back to where it was when the upload started. |
| `ByteArrayInputStream` | Yes | The stream is reset to where it was when the upload started. |
| Any other stream, for example from `Files.newInputStream`, `BufferedInputStream`, or a network response | No | The content is sent once. If that attempt fails, the SDK throws the error. |

If a request with a stream that cannot be rewound fails with a retryable error, such as `429 Too Many Requests` or a `5xx` status, the SDK does not retry it. It throws the original `BoxAPIError` instead, which keeps the response status and headers, including `Retry-After`. For a `401` response, the SDK still refreshes the access token, so your next request uses a valid token.

Uploads to an upload session, such as `uploadBigFile` and `uploadFilePart`, are always retried. Each part is read into memory before it is sent, so it can be sent again regardless of the stream type.

The SDK closes the stream you pass after the request finishes, including all retries, whether it succeeds or fails. Closing it yourself as well, for example with try-with-resources, is safe.

Interceptors run before every attempt, but a request body that reads a stream is built on the first attempt and reused on retries. Changes an interceptor makes to the upload stream on a retry are therefore not applied.

#### Streams That Can Be Retried

To let the SDK retry uploads automatically, pass a `FileInputStream` when uploading from a file:

```java
try (FileInputStream fileStream = new FileInputStream(new File("report.pdf"))) {
client.getUploads().uploadFile(
new UploadFileRequestBody(
new UploadFileRequestBodyAttributesField(
"report.pdf", new UploadFileRequestBodyAttributesParentField("0")),
fileStream));
}
```

If you open files with `Files.newInputStream(path)`, use `new FileInputStream(path.toFile())` instead. The stream returned by `Files.newInputStream` cannot be rewound.

For content that is already in memory, wrap it in a `ByteArrayInputStream`. Do not copy large files into memory only to make them retryable; use a `FileInputStream` instead.

#### Handling Errors for Streams That Cannot Be Retried

If you upload from a stream that cannot be rewound, handle retryable errors in your code and send a new stream on each attempt:

```java
BoxAPIError lastError = null;
for (int attempt = 0; attempt < 3; attempt++) {
try (InputStream stream = openContentStream()) {
return client.getUploads().uploadFile(
new UploadFileRequestBody(
new UploadFileRequestBodyAttributesField(
"report.pdf", new UploadFileRequestBodyAttributesParentField("0")),
stream));
} catch (BoxAPIError error) {
int status = error.getResponseInfo().getStatusCode();
if (status != 429 && status < 500) {
throw error;
}
lastError = error;
String retryAfter = error.getResponseInfo().getHeaders().get("Retry-After");
Thread.sleep(retryAfter != null ? (long) (Double.parseDouble(retryAfter) * 1000) : 1000L);
}
}
throw lastError;
```

`openContentStream()` stands for your own code that opens the content from the beginning, for example a new request to the system the content comes from.

### Customizing Retry Parameters

You can customize all retry parameters by initializing `BoxRetryStrategy` with the desired values and passing it to `NetworkSession`:
Expand Down
14 changes: 14 additions & 0 deletions src/main/java/com/box/sdkgen/box/errors/RequestInfo.java
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
package com.box.sdkgen.box.errors;

import com.box.sdkgen.internal.logging.DataSanitizer;
import com.box.sdkgen.networking.inputstreamrequestbody.InputStreamRequestBody;
import java.util.Map;
import java.util.stream.Collectors;
import okhttp3.MultipartBody;
import okhttp3.Request;
import okhttp3.RequestBody;
import okio.Buffer;
Expand Down Expand Up @@ -120,6 +122,10 @@ public static RequestInfo fromRequest(Request request, String contentType) {
}

private static String getRequestBodyAsString(RequestBody requestBody) {
// Writing a stream body here would read the whole file content into memory.
if (containsStream(requestBody)) {
return null;
}
try {
if (requestBody != null) {
Buffer buffer = new Buffer();
Expand All @@ -132,6 +138,14 @@ private static String getRequestBodyAsString(RequestBody requestBody) {
return null;
}

private static boolean containsStream(RequestBody requestBody) {
if (requestBody instanceof MultipartBody) {
return ((MultipartBody) requestBody)
.parts().stream().anyMatch(part -> containsStream(part.body()));
}
return requestBody instanceof InputStreamRequestBody;
}

String print(DataSanitizer dataSanitizer) {
Map<String, String> sanitizedHeaders =
dataSanitizer == null ? headers : dataSanitizer.sanitizeHeaders(headers);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,14 @@
import com.box.sdkgen.networking.fetchoptions.MultipartItem;
import com.box.sdkgen.networking.fetchoptions.ResponseFormat;
import com.box.sdkgen.networking.fetchresponse.FetchResponse;
import com.box.sdkgen.networking.inputstreamrequestbody.InputStreamRequestBody;
import com.box.sdkgen.networking.network.NetworkSession;
import com.box.sdkgen.networking.networkclient.NetworkClient;
import com.box.sdkgen.networking.proxyconfig.ProxyConfig;
import com.box.sdkgen.networking.timeoutconfig.TimeoutConfig;
import com.fasterxml.jackson.databind.JsonNode;
import java.io.IOException;
import java.io.InputStream;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.net.URI;
Expand All @@ -43,9 +45,6 @@
import okhttp3.Request;
import okhttp3.RequestBody;
import okhttp3.Response;
import okio.BufferedSink;
import okio.Okio;
import okio.Source;

public class BoxNetworkClient implements NetworkClient {

Expand Down Expand Up @@ -132,11 +131,20 @@ public BoxNetworkClient withTimeoutConfig(TimeoutConfig config) {
}

public FetchResponse fetch(FetchOptions options) {
try {
return fetchWithRetries(options);
} finally {
closeFileStreams(options);
}
}

private FetchResponse fetchWithRetries(FetchOptions options) {
NetworkSession networkSession =
options.getNetworkSession() == null ? new NetworkSession() : options.getNetworkSession();

boolean authenticationNeeded = false;
Request request;
RequestBody body = null;
FetchResponse fetchResponse = new FetchResponse.Builder(0, new TreeMap<>()).build();
Exception exceptionThrown = null;

Expand All @@ -146,21 +154,17 @@ public FetchResponse fetch(FetchOptions options) {
boolean shouldRetry = false;

while (true) {
FetchOptions attemptOptions =
(attemptNumber > 1 || numberOfRetriesOnException > 0 || options.getData() == null)
&& options.getDataFactory() != null
? withData(options, options.getDataFactory().create())
: options;
FetchOptions fetchOptions = applyInterceptors(attemptOptions, networkSession);
boolean isFirstAttempt = attemptNumber == 1 && numberOfRetriesOnException == 0;
FetchOptions fetchOptions = applyInterceptors(options, networkSession);
if (fetchOptions.getDataFactory() != null
&& options.getDataFactory() == null
&& (attemptNumber > 1
|| numberOfRetriesOnException > 0
|| fetchOptions.getData() == null)) {
&& (!isFirstAttempt || fetchOptions.getData() == null)) {
fetchOptions = withData(fetchOptions, fetchOptions.getDataFactory().create());
}

request = prepareRequest(fetchOptions, authenticationNeeded, networkSession);
if (isFirstAttempt || !containsFileStream(fetchOptions)) {
body = prepareRequestBody(fetchOptions);
}
request = prepareRequest(fetchOptions, body, authenticationNeeded, networkSession);

Response response = null;
String rawResponseBody = null;
Expand Down Expand Up @@ -222,8 +226,9 @@ public FetchResponse fetch(FetchOptions options) {
networkSession
.getRetryStrategy()
.shouldRetry(fetchOptions, fetchResponse, attemptForRetry);
boolean isRetryable = canBeResent(body);

if (shouldRetry) {
if (shouldRetry && isRetryable) {
double retryDelay =
networkSession
.getRetryStrategy()
Expand Down Expand Up @@ -301,11 +306,13 @@ private static FetchOptions withData(FetchOptions options, JsonNode data) {
}

private static Request prepareRequest(
FetchOptions options, boolean reauthenticate, NetworkSession networkSession) {
FetchOptions options,
RequestBody body,
boolean reauthenticate,
NetworkSession networkSession) {
Request.Builder requestBuilder = new Request.Builder().url(options.getUrl());
Headers headers = prepareHeaders(options, reauthenticate, networkSession);
HttpUrl url = prepareUrl(options);
RequestBody body = prepareRequestBody(options);

requestBuilder.headers(headers);
requestBuilder.url(url);
Expand Down Expand Up @@ -438,23 +445,48 @@ private static int getRetryAfterTimeInSeconds(int attemptNumber, String retryAft
return (int) (Math.pow(2, attemptNumber) * BASE_TIMEOUT * jitter);
}

public static RequestBody createMultipartRequestBody(MultipartItem part) {
return new RequestBody() {
@Override
public MediaType contentType() {
if (part.contentType != null) {
return MediaType.parse(part.contentType);
}
return MediaType.parse("application/octet-stream");
}
/**
* Returns whether the request body reads a file stream. Such a body is built once and reused on
* retries, because rebuilding it would read the stream from where the previous attempt stopped.
*/
private static boolean containsFileStream(FetchOptions options) {
return options.getFileStream() != null
|| (options.getMultipartData() != null
&& options.getMultipartData().stream().anyMatch(part -> part.getFileStream() != null));
}

@Override
public void writeTo(BufferedSink sink) throws IOException {
try (Source source = Okio.source(part.getFileStream())) {
sink.writeAll(source);
}
}
};
private static void closeFileStreams(FetchOptions options) {
if (options.getFileStream() != null) {
closeQuietly(options.getFileStream());
}
if (options.getMultipartData() != null) {
options.getMultipartData().stream()
.map(MultipartItem::getFileStream)
.filter(Objects::nonNull)
.forEach(BoxNetworkClient::closeQuietly);
}
}

/** A failure to close a stream must not replace the result or the error of the request. */
private static void closeQuietly(InputStream stream) {
try {
stream.close();
} catch (IOException ignored) {
}
}

private static boolean canBeResent(RequestBody body) {
if (body instanceof MultipartBody) {
return ((MultipartBody) body).parts().stream().allMatch(part -> canBeResent(part.body()));
}
return !(body instanceof InputStreamRequestBody)
|| ((InputStreamRequestBody) body).canBeResent();
}

public static RequestBody createMultipartRequestBody(MultipartItem part) {
MediaType contentType =
MediaType.parse(part.contentType != null ? part.contentType : "application/octet-stream");
return new InputStreamRequestBody(part.getFileStream(), contentType);
}

public static OkHttpClient.Builder getDefaultOkHttpClientBuilder() {
Expand Down
Loading
Loading