-
|
It does not seem like I have my s3 client configured this way - When I run my application, I can see HTTP responses being logged - However, all HTTP responses are not being logged. For example, the application logs a few of these errors which are logged after the operation response is received from the S3 API - However, it seems like How can I accurately log all HTTP requests and responses made by the S3 APIs? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 5 replies
-
|
Hi @anishsana , A Also, the connection reset error can happen for variety of reasons and would not always be related to a 500 or any error. In a concurrent environment the server can respond with successful code, but because of the connection management on the server side, it might close idle connections which can cause these (sometime successful) responses to never make it over the wire. The correct strategy to handle failed requests caused by connection reset errors is to retry them, which the SDK should do by default. Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @RanVaknin It looks like the SDK retries only when For example, when performing GetObject operations, I see no retries: However, retries are being made when performing MultipartUpload operations: Is this a bug? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @anishsana , The SDK retries any retryable error regardless if its a read or a write. From the output we actually don't see if the request is being retried or not. The tcp error might be manifesting but then might be retried. The error thrown in You have logs enabled, what do the wire logs show? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @RanVaknin The errors I mentioned were taken from the error object response from the GetObject and UploadPart S3 API requests in the application. I confirmed by checking the wire logs that there was only one attempt for the object by the GetObject operation (so no retries made) whereas the UploadPart operation has 3 attempts for the part (retries made). 1 attempt for GetObject: 3 attempts for UploadPart for the same part: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @RanVaknin Looks like there are times when the SDK retries and there are times when it doesn't. Below instance shows a read connection reset that was retried for a Create Multipart Upload operation: Confirmed the same with S3 requests and responses logging: Could the issue be specific to the GetObject API not retrying requests with connection reset errors? Is there anything I can do to ensure that these connection errors, not just connection reset but any connection related errors, are always retried just in case the SDK does not retry them? Maybe a custom retryer for these errors? Any insight with an example would be super helpful. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
https://aws.github.io/aws-sdk-go-v2/docs/making-requests/#responses-with-ioreadcloser
In an operation like GetObject, where the entire HTTP response body is the userdata (the object, in this case), we just pass the underlying This is different from |
Beta Was this translation helpful? Give feedback.
-
|
That makes sense. Thank you! |
Beta Was this translation helpful? Give feedback.
https://aws.github.io/aws-sdk-go-v2/docs/making-requests/#responses-with-ioreadcloser
In an operation like GetObject, where the entire HTTP response body is the userdata (the object, in this case), we just pass the underlying
http.Response.Bodyto you. So you're probably gettingconnection resetin the context of actuallyReading the response body yourself - the SDK isn't going to retry that, you're not inside an SDK operation call (there's no retry loop to begin with).This is different from
PutObject, wherein the SDK is taking object data from you and handling conveying it over the wir…