@aws-sdk/client-s3 Converting circular structure to JSON warning when using custom logger
#4339
Comments
micchickenburger
added
bug
yenfryherrerafeliz
added
investigating
|
Hi @micchickenburger, thanks for opening this issue. I can confirm this bug. I will mark this issue to be reviewed so we can further address it. Steps to reproduce:
import {GetObjectCommand, ListBucketsCommand, S3Client} from "@aws-sdk/client-s3";
const client = new S3Client({
region: 'us-east-2',
logger: {
info: content => {
console.log('INFO: ', JSON.stringify(content))
}
}
});
const response = await client.send(new GetObjectCommand({
Bucket: process.env.TEST_BUCKET,
Key: process.env.TEST_KEY
}));
console.log(response)
Thanks! |
yenfryherrerafeliz
added
needs-review
|
I am having essentially this same issue and it is confounding me. Versions
Error stack trace/bucket/documents.js.getDocument getDocument: async (path) => {
try{
console.log(`bucket.documents.getDocument(${path})`);
const s3cmd = new AWS.GetObjectCommand({ Bucket: process.env.S3_DOCUMENTS_BUCKET, Key: path });
const doc = await client.send(s3cmd);
return doc;
}
catch(err){
console.error(`path=${path}
`, err);
return undefined;
}
},/manager/documents.js.getDocument getDocument: async(path) => {
let document = await documentsRepo.getDocument(path);
if(document){
let stream = document.Body;
return new Promise((resolve, reject) => {
const chunks = []
stream.on('data', chunk => chunks.push(chunk))
stream.once('end', () => resolve(Buffer.concat(chunks)))
stream.once('error', reject)
});
}
return undefined;
},/controller/documents.js calling coderouter.get("/*", async (req, res) => {
let documentKey = decodeURI(req.path.substring(1));
let document = await documentsManager.getDocument(documentKey);
if(!document){
res.status(400).json({ msg: "Requested file does not exist" });
return;
}
res.write(document);
res.end();
});This error is being thrown from this line: const s3cmd = new AWS.GetObjectCommand({ Bucket: process.env.S3_DOCUMENTS_BUCKET, Key: path });This code doesn't always error - it depends on the environment.
The error only happens on QA so I am looking for a difference there but all I can find is serverless and GitHub actions. But I downloaded the GitHub built code and ran it locally using npm and it works fine. |
|
What is the SDK trying to log? I haven't told it to log anything. I event started looking at how to disable the built-in logging ... https://stackoverflow.com/questions/68066813/how-to-disable-aws-sdk-logging/75189203#75189203 |
|
I'm experiencing the exact same issue. Did you find any solution? |
|
Nope. Still waiting for a clue. Had to move on to some other problems.
…On Wed, Jan 25, 2023 at 2:31 PM Samuel Henderson ***@***.***> wrote:
I'm experiencing the exact same issue. Did you find any solution?
—
Reply to this email directly, view it on GitHub
<#4339 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAU2O3V34BNOXSTAAUIF2P3WUGLP3ANCNFSM6AAAAAAT5HKMLE>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
@samzi123 @JasonGloverNZ If you have control over your logger functions you can use node’s |
|
Thanks Micah.
Thing is this ... it's not my code that's doing the logging. In the stack
trace you can see that AWS SDK is doing some logging.
I found references online that suggest that AWS SDK isn't meant to do any
logging unless you specify a logging provider, but other people say it does
it regardless. That's certainly what I am seeing.
…On Wed, Jan 25, 2023 at 10:48 PM Micah Henning ***@***.***> wrote:
@samzi123 <https://github.com/samzi123> @JasonGloverNZ
<https://github.com/JasonGloverNZ> If you have control over your logger
functions you can use node’s util.inspect instead of JSON.stringify. See
the bottom of the original post for an example.
—
Reply to this email directly, view it on GitHub
<#4339 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAU2O3QOEG24OLQS5QKKGZTWUIF3ZANCNFSM6AAAAAAT5HKMLE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Sure, I can see that. When you instantiate your S3 client, you can pass a custom logger that uses |
|
I'm kind of at a loss on this one. This is one of the few results on the
internet for this error in this location.
This code works in local development, only fails when deployed.
Local dev:
- node is executing.
In QA:
- build prepared by GitHub action
- executing through serverless.com framework
- process hosted by AWS lambda
- same node version
I'm wondering if somehow the SDK picks up that there is a logger provided
to it by either the lambda environment (which I am guessing is some
streamlined Linux) or by the serverless framework.
I certainly haven't configured one anywhere.
🤷🏼♂️🧐
On Thu, Jan 26, 2023 at 11:25 AM, Micah Henning ***@***.***> wrote:
Sure, I can see that. When you instantiate your S3 client, you can pass a
custom logger that uses util.inspect as a workaround in the meantime.
That function removes circular references and allows the S3 SDK to log
without error. I agree that the library should not log at all unless a
logger is passed to it.
—
Reply to this email directly, view it on GitHub
<#4339 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAU2O3X5GI3WMML5WYPPJJLWUK6QRANCNFSM6AAAAAAT5HKMLE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Jason Glover
415 530 6975
about.me/jasonglover
|
I had the same issue and was only getting the issue when deploying the code to Lambda with Serverless. I'm guessing it's due to some problem with Serverless' logging. I ended up using v2 of the aws-sdk to call getObject which resolved the issue. |
|
I have managed to get past this problem by disabling the SDK's internal logging. I did so by setting const s3Config = {
endpoint: process.env.S3_ENDPOINT,
region: process.env.S3_REGION,
logger: undefined,
credentials: {
accessKeyId: process.env.S3_KEY,
secretAccessKey: process.env.S3_SECRET
}
}; |
|
Facing the same issue when trying to fetch data from s3. Since the above solution did not work, reverting back to sdk v2 |
|
facing the error |
ajredniwja
added
queued
|
Use Our logger interface has a variable number of arguments like the console logger. Provide a varargs-compatible logging implementation. In the example above, winston only logged the first argument. const s3 = new S3({
region: "us-west-2",
logger: {
...logger,
debug(...args) {
return logger.debug(args.join(" "));
},
info(...args) {
for (const arg of args) {
logger.info(arg);
}
},
},
});example: concat or loop the arguments before passing to winston's single-argument logger. The endpoints debug output looks like this: |
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread. |
Checkboxes for prior research
Describe the bug
When using a custom logger (Winston, in my case) the S3 GetObjectCommand results in a bunch of these outputs:
Then, finishes with:
I didn't test any other commands.
The warning happens because my logger implementation tries to stringify messages passed to it. However, the S3 client library tries to send output with circular references to a logger, which I think is erroneous behavior.
SDK version number
@aws-sdk/[email protected]
Which JavaScript Runtime is this issue in?
Node.js
Details of the browser/Node.js/ReactNative version
v16.19.0
Reproduction Steps
Configuration:
This is the logging pattern I use for all the rest of the AWS libraries with no issue.
Here is my Winston configuration:
Observed Behavior
When using a custom logger (Winston, in my case) the S3 GetObjectCommand results in this output:
This actually breaks the flow; further s3 object processing does not occur.
Expected Behavior
I expect there to be no useless debug messages, and for there to not be a circular reference error during object stringification.
Possible Solution
Not using a custom logger eliminates the issue, as does not stringifying output, but this is the logging pattern I use for all of the aws client libraries without issue. Of course, not stringifying is also not very useful:
EDIT: Using node's util.inspect can be used to remove circular references:
Additional Information/Context
No response
The text was updated successfully, but these errors were encountered: