Increase in memory usage

[clement-s-bridge]

Hello,

I've recently upgraded my Node.js 24 lambda using datadog-lambda-js from 12.137.0 to 12.143.0.

That upgrade caused a big increase in memory usage. Before the lambda was running fine with 256 MB now I have out of memory crash even with 320 MB.

It seems to be caused by some huge traces with hundreds of spans.

I guess the issue might come from dd-trace which has been upgraded from 5.x to 6.x.

Setting DD_TRACE_ENABLED = false "fix" the issue but that's obviously not what we want.

Regards.

[ojproductions]

Hi @clement-s-bridge,

We have some idea what the cause is but to confirm: Is your handler ESM ( i.e it uses import statements, a .mjs file, or "type": "module" in package.json) or CommonJS?

In the meantime, to avoid the crashes you can either:

• increase the lambda's max memory setting
• or set DD_TRACE_PARTIAL_FLUSH_MIN_SPANS to a lower value, such as 100 (the default is 1000). This sends spans in batches during the invocation instead of keeping them all in memory until it ends.

Thanks!

[clement-s-bridge]

Hi @ojproductions

Thanks for your quick response.

I'm using lambda with container image, so I have this in my dockerfile:

# INSTALL DATADOG PACKAGES
RUN cd /tmp && \
    npm install --package-lock false --no-save --install-strategy nested datadog-lambda-js@12.137.0 dd-trace@5.93.0 && \
    mv /tmp/node_modules/* /var/runtime/node_modules/ && \
    # Cleanup old DD endpoint ( will de deleted on DD package when AWS deprecate NodeJS 12 )
    rm /var/runtime/node_modules/datadog-lambda-js/dist/handler.js

# SETUP DATADOG WRAPPER ENDPOINT
CMD ["/var/runtime/node_modules/datadog-lambda-js/dist/handler.handler"]
ENV DD_LAMBDA_HANDLER="main.handler"

And I have "type": "module" in my package.json.

[ojproductions]

Hi again @clement-s-bridge, thanks for the info. The extra spans might be due to #801 released in v142. This essentially started tracing esm native packages that in prior releases were not being loaded by the tracer. To confirm, are you seeing new span types that werent showing in v137 (e.g, http.request, aws.request, or database spans)?
If so, and the new spans are useful to you, keep v12.143.0 and use one of the workarounds from my previous comment (please confirm if they work for you). If you don't need spans from a specific integration, you can turn it off with DD_TRACE_DISABLED_PLUGINS (for example, DD_TRACE_DISABLED_PLUGINS=fs,http)

[joeyzhao2018]

Hi @clement-s-bridge, thank you for reaching out. From 256mb to 320mb yet still crashing is definitely alarming to us. Could you please create a support ticket if possible? That way we can also directly check the traces to understand your setup better.
Meanwhile, I will try to take a look at what @ojproductions suggested above too. But I highly recommend open a support ticket for your case. Thank you.

[clement-s-bridge]

Hello,

@joeyzhao2018 I've opened a support ticket as suggested https://help.datadoghq.com/hc/requests/3023166

@ojproductions Thanks, I'll try to set aside some time next week to confirm your hypothesis.