Problem

When a user follows our doc to set the env var: DD_LOGS_CONFIG_PROCESSING_RULES : [{"type": "exclude_at_match", "name": "exclude_start_and_end_logs", "pattern": "(START|END|REPORT) RequestId"}], we not only excludes START, END and REPORT logs, but some other logs as well, such as (1) extension log and (2) error log from the Lambda handler. As a result, errors can be left unobserved.

Bug

This is because we first apply the log processing rules to the log we receive from telemetry API, to decide whether to send it to Datadog, then use the same result for "orphan log", which may include extension log and user error log.

This PR

For each orphan log, evaluate log processing rules again, instead of reusing the existing result.

Testing

Setup

• Lambda runtime: Node 22
• Handler:

const region = process.env["NO_REGION"];
if (!region) {
  throw new Error("NO_REGION environment variable not set");
}


export const handler = async (event) => {
  const response = {
    statusCode: 200,
    body: JSON.stringify('Hello from Lambda!'),
  };
  return response;
};

• Env var: DD_LOGS_CONFIG_PROCESSING_RULES : [{"type": "exclude_at_match", "name": "exclude_start_and_end_logs", "pattern": "(START|END|REPORT) RequestId"}]

Result before:

No log showed up in Datadog's Log Explorer or Live Tail

Result after:

Custom error log and extension log showed up.

Notes

Thanks @duncanista @astuyve @litianningdatadog for debugging and discussion.

Jira: https://datadoghq.atlassian.net/browse/SLES-2469