OpenTelemetry for .NET AWS Lambda functions

8 minute read

When I write a Lambda function, I don’t want the first useful trace to begin inside my business handler.

By then, quite a lot has already happened.

AWS has invoked the function. The runtime has identified the function model, created scopes, decoded records and dispatched them to handlers. For batched sources such as SQS, Kinesis or DynamoDB Streams, one Lambda invocation may also contain several independently processed records.

That plumbing is exactly where useful observability can disappear if every function has to instrument it by hand.

The functions in this post are created from Kralizek.Lambda.Templates. The generated project references the matching runtime package for the selected function type: the core Kralizek.Lambda.Template package for the common programming model, or one of the source-specific packages built on top of it.

Those runtime packages emit telemetry through the standard .NET System.Diagnostics APIs. When the function runs in an OpenTelemetry-enabled environment, those signals can be collected without moving the instrumentation responsibility into application code.

This post looks at what that means in practice: invocation metadata, one span per record, source-specific tags, bounded metrics, and the application telemetry you can add on top. The same instrumentation can then be observed locally through Aspire or exported from AWS Lambda.

If you want the broader programming model first, I covered it in A programming model for .NET AWS Lambda functions.

Start with an SQS function

Consider a function that processes OrderCreated messages from SQS:

public sealed class Function
    : SqsFunction<OrderCreated, OrderCreatedHandler>;

The SQS template generates a project that references Kralizek.Lambda.Template.Sqs, which builds the SQS programming model on top of the common runtime.

The handler deals with one decoded message:

public sealed class OrderCreatedHandler(IOrderRepository orders)
    : ISqsMessageHandler<OrderCreated>
{
    public async ValueTask<SqsRecordResult> HandleAsync(
        OrderCreated message,
        SqsMessageContext context,
        CancellationToken cancellationToken)
    {
        await orders.CreateAsync(message.OrderId, cancellationToken);

        return SqsRecordResult.Success;
    }
}

There is no OpenTelemetry code here.

But the runtime already knows things that the handler should not need to reconstruct: this is a record-oriented function, which SQS message is being processed, which queue it came from, how long the record took, and whether processing succeeded, failed explicitly, threw, or was canceled.

Those are the signals the runtime packages can emit consistently for functions using the same programming model.

One invocation, one span per record

The Lambda invocation span itself comes from OpenTelemetry.Instrumentation.AWSLambda.

Kralizek.Lambda.Template deliberately does not create another competing invocation span. Instead, it enriches the current Lambda activity with the function model:

kralizek.lambda.function.model = record

The same core package implements RecordFunction, which creates one record.process child activity for every record in the envelope:

AWS Lambda invocation
├─ record.process
├─ record.process
└─ record.process

For an SQS batch with three messages, that means three independently visible pieces of work inside the same Lambda invocation.

This matters when partial batch responses are enabled. The invocation can complete successfully while one record is reported as failed and scheduled for retry. Marking only the entire invocation as success or failure would lose that distinction.

The record activities preserve it.

The record span already knows about SQS

A generic record.process span would not be particularly useful by itself. Kralizek.Lambda.Template.Sqs enriches it with transport metadata it already owns.

For SQS, the span contains attributes such as:

messaging.system = aws_sqs
messaging.operation.name = process
messaging.operation.type = process
messaging.message.id = <message id>
messaging.destination.name = <queue name>
kralizek.aws.sqs.queue.arn = <queue ARN>

Where OpenTelemetry defines an appropriate semantic convention, the package uses it. Metadata without a matching convention stays under the framework-owned kralizek.aws.* namespace rather than pretending to be a standard attribute.

The other source packages follow the same rule while adding the metadata that belongs to their event source:

Kralizek.Lambda.Template.Sns adds SNS message, topic and subscription information;
Kralizek.Lambda.Template.KinesisStreams adds Kinesis event ID, sequence number and partition key;
Kralizek.Lambda.Template.DynamoDbStreams adds DynamoDB Streams event and stream metadata;
Kralizek.Lambda.Template.S3 adds S3 bucket, object, event and region information, and adds S3 Batch task/result metadata for Batch Operations;
Kralizek.Lambda.Template.EventBridge enriches the invocation span with EventBridge event ID, source and detail type;
Kralizek.Lambda.Template.Cognito enriches the invocation span with Cognito trigger, user-pool, user and region information.

The distinction between invocation and record spans is intentional. EventBridge and Cognito represent one logical event/request per invocation, so their packages enrich the existing invocation span instead of creating an artificial record span.

A failed record is not necessarily a failed invocation

The handler result is also part of the telemetry model.

Kralizek.Lambda.Template owns the common record-processing instrumentation, while the source package interprets the result type it exposes to application code.

For sources with explicit per-record outcomes, returning a source-defined failure marks the corresponding record span as Error, even though the handler returned normally.

For SQS, for example:

return SqsRecordResult.Failed("Order could not be processed");

is different from:

throw new InvalidOperationException("Repository unavailable");

Both make the record unsuccessful, but they are different operational outcomes.

The framework keeps four bounded outcomes for record metrics:

success — the handler returned a successful source-specific result;
failure — the handler deliberately returned a failed result;
error — record processing threw an exception;
canceled — Lambda invocation cancellation interrupted processing.

Kralizek.Lambda.Template.Sqs, Kralizek.Lambda.Template.KinesisStreams and Kralizek.Lambda.Template.DynamoDbStreams all map their source-specific failure results into that common telemetry model while producing the partial-batch response Lambda expects for that source.

Kralizek.Lambda.Template.S3 does something similar for S3 Batch Operations: temporary and permanent task failures are failed spans, with an additional bounded attribute describing the S3 Batch result case.

A valid partial-batch response can therefore contain failed records while the parent Lambda invocation span remains successful.

That is the behavior I want from the trace: the record failed; the Lambda invocation correctly handled that failure.

Metrics without high-cardinality baggage

The common Kralizek.Lambda.Template instrumentation exposes framework metrics through Meter:

Instrument	Type	What it measures
`kralizek.lambda.invocations`	Counter	Framework invocations by function model
`kralizek.lambda.records`	Counter	Processed records by outcome
`kralizek.lambda.record.duration`	Histogram	Per-record processing time by outcome

The source-specific metadata contributed by packages such as Kralizek.Lambda.Template.Sqs or Kralizek.Lambda.Template.KinesisStreams does not automatically become metric tags.

Message IDs, object keys, sequence numbers, partition keys, ARNs, user names and failure messages are intentionally kept away from framework metric dimensions. They are useful when investigating an individual trace and expensive or dangerous as unbounded metric cardinality.

The framework can therefore tell me that SQS record failures increased without creating a time series for every message that ever passed through the function.

Application telemetry fits underneath it

Framework telemetry stops at the framework boundary.

Kralizek.Lambda.Template.Sqs can tell me which SQS record is running and how that processing ended. It cannot know that the message represents an order or which domain operation matters inside the handler.

That belongs to the application:

private static readonly ActivitySource ActivitySource =
    new("Orders");

public async ValueTask<SqsRecordResult> HandleAsync(
    OrderCreated message,
    SqsMessageContext context,
    CancellationToken cancellationToken)
{
    using var activity = ActivitySource.StartActivity("order.create");
    activity?.SetTag("order.id", message.OrderId);

    await orders.CreateAsync(message.OrderId, cancellationToken);

    return SqsRecordResult.Success;
}

Because the handler runs while the record activity created by Kralizek.Lambda.Template is current, the application activity naturally becomes its child:

AWS Lambda invocation
└─ record.process
   └─ order.create

This is the division I find useful: the runtime packages describe the Lambda execution model and event source; application code describes the domain work happening inside it.

Neither side needs to duplicate the other’s instrumentation.

Generating the function with OpenTelemetry enabled

OpenTelemetry is exposed by the Kralizek.Lambda.Templates package as a template-time choice:

dotnet new lambda-template-sqs --otel

For this template, the generated project references Kralizek.Lambda.Template.Sqs for the SQS runtime behavior. With --otel, it also references OpenTelemetry.Instrumentation.AWSLambda and the OTLP exporter, subscribes to the shared activity source and meter exposed by Kralizek.Lambda.Template, and wraps the normal handler with the AWS Lambda OpenTelemetry wrapper.

The generated tracing setup is intentionally ordinary OpenTelemetry configuration:

private static TracerProvider ConfigureTracing() =>
    Sdk.CreateTracerProviderBuilder()
        .AddSource(LambdaTelemetry.ActivitySourceName)
        .AddAWSLambdaConfigurations()
        .AddOtlpExporter()
        .Build();

Metrics use the corresponding meter:

private static MeterProvider ConfigureMetrics() =>
    Sdk.CreateMeterProviderBuilder()
        .AddMeter(LambdaTelemetry.MeterName)
        .AddOtlpExporter()
        .Build();

LambdaTelemetry comes from Kralizek.Lambda.Template; source packages contribute to that shared instrumentation identity instead of making every integration register a separate activity source or meter.

The generated handler calls AWSLambdaWrapper.TraceAsync around the base implementation and force-flushes the meter provider at the end of the invocation so buffered measurements are not left behind when Lambda freezes the execution environment.

Without --otel, those OpenTelemetry packages and that setup are not generated at all. The runtime packages still use System.Diagnostics; the application simply chooses not to attach an OpenTelemetry SDK to those signals.

The same signals fit local Aspire development

Using OTLP is useful because the framework instrumentation is not tied to one backend.

Locally, an Aspire AppHost can run a function generated from Kralizek.Lambda.Templates through the AWS Aspire integration and provide an OpenTelemetry environment where the emitted traces and metrics can be inspected in the Aspire dashboard.

That makes the development loop particularly useful for record-oriented functions. Send a batch to the SQS function and the trace immediately exposes the Lambda invocation, the record.process spans from Kralizek.Lambda.Template, the SQS metadata contributed by Kralizek.Lambda.Template.Sqs, and any application spans nested inside them.

The Lambda code does not need an Aspire-specific instrumentation model. Aspire is consuming the same Activity and Meter signals that can later be exported when the function runs in AWS.

In AWS, change the destination, not the instrumentation

The hosted setup is an infrastructure choice.

The project generated by Kralizek.Lambda.Templates --otel already uses the AWS Lambda OpenTelemetry instrumentation and an OTLP exporter. In AWS, that exporter can target the collector or telemetry pipeline chosen for the environment, including an ADOT-based setup.

The important part is that the trace model does not change when the destination does.

A record processed locally under Aspire and the same record processed by a deployed Lambda are described by the same runtime instrumentation: invocation metadata from Kralizek.Lambda.Template, record.process, source-specific tags from the relevant integration package, outcome and duration.

What changes is where those signals go.

What you get before adding domain instrumentation

For a Lambda created from Kralizek.Lambda.Templates, putting the function in an OpenTelemetry context gives you useful structure before the handler emits any domain-specific telemetry:

the standard AWS Lambda invocation span from OpenTelemetry.Instrumentation.AWSLambda;
the function model and common record-processing telemetry from Kralizek.Lambda.Template;
one child span per record for record-oriented sources;
source-specific metadata from Kralizek.Lambda.Template.Sqs, .Sns, .KinesisStreams, .DynamoDbStreams, .S3, .EventBridge or .Cognito, depending on the function;
record success, explicit failure, exception and cancellation semantics;
invocation, record and duration metrics with bounded dimensions.

Then application activities and metrics can fill in the part only the application understands.

That is the useful boundary for me. The runtime packages can explain which Lambda work is happening, for which record, and how it ended. The handler can explain what that work means to the application.

And because both sides use the normal .NET diagnostics model, the surrounding OpenTelemetry environment can compose them into the same trace whether I am looking at it locally in Aspire or after the function has been deployed to AWS.

Support this blog

If you liked this article, consider supporting this blog by buying me a pizza!

Twitter Facebook LinkedIn

Renato Golia