OpenTelemetry: Instrumentation

 

OpenTelemetry C++ API

header only library

C++11 이후 최신 version들 모두 지원

stable ABI를 제공

opentelemetry::nosted namespace 사용

 

Getting started

Tracing

Obtain a tracer

auto provider = opentelemetry::trace::Provider::GetTracerProvider();
auto tracer = provider->GetTracer("foo_library", "1.0.0");

 

Start a span

auto span = tracer->StartSpan("HandleRequest");

 

Mark a span as active

auto scope = tracer->WithActiveSpan(span);

span을 active하게 표시(mark) 하고 Scope 객체를 반환

Scope 객체는 span의 active을 얼마나 할지를 제어

Span은 Scope 객체의 lifetime 동안만 active하게 유지됨

 

span의 부모의 명시 없이 생성된다면, 이 span은 현재 active span의 부모가 됨

이렇게 부모가 없는 span이 root span임

 

 

Create nested Spans

auto outer_span = tracer->StartSpan("Outer operation");
auto outer_scope = tracer->WithActiveSpan(outer_span);
{
    auto inner_span = tracer->StartSpan("Inner operation");
    auto inner_scope = tracer->WithActiveSpan(inner_span);
    // ... perform inner operation
    inner_span->End();
}
// ... perform outer operation
outer_span->End();

nested span

parent-chlid relationship with other spans.

주어진 span이 active하면, 새롭게 생성되는 sapn은 active span의 trace ID와 다른 context attribute를 상속받음

 

 

Context Propagation

// set global propagator
opentelemetry::context::propagation::GlobalTextMapPropagator::SetGlobalPropagator(
    nostd::shared_ptr<opentelemetry::context::propagation::TextMapPropagator>(
        new opentelemetry::trace::propagation::HttpTraceContext()));

// get global propagator
HttpTextMapCarrier<opentelemetry::ext::http::client::Headers> carrier;
auto propagator =
    opentelemetry::context::propagation::GlobalTextMapPropagator::GetGlobalPropagator();

//inject context to headers
auto current_ctx = opentelemetry::context::RuntimeContext::GetCurrent();
propagator->Inject(carrier, current_ctx);

//Extract headers to context
auto current_ctx = opentelemetry::context::RuntimeContext::GetCurrent();
auto new_context = propagator->Extract(carrier, current_ctx);
auto remote_span = opentelemetry::trace::propagation::GetSpan(new_context);

 

context는 현재 active한 span의 spa id, trace Id, flag등과 같은 metadata를 지님

Context Propagation은 이런 context를 HTTP header를 통해 다른 service로 분배하는 매커니즘

OpenTelemetry는 text 기반의 전송 방식을 사용하며 이는 W3C Trace Context HTTP header를 사용함

 

 

Getting started

OpenTelemetry C++ SDK는 OpenTelemetry C++ API의 reference 구현을 제공

Processor, Sampler, core Exporter도 제공

 

Exporter

telemetry data를 특정 backend로 전달하는 역할을 수행

6개의 exporter들을 제공

 

1) in-memory exporter: memory에 저장하기에 debugging에 유용

2) Jaeger Exporter: UDP, HTTP를 통해 전달

3) Zipkin Exporter: Zipkin backend로 전달

4) Logging Exporter: log stream에 전달

5) OpenTelemetry(OTLP) Exporter: protobuf/gRPC 혹은 protobuf/HTTP로 전달

6) ETW exporter: Event Tracing for Windows를 통해 전달

 

//namespace alias used in sample code here.
namespace sdktrace   = opentelemetry::sdk::trace;

// logging exporter
auto ostream_exporter =
    std::unique_ptr<sdktrace::SpanExporter>(new opentelemetry::exporter::trace::OStreamSpanExporter);

// memory exporter
auto memory_exporter =
    std::unique_ptr<sdktrace::SpanExporter>(new opentelemetry::exporter::memory::InMemorySpanExporter);

// zipkin exporter
opentelemetry::exporter::zipkin::ZipkinExporterOptions opts;
opts.endpoint = "http://localhost:9411/api/v2/spans" ; // or export OTEL_EXPORTER_ZIPKIN_ENDPOINT="..."
opts.service_name = "default_service" ;
auto zipkin_exporter =
    std::unique_ptr<sdktrace::SpanExporter>(new opentelemetry::exporter::zipkin::ZipkinExporter(opts));

// Jaeger UDP exporter
opentelemetry::exporter::jaeger::JaegerExporterOptions opts;
opts.endpoint = "localhost";
opts.server_port =  6831;
auto jaeger_udp_exporter =
    std::unique_ptr<sdktrace::SpanExporter>(new opentelemetry::exporter::jaeger::JaegerExporter(opts));

// Jaeger HTTP exporter
opentelemetry::exporter::jaeger::JaegerExporterOptions opts;
opts.transport_format  = opentelemetry::exporter::jaeger::TransportFormat::kThriftHttp;
opts.endpoint = "localhost";
opts.server_port = 14268;
opts.headers = {{}}; // optional headers
auto jaeger_http_exporter =
    std::unique_ptr<sdktrace::SpanExporter>(new opentelemetry::exporter::jaeger::JaegerExporter(opts));

// otlp grpc exporter
opentelemetry::exporter::otlp::OtlpGrpcExporterOptions opts;
opts.endpoint = "localhost:4317";
opts.use_ssl_credentials = true;
opts.ssl_credentials_cacert_as_string = "ssl-certificate";
auto otlp_grpc_exporter =
    std::unique_ptr<sdktrace::SpanExporter>(new opentelemetry::exporter::otlp::OtlpGrpcExporter(opts));

// otlp http exporter
opentelemetry::exporter::otlp::OtlpHttpExporterOptions opts;
opts.url = "http://localhost:4318/v1/traces";
auto otlp_http_exporter =
    std::unique_ptr<sdktrace::SpanExporter>(new opentelemetry::exporter::otlp::OtlpHttpExporter(opts));

 

Span Processor

Exporter를 가지고 초기화 됨

 

 

// simple processor
auto simple_processor = std::unique_ptr<sdktrace::SpanProcessor>(
    new sdktrace::SimpleSpanProcessor(std::move(ostream_exporter)));

즉각적으로 exporter로 전달

 

// batch processor
sdktrace::BatchSpanProcessorOptions options{};
auto batch_processor = std::unique_ptr<sdktrace::SpanProcessor>(
    new sdktrace::BatchSpanProcessor(std::move(memory_exporter), options));

ended span을 batch 처리 후 bulk로 exporter에 전달

 

// multi-processor
std::vector<std::unique_ptr<SpanProcessor>>
    processors{std::move(simple_processor), std::move(batch_processor)};
auto multi_processor = std::unique_ptr<sdktrace::SpanProcessor>(
    new sdktrace::MultiSpanProcessor(std::move(processors));

여러 span processor들이 동시에 active되고 설정되도록 함

 

Resource

A Resource is an immutable representation of the entity producing telemetry as key-value pair. The OpenTelemetry C++ SDK allow for creation of Resources and for associating them with telemetry.

auto resource_attributes = opentelemetry::sdk::resource::ResourceAttributes
    {
        {"service.name", "shoppingcart"},
        {"service.instance.id", "instance-12"}
    };
auto resource = opentelemetry::sdk::resource::Resource::Create(resource_attributes);
auto received_attributes = resource.GetAttributes();
// received_attributes contains
//      - service.name = shoppingcart
//      - service.instance.id = instance-12
//      - telemetry.sdk.name = opentelemetry
//      - telemetry.sdk.language = cpp
//      - telemetry.sdk.version = <current sdk version>

 

Sampler

sample을 제어하고 줄이는 매커니즘

4가지 방식을 지원

AlwaysOnSampler: 모두 sample

AlwaysOffSampler: 모두 sample 하지 않음

ParentBased: paranet의 span을 사용해 sampling 결정을 수행

TraceIdRatioBaed: trace의 설정 percentage대로 sample

//AlwaysOnSampler
auto always_on_sampler = std::unique_ptr<sdktrace::AlwaysOnSampler>
    (new sdktrace::AlwaysOnSampler);

//AlwaysOffSampler
auto always_off_sampler = std::unique_ptr<sdktrace::AlwaysOffSampler>
    (new sdktrace::AlwaysOffSampler);

//ParentBasedSampler
auto parent_based_sampler = std::unique_ptr<sdktrace::ParentBasedSampler>
    (new sdktrace::ParentBasedSampler);

//TraceIdRatioBasedSampler - Sample 50% generated spans
double ratio       = 0.5;
auto always_off_sampler = std::unique_ptr<sdktrace::TraceIdRatioBasedSampler>
    (new sdktrace::TraceIdRatioBasedSampler(ratio));

 

TracerContext

SDK 설정은 TraceProvider와 이를 통해 생성되는 모든 Tracer instance와 TracerContext를 통해 공유됨

auto tracer_context = std::make_shared<sdktrace::TracerContext>
    (std::move(multi_processor), resource, std::move(always_on_sampler));

 

TracerPvovider

TracerProvider instance holds the SDK configurations ( Span Processors, Samplers, Resource). There is single global TracerProvider instance for an application, and it is created at the start of application. There are two different mechanisms to create TraceProvider instance

• Using constructor which takes already created TracerContext shared object as parameter.
• Using consructor which takes SDK configurations as parameter.

// Created using `TracerContext` instance
auto tracer_provider = nostd::shared_ptr<sdktrace::TracerProvider>
    (new sdktrace::TracerProvider(tracer_context));

// Create using SDK configurations as parameter
auto tracer_provider = nostd::shared_ptr<sdktrace::TracerProvider>
    (std::move(simple_processor), resource, std::move(always_on_sampler));

// set the global tracer TraceProvider
opentelemetry::trace::Provider::SetTracerProvider(tracer_provider);

 

Logging and Error Handling

OpenTelemetry C++ SDK provides mechanism for application owner to add customer log and error handler. The default log handler is redirected to standard output ( using std::cout ).

The logging macro supports logging using C++ stream format, and key-value pair. The log handler is meant to capture errors and warnings arising from SDK, not supposed to be used for the application errors. The different log levels are supported - Error, Warn, Info and Debug. The default log level is Warn ( to dump both Error and Warn) and it can be changed at compile time.

OTEL_INTERNAL_LOG_ERROR(" Connection failed. Error string " << error_str << " Error Num: " << errorno);
opentelemetry::sdk::common::AttributeMap error_attributes = {
  {"url", url}, {"content-length", len}, {"content-type", type}};
OTEL_INTERNAL_LOG_ERROR(" Connection failed." , error_attributes);
opentelemetry::sdk::common::AttributeMap http_attributes = {
  {"url", url}, {"content-length", len}, {"content-type", type}};
OTEL_INTERNAL_LOG_DEBUG(" Connection Established Successfully. Headers:", http_attributes);

The custom log handler can be defined by inheriting from opentelemetry::sdk::common::internal_log::LogHandler class.

class CustomLogHandler : public opentelemetry::sdk::common::internal_log::LogHandler
{
    void Handle(opentelemetry::sdk::common::internal_log::LogLevel level,
                const char \*file,
                int line,
                const char \*msg,
                const opentelemetry::sdk::common::AttributeMap &attributes) noexcept override

    {
        // add implementation here
    }
};
opentelemetry::sdk::common::internal_log::GlobalLogHandler::SetLogHandler(CustomLogHandler());
opentelemetry::sdk::common::internal_log::GlobalLogHandler::SetLogLevel(opentelemetry::sdk::common::internal_log::LogLevel::Debug);