The Complete Datadog Course

Datadog is a cloud observability and monitoring platform. It collects telemetry from every layer of your stack — servers, containers, databases, cloud services, and application code — and turns it into dashboards, alerts, and searchable data you can use to understand and debug production systems.

Everything sits on three pillars of observability:

• Metrics — numeric time-series like CPU, request rate, latency, and error count.
• Traces — the path of a single request as it moves through your services (APM).
• Logs — the text events your apps and infrastructure emit, collected and indexed for search.

On top of those pillars, Datadog layers products for infrastructure, security, digital experience, and — the focus of this course — AI. This whole course is organized into those product modules so you can learn (or jump to) exactly what you need.

Data gets into Datadog through the Datadog Agent — a lightweight process you run on your hosts (or as a container) that gathers metrics, traces, and logs and ships them to Datadog. Sign up for the free 14-day trial, pick your site (e.g. datadoghq.com for US1, datadoghq.eu for EU), grab your API key, and install:

# One-line Datadog Agent install (Linux)
DD_API_KEY=<your_api_key> DD_SITE="datadoghq.com" \
  bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

Within a minute or two your host appears under Infrastructure → Host Map. Run the Agent as a Kubernetes DaemonSet and it collects from every node automatically.

A metric is a time-series: a name (like system.cpu.user), a value, a timestamp, and tags (key:value labels such as env:prod or service:checkout). Tags are the superpower — they let you slice one metric by host, region, or service on the fly.

In the metric query editor you pick a metric, filter by tags (from), and aggregate (avg, sum, p95) — optionally grouping by a tag. Functions let you compute rates, rolling averages, and arithmetic across metrics.

A dashboard is a canvas of widgets (timeseries, top lists, heatmaps, query values) you build from your metrics, traces, and logs. Use template variables (e.g. $env, $service) to make one dashboard reusable across services and environments.

Notebooks are the narrative cousin of dashboards — mixed text and graphs, perfect for incident post-mortems, runbooks, and sharing an investigation as a story.

A monitor is an alert: pick a metric, set a condition and threshold, and route notifications to Slack, email, PagerDuty, or Datadog On-Call. Monitor types include threshold, change, composite (combine monitors), and the ML types in the next lesson.

Good alerts are actionable: use tags to scope them, add clear messages with @-mentions and runbook links, and avoid alert fatigue by alerting on symptoms users feel (latency, errors) rather than every raw metric.

Some thresholds can't be hard-coded — traffic that's normal at noon is alarming at 3am. Datadog's ML-based monitors handle this:

• Anomaly — learns a metric's seasonal pattern and alerts when it deviates from the expected band.
• Forecast — projects a metric forward and alerts before it will cross a threshold (e.g. "disk full in 3 days").
• Outlier — alerts when one member of a group (one host, one pod) behaves unlike its peers.

An SLO sets a reliability target (e.g. "99.9% of checkout requests succeed over 30 days"). Datadog tracks the target against a Service Level Indicator (a metric or monitor) and shows your remaining error budget — how much unreliability you can still spend before you breach.

SLOs shift the conversation from "is this graph red?" to "are we keeping our promise to users?" — and the error budget tells you when to slow down and stabilize versus ship features.

APM (Application Performance Monitoring) captures a trace for each request — the full path as it hops across your services. Each step is a span with its own timing. You instrument your app with a Datadog tracing library (Python, Java, Node, Go, …), often with zero code changes via auto-instrumentation.

The Service Map draws your architecture live from traces — every service as a node, dependencies as edges, colored by health. It's the fastest way to see what talks to what and where latency or errors originate.

Open a trace and you get a flame graph — a waterfall of spans showing exactly where time went (a slow DB query, an N+1 loop, a downstream call). Error Tracking groups related errors into issues so one spike of 10,000 errors becomes a single, triageable problem with a first-seen/last-seen timeline.

Watchdog RCA (root cause analysis) runs on your APM data automatically, correlating anomalies across services to suggest a likely cause — a preview of the AI module.

APM tells you which service is slow; the Continuous Profiler tells you which line of code is burning CPU, allocating memory, or holding locks — in production, continuously, at low overhead. You can compare profiles across deploys to catch a regression the moment it ships.

The Agent (or a cloud integration) ships logs to Datadog, where pipelines process them: parse raw text into structured fields (the Grok parser), remap attributes, and enrich. Structured logs become facets you can filter and aggregate on — turning a wall of text into queryable data.

To control cost, indexes and filters decide which logs are retained for search, while everything can still be archived cheaply to cloud storage and rehydrated on demand.

The Log Explorer lets you search across all logs, filter by facet, and switch from raw events to analytics — group by fields, count, and graph (e.g. "5xx responses per service over time"). Because logs share tags with metrics and traces, you can pivot from a spiking metric straight to the exact logs behind it.

The Host Map visualizes every host as a tile, colored and grouped by any tag — a fast read on fleet health. Datadog's 900+ integrations (PostgreSQL, NGINX, Redis, Kafka, AWS, GCP, Azure…) are usually a small YAML config that tells the Agent what to collect, instantly adding curated metrics and an out-of-the-box dashboard.

Run the Agent as a DaemonSet and Datadog auto-discovers pods and services, collecting container metrics, kube-state data, and logs. The Container Map and Live Containers views show utilization down to the pod, while Autodiscovery attaches the right integration config to containers as they start and stop.

Cloud Cost Management pulls your AWS/GCP/Azure billing into Datadog and puts spend next to utilization — so you can ask "what did this service cost, and was it even used?" in one place. Tag-based breakdowns make it easy to attribute cost to teams and find waste.

Watchdog is Datadog's always-on, ML-driven detection engine. No configuration: it continuously learns normal behavior across your metrics, traces, and logs, then surfaces the signals that matter. It does automatic anomaly detection, root cause analysis, impact assessment (how many users are affected), and faulty-deployment detection.

Where a monitor answers a question you knew to ask, Watchdog catches the "unknown unknowns" you never wrote a monitor for.

Bits AI is Datadog's AI teammate. It's agentic: chat with it in real time, or delegate a whole task — investigate an alert, remediate code, triage a security signal — and let it run autonomously and report back. It lives inside the Datadog app and reasons over your observability data.

Bits runs on an AI Credits billing model, and is available in the app, on mobile, and via Slack. (Note: not available on Datadog's US government sites.)

The flagship capability. Bits Investigation (Bits AI SRE) activates the moment an alert fires and starts root-cause analysis without a prompt — often finding a likely cause, and even proposing a fix, before an engineer finishes reading their page.

How it investigates — parallel hypotheses

1. Generates multiple hypotheses about the cause, all at once.
2. Systematically tests each by querying your logs, traces, metrics, and infra.
3. Marks each validated, invalidated, or inconclusive, with the evidence.
4. Follows the promising leads and drops the dead ends — narrowing to a root cause.

Findings post to Slack, mobile, and the app, and integrate with On-Call & Case Management. You can chat to refine, and your corrections make future investigations better.

Not every question is an incident. Bits Chat lets you explore observability data in plain language — "which services had the highest error rate in the last hour?" or "show p99 latency for checkout today vs. yesterday" — no query syntax required.

Bits Data Analysis extends this to business data you've brought into Datadog, so you can interrogate it conversationally instead of writing SQL or building a report.

This is the piece that connects Datadog to your AI tools. The Datadog MCP Server is a bridge between your observability data and any AI agent that speaks the Model Context Protocol (MCP) — Claude Code, Cursor, OpenAI Codex, and more. It exposes structured tools for APM, logs, metrics, traces, monitors, dashboards, and security signals, so your coding agent can pull real telemetry while it debugs or writes code.

Connect it to Claude Code

The quickest path is the official plugin; you can also configure the remote server manually.

# Option A — official plugin
/plugin install datadog@claude-plugins-official
/ddsetup        # pick your Datadog site + OAuth login
/ddtoolsets     # enable APM / Logs / Metrics tool groups

// Option B — manual remote MCP server (endpoint varies by site)
{
  "mcpServers": {
    "datadog": {
      "type": "http",
      "url": "<YOUR_MCP_SERVER_ENDPOINT>"
    }
  }
}

You'll need mcp_read (or mcp_write) permission plus access to the resources you query. Teams use it to build automated code-change proposals, group and analyze debugging logs, and pull code-in-context-with-telemetry to speed up incident investigation. Usage has generous rate limits and is tracked in the Audit Trail.

Autonomous AI agents fail in ways a single API call never does — infinite tool-call loops, runaway cost, a wrong decision three steps deep. AI Agent Monitoring is observability built for that: it visualizes the agent's execution graph, every tool call and decision as a span, with latency, cost, and errors at each step.

It answers the questions unique to agents: Did it pick the right tool? Is it stuck in a loop? Where did the cost blow up? Which decision led it wrong?

The Bits AI Dev Agent closes the loop from problem to fix. When Datadog surfaces an issue (a bug, a Watchdog anomaly, a flaky test), the Dev Agent can generate a code fix and propose a pull request — turning an alert into a reviewable change instead of a ticket.

It's part of a family of domain-expert Bits agents (incident response, product development, security) built on shared tasks: Bits AI SRE finds the root cause, the Dev Agent drafts the code change, a human reviews and merges.

So far the AI has been Datadog watching your systems. LLM Observability flips it: it's how you monitor the AI applications you build — chatbots, RAG pipelines, and agents. It helps you debug root causes, measure performance and cost, and evaluate quality, privacy, and safety.

The core model: traces and spans

Every request produces a trace — the request's full journey — which can be a single LLM call, a predetermined workflow (LLM calls + tools + preprocessing), or an agent-executed workflow. Inside are spans: individual steps or agent decisions, each carrying inputs, outputs, latency, errors, and privacy metadata.

It monitors performance & cost (tokens, latency, spend), quality via topic clustering ("Patterns"), and security & privacy (auto-masking sensitive data, detecting prompt injection).

Datadog's Python SDK ships inside ddtrace and auto-instruments OpenAI, LangChain, Anthropic, and Bedrock — capturing latency, errors, and tokens without code changes.

pip install ddtrace   # Python 3.7+

# Run your app with LLM Observability enabled
DD_SITE=datadoghq.com DD_API_KEY=<your_key> \
DD_LLMOBS_ENABLED=1 DD_LLMOBS_ML_APP=support-bot \
DD_LLMOBS_AGENTLESS_ENABLED=1 \
  ddtrace-run python app.py

Auto-instrumentation captures raw calls; to model your app's workflow, use the decorators @workflow, @task, and @llm with LLMObs.annotate():

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import workflow, task, llm

@workflow
def process_request(question):
    clean = sanitize_input(question)
    return call_model(clean)

@task
def sanitize_input(text):
    return text.strip()

@llm(model_name="gpt-4", model_provider="openai")
def call_model(prompt):
    answer = "AI is..."   # your real model call
    LLMObs.annotate(
        input_data=[{"role": "user", "content": prompt}],
        output_data=[{"role": "assistant", "content": answer}],
        metrics={"input_tokens": 12, "output_tokens": 24},
    )
    return answer

Open LLM Observability and you'll see the trace: the process_request workflow, the sanitize_input task, and the call_model span with its tokens and latency.

Traces tell you what happened; evaluations tell you whether it was any good. Datadog clusters user inquiries into topics ("Patterns") so you can see what people actually ask, spot coverage gaps, and track whether answer quality is improving or drifting.

Safety is built in: it automatically masks sensitive data so PII doesn't leak into traces, flags prompt-injection attempts, and its Agent Observability Insights surface unexpected behavior changes (a quality regression, a cost spike, a new failure mode).

Real User Monitoring (RUM) instruments your web/mobile front end to capture real users' page loads, route changes, errors, and Core Web Vitals — from actual devices, not a lab. Session Replay reconstructs a user's session like a video, so you can watch the rage-click or broken form that a metric only hinted at.

Because RUM connects to backend traces, you can jump from a slow page a user saw straight into the APM trace behind it — front end to database in two clicks.

Synthetic tests proactively check your app from Datadog's global locations before users hit a problem. API tests verify endpoints and uptime; browser tests record multi-step user journeys (log in → search → checkout) and alert if any step breaks. Great for SLAs and catching regressions in CI.

Datadog reuses your existing logs and traces for security. Cloud SIEM applies detection rules to your logs to surface threats (suspicious logins, privilege escalation) as security signals. Cloud Security Management (CSM) flags misconfigurations and risks in your cloud posture. App and API Protection (AAP) detects and blocks attacks (like the OWASP Top 10) in-app, using the traces you already collect.

The Sensitive Data Scanner scans logs, traces, RUM events, and more for PII and secrets (emails, credit cards, API keys) using pattern-based rules, then redacts or hashes them before they're stored. It's how you keep compliance clean across all that telemetry.

Click-ops doesn't scale. The Datadog Terraform provider lets you define monitors, dashboards, SLOs, and more as version-controlled code — reviewed in PRs, rolled out consistently, and reproducible across teams.

resource "datadog_monitor" "high_5xx" {
  name    = "High 5xx on checkout"
  type    = "metric alert"
  query   = "sum(last_5m):sum:trace.http.request.errors{service:checkout} > 50"
  message = "Checkout errors elevated @slack-oncall"
}

Workflow Automation runs actions in response to signals — auto-remediate (restart a service, scale up), enrich an incident, or open a ticket — with hundreds of integration steps and no glue code. Combined with the AI module, Datadog can detect → investigate → act with minimal human toil.

Datadog bills per product, each with its own unit: Infrastructure per host, APM per host, Logs per GB ingested and per million events indexed, RUM/Synthetics per session/test run. The AI features (Bits AI) use an AI Credits model — you consume credits as agents do work — so keep an eye on it as you delegate more.

Two credentials to know: an API key (identifies your org, used by Agents and the API) and an Application key (identifies a user, used for read/query API calls). Manage them, and per-user permissions like mcp_read, in the org settings.

Here's how the pieces chain into one loop — the payoff of using Datadog end to end:

1. Watchdog detects. Checkout latency creeps past its learned baseline. No one wrote this monitor — Watchdog flags it and estimates user impact. (L16)
2. Bits AI SRE investigates. It spins up hypotheses in parallel, queries logs/traces/metrics, and finds a recent deploy changed a DB query — marked validated with evidence. (L18)
3. The fix gets drafted. Findings post to Slack and the On-Call incident; the Bits AI Dev Agent proposes a PR. (L22)
4. From your editor. A dev opens Claude Code, and via the MCP Server pulls the exact traces into context to confirm the fix. (L20)
5. Your AI features stay observed too. Meanwhile LLM Observability traces your support bot — cost, latency, quality, and prompt-injection — so a regression there surfaces the same way. (L23)
6. Automation acts. A Workflow scales the service and files the ticket; an SLO tracks whether the error budget held. (L31, L7)

Best practices to take with you

• Let Watchdog run — use monitors for known thresholds, Watchdog for the unknowns.
• Start with Bits collaboratively, then delegate more as you trust it.
• Connect the MCP Server to your editor early — observability where you write code is a step change.
• Instrument AI apps with @workflow/@task/@llm from day one.
• Manage monitors/dashboards as code, and watch your AI Credits.