🔍 Lab 2: Exploring AI Traces in Dynatrace

💻 Developer: “Why is my RAG giving bad answers?”

Your story: You’ve deployed a RAG-powered chatbot, but users are complaining that sometimes it gives irrelevant or incomplete answers. You need to understand:

• Is the vector search retrieving the right documents?
• Is the context being formatted correctly for the LLM?
• What prompts are actually being sent to the model?

Your goal: Learn to trace a request end-to-end, inspect prompts/completions, and identify where your RAG pipeline might be breaking down.

Focus on: Steps 4, 5, and 6 (marked with 💻)

💻 Step 4: Analyze an AI Trace

4.1 Understanding the Trace Structure

A typical RAG request trace includes these spans:

📍 rag_chat_pipeline.workflow (Main RAG pipeline)
  └── 📍 analyze_query_intent.task (Classify user query type)
      └── 📍 AzureChatOpenAI.chat (LLM call for classification)
  └── 📍 retrieve_documents.task (Document retrieval)
      └── 📍 openai.embeddings (Generate query embedding)
      └── 📍 chroma.query (Vector store search)
  └── 📍 generate_context.task (Format retrieved docs)
  └── 📍 generate_response.task (Generate final answer)
      └── 📍 AzureChatOpenAI.chat (LLM completion call)

4.2 Examine the LLM Span

Click on the azure_openai.chat span under the analyze_query_intent.task to see:

4.3 View Prompts and Responses

Note: Depending on configuration, you may see:

• gen_ai.prompt.0.content - The input prompt content
• gen_ai.prompt.0.role - The prompt role (user, system)
• gen_ai.completion.0.content - The generated response content
• gen_ai.completion.0.role - The completion role (assistant)
• gen_ai.completion.0.finish_reason - Why generation stopped (stop, length)

This visibility is crucial for debugging AI applications!

💻 Step 5: Analyze Embedding Spans

5.1 Find the Embedding Span

In the trace view, locate the openai.embeddings span.

5.2 Examine Embedding Details

Key attributes include:

💻 Step 6: Vector Store Spans

6.1 Find the Vector Store Span

Look for chroma.query or similar vector database spans.

6.2 Key Insights

Click on the chroma.query span to see database attributes:

🔧 Step 7: Token Optimization

Understanding Token Limits

All AI models have maximum input and output tokens that they can accomodate. In our case, we’re using GPT-4o with the following limits:

7.1 Create a New Notebook

1. Navigate to Notebooks in the left-hand menu
2. Click + Notebook on the top to create a new notebook
3. Name it: AI Observability - {YOUR_ATTENDEE_ID}
4. For each DQL query, create a new DQL tile in your Notebook.

Lookup Tables

For this lab, we’ve made use of lookup tables. Lookup tables allow us to upload referencable tables that we can use to enrich our data in Dynatrace. In this case, we’ve created a lookup table with the maximum input/output tokens for our LLM model to make our following DQL queries more dynamic and robust in case prices ever change in the future.

To see the table for this lab, run the following DQL query:

load "/lookups/ai/azure-openai/model-max-tokens"

To see all lookup tables, run the following DQL query:

fetch dt.system.files

Documentation

7.2 Find the Biggest Token Spenders and Understand What Percentage of Token Limits are Used

//Find the Biggest Token Spenders and Understand What Percentage of Token Limits are Used
fetch spans
| filter service.name == "ai-chat-service-{YOUR_ATTENDEE_ID}"
| filter isNotNull(gen_ai.usage.input_tokens)
| summarize 
    total_input = sum(gen_ai.usage.input_tokens),
    total_output = sum(gen_ai.usage.output_tokens),
    avg_input = avg(gen_ai.usage.input_tokens),
    avg_output = avg(gen_ai.usage.output_tokens),
    request_count = count(),
  by: {gen_ai.response.model}
| fieldsAdd total_tokens = total_input + total_output
| lookup [load "/lookups/ai/azure-openai/model-max-tokens"], sourceField:gen_ai.response.model, lookupField:model
| filter isNotNull(lookup.model)
| fieldsAdd input_token_usage_percent = (avg_input / lookup.max.tokens.input)*100
| fieldsAdd output_token_usage_percent = (avg_output / lookup.max.tokens.output)*100
| fieldsRemove "lookup*"
| fields gen_ai.response.model, request_count, total_input, total_output, avg_input, avg_output, input_token_usage_percent, output_token_usage_percent

💻 Developer Takeaways

You now know how to debug your RAG pipeline using traces:

1. ✅ Navigate the trace structure to understand your RAG workflow
2. ✅ Inspect LLM spans to see prompts, completions, and model parameters
3. ✅ Analyze embedding spans to verify query vectorization
4. ✅ Check vector store spans to confirm document retrieval
5. ✅ Use span attributes to debug why your AI gives certain responses

Next time your RAG gives a bad answer: Open the trace, check the retrieved documents, and inspect what prompt was actually sent to the LLM.