Building a Retrieval-Augmented Generation (RAG) Agent with Azure AI Foundry and Azure AI Search

One of the biggest challenges when building AI applications is enabling them to answer questions using your own business data rather than relying solely on information learned during training.

Retrieval-Augmented Generation (RAG) solves this problem by combining the reasoning capabilities of large language models with the ability to search and retrieve information from trusted data sources.

In this tutorial, we'll build a complete RAG solution using Azure AI Foundry and Azure AI Search. We'll create a searchable knowledge base, configure an AI agent, and connect the two together so users can ask natural language questions and receive responses grounded in their indexed documents.

The following architecture illustrates the solution we will build.

To build the RAG solution, we will complete the following stages:

1. Create an Azure Resource Group
2. Create an Azure Storage Account
3. Upload the sample sales data to Azure Storage
4. Create an Azure AI Foundry resource
5. Deploy an embedding model
6. Create and populate an Azure AI Search index
7. Create an Azure AI Foundry Agent using GPT-4.1-mini
8. Connect the agent to Azure AI Search
9. Test the completed RAG solution

Prerequisites

Before starting this tutorial, ensure you have:

1. An active Azure subscription
2. Permission to create Azure resources within your subscription

Note: Microsoft periodically updates the Azure Portal interface. The screenshots and navigation paths shown in this guide were accurate when the article was written, but minor differences may exist in newer versions of the portal.

Step 1 - Create an Azure Resource Group

This step creates a resource group that allows all Azure resources used throughout the tutorial to be managed together.

1. Go to Azure Portal
2. Search for 'Resource Group' and navigate to it.
3. Click Create.
   1. Choose a subscription.
   2. Give the resource a name. (I used 'Rag-Test02')
   3. Choose a region.
   4. Click Create

Step 2 - Create an Azure Storage Account

Before the sales data can be indexed and queried by our AI agent, it must first be uploaded to Azure Storage.

1. Search for 'Storage Account' and navigate to it.
2. Click Create.
   1. Choose a subscription.
   2. Give the account a name (I used ragtest02storage)
   3. Select the resource group we created in step 1.
   4. Choose a region.
   5. Click Create

Step 3 - Upload the sample sales data to Azure Storage

Now that the storage account has been created we can create a container to store our sales data.

Note : I used Video game sales data as a .csv which I got from Find Open Datasets and Machine Learning Projects | Kaggle

1. Click on the newly created Storage Account.
2. Navigate on the left menu to containers.
3. Click 'Add Container'.
   1. Give the Container a name.
   2. Click Create
4. Click on newly created container.
5. Click Upload -> Browse to file. Click upload. (I uploaded the file Video_Games_Sales_Cleaned.csv dataset into my container I created called videoGamesales)

Step 4 - Create an Azure AI Foundry resource

1. In the Azure Portal, search for 'AI Foundry Resource'
2. Click 'Create a Foundry Resource'
   1. Select the Resource group we created in step 1.
   2. Give a name to our AI Foundry Resource (I used RagTestVideoGame02)
   3. Select a region.
   4. Give a project name.
   5. Click 'Review + create'
3. Once the resource has been created click on 'Go To Resource'
4. Then click on Go To Foundry Portal

Step 5 - Deploy an Embedding Model

Large Language Models do not have direct access to external datasets. To enable semantic search, we create an embedding model that converts our source data into vectors that Azure AI Search can index and retrieve.

1. Click on 'View Deployments'
2. Then click 'Deploy a base model'
3. Once selected Click 'Deploy'
4. For this tutorial, we'll use the 'text-embedding-3-small' model. Search for it and then click on it. This will take you to a page describing the model.

Step 6 - Create and populate an Azure AI Search index

Azure AI Search acts as the retrieval layer of our RAG solution. It indexes the vectorised data and returns relevant information when the agent receives a question.

1. Go to Azure Portal.
2. Search for 'Azure AI Search'
3. Click 'Create'
4. Go to the created AI Search resource.
5. Click 'Import Data'
   1. Select 'Azure Blob storage'
   2. Select 'Rag Scenario'
   3. Select Storage Account and the container we created in Step 2
   4. Select the Foundry Project created in Step 4d.
   5. Then select the 'text embedding' model we created in step 5.
   6. Review and Create.

Step 7 - Create an Azure AI Foundry Agent using GPT-4.1-mini

This step creates an Azure AI Foundry Agent that can be queried either through the Foundry chat interface or programmatically using C#.

1. Go to Azure AI Foundry Portal.
2. Select the AI Foundry project created in Step 4d.
3. Deploy Model
   1. Click on the 'View Deployments' button
   2. Then click on the 'Deploy Base model' button
   3. Pick the AI model to use. In this instance we pick 'gpt-4.1-mini'
   4. Click Deploy
   5. Configure Model
      1. Add instructions to help guide the model to give responses using the uploaded sales data.
      
   6. Click 'Save as Agent'.

Step 8 - Connect the agent to Azure AI Search

This step connects the agent to the Azure AI Search index created earlier, allowing it to retrieve relevant information when responding to user queries.

1. Select Knowledge within the newly created Agent.
2. Create a Connection
   1. Select the Foundry IQ Resource that we created in step 4 2b and Select API Key for authentication type.
   2. Create a Knowledge Base the Agent can use by click on 'Create a knowledge base'
   3. Provide a name, select a model and output mode to 'Extractive data'
   4. Click 'Add Source' and select the Azure AI Search Index created in Step 6. Select 'Azure AI Search Index'
   5. Select the Search Index we created in step 6. Note this is the name of the search index created within our azure AI Search Service.

Step 9 - Test the completed RAG solution

Once the knowledge base has been connected, open the agent chat window and ask a question about the uploaded sales data.

Summary

In this tutorial we built a complete Retrieval-Augmented Generation (RAG) solution using Azure AI Foundry and Azure AI Search.

We created an Azure Storage Account, uploaded source data, generated embeddings, built a searchable index, and configured an Azure AI Foundry Agent capable of retrieving and reasoning over that data.

This architecture enables AI applications to provide responses based on trusted business data rather than relying solely on the model's training knowledge. It also forms the foundation for more advanced solutions such as enterprise knowledge bases, internal support agents, and document search applications.

Further Reading

• Microsoft Learn - Azure AI Search Documentation
• Microsoft Learn - Azure AI Foundry Documentation
• Microsoft Learn - Vector Search in Azure AI Search
• Microsoft Learn - Azure AI Foundry Agents Overview

Related Articles

• Building a C# Client for an Azure AI Foundry RAG Agent
• A high-level introduction to Foundry for engineering teams.
• A practical workflow for building AI solutions using Foundry.
• How to Modernise Legacy NET Systems Using Azure
• When Should a Small Business Invest in AI?