Setting up Azure DevOps Private Agents in Azure Kubernetes Service (AKS)
Introduction
The conventional method for configuring Azure DevOps private agents involves deploying them on virtual machines (VMs). However, an interesting alternative is to utilize Azure Kubernetes Service (AKS) for this purpose.
This article serves as a guide for setting up Azure DevOps private agents within an AKS cluster using Helm charts, providing step-by-step instructions for the process.
Opting to deploy Azure DevOps private agents in AKS instead of provisioning new VMs can significantly impact cost-efficiency, scalability, and manageability of CI/CD pipelines. While VMs are easy to set up, they incur higher costs and may suffer from resource underutilization due to each VM requiring a full operating system and dedicated resources regardless of workload. This setup introduces complexity in scaling and maintenance, alongside increased compute and storage expenses.
In contrast, hosting Azure DevOps private agents in AKS offers a more dynamic and cost-effective solution. AKS abstracts much of the complexity in managing containerized applications and leverages Kubernetes' orchestration capabilities. By deploying DevOps agents as containers within AKS, organizations benefit from automatic scaling based on workload, faster startup times compared to VMs, and simplified management of agent fleets. This article aims to explore the advantages of using AKS over VM-based approaches and provide a comprehensive guide to efficiently implementing Azure DevOps private agents within AKS, enhancing DevOps workflows for improved performance, reliability, and cost-effectiveness.
What is an Azure DevOps Private Agent?
At its core, an Azure DevOps private agent is a software agent that runs on a user-provided infrastructure, whether that's on-premises servers, virtual machines, or within containerized environments like Azure Kubernetes Service (AKS). These agents are responsible for executing the tasks defined in Azure DevOps pipelines, such as compiling code, running tests, and deploying applications. Each private agent is registered to a specific Azure DevOps organization and can be further associated with one or more projects within that organization.
Here are some advantages of Azure DevOps private Agents:
-
Self-Hosted: Unlike the hosted agents provided by Azure DevOps, which are managed by Microsoft and run in a shared environment, private agents are self-hosted by you or your organization. You install, configure, and manage these agents on your own infrastructure, giving you more control over the execution environment.
-
Customization: Private agents can be customized to match the specific requirements of your build and release tasks. You can install additional software, configure environment variables, and modify the agent to your project's needs.
-
Security and Isolation: Private agents are often used when you need to execute CI/CD jobs in an environment that is not accessible from the public internet or when you want to maintain greater control over security. Private agents can operate within your private network, enhancing security and isolation.
-
Access to Private Resources: If your CI/CD processes need access to private resources, databases, or services that are not publicly accessible, private agents can be configured to reach these resources within your network.
Technical Scenario
When you're operating Azure services within a private network that is not accessible via the public internet, and you need connectivity from an Azure DevOps pipeline located in a public network, the solution is to deploy a private build machine within the same virtual network as your private services. This private build machine acts as a bridge, facilitating seamless connections to resources within the private network from Azure DevOps pipelines.
For instance, if you've established a private AKS cluster to run your applications, you'll need a dedicated private self-hosted agent within the same virtual network to facilitate the deployment process. Similarly, consider a scenario where you've configured a PostgreSQL server with a private DNS zone, and your Azure DevOps pipeline resides in a public network. To establish a connection to this database, it is necessary to deploy your own private agent within the virtual network housing the PostgreSQL server. This solution enables secure and efficient communication with your private resources.
Objective
In this exercise, our objective is to accomplish and learn the following tasks:
- Step-1: Installing Private Agent Helmchart in Kubernetes
- Step-2: Verify private agent resources in AKS
- Step-3: Create new agent pool in Azure DevOps
- Step-4: Create new Personal Access Token (PAT)
- Step-5: Register the Self-Hosted Agent
- Step-6: Update Build pipeline with Private Agent.
- Step-7: Test the new Private Agent
Prerequisites
Before we begin, ensure you have the following prerequisites in place:
-
Azure Subscription: You must have an active Azure subscription to create and manage an AKS cluster.
-
Azure Kubernetes Service (AKS) Cluster: An operational AKS cluster where you will deploy the Azure Pipelines agent. If you do not have one, you must set it up beforehand.
-
Azure DevOps Organization: Access to an Azure DevOps organization is required for creating agent pools and pipelines.
-
Azure CLI: The Azure Command Line Interface (CLI) installed on your workstation for managing Azure resources, including AKS.
-
kubectl: The Kubernetes command-line tool,
kubectl, must be installed and configured to communicate with your AKS cluster. This tool is essential for deploying and managing applications on Kubernetes. -
Helm: Helm, the package manager for Kubernetes, is required for installing and managing the private agent Helm chart. Ensure Helm is installed and ready to use on your system.
-
Permissions in Azure DevOps: Adequate permissions within your Azure DevOps organization to create and manage agent pools, personal access tokens (PATs), and pipelines.
Implementation Details
login to Azure
Verify that you are logged into the right Azure subscription before start anything in visual studio code
# Login to Azure
az login
# Sets Azure subscription to desired subscription using ID
az account set -s "anji.keesari"
Connect to Cluster
To interact with your Azure Kubernetes Service (AKS) cluster, you need to establish a connection. Depending on your role, you can use either the User or Admin credentials:
# Azure Kubernetes Service Cluster User Role
az aks get-credentials -g "rg-aks-dev" -n "aks-cluster1-dev"
# Azure Kubernetes Service Cluster Admin Role
az aks get-credentials -g "rg-aks-dev" -n "aks-cluster1-dev" --admin
# verify the aks connection by running following commands
kubectl get no
kubectl get namespace -A
Deployment Methods
Deploying Azure DevOps private agents within an Azure Kubernetes Service (AKS) cluster offers a modern approach to managing build and deployment infrastructure, providing scalability and cost efficiency. There are two primary methods to achieve this deployment, each with its own set of procedures and advantages.
Method 1: Creating and Deploying Your Own Docker Container to AKS
This method involves packaging the Azure DevOps agent into a custom Docker container and then deploying that container to your AKS cluster. It allows for maximum customization and control over the agent environment.
- Prepare the Azure DevOps Agent Dockerfile
- Build the Docker Image
- Push the Image to a Container Registry
- Deploy the Container to AKS
- Configure and Run the Azure DevOps Agent
Advantages:
- Customizability: This method offers the most flexibility, allowing you to include any tools, scripts, or configurations needed for your builds.
- Control: You have complete control over the agent's environment, updates, and lifecycle management.
Option 2: Deploying Private Agent Using Helm Charts to AKS
Helm charts offer a simplified and declarative way to deploy and manage applications in Kubernetes, including Azure DevOps agents.
Advantages:
- Simplicity: Helm charts abstract away much of the complexity associated with Kubernetes deployments, making it easier to deploy and manage Azure DevOps agents.
- Scalability: Easily scale the number of agents up or down by adjusting the Helm release.
- Reusability: Helm charts can be shared and reused across different environments or projects, promoting consistency and saving time.
Both methods provide a way to implement in AKS for Azure DevOps agent deployment, with the choice depending on your specific needs, expertise, and preferences for customization and management.
In this article, we will be utilizing Method 2 to install the private agent, which involves the installation of Private agent Helm charts in AKS, However in case if you want to test docker container locally before using helmchart, here are commands:
docker run -d -e AZP_AGENT_NAME="<agent name>" -e AZP_URL="https://dev.azure.com/<your org.>" -e AZP_POOL="<agent pool>" -e AZP_TOKEN="<PAT>" emberstack/azure-pipelines-agent
verify docker image and container
Step-1: Installing Private Agent Helmchart in Kubernetes
Find the azure-pipelines-agent Helmchart from emberstack on the ArtifactHUB website:
Click on the "Install" button to retrieve the necessary details for this Helmchart installation
For this installation, I am going to use the following Helmchart. You can find detailed information about this Helm chart on the following website:
azure-pipelines-agent Helm chart
Add helm repo
Update helm repo
Now that you have a Helm chart for your Azure Pipelines agent, it's time to deploy it to your AKS cluster.
Install helmchart
output
NAME: azure-pipelines-agent
LAST DEPLOYED: Mon Feb 5 09:01:21 2024
NAMESPACE: build-agent
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Check your Azure DevOps portal to manage the Azure Pipelines Agent.
List helm chart
output
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
azure-pipelines-agent build-agent 2 2024-02-05 09:02:09.946719 -0800 PST deployed azure-pipelines-agent-2.2.26 2.2.26
Lsit helm chart history
output
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Mon Feb 5 09:01:21 2024 superseded azure-pipelines-agent-2.2.26 2.2.26 Install complete
2 Mon Feb 5 09:02:09 2024 deployed azure-pipelines-agent-2.2.26 2.2.26 Upgrade complete
Show helm chart status
output
NAME: azure-pipelines-agent
LAST DEPLOYED: Mon Feb 5 09:02:09 2024
NAMESPACE: build-agent
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
Check your Azure DevOps portal to manage the Azure Pipelines Agent.
Step 2. Verify private agent resources in AKS
Validate to make sure all the deployments / services created and running as expected.
Run the following kubectl commands to verify the installation in the AKS cluster.
or
kubectl get namespace build-agent
kubectl get deployments -n build-agent
kubectl get pods -n build-agent
kubectl get services -n build-agent
expected output
Ensure that the agent deployment in AKS is successful and verify its connectivity to Azure DevOps. You can also configure auto-scaling rules for the agent pool as needed.
Verify the pod deployment by checking pod logs.
output
Step-3: Create new agent pool in Azure DevOps
To group and manage your private agents effectively, create a new agent pool in Azure DevOps. This information will be used during agent registration.
Here are the steps to create a new agent pool in Azure DevOps :
- Sign in to Azure DevOps.
- Access your organization or Project depnding on requirement.
- Navigate to Project Settings.
- Click on "Agent pools."
- Create a new agent pool by clicking "+ New agent."
- Configure the agent pool with a name, description, visibility, and security settings.
- Click "Create" to complete the process.
- The new agent pool is now ready for use.
Step-4: Create new Personal Access Token (PAT)
Generate a new Personal Access Token (PAT) with required permissions to authenticate the agent with Azure DevOps. This token will be used during agent registration.
here are the steps to create a new Personal Access Token (PAT) in Azure DevOps:
- Sign in to Azure DevOps.
- Access your organization.
- Go to User Settings > Security.
- Click "New token" under Personal access tokens.
- Configure the PAT: name, expiration, access scope.
- Click "Create" to generate the token.
- Copy and securely store the PAT.
- Confirmation message will appear.
- Use the PAT for authentication in Azure DevOps services.
Step-5: Register the Self-Hosted Agent
After deploying the agent to your AKS cluster using the Helm chart, the next crucial step is to register the agent with Azure DevOps. This process involves configuring the agent with essential details such as your Personal Access Token (PAT), the agent pool for assignment, and any specific capabilities the agent should possess. Successfully completing this step links your self-hosted agent to Azure DevOps, enabling it to execute pipelines and tasks.
Updating Parameters in the Helm Chart
To register the agent, you'll need to update the Helm chart with specific parameters related to your Azure DevOps setup. These include:
pipelines.url: The base URL for your Azure DevOps organization.pipelines.pat.value: The Personal Access Token (PAT) that the agent will use to authenticate with Azure DevOps.pipelines.pool: The name of the agent pool to which the agent should be registered.
Ensure that you replace placeholders like your-organization, your-personal-access-token, your-agent-pool-name, and any other specific values with the actual information relevant to your setup.
Command to Update Helm Chart Values:
Execute the following command to update the Helm chart values and register the agent with Azure DevOps:
helm upgrade azure-pipelines-agent emberstack/azure-pipelines-agent --set pipelines.url=https://dev.azure.com/orgname,pipelines.pat.value=lsir5gjt2djieulvmlmgv66jdrbmcaeww4oydtsxf25ap52ztpyq,pipelines.pool=azure-pipelines-agent --namespace "build-agent"
Remember to replace your-organization, YOUR_PERSONAL_ACCESS_TOKEN, and your-agent-pool-name with your specific details.
Expected Output:
Release "azure-pipelines-agent" has been upgraded. Happy Helming!
NAME: azure-pipelines-agent
LAST DEPLOYED: Mon Feb 5 09:51:23 2024
NAMESPACE: build-agent
STATUS: deployed
REVISION: 9
TEST SUITE: None
NOTES:
Check your Azure DevOps portal to manage the Azure Pipelines Agent.
This output confirms that the Helm chart has been successfully updated and the agent is registered with Azure DevOps. You can now navigate to the Azure DevOps portal to manage and utilize your self-hosted agent within your CI/CD pipelines.
Step-6: Update Build pipeline with Private Agent
Once your self-hosted private agent is successfully registered and operational within Azure DevOps, the subsequent step involves integrating this agent into your build pipelines. By specifying the agent pool containing your private agent in the pipeline configuration, you can direct your builds to run on this self-managed infrastructure, leveraging its benefits for your CI/CD processes.
Modifying Azure Pipelines Configuration
To utilize your self-hosted private agent, you need to modify the configuration of your Azure Pipelines. This adjustment involves specifying the agent pool that your private agent belongs to. By doing so, you ensure that the pipeline runs on your private infrastructure.
How to Update Your Build Pipeline with the New Agent Pool:
You can update the YAML file defining your build pipeline by including the pool property with the name of your agent pool. Below is an example of how to specify the agent pool within a job in your Azure Pipelines YAML configuration:
Step-7: Test the new Private Agent
To verify that your self-hosted agent is working as expected, you can queue a build or release pipeline that targets the agent pool where your AKS agent is registered. Azure Pipelines will automatically route the job to your self-hosted agent, and you can monitor the job's progress in the Azure DevOps portal.
Conclusion
Setting up a self-hosted Azure Pipelines agent in an AKS cluster using Helm charts offers control, scalability, and resource efficiency for your CI/CD workflows. By following the steps outlined in this article, you can identify a Helm chart, deploy it to your AKS cluster, and register the agent with Azure DevOps seamlessly.