Category: Technology

The Hidden Cost of “Simple” Search Apps

Many teams we talk to already use Azure AI Search. It’s a powerful service for making text and documents searchable with semantic and vector search.

But here’s the pattern we see over and over:

• A new web app is built (maybe Streamlit, maybe a custom React app).
• It’s hosted on Azure App Service or VMs.
• It duplicates authentication, hosting, monitoring, DevOps pipelines…
• And at the end of the day, users just get a search box + results table.

The business value is real, but the delivery is complex and costly for what it achieves.

Imagine if your users could

• Type a search query,
• Get semantic results with highlights and summaries,
• And see them right inside the Power BI dashboards they already use every day.

No new app.
No separate portal.
No extra infra to maintain.

Just a familiar search box in Power BI, powered by Azure AI Search + AI summarization.

Also note that this approach requires no PowerApps or Power Automate — it runs entirely within Power BI.

What does it include

Power BI report provides:

• Free text input for users to type search queries.
• Native Power BI filtering (date, region, product, etc.) alongside semantic search results.
• Paging support to navigate through large result sets.
• Export options to Excel, CSV, PDF, and more.
• Multiple report pages to query across different indexes, documents or datasets.
• Power BI native authentication to the report

High-Impact Use Cases

Organizations like yours can quickly benefit from this approach for scenarios such as

• Customer Service: Mine complaint text for themes (refunds, delivery delays, product defects).
• Compliance & Legal: surface contract clauses or policy excerpts directly in dashboards.
• Ops & IT: search across incident logs and root cause notes.
• HR & Internal Comms: make policies instantly discoverable by employees.

All without building another app that IT must support.

Why it matters?

• Cost savings: no Azure App Service, no custom UI hosting, no redundant auth.
• User adoption: everyone already knows Power BI. No training needed.
• Speed: what used to take weeks of app dev can now be delivered in days.

Where This Approach Fits Best

This design is intentionally simple and focused. It excels when you need:

• A single query to retrieve relevant results.
• Clear insights (summaries, highlights, tags) displayed directly in Power BI.
• Seamless integration with existing dashboards and metrics.

It’s built for search + analytics, not for chatbot-style experiences.

So, if your scenario requires things like:

• Conversational Q&A with follow-up questions,
• Multi-turn history or context retention,
• Uploading and reasoning over new documents during query,

Those are better served by a different architecture.

Think of this as “one search → one set of insights → shown in Power BI” — fast, clean, and highly effective for dashboards.

Your Next Step Toward Smarter Search

The exciting part here isn’t that it’s impossible tech or solution — it’s that the approach is far simpler than many expect, and that simplicity makes it faster, cheaper, and easier to adopt.

If you’re juggling multiple initiatives, this approach is lightweight and ideal for proving value before scaling further. You don’t need to build a full-blown application — instead, you can get something off the ground in days, not weeks. The sooner you see it running on your own data, the faster you’ll recognize its value.

Bring us your use case, and we’ll show you results fast (POC/implementation). You can skip the app dev overhead and light it up search inside Power BI directly. You’ll be surprised how simple (and cost-effective) it can be.

Data resilience is a cornerstone of modern analytics platforms. In Microsoft Fabric, maintaining backups and implementing automated policies for retention and restoration can elevate data management.

While Fabric is a robust platform, disaster recovery (DR) is not designed to address operational issues like data refresh failures or accidental deletions, which necessitate an automated approach to bridge the gaps and ensure operational continuity.

Effective backup, retention, restoration strategies are essential to maintaining reliable data platform, particularly in scenarios involving refresh failures or data corruption.

Note: This is not a substitute for disaster recovery feature of Microsoft Fabric, but a complementary approach to enhance resilience, streamline restoration processes, and minimize downtime through automation and proactive configurations.

Here’s an overview of setting up, configuring, and automating these processes while addressing challenges and their solutions.

Setting Up Backup and Retention Policies

Microsoft Fabric’s Lakehouse and OneLake provide unique capabilities for handling data. Backing up data involves:

• Daily Incremental Backups: Ensuring minimal data loss by creating daily snapshots.
• Retention Policy Configuration: Establishing tiers like daily, weekly, monthly, and yearly retention to balance storage costs and compliance.
• Automation with Notebooks: Using Fabric notebooks to schedule backups and enforce retention policies, such as retaining the last 7 daily backups or 6 monthly backups and cleaning up obsolete ones.

Automation Highlights:

1. Backup Creation: Scheduled scripts create snapshots at specific intervals. For example, Spark jobs can efficiently copy data using APIs like mssparkutils.
2. Retention Enforcement: A policy-driven approach automatically removes outdated backups while preserving critical ones for auditing or recovery.
3. Logging and Monitoring: Every backup, cleanup, and restoration action is logged to ensure transparency and auditability.

Restoration: Recovering from Data Loss

Fabric allows for full or selective restoration of data from backups. Restoration tasks involve:

• Restoring entire Lakehouse or specific tables from a backup.
• Using structured logs to identify and resolve errors during the restoration process.
• Minimizing downtime by enabling rapid data recovery with scripts or automation tools.

Why Automate Backup and DR in Microsoft Fabric?

Automation mitigates risks and improves efficiency:

• Data Integrity: Automated backups ensure all critical data is consistently safeguarded.
• Operational Continuity: Quick restoration scripts minimize business downtime.
• Cost Optimization: Automating cleanup eliminates outdated backups, reducing unnecessary storage expenses.
• Scalability: Structured policies can accommodate growing datasets without additional manual effort.

Conclusion

While Microsoft Fabric is a promising data platform, addressing data corruptions, accidental deletion challenges require a proactive and automated approach. By leveraging our automation for backup, retention, cleanup, and restoration, organizations can safeguard data, ensure business continuity, it provides significant value for the business.

Executive Summary

Retrieval-Augmented Generation (RAG) has quickly become one of the most glorified terms in enterprise AI. RAG is typically showcased over a handful of PDFs which seems to be easy and simple. But operating at enterprise scale—lakhs (hundreds of thousands) of records, low-latency retrieval, strict accuracy, and predictable costs—is hard. Real limits show up in four places: embeddings, vector indexes (e.g., Azure AI Search), retrieval/filters, and LLMs themselves. You’ll need hybrid architectures, careful schema/ops, observability, and strict cost controls to get beyond prototypes.

Embedding Challenges at Scale

Cost & volume. Embedding large datasets quickly runs into scale issues. Even a moderately sized corpus—each record carrying a few hundred to thousands of tokens—translates into tens of millions of tokens overall. The embedding phase alone can run into significant dollar costs per cycle, and this expense only grows as data is refreshed or re-processed.

Throughput & reliability. APIs enforce token and RPM limits. Production pipelines need:

• Token-aware dynamic batching
• Retry with backoff + resume checkpoints
• Audit logs for failed batches

Chunking trade-offs.

• Too fine → semantic context is lost
• Too coarse → token bloat + noisy matches Use header/paragraph-aware chunking with small overlaps; expect to tune per source type.

Domain relevance gaps. General-purpose embeddings miss subtle, domain-specific meaning (biomedical, legal, financial). Dimensionality isn’t the cure; domain representation is. Without specialization, recall will feel “lexical” rather than truly semantic.

Vector Databases: Strengths… and Constraints for RAG

Immutable schema. Once an index is created, fields can’t be changed. Adding a new filterable/tag field → recreate the index.

Full reloads. Schema tweaks or chunking updates often require re-embedding and re-indexing everything—expensive and time-consuming (parallelism will still hit API quotas).

Operational sprawl. Multiple teams/use cases → multiple indexes → fragmented pipelines and higher latency. Unlike a DB with views/joins, AI Search pushes you toward rigid, static definitions.

(Reference: Microsoft docs on Azure AI Search vector search and integrated vectorization.)

Retrieval & Filtering Limits

Shallow top-K. Even at top-K, relevant items can fall just outside the cut-off. In regulated domains, a single miss matters.

Context window pressure. As best practice we should send only required fields, then join externally on predicted Answers which will have key identifiers Like ID to finish the answer. (In our Scenario we sent 1-2 columns having Id, descriptions out of 25 columns in a table)

Filter logic ceiling. Basic metadata filters work, but you’ll miss nested conditions, dynamic role-based filters, and cross-field joins that are trivial in SQL.

LLM Limitations You Will Hit

Input limits. Even with large-context models (450k+ tokens),However, even within this boundary, we observed that certain records — particularly those in lower relevance ranges may be implicitly skipped during reasoning .This behaviour is non-deterministicand poses serious challenges for enterprise-grade tasks where every data point is critical

Output limits.Many models cap useful output (e.g., 2k–16k tokens). This affects multi-record responses, structured summaries, and complex decision-making outputs — leading to truncated or incomplete responses, particularly when returning JSON, tables, or lists.

Latency & variance. Complex prompts over hundreds of records can take minutes. Stochastic ranking creates run-to-run differences—tough for enterprise SLAs.

Concurrency & quotas. Enterprises face quota exhaustion due to shared token pools, Concurrent usage by multiple users or batch agents can quickly consume limits Smaller organizations with access to 8K input / 2K outputtoken models (e.g., LLaMA, Mistral) face even tighter ceilings — making RAG challenging beyond pilot projects

Productionisation, Observability — and Evaluation That Actually Matters

While most RAG demos end at a “right-looking” answer, real deployments must be observable, traceable, resilient, and continuously evaluated. GenAI pipelines often underinvest in these layers, especially with multiple async retrievals and LLM hops.

Operational pain points

• Sparse/unstructured LLM logs → hard to reproduce issues or inspect reasoning paths.
• Thin vector/AI Search telemetry → silent filter failures or low-recall cases go unnoticed.
• Latency tracing across hybrids (SQL + vector + LLM) is messy without end-to-end spans.
• Failure isolation is non-trivial: embed vs ranker vs LLM vs truncation?

What good observability looks like

• End-to-end tracing (embed → index → retrieve → rerank → prompt → output).
• Structured logs for: retrieval sets & scores, prompt/response token usage, cost, latency, confidence, and final joins.
• Quality gates and alerts on recall@K, latency budgets, cost per query, and hallucination/citation signals.

Evaluation: beyond generic metrics

• Partner with domain experts to define what “good” means. Automatic scores alone aren’t enough in regulated or domain-heavy settings.
• Build a domain ground-truth set (gold + “acceptable variants”) curated by SMEs; refresh it quarterly.
• Establish a human-in-the-loop loop: double-blind SME review on sampled traffic; escalate low-confidence or low-evidence answers by policy.
• Maintain an error taxonomy (missed retrieval, wrong join, truncation, unsupported query, hallucination) with severity labels; track trends over time.
• Run canaries/A-B tests in prod; compare quality, latency, cost, and SME acceptance before full rollout.
• Log evaluation metadata (query id, versioned index, chunking config, model/runtime version) so you can pinpoint regressions.

In practice (our pattern)

• We co-defined a gold set with SMEs and require evidence-backed answers; low-evidence responses are auto-routed to review.
• We track recall@K + citation coverage for retrieval, and field-level precision/recall for extraction tasks, alongside cost/latency dashboards.

Bottom line Without observability + domain-grounded evaluation, production RAG stays brittle and opaque. With them, you get a system you can debug, trust, and scale—not just a demo that looks good once.

Conclusion: Beyond the Buzzwords

While Retrieval-Augmented Generation (RAG) has gained mainstream attention as the future of enterprise AI, real-world adoption reveals a wide gap between expectation and execution. From embedding inconsistencies and rigid vector schema constraints to LLM context bottlenecks and high operational latency, the challenges compound rapidly at production scale. Even in enterprise environments with access to high-token models, limitations around completeness, determinism, and runtime stability remain unresolved.

This doesn’t mean RAG is fundamentally flawed—it’s a powerful paradigm when paired with the right retrieval tuning, agent orchestration, hybrid pipelines, and system-level observability. But as engineers and architects, it’s time we shift the conversation from aspirational posts to grounded, production-aware designs. Until embedding models evolve to be truly domain-specific, vector systems allow dynamic schemas, and LLMs deliver predictable performance at scale, RAG should be treated not as a plug-and-play solution—but as a custom-engineered pipeline with domain, data, and budget constraints at its core

This blog represents the collective strength of our AI team—where collaboration, innovation, and expertise come together to create meaningful insights. It is a testament to the value SNP delivers through its AI practice, showcasing how we help our customers turn possibilities into impact.

Data estates are becoming increasingly heterogeneous as data grows exponentially and spreads across data centers, edge devices, and multiple public clouds. In addition to the complexity of managing data across different environments, the lack of a unified view of all the assets, security and governance presents an additional challenge.

Leveraging the cloud for your SQL infrastructure has many benefits like cost reduction, driving productivity, accelerating insights and decision-making can make a measurable impact on an organization’s competitiveness, particularly in uncertain times. While infrastructure, servers, networking, etc. all by default are maintained by the cloud provider.

With SQL servers 2008 and 2012 reaching their end-of-life, it is advisable to upgrade them or migrate them to Azure cloud services. Modernizing any version of SQL server to Azure brings up many added benefits, including:

• Azure PaaS provides 99.99% availability
• Azure IaaS provides 99.95% availability
• Extended security updates for 2008, 2012 servers
• Backing up SQL Server running in Azure VMs is made easy with Azure Backup, a stream-based, specialized solution. The solution aligns with Azure Backup’s long-term retention, zero infrastructure backup, and central management features.

Tools leveraged

For modernizing the SQL infrastructure, SNP leveraged a variety of tools from Microsoft, such as the following.

• The Azure Database Migration Service has been used since the beginning to modernize on-premises SQL servers. Using this tool, you can migrate your data, schema, and objects from multiple sources to Azure at scale, while simplifying, guiding, and automating the process.
• Azure Data Studio is one of the newest tools for modernizing SQL infrastructure with an extension of Azure SQL Migration. It’s designed for data professionals who run SQL Server and Azure databases on-premises and in multi cloud environments.

Potential reference architecture diagram

Let’s take a closer look at the architecture, what components are involved and what is being done in Azure Data Studio to migrate or modernize the on-premises SQL infrastructure.

Among the components of Azure data studio are the source to be modernized, the destination where the on-premises SQL must be moved, and the staging layer for the backup files. Backup files are a major component of modernization.

There are various components that are involved in the Azure Data Studio migration or modernization- Source SQL server. The on-premise SQL server which is to be modernized/migrated, Destination Server- The Azure SQL VM to which the on-prem SQL server will be moved, and the staging layer (Storage Account or the Network Share Folder) for the backup files. Backup files are a major component of modernization.

Azure Data Studio and Azure SQL Migration primarily rely on backup files. It uses a full backup of the database as well as transactional log backups. Another important component is the staging layer, where backup files will be stored.

Microsoft Azure Data Studio uses a network share folder, an Azure storage container, or an Azure file. There must be a specific structure or order in which backup files are placed in either of the places. As shown in the below architecture, backup files specific to the Database must be placed in their own folders or containers.

As part of the migration to Azure, Azure Data Studio along with the Azure SQL Migration extension utilizes a technology called Data Migration Service, which is the core technology behind the scenes. It has also been integrated with Azure Data Factory, which runs the pipeline at regular intervals to copy the backup files from the on-prem network share folder to Azure thereby restoring them on the target or restoring them if they are in containers.

When the backup files are in a network share folder, Azure Data Studio uses Self Hosted Integration Run time to establish a connection between on-premises and Azure. After the connection has been established, the Azure Data Studio begins the modernization process leveraging Azure DMS.

Initially, all full and subsequent transactional log backup files of the databases are placed in a specified database folder or database container. Azure Data Studio copies backup files from network share folders to Azure storage containers if the backup files are in a network share folder.

Following this, Azure Data Studio restores them to the target Azure SQL VM or Azure SQL Managed Instance while Azure Data Studio directly restores backup files from the storage account to the Azure target if the backup files are stored in the storage account.

Following the completion of the last log restoration on the target Azure SQL database, we need to cut over the database and bring it online on the target. The databases will be placed in the Restoring mode during the restoration of the backup files, which means that we will not be able to access them until the cutover has been completed.

Your next steps

If you like what you have read so far, let’s move forward together with confidence. We are here to help at every step. Contact SNP’s migration experts.

Modernizing and migrating SQL Server is just one of the many reasons why a company might want to migrate its data. Other common reasons may include mergers, hardware upgrades or moving to the cloud. In most cases, however, data migrations are associated with downtime, data loss, operational disruptions, and compatibility problems.

With SNP Technologies Inc., these concerns are alleviated, and the migration process is simplified. We help businesses migrate complete workloads seamlessly through real-time, byte-level replication and orchestration. For enhanced agility with little to no downtime, we can migrate data and systems between physical, virtual and cloud-based platforms.

When it comes to the modernization of SQL Server, we can migrate and upgrade your workload simultaneously. Production sources can be lower versions of SQL Server that are then upgraded to newer versions, for example, SQL 2008, 2008R2, and 2012 can be moved to a newer version of Windows and SQL or to Azure.

 

Some key benefits of modernizing or migrating your sql workloads include:

• Built-in high Availability and disaster recovery for Azure SQL PaaS with 99.99% availability
• Automatic backups for Azure SQL PaaS services
• High availability with 99.95% for Azure IaaS
• Can leverage the azure automatic backups or Azure Backup for SQL Server on Azure VM

Listed below are the various steps SNP follows to migrate an on-premises SQL server to Azure PaaS or IaaS

• Assessment to determine what is the most appropriate target and their Azure sizing.
• A performance assessment will be conducted before the migration to determine potential issues with the modernization.
• A performance assessment will be conducted post-migration to determine if there is any impact on performance.
• Migration to the designated target.

 

As part of our modernization process, we utilize a variety of tools that Microsoft provides. The following are various tools or services we leverage during the modernization process.

Assessment to determine what is the most appropriate target and their Azure sizing with Azure Migrate:

Azure migrate is a service in Azure that uses Azure SQL Assessment to assess the customer’s on-premises SQL infrastructure. In Azure Migrate, all objects on the SQL server are analyzed against the target (whether it’s Azure SQL Database, Azure SQL Managed Instance or SQL Server on Azure VM) and the target is calculated by considering all performance parameters such as IOPS, CPU, Memory, Costing etc., along with the appropriate Azure size. Following the assessment, SNP gets a better idea of what needs to be migrated, while the assessment report recommends the most appropriate migration solution.

This assessment generates four types of reports:

• Recommended type (it gives us the best option by comparing all the available options (best fit)– If the SQL server is ready for all the targets, it will give us the best fit considering all the factors like performance, cost, etc
• Recommendation of instances to Azure SQL MI– It gives the information If the SQL server is ready for MI. If the SQL server is ready for MI, it gives us a target recommendation size. If the Server has any issues with SQL MI, it shows us all the various issues it has and its corresponding recommendations
• Recommendation of Instances to Azure SQL VM– It will assess individual instance and provides us with the suitable configuration specific to individual instance
• Recommendation of Servers to SQL Server on Azure VM– If the server is ready to move to the SQL server on Azure, it will give us the appropriate recommendation

Our assessment checks if there are any performance impacts post migration with Microsoft’s data migration assistant

To prepare for modernizing our SQL infrastructure to Azure, we need to know what objects will be impacted post-migration, so we can plan what steps to take post-migration. A second assessment is performed using a Microsoft Data Migration Assistant (DMA) tool to identify all the objects that will be impacted after migration. This tool can be used to determine which objects are going to be impacted post-migration during this phase. The DMA categorizes the objects into five/ four categories for modernizing to SQL Server on Azure VM.

Some key factors considered at this stage include:

1. Breaking Changes:  These are the changes that will impact the performance of a particular object. Following a migration, we will need to ensure that breaking changes are addressed.
2. Behavior Changes: There are changes that may impact query performance and should be addressed for optimal results.
3. Informational issues: We can use this information to identify issues that might affect post-migration
4. Deprecated Feature: These are the features that are going to be deprecated
5. Migration blockers: These are the objects that are going to block the migration, either we remove them prior to migration or change them as per the business requirements.

Note: Migration blockers are specific to the Modernization of SQL Server to Azure SQL PaaS

 

Modernization using Azure Data Studio:

Once we have an Azure target along with the Azure size and a list of affected objects, we can move on to modernization, where we migrate our SQL infrastructure to the Azure target. In this phase, the SQL infrastructure is modernized using a tool called Azure Data Studio, which uses an extension called Azure SQL Migration, leveraging Azure Data Migration Service (Azure DMS).

In Azure Data Studio, you will be able to perform a modernization of the SQL server infrastructure by using the Native SQL backups (the latest full back up as well as the transactional log backups from the previous backup). In this method, backup files of SQL server databases are copied and restored on the target. Using Azure Data Studio, we can automate the backup and restore process. All we must do is manually place the backup files into a shared network folder or Azure storage container so that the tool recognizes the backups and restores them automatically.

Post Migration:

Upon completion of modernization, all objects impacted by the modernization should be resolved for optimal performance. DMA provides information regarding all impacted objects and offers recommendations on how to address them.

Your Next Steps:

If you like what you’ve read so far, let’s move forward together with confidence. We’re here to help at every step. Contact SNP’s migration experts here

 

 

Azure Arc-enabled Kubernetes extends Azure’s management capabilities to Kubernetes clusters running anywhere, whether in public clouds or on-premises data centers. This integration allows customers to leverage Azure features such as Azure Policy, GitOps, Azure Monitor, Microsoft Defender, Azure RBAC, and Azure Machine Learning.

Key features of Azure Arc-enabled Kubernetes include:

1. Centralized Management: Attach and configure Kubernetes clusters from diverse environments in Azure, facilitating a unified management experience.
2. Governance and Configuration: Apply governance policies and configurations across all clusters to ensure compliance and consistency.
3. Integrated DevOps: Streamline DevOps practices with integrated tools that enhance collaboration and deployment efficiency.
4. Inventory and Organization: Organize clusters through inventory, grouping, and tagging for better visibility and management.
5. Modern Application Deployment: Enable the deployment of modern applications at scale across any environment.

In this blog, we will follow a step by step approach and learn how to:

1. Connect Kubernetes clusters running outside of Azure

2. GitOps – to define applications and cluster configuration in source control

3. Azure Policy for Kubernetes

4. Azure Monitor for containers

 

1. Connect Kubernetes clusters

Prerequisites

• Azure account with an active subscription.
• Identity – User or service principal
• Latest Azure CLI
• Extensions – connectedk8s and k8sconfiguration
• An up-and-running Kubernetes cluster
• Resource providers – Microsoft.Kubernetes, Microsoft.KubernetesConfiguration, Microsoft.ExtendedLocation

Create a Resource Group

Create a Resource Group using below command in Azure portal choose your desired location. Azure Arc for Kubernetes supports most of the azure regions. Use this page Azure products by region to know the supported regions.

* az group create –name AzureArcRes -l EastUS -o table

For example: az group create –name AzureArcK8sTest –location EastUS –output table

Connect to the cluster with admin access and attach it with Azure Arc

We use az connectedk8s connect cli extension to attach our Kubernetes clusters to Azure Arc.

This command verify the connectivity to our Kubernetes clusters via kube-config (“~/.kube/config”) file and deploy Azure Arc agents to the cluster into the “azure-arc” namespace and installs Helm v3 to the .azure folder.

For this demonstration we connect and attach AWS – Elastic Kubernetes service and Google cloud – Kubernetes engine. Below, we step through the commands used to connect and attach to each cluster.

 

AWS – EKS

* aws eks –region <Region> update-kubeconfig –name <ClusterName>

* kubectl get nodes

* az connectedk8s connect –name <ClusterName> –resource-group AzureArcRes

GCLOUD- GKE

* gcloud container clusters get-credentials <ClusterName> –zone <ZONE> –project <ProjectID>

* kubectl get no

* az connectedk8s connect –name <ClusterName> –resource-group AzureArcRes

Verify Connected Clusters

* az connectedk8s list -g AzureArcRes -o table

 

2. Using GitOps to define applications & clusters

We use the connected GKE cluster for our example to deploy a simple application.

Create a configuration to deploy an application to kubernetes cluster.
We use “k8sconfiguration” extension to link our connected cluster to an example git repository provided by SNP.

* export KUBECONFIG=~/.kube/gke-config

* az k8sconfiguration create \

–name app-config \

–cluster-name <ClusterName> –resource-group <YOUR_RG_NAME>\

–operator-instance-name app-config –operator-namespace cluster-config \

–repository-url https://github.com/gousiya573-snp/SourceCode/tree/master/Application \

–scope cluster –cluster-type connectedClusters

Check to see that the namespaces, deployments, and resources have been created:

* kubectl get ns –show-labels

We can see that cluster-config namespace have been created.

* kubectl get po,svc

The flux operator has been deployed to cluster-config namespace, as directed by our sourceControlConfig and application deployed successfully, we can see the pods are Running and Service LoadBalancer IP also created.

Access the EXTERNAL-IP to see the output page:

Please Note:

Supported repository-url Parameters for Public & Private repos:

* Public GitHub Repo   –  http://github.com/username/repo  (or) git://github.com/username/repo

* Private GitHub Repo –  https://github.com/username/repo (or) git@github.com:username/repo

* For the Private Repos – flux generates a SSH key and logs the public key as shown below:

For this demonstration we connect and attach AWS – Elastic Kubernetes service and Google cloud – Kubernetes engine. Below, we step through the commands used to connect and attach to each cluster.

3. Azure Policy for Kubernetes

Use Azure Policy to enforce that each Microsoft.Kubernetes/connectedclusters resource or Git-Ops enabled Microsoft.ContainerService/managedClusters resource has specific Microsoft.KubernetesConfiguration/sourceControlConfigurations applied on it.

Assign Policy:

To create the policy navigate to Azure portal and Policy, in the Authoring section select the Definitions.
Click on Initiative definition to create the policy and search for gitops in the Available Definitions, click on Deploy GitOps to Kubernetes clusters policy to add.
Select the subscription in the Definition locations, Give the Policy assignment Name and Description.

Choose the Kubernetes in the existing Category list and scroll-down to fill the Configuration related details of an application.

Select the policy definition and click on Assign option above and set the scope for the assignment. Scope can be Azure resource group level or subscription and complete the other basics steps – Assignment name, Exclusions, remediation etc.

Click on parameters and provide name for the Configuration resource, Operator instance, Operator namespace and set the Operator scope to cluster level or namespace, Operator type is Flux and provide your application github repo url (public or private) in the Repository Url field. Now, additionally pass the Operator parameters such as “–git-branch=master –git-path=manifests –git-user=your-username –git-readonly=false” finally click on Save option and see the policy with the given name is created in the Assignments.

Once the assignment is created the Policy engine will identify all connectedCluster or managedCluster resources that are located within the scope and will apply the sourceControlConfiguration on them.

–git-readonly=false enables the CI/CD for the repo and creates the Auto releases for the commits.

 

 

Verify a Policy Assignment

Go to Azure portal and click on connected Cluster resources to check the Compliant Status, Compliant: config-agent was able to successfully configure the cluster and deploy flux without error.

We can see the policy assignment that we created above, and the Compliance state should be Compliant.

4. Azure Monitor for Containers

It provides rich monitoring experience for the Azure Kubernetes Service (AKS) and AKS Engine clusters. This can be enabled for one or more existing deployments of Arc enabled Kubernetes clusters using az cli, azure portal and resource manager.

Create Azure Log Analytics workspace or use an existing one to configure the insights and logs. Use below command to install the extension and configure it to report to the log analytics workspace.

*az k8s-extension create –name azuremonitor-containers –cluster-name <cluster-name> –resource-group <resource-group> –cluster-type connectedClusters –extension-type Microsoft.AzureMonitor.Containers –configuration-settings logAnalyticsWorkspaceResourceID=<armResourceIdOfExistingWorkspace

It takes about 10 to 15 minutes to get the health metrics, logs, and insights for the cluster. You can check the status of extension in the Azure portal or through CLI. Extension status should show as “Installed”.

We can also scrape and analyze Prometheus metrics from our cluster.

Clean Up Resources

To delete an extension:

* az k8s-extension delete –name azuremonitor-containers –cluster-type connectedClusters –cluster-name <cluster-name> –resource-group <resource-group-name>

To delete a configuration:

*az k8sconfiguration delete –name ‘<config name>‘ -g ‘<resource group name>‘ –cluster-name ‘<cluster name>‘ –cluster-type connectedClusters

To disconnect a connected cluster:

* az connectedk8s delete –name <cluster-name> –resource-group <resource-group-name>

 

Conclusion:

This blog provides an overview of Azure Arc-enabled Kubernetes, highlighting how SNP assists its customers in setting up Kubernetes clusters with Azure Arc for scalable deployment. It emphasizes the benefits of Azure Arc in managing Kubernetes environments effectively.

SNP offers subscription services to accelerate your Kubernetes journey, enabling the installation of production-grade Kubernetes both on-premises and in Microsoft Azure. For more information or to get assistance from SNP specialists, you can reach out through the provided contact options. Contact SNP specialists here.