Greetings! My name is Ksenia, and I am a DevOps developer at Loovatech.

I love creating infrastructure and processes that developers and operations teams can use to be more productive and deliver better results.

So if your day-to-day working routine is closely tied to numerous self-same microservices and diverse environments, I hope this article could act as a useful tool for you. The material will mostly suit Junior+ and Middle-grade DevOps specialists.

Case:

We were contacted by a client about a project related to the operation of cash register systems of a grocery store chain, including goods list management, prices, and other data. The client asked our team to help migrate a monolithic app from its in-house infrastructure running without containerization to Kubernetes in the public cloud and to spin up three environments for development, testing, and production. 

The first step was to rewrite the app to a microservice architecture. Aiming to offer a cost-effective solution and not put too much unnecessary work on the development team, we decided to create an infrastructure that will allow developers to deploy any number of microservices and development environments by themselves. The task implied that the development of any new microservice should happen with either none or some very limited DevOps involvement.

We already had several microservices prepared by the time of the project kick-off. With the help of Terraform we were able to successfully configure all the necessary infrastructure. It included a virtual network, a subnet, a managed Kubernetes cluster, a managed PostgreSQL database, and a managed secrets storage tool. 

We also deployed the Nginx ingress web server and ArgoCD using Terraform. All the services and microservices of the app, including utility Helm charts, were deployed through ArgoCD. Though we also faced several shortcomings that we wanted to eliminate before launching the production environment.

Here is the text stack we decided to go with:

The microservices setup was implemented through ArgoCD using the "app-of-apps" approach. The "App of Apps" concept in Argo CD refers to an approach allowing management of  multiple Kubernetes applications through a single application called "App of Apps."

This approach enables a unified way to manage and deploy multiple apps, greatly simplifying automation and management of the entire infrastructure while also ensuring all changes happen in their related environments. Without an “app of apps,” launching each service requires DevOps input and conflicts with business requirements.

The key benefits of using "App of Apps" concept in ArgoCD include:

• Ability to effortlessly manage multiple applications at once. One application describes other applications and manages them as a group. This makes it easier to deploy and manage multiple microservices.

• Improved organization through clearer hierarchy, making application management process much easier.

• Convenient automation of deployment and updating of all applications included in the "App of Apps" via ArgoCD.

• Reduced errors through centralizing configuration and higher control levels.

We have created 3 Apps of Apps, one for each environment. An Automated Sync Policy was configured for each Application in Argo CD.

This approach enables automatic synchronization of the application's actual state with its desired state in the Git repository. Therefore, a commit to the helm charts git repository, specified as the source for the application, acts as a trigger to start synchronization.

We also created a separate Helm chart with the same templates for each microservice. Weadded values ​​for 3 environments - development, pre-production (test) and production, alongside one branch for all 3 environments. This is how the git repository structure packed with helm chart templates looked like at this step:

Right after the development environment launch, we faced the following issues:

1. Awkward management The risk of errors increased without centralized management of identical templates for helm charts. Since we copied many templates with different edits, we quickly realized that this practice is error-prone.

2. We had to create a branch for each environment since the Automated Sync Policy in ArgoCD for production can lead to the rollout of an incorrect template. Another way was to create separate directories for each environment. The amount of manual labor did not correspond to the initially chosen concept.

3. Chart versioning is not followed. To be more specific – the version was specified manually, meaning no one really ever paid any attention to it. Properly working automated chart versioning simplifies lifecycle management.

Although the infrastructure met the developer’s requirements, administering helm charts was a complex time-consuming manual task. We decided the project required a chart repository. Specialized chart repositories are designed to solve versioning problems. Our choice fell on ChartMuseum.

ChartMuseum is one of the leading repositories for Helm charts, offering a range of benefits for managing and deploying applications using Helm charts.

The key reasons for choosing CharMuseum for us were:

1. This repository provides local storage of Helm charts, granting full control and easy management of chart repositories within their own infrastructure. The repository is stored directly inside the cluster, administered together with its infrastructure and is always at hand.

2. ChartMuseum allows one to flexibly manage versions, updates, and releases of Helm charts.

3. Flexible authentication and authorization methods allow teams to control access to repositories and charts, maintaining advanced security.

The following steps were taken to implement the solution:

• Deployed chartmuseum from https://github.com/helm/chartmuseum

• Created a directory with a new helm chart, which should have served as a source for all microservices, and called it library-chart.

• Pushed library-chart into chartmuseum

  
  

helm package library-chart -d library-chart/
curl --data-binary "@library-1.0.11.tgz" https://my-chart-museum/api/charts

The next step taken was adjusting the Chart.yaml version for microservices, which was then transferred to a new version of the chart. We also updated the dependency for them after each change:

apiVersion: v2
appVersion: 0.0.3
description: Chart for publishing  microservice1 to Kubernetes
name: microservice1
type: application
version: 1.0.9
dependencies:
- name: library
  version: "1.0.11"
  repository: https://my-chart-museum

             helm dependency update

Although this is a pretty common solution, for our project it solved the issue of the lack of versioning.Templates now have a centralized source, though there are additional administration steps to be taken.

That's when we came up with a decision to use Multiple Sources for an Application + ChartMuseum, where we obtain the data for setting up the ArgoCD application from several sources. ArgoCD accesses Chartmuseum directly, the charts directory disappears from the git repository.

This feature appeared in ArgoCD 2.6. At the time of writing of the article, the functionality is still marked as a Beta Feature: https://argo-cd.readthedocs.io/en/stable/user-guide/multiple_sources/

Here are some examples of using the multiple sources feature:

• pooling resources from different repositories for microservices-based apps;

• managing shared libraries or shared resources between multiple teams or repositories;

• merging resources from different branches of a Git repository for testing or experimentation purposes;

• managing resources for different environments (e.g., development, pre-production, production) in separate Git repositories;

• specifying Helm value files from an external Git repository — ChartMuseum contains the Helm, and the Git repository contains values*.yaml files.

The app’s manifest in ArgoCD looks like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: microcervice1
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: 'https://my-chart-museum/'
      chart: library
      targetRevision: 1.0.11
      helm:
        valueFiles:
          - $values/myproject/helm/microservice1/values.yaml
          - $values/myproject/helm/microservice2/values-production.yaml
    - repoURL: 'https://my-gitlab.com/active/my_repo.git'
      targetRevision: master
      ref: values
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: mynamespace
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
    automated:
      selfHeal: true

The $values ​​tag refers to the root directory of the my_repo repository, which is pointed to by the ref key in the sources list. The $values ​​variable can only be specified at the start of the values ​​file path. Accordingly, my_repo acts as a source of values ​​files for microservices.

The structure becomes clearer and more understandable; only the library chart needs to be edited, and a single entry point for configurations appears:

Here's how changing the helm chart version works in our current implementation:

1. Fixing the the Chart.yaml version and pushing it to chartmuseum after changing the templates

helm package library-chart -d library-chart/
cd library-chart && curl --data-binary "@library-1.0.12.tgz"  https:/my-chart-museum/api/charts

This step was performed manually since after the launch these changes immediately became extremely rare. However, this step can still be easily automated via CI/CD per commit.

2. Correct the version in the application manifest.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: microcervice1
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: 'https://my-chart-museum/'
      chart: library
      targetRevision: 1.0.12

3. The new chart version is deployed after completing the auto sync.

Here are some notable features:

1. Application parameters cannot be checked and corrected via ArgoCD. For some, the lack of such an opportunity is a deal breaker. For me, this is a plus, as even if my hands would itch to adjust the parameters via ArgoCD, I won’t be able to do it. For the GitOps concept, this feature is a benefit as the only source of parameters will be the Git repository.

2. It's impossible to make Rollback through the Argo CD UI - it just states that “Rollback is not supported for apps with multiple sources”. Luckily, it wasn't a big deal in our case since we were using Argo Rollout with blue-green deployment. Traffic switches to pod only after passing the analysis. 

Though this still could be a no-no for many. A pull request has been opened to correct this limitation: https://github.com/argoproj/argo-cd/pull/14124

So, here are the benefits we have gained:

1. We accelerated the process of adding a new microservice. We added a microservice directory and values ​​files for 3 environments on the Gitlab side of the repository, alongside implementation of the ArgoCD application manifest for each environment. For all environments, the master branch acts as the source.

2. We simplified the process of administering helm charts and eliminated the need for a  separate helm for different environments into different branches or directories. 

3. We reduced the likelihood of an error occurring when making changes to the helm charts.

As a result, the multiple-source Argo CD + Chartmuseum solution used to manage microservices on our project helped optimize development processes, reduce required investments and resources, and reduce the risk of configuration errors. Developers were able to add new microservices without involving DevOps and with low chances of error, meeting the key project requirements and client expectations.