StreamNative Pulsar Operator Tutorial Part 3

Photo by Caleb Angel on Unsplash

In this part, I demo how to use ArgoCD to control Pulsar Custom Resource (CR) by monitoring the GitHub branch/tag.

In part 1, I created 4 pulsar CRs with pulsar version 2.9. I create a GitHub repository, pulsar-ops, to keep track of the CRs. ArgoCD will trace the changes in a folder of a branch, so I put all the CRs under the default folder.

➜  deployment git:(main) tree
.
├── README.md
└── default
    ├── bk-cluster.yaml
    ├── br-cluster.yaml
    ├── px-cluster.yaml
    └── zk-cluster.yaml
1 directory, 5 files 

Once all the changes are committed, I add the remote to my GitHub target repo, tag the pulsar version, then push the CRs upstream.

➜  deployment git:(main) git remote add origin git@github.com:yuweisung/pulsar-ops.git
➜  deployment git:(main) git tag -a v2.9 -m "pulsar-2.9"
➜  deployment git:(main) git push --set-upstream origin v2.9

Now I can go to ArgoCD WebUI to create a pulsar cluster app. Login to ArgoCD, and create a new app with the following info, then click CREATE.

• Application Name: sn-platform
• Project: default
• SYNC POLICY: Automatic
• Repository URL: https://github.com/yuweisung/pulsar-ops.git
• Revision: v2.9 Tags
• Path: default
• Destination: https://kubernetes.default.svc
• Namespace: sn-platform

After the app is created, ArgoCD will scan the specific tag and syn the CRs to deploy the pulsar cluster as we did in part 1 using “kubectl apply -f”. The app detail shows the progress of the deployment.

A few minutes later, ArgoCD shows status synced.

Next, I use git push and pull requests to control the CR details, e.g., adding JVM options, changing replicas of CR, and letting ArgoCD detect the changes and auto-apply the operation changes for me.

Open vs code and change the image version from 2.9.2.15 to 2.9.2.17 and commit the changes.

After committing the changes, I retag the commit to v2.9 and push with the force option.

➜  deployment git:(main) git tag -a -f v2.9 1ee752a
Updated tag 'v2.9' (was e40f3b6)
➜  deployment git:(main) git push origin v2.9 --force
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 165 bytes | 165.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:yuweisung/pulsar-ops.git
 + e40f3b6...fc5fb8d v2.9 -> v2.9 (forced update)

ArgoCD detects the commit changes and syncs the status automatically. From the WebUI, you can see that some components are in “spinning” status as it follows the statefulset upgrade strategy.

Click on the zookeeper pod and you can see the image version updated.

Next, I want to use the same GitOps process to scale up/down brokers. Go back to the vs-code and modify the “br-cluster.yaml” file.

apiVersion: pulsar.streamnative.io/v1alpha1
kind: PulsarBroker
metadata:
  name: my
  namespace: sn-platform
spec:
  image: streamnative/pulsar:2.9.2.17
  pod:
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
    terminationGracePeriodSeconds: 30
  config:
    custom:
      webSocketServiceEnabled: "true"
  replicas: 3
  zkServers: my-zk-headless:2181

Then run the git commands to force retagging the new commit to v2.9 and to force pushing to the remote origin branch.

➜  deployment git:(main) git add default/br-cluster.yaml
➜  deployment git:(main) git commit -m 'scale up broker to 3'
[main 32e0468] scale up broker to 3
 1 file changed, 1 insertion(+), 1 deletion(-)
➜  deployment git:(main) git tag -a -f v2.9 32e0468
➜  deployment git:(main) git push origin v2.9 --force
Counting objects: 100% (8/8), done.
Delta compression using up to 10 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 516 bytes | 516.00 KiB/s, done.
Total 5 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:yuweisung/pulsar-ops.git
 + fc5fb8d...13bf247 v2.9 -> v2.9 (forced update)

Back to the ArgoCD WebUI and the new broker (my-broker-2) is spinning up.

With ArgoCD and GitHub, you can control pulsar CR changes. Next part, I will discuss how to add client-side (pulsar consume/producer) to the picture.
Stay tuned.