Tutorials

• 1: Kiali and Grafana Tempo Query integration
• 1.1: Introduction
• 1.2: Kiali and Tempo setup
• 2: Travels Demo - Multicluster
• 2.1: Introduction
• 2.2: Prerequisites
• 2.3: Deploy East cluster
• 2.4: Install Istio on East cluster
• 2.5: Install Kiali
• 2.6: Install Travels on East cluster
• 2.7: Deploy West cluster
• 2.8: Install Istio on West cluster
• 2.9: Configure Kiali for multicluster
• 2.10: Install Travels on West cluster
• 3: Travels Demo Tutorial
• 3.1: Prerequisites
• 3.2: Install Travel Demo
• 3.3: First Steps
• 3.4: Observe
• 3.5: Connect
• 3.6: Secure
• 3.7: Uninstall Travel Demo

1 - Kiali and Grafana Tempo Query integration

This tutorial goes throw the process to setup up Grafana Tempo Query as the Kiali default distribution tracing system.

1.1 - Introduction

Introduction

Kiali uses Jaeger as a default distributed tracing backend. In this tutorial, we will replace it for Grafana Tempo.

We will setup a local environment in minikube, and install Kiali with Tempo as a distributed backend. This is a simplified architecture diagram:

• We will install Tempo with the Tempo Operator and enable Jaeger query frontend to be compatible with Kiali in order to query traces.
• We will setup Istio to send traces to the Tempo collector using the zipkin protocol. It is enabled by default from version 3.0 or higher of the Tempo Operator.
• We will install MinIO and setup it up as object store, S3 compatible.

Environment

We use the following environment:

• Istio 1.18.1
• Kiali 1.72
• Minikube 1.30
• Tempo operator TempoStack v3.0

There are different installation methods for Grafana Tempo, but in this tutorial we will use the Tempo operator.

1.2 - Kiali and Tempo setup

Steps to install Kiali and Grafana Tempo

We will start minikube:

minikube start

It is a requirement to have cert-manager installed:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml

Install the operator. It is important to download a version 3.0 or higher. In previous versions, the zipkin collector was not exposed, there was no way to change it as it was not defined in the CRD.

kubectl apply -f https://github.com/grafana/tempo-operator/releases/download/v0.3.0/tempo-operator.yaml

We will create the tempo namespace:

kubectl create namespace tempo

We will deploy minio, this is a sample minio.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # This name uniquely identifies the PVC. Will be used in deployment below.
  name: minio-pv-claim
  labels:
    app: minio-storage-claim
spec:
  # Read more about access modes here: http://kubernetes.io/docs/user-guide/persistent-volumes/#access-modes
  accessModes:
    - ReadWriteOnce
  resources:
    # This is the request for storage. Should be available in the cluster.
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  selector:
    matchLabels:
      app: minio
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        # Label is used as selector in the service.
        app: minio
    spec:
      # Refer to the PVC created earlier
      volumes:
        - name: storage
          persistentVolumeClaim:
            # Name of the PVC created earlier
            claimName: minio-pv-claim
      initContainers:
        - name: create-buckets
          image: busybox:1.28
          command:
            - "sh"
            - "-c"
            - "mkdir -p /storage/tempo-data"
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
      containers:
        - name: minio
          # Pulls the default Minio image from Docker Hub
          image: minio/minio:latest
          args:
            - server
            - /storage
            - --console-address
            - ":9001"
          env:
            # Minio access key and secret key
            - name: MINIO_ACCESS_KEY
              value: "minio"
            - name: MINIO_SECRET_KEY
              value: "minio123"
          ports:
            - containerPort: 9000
            - containerPort: 9001
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
---
apiVersion: v1
kind: Service
metadata:
  name: minio
spec:
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
      name: api
    - port: 9001
      targetPort: 9001
      protocol: TCP
      name: console
  selector:
    app: minio

And apply the yaml:

kubectl apply -n tempo -f minio.yaml

We will create a secret to access minio:

kubectl create secret generic -n tempo tempostack-dev-minio \
--from-literal=bucket="tempo-data" \
--from-literal=endpoint="http://minio:9000" \
--from-literal=access_key_id="minio" \
--from-literal=access_key_secret="minio123"

Install Grafana tempo with the operator. We will use the secret created in the previous step:

kubectl apply -n tempo -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: smm
spec:
  storageSize: 1Gi
  storage:
    secret:
      type: s3
      name: tempostack-dev-minio
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
        ingress:
          type: ingress
EOF

As an optional step, we can check if all the deployments have started correctly, and the services distributor has the port 9411 and the query frontend 16686:

kubectl get all -n tempo

(Optional) We can test if minio is working with a batch job to send some traces, in this case, to the open telemetry collector:

kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: tracegen
spec:
  template:
    spec:
      containers:
        - name: tracegen
          image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/tracegen:latest
          command:
            - "./tracegen"
          args:
            - -otlp-endpoint=tempo-smm-distributor.tempo.svc.cluster.local:4317
            - -otlp-insecure
            - -duration=30s
            - -workers=1
      restartPolicy: Never
  backoffLimit: 4
EOF

And access the minio console:

kubectl port-forward --namespace istio-system service/minio 9001:9001

Install Istio with helm (Option I)

Istio can be installed with Helm following the instructions. The zipkin address needs to be set:

--set values.meshConfig.defaultConfig.tracing.zipkin.address="tempo-smm-distributor.tempo:9411"

And then, install Jaeger as Istio addon.

Install Istio using Kiali source code (Option II)

For development purposes, if we have Kiali source code, we can use the kiali hack scripts:

hack/istio/install-istio-via-istioctl.sh -c kubectl -a "prometheus grafana" -s values.meshConfig.defaultConfig.tracing.zipkin.address="tempo-smm-distributor.tempo:9411"

Install Kiali and bookinfo demo with some traffic generation

Install kiali:

helm install \
    --namespace istio-system \
    --set external_services.tracing.in_cluster_url=http://tempo-smm-query-frontend.tempo:16685 \
    --set external_services.tracing.url=http://localhost:16686 \
    --set auth.strategy=anonymous \
    kiali-server \
    kiali/kiali-server

Install bookinfo with traffic generator

curl -L -o install-bookinfo.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/install-bookinfo-demo.sh
chmod +x install-bookinfo.sh
./install-bookinfo.sh -c kubectl -tg -id ${ISTIO_DIR}

And access Kiali:

kubectl port-forward svc/kiali 20001:20001 -n istio-system

2 - Travels Demo - Multicluster

This tutorial will demonstrate Kiali capabilities for Istio multicluster, particulary for the primary-remote cluster model.

For more information, check our documentation for multicluster.

2.1 - Introduction

So far, we know how good Kiali can be to understand applications, their relationships with each other and with external applications.

In the previous tutorial, Kiali was setup to observe just a single cluster. Now, we will expand its capabilities to observe more than one cluster. The extra clusters are remotes, meaning that there is not a control plane on them, they only have user applications.

This topology is called primary-remote and it is very useful to spread applications into different clusters having just one primary cluster, which is where Istio and Kiali are installed.

This scenario is a good choice when as an application administrator or architect, you want to give a different set of clusters to different sets of developers and you also want all these applications to belong to the same mesh. This scenario is also very helpful to give applications high availability capabilities while keeping the observability together (we are referring to just applications in terms of high availability, for Istio, we might want to install a multi-primary deployment model, which is on the roadmap for the multicluster journey for Kiali).

In this tutorial we will be deploying Istio in a primary-remote deployment. At first, we will install the “east” cluster with Istio, then we will add the “west” remote cluster and join it to the mesh. Then we will see how Kiali allows us to observe and manage both clusters and their applications. Metrics will be aggregated into the “east” cluster using Prometheus federation and a single Kiali will be deployed on the “east” cluster.

If you already have a primary-remote deployment, you can skip to instaliing Kiali.

2.2 - Prerequisites

This tutorial is a walkthrough guide to install everything. For this reason, we will need:

• minikube
• istioctl
• helm

This tutorial was tested on:

• Minikube v1.30.1
• Istio v1.18.1
• Kiali v1.70

Clusters are provided by minikube instances, but this tutorial should work on on any Kubernetes environment.

We will set up some environment variables for the following commands:

CLUSTER_EAST="east"
CLUSTER_WEST="west"
ISTIO_DIR="absolute-path-to-istio-folder"

As Istio will be installed on more than one cluster and needs to communicate between clusters, we need to create certificates for the Istio installation. We will follow the Istio documentation related to certificates to achieve this:

mkdir -p certs
pushd certs

make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk root-ca

make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk $CLUSTER_EAST-cacerts
make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk $CLUSTER_WEST-cacerts

popd

The result is two certificates for then use when installing Istio in the future.

2.3 - Deploy East cluster

Run the following commands to deploy the first cluster:

minikube start -p $CLUSTER_EAST --network istio --memory 8g --cpus 4

For both clusters, we need to configure MetalLB, which is a load balancer. This is because we need to assign an external IP to the required ingress gateways to enable cross cluster communication between Istio and the applications installed.

minikube addons enable metallb -p $CLUSTER_EAST

We set up some environment variables with IP ranges that MetalLB will then assign to the services:

MINIKUBE_IP=$(minikube ip -p $CLUSTER_EAST)
MINIKUBE_IP_NETWORK=$(echo $MINIKUBE_IP | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1/')
MINIKUBE_LB_RANGE="${MINIKUBE_IP_NETWORK}.20-${MINIKUBE_IP_NETWORK}.29"

cat <<EOF | kubectl --context $CLUSTER_EAST apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses: [${MINIKUBE_LB_RANGE}]
EOF

We should have the first cluster deployed and ready to use.

2.4 - Install Istio on East cluster

The east cluster is the primary one, consequently is where the istiod process will be installed alongside other applications like Kiali.

Run the following commands to install Istio:

kubectl create namespace istio-system --context $CLUSTER_EAST

kubectl create secret generic cacerts -n istio-system --context $CLUSTER_EAST \
      --from-file=certs/$CLUSTER_EAST/ca-cert.pem \
      --from-file=certs/$CLUSTER_EAST/ca-key.pem \
      --from-file=certs/$CLUSTER_EAST/root-cert.pem \
      --from-file=certs/$CLUSTER_EAST/cert-chain.pem

kubectl --context=$CLUSTER_EAST label namespace istio-system topology.istio.io/network=network1

cat <<EOF > $CLUSTER_EAST.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: $CLUSTER_EAST
      network: network1
EOF

istioctl install -y --set values.pilot.env.EXTERNAL_ISTIOD=true --context=$CLUSTER_EAST -f $CLUSTER_EAST.yaml

After the installation, we need to create what we called an “east-west” gateway. It’s an ingress gateway just for the cross cluster configuration as we are opting to use the installation for different networks (this will be the case in the majority of the production scenarios).

$ISTIO_DIR/samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster $CLUSTER_EAST --network network1 | \
    istioctl --context=$CLUSTER_EAST install -y -f -

Then, we need to expose the istiod service as well as the applications for the cross cluster communication:

kubectl apply --context=$CLUSTER_EAST -n istio-system -f \
    $ISTIO_DIR/samples/multicluster/expose-istiod.yaml

kubectl --context=$CLUSTER_EAST apply -n istio-system -f \
    $ISTIO_DIR/samples/multicluster/expose-services.yaml

export DISCOVERY_ADDRESS=$(kubectl \
    --context=$CLUSTER_EAST \
    -n istio-system get svc istio-eastwestgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Finally, we need to install Prometheus, which is important and required for Kiali to operate:

kubectl --context $CLUSTER_EAST -n istio-system apply -f $ISTIO_DIR/samples/addons/prometheus.yaml

2.5 - Install Kiali

Run the following command to install Kiali using the Kiali operator:

kubectl config use-context $CLUSTER_EAST

helm upgrade --install --namespace istio-system --set auth.strategy=anonymous --set deployment.logger.log_level=debug --set deployment.ingress.enabled=true --repo https://kiali.org/helm-charts kiali-server kiali-server 

Verify that Kiali is running with the following command:

istioctl dashboard kiali

There are other alternatives to expose Kiali or other Addons in Istio. Check Remotely Accessing Telemetry Addons for more information.

2.6 - Install Travels on East cluster

Run the following commands to install Travels application on east cluster:

kubectl create namespace travel-agency --context $CLUSTER_EAST
kubectl create namespace travel-portal --context $CLUSTER_EAST
kubectl create namespace travel-control --context $CLUSTER_EAST

kubectl label namespace travel-agency istio-injection=enabled --context $CLUSTER_EAST
kubectl label namespace travel-portal istio-injection=enabled --context $CLUSTER_EAST
kubectl label namespace travel-control istio-injection=enabled --context $CLUSTER_EAST

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_agency.yaml) -n travel-agency --context $CLUSTER_EAST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_portal.yaml) -n travel-portal --context $CLUSTER_EAST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_control.yaml) -n travel-control --context $CLUSTER_EAST

After the installation, we can see that the Travels application is running on the east cluster:

It is important to note that Kiali only observes one istio-system namespace as we did not configure it for multicluster yet.

Go to the Graph page and select the three namespaces related to the Travels demo in the namespace dropdown menu. This shows you the in-cluster traffic:

So far, we installed everything on one cluster, similarly to the Travels tutorial for a single cluster.

Now we will expand this topology to include a remote cluster. As we commented this situation can be very common in a production scenario, either because we might want to split some applications into different clusters, generally because they are maintained by different developers or for high availability or just making applications available in other zones to reduce latencies.

2.7 - Deploy West cluster

Run the following commands to deploy the second cluster:

minikube start -p $CLUSTER_WEST --network istio --memory 8g --cpus 4

Similar to the east cluster, we configure MetalLB:

minikube addons enable metallb -p $CLUSTER_WEST

MINIKUBE_IP=$(minikube ip -p $CLUSTER_WEST)
MINIKUBE_IP_NETWORK=$(echo $MINIKUBE_IP | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1/')
MINIKUBE_LB_RANGE="${MINIKUBE_IP_NETWORK}.30-${MINIKUBE_IP_NETWORK}.39"

cat <<EOF | kubectl --context $CLUSTER_WEST apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses: [${MINIKUBE_LB_RANGE}]
EOF

2.8 - Install Istio on West cluster

This installation will be different as this cluster will be a remote. In a remote cluster, it won’t be an Istio control plane. Istio will install some resources that allows the primary control plane to configure the workloads in the remote cluster like injecting the sidecars and configuring the low level routing.

kubectl create namespace istio-system --context $CLUSTER_WEST

kubectl create secret generic cacerts -n istio-system --context $CLUSTER_WEST \
      --from-file=certs/$CLUSTER_WEST/ca-cert.pem \
      --from-file=certs/$CLUSTER_WEST/ca-key.pem \
      --from-file=certs/$CLUSTER_WEST/root-cert.pem \
      --from-file=certs/$CLUSTER_WEST/cert-chain.pem

kubectl --context=$CLUSTER_WEST annotate namespace istio-system topology.istio.io/controlPlaneClusters=$CLUSTER_EAST
kubectl --context=$CLUSTER_WEST label namespace istio-system topology.istio.io/network=network2

cat <<EOF > $CLUSTER_WEST.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: remote
  values:
    istiodRemote:
      injectionPath: /inject/cluster/$CLUSTER_WEST/net/network2
    global:
      remotePilotAddress: ${DISCOVERY_ADDRESS}
EOF

istioctl install -y --context=$CLUSTER_WEST -f $CLUSTER_WEST.yaml

We will also install a Prometheus instance on the remote. We will federate both Prometheus, with the east’s one being the place where all metrics will be gathered together:

kubectl apply -f $ISTIO_DIR/samples/addons/prometheus.yaml --context $CLUSTER_WEST

An important step is to create a secret on east cluster allowing it to fetch information of the remote cluster:

istioctl x create-remote-secret \
    --context=$CLUSTER_WEST \
    --name=$CLUSTER_WEST | \
    kubectl apply -f - --context=$CLUSTER_EAST

Finally, we create the east-west gateway

$ISTIO_DIR/samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster $CLUSTER_WEST --network network2 | \
    istioctl --context=$CLUSTER_WEST install -y -f -

Prometheus federation

Kiali requires unified metrics from a single Prometheus endpoint for all clusters, even in a multi-cluster environment. In this tutorial, we will federate the two Prometheus instances, meaning that all the remote’s metrics should be fetched by the main Prometheus.

We will configure east’s Prometheus to fetch west’s metrics:

kubectl patch svc prometheus -n istio-system --context $CLUSTER_WEST -p "{\"spec\": {\"type\": \"LoadBalancer\"}}"

WEST_PROMETHEUS_ADDRESS=$(kubectl --context=$CLUSTER_WEST -n istio-system get svc prometheus -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

curl -L -o prometheus.yaml https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/prometheus.yaml

sed -i "s/WEST_PROMETHEUS_ADDRESS/$WEST_PROMETHEUS_ADDRESS/g" prometheus.yaml

kubectl --context=$CLUSTER_EAST apply -f prometheus.yaml -n istio-system

2.9 - Configure Kiali for multicluster

We will configure Kiali to access the remote cluster. This will require a secret (similar to the Istio secret) containing the credentials for Kiali to fetch information from the remote cluster:

curl -L -o kiali-prepare-remote-cluster.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/kiali-prepare-remote-cluster.sh

chmod +x kiali-prepare-remote-cluster.sh

./kiali-prepare-remote-cluster.sh --kiali-cluster-context $CLUSTER_EAST --remote-cluster-context $CLUSTER_WEST

Finally, upgrade the installation for Kiali to pick up the secret:

kubectl config use-context $CLUSTER_EAST

helm upgrade --install --namespace istio-system --set auth.strategy=anonymous --set deployment.logger.log_level=debug --set deployment.ingress.enabled=true --repo https://kiali.org/helm-charts kiali-server kiali-server

As result, we can quickly see that a new namespace appear in the Overview, the istio-system namespace from west cluster:

2.10 - Install Travels on West cluster

We are going to deploy two new services just to distribute traffic on the new cluster. These services are travels v2 and v3:

kubectl create ns travel-agency --context $CLUSTER_WEST

kubectl label namespace travel-agency istio-injection=enabled --context $CLUSTER_WEST

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v2.yaml) -n travel-agency --context $CLUSTER_WEST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v3.yaml) -n travel-agency --context $CLUSTER_WEST

cat <<EOF | kubectl -n travel-agency --context $CLUSTER_WEST apply -f -
apiVersion: v1
kind: Service
metadata:
  name: travels
  labels:
    app: travels
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: travels
---
apiVersion: v1
kind: Service
metadata:
  name: insurances
  labels:
    app: insurances
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: insurances
---
apiVersion: v1
kind: Service
metadata:
  name: hotels
  labels:
    app: hotels
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: hotels
---
apiVersion: v1
kind: Service
metadata:
  name: flights
  labels:
    app: flights
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: flights
---
apiVersion: v1
kind: Service
metadata:
  name: discounts
  labels:
    app: discounts
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: discounts
---
apiVersion: v1
kind: Service
metadata:
  name: cars
  labels:
    app: cars
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: cars
EOF

After the installation, we can see that traffic is flowing to the remote cluster too:

This is happening automatically, Istio balances the traffic to both services. The key thing to notice here is that there is a concept called namespace sameness in Istio that is very important when planning our multicluster setup.

In both clusters, we can see that we have the same namespaces. They are called the same in both. Also, we can see that the services in both clusters need to exist and be called the same.

When we created the west’s namespaces, they are called the same, and also notice that even if we do not have instances of insurances or cars, we created the services. This is because travel services from the cluster will try to communicate with these services, not caring at all if the applications are in the west or east cluster. Istio will handle the routing in the back.

From this moment, we can start playing with Kiali to introduce some scenarios previously seen in the Travels tutorial.

3 - Travels Demo Tutorial

This tutorial uses the Kiali Travels Demo to teach Kiali and Istio features.

3.1 - Prerequisites

Platform Setup

This tutorial assumes you will have access to a Kubernetes cluster with Istio installed.

This tutorial has been tested using:

• a Minikube installation.
• an OpenShift installation.

Install Istio

Once you have your Kubernetes cluster ready, follow the Istio Getting Started to install and setup a demo profile that will be used in this tutorial.

DNS entries can be added in a basic way to the /etc/hosts file but you can use any other DNS service that allows to resolve a domain with the external Ingress IP.

Update Kiali

Istio defines a specific Kiali version as an addon.

In this tutorial we are going to update Kiali to the latest release version.

Assuming you have installed the addons following the Istio Getting Started guide, you can uninstall Kiali with the command:

kubectl delete -f ${ISTIO_HOME}/samples/addons/kiali.yaml --ignore-not-found

There are multiple ways to install a recent version of Kiali, this tutorial follows the Quick Start using Helm Chart.

helm install \
  --namespace istio-system \
  --set auth.strategy="anonymous" \
  --repo https://kiali.org/helm-charts \
  kiali-server \
  kiali-server

Access the Kiali UI

The Istio istioctl client has an easy method to expose and access Kiali:

${ISTIO_HOME}/bin/istioctl dashboard kiali

There are other alternatives to expose Kiali or other Addons in Istio. Check Remotely Accessing Telemetry Addons for more information.

After the Prerequisites you should be able to access Kiali. Verify its version by clicking the “?” icon and selecting “About”:

3.2 - Install Travel Demo

Deploy the Travel Demo

This demo application will deploy several services grouped into three namespaces.

Note that at this step we are going to deploy the application without any reference to Istio.

We will join services to the ServiceMesh in a following step.

To create and deploy the namespaces perform the following commands:

kubectl create namespace travel-agency
kubectl create namespace travel-portal
kubectl create namespace travel-control

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_agency.yaml) -n travel-agency
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_portal.yaml) -n travel-portal
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_control.yaml) -n travel-control

Check that all deployments rolled out as expected:

$ kubectl get deployments -n travel-control
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
control   1/1     1            1           85s

$ kubectl get deployments -n travel-portal
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
travels   1/1     1            1           91s
viaggi    1/1     1            1           91s
voyages   1/1     1            1           91s

$ kubectl get deployments -n travel-agency
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
cars-v1         1/1     1            1           96s
discounts-v1    1/1     1            1           96s
flights-v1      1/1     1            1           96s
hotels-v1       1/1     1            1           96s
insurances-v1   1/1     1            1           96s
mysqldb-v1      1/1     1            1           96s
travels-v1      1/1     1            1           96s

Understanding the demo application

Travel Portal namespace

The Travel Demo application simulates two business domains organized in different namespaces.

In a first namespace called travel-portal there will be deployed several travel shops, where users can search for and book flights, hotels, cars or insurance.

The shop applications can behave differently based on request characteristics like channel (web or mobile) or user (new or existing).

These workloads may generate different types of traffic to imitate different real scenarios.

All the portals consume a service called travels deployed in the travel-agency namespace.

Travel Agency namespace

A second namespace called travel-agency will host a set of services created to provide quotes for travel.

A main travels service will be the business entry point for the travel agency. It receives a destination city and a user as parameters and it calculates all elements that compose a travel budget: airfare, lodging, car reservation and travel insurance.

Each service can provide an independent quote and the travels service must then aggregate them into a single response.

Additionally, some users, like registered users, can have access to special discounts, managed as well by an external service.

Service relations between namespaces can be described in the following diagram:

Travel Portal and Travel Agency flow

A typical flow consists of the following steps:

. A portal queries the travels service for available destinations. . Travels service queries the available hotels and returns to the portal shop. . A user selects a destination and a type of travel, which may include a flight and/or a car, hotel and insurance. . Cars, Hotels and Flights may have available discounts depending on user type.

Travel Control namespace

The travel-control namespace runs a business dashboard with two key features:

• Allow setting changes for every travel shop simulator (traffic ratio, device, user and type of travel).
• Provide a business view of the total requests generated from the travel-portal namespace to the travel-agency services, organized by business criteria as grouped per shop, per type of traffic and per city.

3.3 - First Steps

Missing Sidecars

The Travel Demo has been deployed in the previous step but without installing any Istio sidecar proxy.

In that case, the application won’t connect to the control plane and won’t take advantage of Istio’s features.

In Kiali, we will see the new namespaces in the overview page:

But we won’t see any traffic in the graph page for any of these new namespaces:

If we examine the Applications, Workloads or Services page, it will confirm that there are missing sidecars:

Enable Sidecars

In this tutorial, we will add namespaces and workloads into the ServiceMesh individually step by step.

This will help you to understand how Istio sidecar proxies work and how applications can use Istio’s features.

We are going to start with the control workload deployed into the travel-control namespace:

Understanding what happened:

(i) Sidecar Injection

(ii) Automatic Sidecar Injection

Open Travel Demo to Outside Traffic

The control workload now has an Istio sidecar proxy injected but this application is not accessible from the outside.

In this step we are going to expose the control service using an Istio Ingress Gateway which will map a path to a route at the edge of the mesh.

For minikube we will check the External IP of the Ingress gateway:

$ kubectl get services/istio-ingressgateway -n istio-system
NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                                                                      AGE
istio-ingressgateway   LoadBalancer   10.101.6.144   10.101.6.144   15021:30757/TCP,80:32647/TCP,443:30900/TCP,31400:30427/TCP,15443:31072/TCP   19h

And we will add a simple entry to the /etc/hosts of the tutorial machine with the desired DNS entry:

...
10.101.6.144 control.travel-control.istio-cluster.org
...

Then, from this machine, the url control.travel-control.istio-cluster.org will resolve to the External IP of the Ingress Gateway of Istio.

For OpenShift we will expose the Ingress gateway as a service:

$ oc expose service istio-ingressgateway -n istio-system
$ oc get routes -n istio-system
NAME                   HOST/PORT                                  PATH   SERVICES               PORT    TERMINATION          WILDCARD
istio-ingressgateway   <YOUR_ROUTE_HOST>                                 istio-ingressgateway   http2                        None

Then, from this machine, the host <YOUR_ROUTE_HOST> will resolve to the External IP of the Ingress Gateway of Istio. For OpenShift we will not define a DNS entry, instead, where you see control.travel-control.istio-cluster.org in the steps below, subsitute the value of <YOUR_ROUTE_HOST>.

Use “Add Route Rule” button to add a default rule where any request will be routed to the control workload.

Use the Advanced Options and add a gateway with host control.travel-control.istio-cluster.org and create the Istio config.

Verify the Istio configuration generated.

Understanding what happened:

• External traffic enters into the cluster through a Gateway
• Traffic is routed to the control service through a VirtualService
• Kiali Graph visualizes the traffic telemetry reported from the control sidecar proxy
  • Only the travel-control namespace is part of the mesh

(i) Istio Gateway

(ii) Istio Virtual Service

3.4 - Observe

Enable Sidecars in all workloads

An Istio sidecar proxy adds a workload into the mesh.

Proxies connect with the control plane and provide Service Mesh functionality.

Automatically providing metrics, logs and traces is a major feature of the sidecar.

In the previous steps we have added a sidecar only in the travel-control namespace’s control workload.

We have added new powerful features but the application is still missing visibility from other workloads.

That control workload provides good visibility of its traffic, but telemetry is partially enabled, as travel-portal and travel-agency workloads don’t have sidecar proxies.

In the First Steps of this tutorial we didn’t inject the sidecar proxies on purpose to show a scenario where only some workloads may have sidecars.

Typically, Istio users annotate namespaces before the deployment to allow Istio to automatically add the sidecar when the application is rolled out into the cluster. Perform the following commands:

kubectl label namespace travel-agency istio-injection=enabled
kubectl label namespace travel-portal istio-injection=enabled

kubectl rollout restart deploy -n travel-portal
kubectl rollout restart deploy -n travel-agency

Verify that travel-control, travel-portal and travel-agency workloads have sidecars deployed:

Graph walkthrough

The graph provides a powerful set of Graph Features to visualize the traffic topology of the service mesh.

In this step, we will show how to use the Graph to show relevant information in the context of the Travel Demo application.

Our goal will be to identify the most critical service of the demo application.

Review the status of the mesh, everything seems healthy, but also note that hotels service has more load compared to other services inlcuded in the travel-agency namespace.

Combining telemetry and tracing information will show that there are traces started from a portal that involve multiple services but also other traces that only consume the hotels service.

The graph can focus on an element to study a particular context in detail. Note that a contextual menu is available using right-click, to easily shortcut the navigation to other sections.

Application details

Kiali provides Detail Views to navigate into applications, workloads and services.

These views provide information about the structure, health, metrics, logs, traces and Istio configuration for any application component.

In this tutorial we are going to learn how to use them to examine the main travels application of our example.

An application is an abstract group of workloads and services labeled with the same “application” name.

From Service Mesh perspective this concept is significant as telemetry and tracing signals are mainly grouped by “application” even if multiple workloads are involved.

At this point of the tutorial, the travels application is quite simple, just a travels-v1 workload exposed through the travels service. Navigate to the travels-v1 workload detail by clicking the link in the travels application overview.

The Metrics tab provides a powerful visualization of telemetry collected by the Istio proxy sidecar. It presents a dashboard of charts, each of which can be expanded for closer inspection. Expand the Request volume chart:

Metrics Settings provides multiple predefined criteria out-of-the-box. Additionally, enable the spans checkbox to correlate metrics and tracing spans in a single chart.

We can see in the context of the Travels application, the hotels service request volume differs from that of the other travel-agency services.

By examining the Request Duration chart also shows that there is no suspicious delay, so probably this asymmetric volume is part of the application business’ logic.

The Logs tab provides a unified view of application container logs with the Istio sidecar proxy logs. It also offers a spans checkbox, providing a correlated view of both logs and tracing, helping identify spans of interest.

From the application container log we can spot that there are two main business methods: GetDestinations and GetTravelQuote.

In the Istio sidecar proxy log we see that GetDestinations invokes a GET /hotels request without parameters.

However, GetTravelQuote invokes multiple requests to other services using a specific city as a parameter.

Then, as discussed in the Travel Demo design, an initial query returns all available hotels before letting the user choose one and then get specific quotes for other destination services.

That scenario is shown in the increase of the hotels service utilization.

Now we have identified that the hotels service has more use than other travel-agency services.

The next step is to get more context to answer if some particular service is acting slower than expected.

The Traces tab allows comparison between traces and metrics histograms, letting the user determine if a particular spike is expected in the context of average values.

In the same context, individual spans can be compared in more detail, helping to identify a problematic step in the broader scenario.

3.5 - Connect

Request Routing

The Travel Demo application has several portals deployed on the travel-portal namespace consuming the travels service deployed on the travel-agency namespace.

The travels service is backed by a single workload called travels-v1 that receives requests from all portal workloads.

At a moment of the lifecycle the business needs of the portals may differ and new versions of the travels service may be necessary.

This step will show how to route requests dynamically to multiple versions of the travels service.

To deploy the new versions of the travels service execute the following commands:

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v2.yaml) -n travel-agency
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v3.yaml) -n travel-agency

As there is no specific routing defined, when there are multiple workloads for travels service the requests are uniformly distributed.

The Traffic Management features of Istio allow you to define Matching Conditions for dynamic request routing.

In our scenario we would like to perform the following routing logic:

• All traffic from travels.uk routed to travels-v1
• All traffic from viaggi.it routed to travels-v2
• All traffic from voyages.fr routed to travels-v3

Portal workloads use HTTP/1.1 protocols to call the travels service, so one strategy could be to use the HTTP headers to define the matching condition.

But, where to find the HTTP headers ? That information typically belongs to the application domain and we should examine the code, documentation or dynamically trace a request to understand which headers are being used in this context.

There are multiple possibilities. The Travel Demo application uses an Istio Annotation feature to add an annotation into the Deployment descriptor, which adds additional Istio configuration into the proxy.

In our example the HTTP Headers are added as part of the trace context.

Then tracing will populate custom tags with the portal, device, user and travel used.

We will define three “Request Matching” rules as part of this request routing. Define all three rules before clicking the Create button.

In the first rule, we will add a request match for when the portal header has the value of travels.uk.

Define the exact match, like below, and click the “Add Match” button to update the “Matching selected” for this rule.

Move to “Route To” tab and update the destination for this “Request Matching” rule. Then use the “Add Route Rule” to create the first rule.

Add similar rules to route traffic from viaggi.it to travels-v2 workload and from voyages.fr to travels-v3 workload.

When the three rules are defined you can use “Create” button to generate all Istio configurations needed for this scenario. Note that the rule ordering does not matter in this scenario.

The Istio config for a given service is found on the “Istio Config” card, on the Service Details page.

Once the Request Routing is working we can verify that outbound traffic from every portal goes to the single travels workload. To see this clearly use a “Workload Graph” for the “travel-portal” namespace, enable “Traffic Distribution” edge labels and disable the “Service Nodes” Display option:

Note that no distribution label on an edge implies 100% of traffic.

Examining the “Inbound Traffic” for any of the travels workloads will show a similar pattern in the telemetry.

Using a custom time range to select a large interval, we can see how the workload initially received traffic from all portals but then only a single portal after the Request Routing scenarios were defined.

Kiali Wizards allow you to define high level Service Mesh scenarios and will generate the Istio Configuration needed for its implementation (VirtualServices, DestinationRules, Gateways and PeerRequests). These scenarios can be updated or deleted from the “Actions” menu of a given service.

To experiment further you can navigate to the travels service and update your configuration by selecting “Request Routing”, as shown below. When you have finished experimenting with Routing Request scenarios then use the “Actions” menu to delete the generated Istio config.

Fault Injection

The Observe step has spotted that the hotels service has additional traffic compared with other services deployed in the travel-agency namespace.

Also, this service becomes critical in the main business logic. It is responsible for querying all available destinations, presenting them to the user, and getting a quote for the selected destination.

This also means that the hotels service may be one of the weakest points of the Travel Demo application.

This step will show how to test the resilience of the Travel Demo application by injecting faults into the hotels service and then observing how the application reacts to this scenario.

Select an HTTP Delay and specify the “Delay percentage” and “Fixed Delay” values. The default values will introduce a 5 seconds delay into 100% of received requests.

Telemetry is collected from proxies and it is labeled with information about the source and destination workloads.

In our example, let’s say that travels service (“Service A” in the Istio diagram below) invokes the hotels service (“Service B” in the diagram). Travels is the “source” workload and hotels is the “destination” workload. The travels proxy will report telemetry from the source perspective and hotels proxy will report telemetry from the destination perspective. Let’s look at the latency reporting from both perspectives.

The travels workload proxy has the Fault Injection configuration so it will perform the call to the hotels service and will apply the delay on the travels workload side (this is reported as source telemetry).

We can see in the hotels telemetry reported by the source (the travels proxy) that there is a visible gap showing 5 second delay in the request duration.

But as the Fault Injection delay is applied on the source proxy (travels), the destination proxy (hotels) is unaffected and its destination telemetry show no delay.

The injected delay is propagated from the travels service to the downstream services deployed on travel-portal namespace, degrading the overall response time. But the downstream services are unaware, operate normally, and show a green status.

As part of this step you can update the Fault Injection scenario to test different delays. When finished, you can delete the generated Istio config for the hotels service.

Traffic Shifting

In the previous Request Routing step we have deployed two new versions of the travels service using the travels-v2 and travels-v3 workloads.

That scenario showed how Istio can route specific requests to specific workloads. It was configured such that each portal deployed in the travel-portal namespace (travels.uk, viaggi.it and voyages.fr) were routed to a specific travels workload (travels-v1, travels-v2 and travels-v3).

This Traffic Shifting step will simulate a new scenario: the new travels-v2 and travels-v3 workloads will represent new improvements for the travels service that will be used by all requests.

These new improvements implemented in travels-v2 and travels-v3 represent two alternative ways to address a specific problem. Our goal is to test them before deciding which one to use as a next version.

At the beginning we will send 80% of the traffic into the original travels-v1 workload, and will split 10% of the traffic each on travels-v2 and travels-v3.

Create a scenario with 80% of the traffic distributed to travels-v1 workload and 10% of the traffic distributed each to travels-v2 and travels-v3.

Istio Telemetry is grouped per logical application. That has the advantage of easily comparing different but related workloads, for one or more services.

In our example, we can use the “Inbound Metrics” and “Outbound Metrics” tabs in the travels application details, group by “Local version” and compare how travels-v2 and travels-v3 are working.

The charts show that the Traffic distribution is working accordingly and 80% is being distributed to travels-v1 workload and they also show no big differences between travels-v2 and travels-v3 in terms of request duration.

As part of this step you can update the Traffic Shifting scenario to test different distributions. When finished, you can delete the generated Istio config for the travels service.

TCP Traffic Shifting

The Travel Demo application has a database service used by several services deployed in the travel-agency namespace.

At some point in the lifecycle of the application the telemetry shows that the database service degrades and starts to increase the average response time.

This is a common situation. In this case, a database specialist suggests an update of the original indexes due to the data growth.

Our database specialist is suggesting two approaches and proposes to prepare two versions of the database service to test which may work better.

This step will show how the “Traffic Shifting” strategy can be applied to TCP services to test which new database indexing strategy works better.

To deploy the new versions of the mysqldb service execute the commands:

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/mysql-v2.yaml) -n travel-agency
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/mysql-v3.yaml) -n travel-agency

Create a scenario with 80% of the traffic distributed to mysqldb-v1 workload and 10% of the traffic distributed each to mysqldb-v2 and mysqldb-v3.

Note that TCP telemetry has different types of metrics, as “Traffic Distribution” is only available for HTTP/gRPC services, for this service we need to use “Traffic Rate” to evaluate the distribution of data (bytes-per-second) between mysqldb workloads.

TCP services have different telemetry but it’s still grouped by versions, allowing the user to compare and study pattern differences for mysqldb-v2 and mysqldb-v3.

The charts show more peaks in mysqldb-v2 compared to mysqldb-v3 but overall a similar behavior, so it’s probably safe to choose either strategy to shift all traffic.

As part of this step you can update the TCP Traffic Shifting scenario to test a different distribution. When finished, you can delete the generated Istio config for the mysqldb service.

Request Timeouts

In the Fault Injection step we showed how we could introduce a delay in the critical hotels service and test the resilience of the application.

The delay was propagated across services and Kiali showed how services accepted the delay without creating errors on the system.

But in real scenarios delays may have important consequences. Services may prefer to fail sooner, and recover, rather than propagating a delay across services.

This step will show how to add a request timeout for one of the portals deployed in travel-portal namespace. The travel.uk and viaggi.it portals will accept delays but voyages.fr will timeout and fail.

Repeat the Fault Injection step to add delay on hotels service.

Add a rule to add a request timeout only on requests coming from voyages.fr portal:

• Use the Request Matching tab to add a matching condition for the portal header with voyages.fr value.
• Use the Request Timeouts tab to add an HTTP Timeout for this rule.
• Add the rule to the scenario.

A first rule should be added to the list like:

Add a second rule to match any request and create the scenario. With this configuration, requests coming from voyages.fr will match the first rule and all others will match the second rule.

Create the rule. The Graph will show how requests coming from voyages.fr start to fail, due to the request timeout introduced.

Requests coming from other portals work without failures but are degraded by the hotels delay.

This scenario can be visualized in detail if we examine the “Inbound Metrics” and we group by “Remote app” and “Response code”.

As expected, the requests coming from voyages.fr don’t propagate the delay and they fail in the 2 seconds range, meanwhile requests from other portals don’t fail but they propagate the delay introduced in the hotels service.

As part of this step you can update the scenarios defined around hotels and travels services to experiment with more conditions, or you can delete the generated Istio config in both services.

Circuit Breaking

Distributed systems will benefit from failing quickly and applying back pressure, as opposed to propagating delays and errors through the system.

Circuit breaking is an important technique used to limit the impact of failures, latency spikes, and other types of network problems.

This step will show how to apply a Circuit Breaker into the travels service in order to limit the number of concurrent requests and connections.

In this example we are going to deploy a new workload that will simulate an important increase in the load of the system.

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_loadtester.yaml) -n travel-portal

The loadtester workload will try to create 50 concurrent connections to the travels service, adding considerable pressure to the travels-agency namespace.

The Travel Demo application is capable of handling this load and in a first look it doesn’t show unhealthy status.

But in a real scenario an unexpected increase in the load of a service like this may have a significant impact in the overall system status.

Use the “Traffic Shifting” Wizard to distribute traffic (evenly) to the travels workloads and use the “Advanced Options” to add a “Circuit Breaker” to the scenario.

The “Connection Pool” settings will indicate that the proxy sidecar will reject requests when the number of concurrent connections and requests exceeds more than one.

The “Outlier Detection” will eject a host from the connection pool if there is more than one consecutive error.

In the loadtester versioned-app Graph we can see that the travels service’s Circuit Breaker accepts some, but fails most, connections.

Remember, that these connections are stopped by the proxy on the loadtester side. That “fail sooner” pattern prevents overloading the network.

Using the Graph we can select the failed edge, check the Flags tab, and see that those requests are closed by the Circuit breaker.

If we examine the “Request volume” metric from the “Outbound Metrics” tab we can see the evolution of the requests, and how the introduction of the Circuit Breaker made the proxy reduce the request volume.

As part of this step you can update the scenarios defined around the travels service to experiment with more Circuit Breaker settings, or you can delete the generated Istio config in the service.

Understanding what happened:

(i) Circuit Breaking

(ii) Outlier Detection

(iii) Connection Pool Settings

(iv) Envoy’s Circuit breaking Architecture

Mirroring

This tutorial has shown several scenarios where Istio can route traffic to different versions in order to compare versions and evaluate which one works best.

The Traffic Shifting step was focused on travels service adding a new travels-v2 and travels-v3 workloads and the TCP Traffic Shifting showed how this scenario can be used on TCP services like mysqldb service.

Mirroring (or shadowing) is a particular case of the Traffic Shifting scenario where the proxy sends a copy of live traffic to a mirrored service.

The mirrored traffic happens out of band of the primary request path. It allows for testing of alternate services, in production environments, with minimal risk.

Istio mirrored traffic is only supported for HTTP/gRPC protocols.

This step will show how to apply mirrored traffic into the travels service.

We will simulate the following:

• travels-v1 is the original traffic and it will keep 80% of the traffic
• travels-v2 is the new version to deploy, it’s being evaluated and it will get 20% of the traffic to compare against travels-v1
• But travels-v3 will be considered as a new, experimental version for testing outside of the regular request path. It will be defined as a mirrored workload on 50% of the original requests.

Note that Istio does not report mirrored traffic telemetry from the source proxy. It is reported from the destination proxy, although it is not flagged as mirrored, and therefore an edge from travels to the travels-v3 workload will appear in the graph. Note the traffic rates reflect the expected ratio of 80/20 between travels-v1 and travels-v2, with travels-v3 at about half of that total.

This can be examined better using the “Source” and “Destination” metrics from the “Inbound Metrics” tab.

The “Source” proxy, in this case the proxies injected into the workloads of travel-portal namespace, won’t report telemetry for travels-v3 mirrored workload.

But the “Destination” proxy, in this case the proxy injected in the travels-v3 workload, will collect the telemetry from the mirrored traffic.

As part of this step you can update the Mirroring scenario to test different mirrored distributions.

When finished you can delete the generated Istio config for the travels service.

3.6 - Secure

Authorization Policies and Sidecars

Security is one of the main pillars of Istio features.

The Istio Security High Level Architecture provides a comprehensive solution to design and implement multiple security scenarios.

In this tutorial we will show how Kiali can use telemetry information to create security policies for the workloads deployed in a given namespace.

Istio telemetry aggregates the ServiceAccount information used in the workloads communication. This information can be used to define authorization policies that deny and allow actions on future live traffic communication status.

Additionally, Istio sidecars can be created to limit the hosts with which a given workload can communicate. This improves traffic control, and also reduces the memory footprint of the proxies.

This step will show how we can define authorization policies for the travel-agency namespace, in the Travel Demo application, for all existing traffic in a given time period.

Once authorization policies are defined, a new workload will be rejected if it doesn’t match the security rules defined.

In this example we will use the loadtester workload as the “intruder” in our security rules.

If we have followed the previous tutorial steps, we need to undeploy it from the system.

kubectl delete -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_loadtester.yaml) -n travel-portal

We should validate that telemetry has updated the travel-portal namespace and “Security” can be enabled in the Graph Display options.

Every workload in the cluster uses a Service Account.

travels.uk, viaggi.it and voyages.fr workloads use the default cluster.local/ns/travel-portal/sa/default ServiceAccount defined automatically per namespace.

This information is propagated into the Istio Telemetry and Kiali can use it to define a set of AuthorizationPolicy rules, and Istio Sidecars.

The Sidecars restrict the list of hosts with which each workload can communicate, based on the current traffic.

The “Create Traffic Policies” action, located in the Overview page, will create these definitions.

This will generate a main DENY ALL rule to protect the whole namespace, and an individual ALLOW rule per workload identified in the telemetry.

It will create also an individual Sidecar per workload, each of them containing the set of hosts.

As an example, we can see that for the travels-v1 workload, the following list of hosts are added to the sidecar.

If the loadtester workload uses a different ServiceAccount then, when it’s deployed, it won’t comply with the AuthorizationPolicy rules defined in the previous step.

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_loadtester.yaml) -n travel-portal

Now, travels workload will reject requests made by loadtester workload and that situation will be reflected in Graph:

This can also be verified in the details page using the Outbound Metrics tab grouped by response code (only the 403 line is present).

Inspecting the Logs tab confirms that loadtester workload is getting a HTTP 403 Forbidden response from travels workloads, as expected.

AuthorizationPolicy resources are defined per workload using matching selectors.

As part of the example, we can show how a ServiceAccount can be added into an existing rule to allow traffic from loadtester workload into the travels-v1 workload only.

As expected, now we can see that travels-v1 workload accepts requests from all travel-portal namespace workloads, but travels-v2 and travels-v3 continue rejecting requests from loadtester source.

Using “Outbound Metrics” tab from the loadtester workload we can group per “Remote version” and “Response code” to get a detailed view of this AuthorizationPolicy change.

According to Istio Sidecar documentation, Istio configures all mesh sidecar proxies to reach every mesh workload. After the sidecars are created, the list of hosts is reduced according to the current traffic. To verify this, we can look for the clusters configured in each proxy.

As an example, looking into the cars-v1 workload, we can see that there is a reduced number of clusters with which the proxy can communicate.

As part of this step, you can update the AuthorizationPolicies and Istio Sidecars generated for the travel-agency namespace, and experiment with more security rules. Or, you can delete the generated Istio config for the namespace.

3.7 - Uninstall Travel Demo

To uninstall the Travel Demo application perform the following commands:

kubectl delete namespace travel-agency
kubectl delete namespace travel-portal
kubectl delete namespace travel-control