# 8주차 - Hubble

## 1. 목적성

* Hubble에 대해 알아봅니다.
* Hubble 의 가시성을 이용해 cillium 내의 네트워크 통신흐름들을 확인합니다.

## 2. Hubble 이란?

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FnKxfJa1CpM37O7porafI%2Fimage.png?alt=media&#x26;token=ef1dcd67-f296-4c91-9a17-bde6d04522e8" alt=""><figcaption><p><a href="https://cilium.io/blog/2020/06/22/cilium-18">https://cilium.io/blog/2020/06/22/cilium-18</a></p></figcaption></figure>

* Hubble은 네트워킹 인프라에 대한 동작에 가시성을 제공합니다.

* 애플리케이션의 코드 수정 없이 설정되며 IP 기반이 아니라, 서비스/파드 기반으로 모니터링을 제공합니다.

* 다양한 메트릭과 서비스 종속성을 ui로 나타내줍니다.

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FdE5aj2iegalOYMVA0Xwf%2Fimage.png?alt=media&#x26;token=514f2f9c-30a6-4df1-a866-52e21ce7a4bc" alt=""><figcaption></figcaption></figure>

## 3. 노드간 파드 통신

* Hubble CLI 확인

```
# 확인
cilium status

# UI 파드 정보 확인
kubectl get pod -n kube-system -l k8s-app=hubble-ui -o wide

# Hubble UI 웹 접속
kubectl patch -n kube-system svc hubble-ui -p '{"spec": {"type": "NodePort"}}'
HubbleUiNodePort=$(kubectl get svc -n kube-system hubble-ui -o jsonpath={.spec.ports[0].nodePort})
echo -e "Hubble UI URL = http://$(curl -s ipinfo.io/ip):$HubbleUiNodePort"

## Service NodePort 생성 후 아래 정보 확인!
iptables -t nat -S
conntrack -L
conntrack -L |grep -v 2379

# Install Hubble Client
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}

# Hubble API Access : localhost TCP 4245 Relay 를 통해 접근, observe 를 통해서 flow 쿼리 확인 가능!
cilium hubble port-forward &

# CLI 로 Hubble API 상태 확인
hubble status

# query the flow API and look for flows
hubble observe

```

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FPVvChbAouJzb8rWrJMPr%2Fimage.png?alt=media&#x26;token=6cca92f0-0c09-46a4-8291-1b3e221f6cfd" alt=""><figcaption></figcaption></figure>

* netpod 를 하나 생성해서 node 간 파드 통신을 확인해 봅시다.

```
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: netpod
  labels:
    app: netpod
spec:
  nodeName: k8s-s
  containers:
  - name: netshoot-pod
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
---
apiVersion: v1
kind: Pod
metadata:
  name: webpod1
  labels:
    app: webpod
spec:
  nodeName: k8s-w1
  containers:
  - name: container
    image: traefik/whoami
  terminationGracePeriodSeconds: 0
---
apiVersion: v1
kind: Pod
metadata:
  name: webpod2
  labels:
    app: webpod
spec:
  nodeName: k8s-w2
  containers:
  - name: container
    image: traefik/whoami
  terminationGracePeriodSeconds: 0
EOF
```

* 실제로 통신을 발생시켜서 Hubble UI의 변동을 확인해봅시다.

```
# 테스트 파드들 IP
NETPODIP=$(kubectl get pods netpod -o jsonpath='{.status.podIP}')
WEBPOD1IP=$(kubectl get pods webpod1 -o jsonpath='{.status.podIP}')
WEBPOD2IP=$(kubectl get pods webpod2 -o jsonpath='{.status.podIP}')

# 단축키(alias) 지정
alias p0="kubectl exec -it netpod  -- "
alias p1="kubectl exec -it webpod1 -- "
alias p2="kubectl exec -it webpod2 -- "
```

```
p0 ping -c 1 $WEBPOD1IP && p0 ping -c 1 $WEBPOD2IP
p0 curl -s $WEBPOD1IP && p0 curl -s $WEBPOD2IP
p0 curl -s $WEBPOD1IP:8080 ; p0 curl -s $WEBPOD2IP:8080
```

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2Ffq5IrDRaz1OURKPEdAex%2Fimage.png?alt=media&#x26;token=88a8825b-e939-4266-8842-bb6f78684aa5" alt=""><figcaption></figcaption></figure>

* eBPF 부착을 확인합니다.

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FcePLtZNyIoFOmFO94TYS%2Fimage.png?alt=media&#x26;token=b19aad8a-982d-4e53-b868-9722c9bcc93f" alt=""><figcaption></figcaption></figure>

* eBPF 가 수행하게될 MAP LIST가 여기에 나와있습니다.

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2F54Ojex1v7ACXjYaz6mRY%2Fimage.png?alt=media&#x26;token=838c3993-941b-487c-93de-3bfb76b622c2" alt=""><figcaption></figcaption></figure>

## 4. 서비스 통신

* Socket-Based LoadBalancing은 어떻게 이뤄질까요?
  * 왼쪽에서는 dnat 을 거치는 네트워크 기반이고
  * 오른쪽은 socket쪽에서 바로 dnat처리하는 로드 밸런싱입니다.

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FHfNINv9q9yIv5tklZ7vI%2Fimage.png?alt=media&#x26;token=b4326fc1-6193-4c1b-b2df-a6fa013c1c85" alt=""><figcaption></figcaption></figure>

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FmJxHVpvoDDzrswEl9Urk%2Fimage.png?alt=media&#x26;token=d4742ade-936f-4152-bf0f-7e73353ca010" alt=""><figcaption></figcaption></figure>

* BPF socket operations program 은 root cgroup 에 연결되며 TCP event(ESTABLISHED) 에서 실행한다.
* connect() 와 sendto() 소켓 함수에 연결된 프로그램(connect4, sendmsg4)에서는 소켓의 목적지 주소를 백엔드 주소와 포트로 변환하고, cilium\_lb4\_backends 맵에 백엔드 주소와 포트를 등록해놓는다. 이후 recvmsg() 소켓 함수에 연결된 프로그램(recvmsg4)에서는 cilium\_lb4\_reverse\_nat 맵을 이용해서 목적지 주소와 포트를 다시 서비스 주소와 포트로 변환함.

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2F1z16wFkhyFMXk7SIZ7GI%2Fimage.png?alt=media&#x26;token=685eaa86-9732-49b7-91e8-0d046d15bc86" alt=""><figcaption></figcaption></figure>

* Socket의 Namespace에서 목적지 주소를 바로 변경합니다.

### 4.a 서비스 생성 및 접속 확인

```
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: svc
spec:
  ports:
    - name: svc-webport
      port: 80
      targetPort: 80
  selector:
    app: webpod
  type: ClusterIP
EOF

// 서비스 확인
kubectl get svc,ep svc


# 서비스IP를 변수에 지정
SVCIP=$(kubectl get svc svc -o jsonpath='{.spec.clusterIP}')

# Pod1 에서 Service(ClusterIP) 접속 트래픽 발생
kubectl exec netpod -- curl -s $SVCIP
kubectl exec netpod -- curl -s $SVCIP | grep Hostname

# 지속적으로 접속 트래픽 발생
SVCIP=$(kubectl get svc svc -o jsonpath='{.spec.clusterIP}')
while true; do kubectl exec netpod -- curl -s $SVCIP | grep Hostname;echo "-----";sleep 1;done

# 파드에서 SVC(ClusterIP) 접속 시 tcpdump 로 확인 >> 파드 내부 캡쳐인데, SVC(10.108.12.195)는 보이지 않고, DNAT 된 web-pod 의 IP가 확인! Magic!
kubectl exec netpod -- tcpdump -enni any -q
	08:54:55.454271 eth0  Out ifindex 14 92:1a:b9:94:94:37 172.16.0.162.44718 > 172.16.1.234.80: tcp 0
	08:54:55.454798 eth0  In  ifindex 14 8a:0c:cc:a9:21:1a 172.16.1.234.80 > 172.16.0.162.44718: tcp 0
	08:54:55.455030 eth0  Out ifindex 14 92:1a:b9:94:94:37 172.16.0.162.44718 > 172.16.1.234.80: tcp 77
...

kubectl exec netpod -- sh -c "ngrep -tW byline -d eth0 '' 'tcp port 80'"
T 2024/10/20 08:07:36.663329 172.16.0.132:59964 -> 172.16.1.53:80 [AP] #34
GET / HTTP/1.1.
Host: 10.10.124.15.
User-Agent: curl/8.7.1.
Accept: */*.


# 서비스 정보 확인
c0 service list
ID   Frontend              Service Type   Backend
16   10.108.12.195:80      ClusterIP      1 => 172.16.2.157:80
                                          2 => 172.16.1.234:80
c0 bpf lb list
SERVICE ADDRESS       BACKEND ADDRESS
10.108.12.195:80      0.0.0.0:0 (16) [ClusterIP, non-routable]
                      172.16.1.234:80 (16)
                      172.16.2.157:80 (16)
# BPF maps
c0 map list --verbose
c0 map list --verbose | grep lb
c0 map get cilium_lb4_services_v2
c0 map get cilium_lb4_backends_v3
c0 map get cilium_lb4_reverse_nat
c0 map get cilium_lb4_reverse_sk
c0 map get cilium_lxc
c0 map get cilium_ipcache
```

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2F1hHRiXRuKELe3fB2uIUy%2Fimage.png?alt=media&#x26;token=e8193d70-807a-4719-be6c-b6d99c0c2f22" alt=""><figcaption><p>kube-svc rule 생성 안되는 것 확인</p></figcaption></figure>

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2F4JPNJj12sM05pqrInPOy%2Fimage.png?alt=media&#x26;token=351a7703-f0ac-441d-8972-f430dfd822ef" alt=""><figcaption></figcaption></figure>

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FFDbVPjp2b1IEjdbhXZr9%2Fimage.png?alt=media&#x26;token=442f3568-27f1-402a-93c8-772f26bb8709" alt=""><figcaption></figcaption></figure>

```
//syscall 목록들을 확인합니다.
kubectl exec netpod -- strace -c curl -s $SVCIP
```

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FH0NxgVSggKIjCPOgY88V%2Fimage.png?alt=media&#x26;token=14dae62f-6f72-43ac-8d21-8aff719febb8" alt=""><figcaption></figcaption></figure>

* socket 주소 가져오는 동작 확인 할 수 있습니다.

## 5. Prometheus 와 Grafana 모니터링

```
# 배포
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.16.3/examples/kubernetes/addons/prometheus/monitoring-example.yaml
kubectl get all -n cilium-monitoring

# 파드와 서비스 확인
kubectl get pod,svc,ep -o wide -n cilium-monitoring

# NodePort 설정
kubectl patch svc grafana -n cilium-monitoring -p '{"spec": {"type": "NodePort"}}'
kubectl patch svc prometheus -n cilium-monitoring -p '{"spec": {"type": "NodePort"}}'

# Grafana 웹 접속
GPT=$(kubectl get svc -n cilium-monitoring grafana -o jsonpath={.spec.ports[0].nodePort})
echo -e "Grafana URL = http://$(curl -s ipinfo.io/ip):$GPT"

# Prometheus 웹 접속 정보 확인
PPT=$(kubectl get svc -n cilium-monitoring prometheus -o jsonpath={.spec.ports[0].nodePort})
echo -e "Prometheus URL = http://$(curl -s ipinfo.io/ip):$PPT"
```

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FE93ZTkDrNwasFSxpeAY2%2Fimage.png?alt=media&#x26;token=6c650310-4e41-4ce7-9c72-53cac585cac5" alt=""><figcaption><p>grafana 대시보드 </p></figcaption></figure>

## 6. Network Policy 설정하기

### 6.1 Cillium의 보안

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2F75mC2aRdV2MUEQiYPzA4%2Fimage.png?alt=media&#x26;token=5d3bb6d6-0779-42e2-8d71-249016af78f6" alt=""><figcaption></figcaption></figure>

* L3, L4 ,L7 모두 네트워크 정책을 지원합니다.
* ID 기반
  * L3 기반으로 ROLE을 정해두면, 엔드포인트 간 통신이 가능해집니다.
* 포트기반
* HTTP기반

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2F8V0w1T9VWBWN00Fqwnon%2Fimage.png?alt=media&#x26;token=f331dfa2-64c5-4ae2-817f-589e98fd8cf9" alt=""><figcaption></figcaption></figure>

```
// 위의 구조를 배포합니다.
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.16.3/examples/minikube/http-sw-app.yaml
kubectl get all
# cilium endpoint 확인
kubectl get ciliumendpoints
c1 endpoint list
c2 endpoint list

```

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FxejNYDgXTh0GnEWam9K6%2Fimage.png?alt=media&#x26;token=ea612eb9-1a3c-4b5d-8d73-465736ddde1f" alt=""><figcaption></figcaption></figure>

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FXWQF3BZaJvLaocN5mV6o%2Fimage.png?alt=media&#x26;token=afb4b600-343d-49a4-8b27-3ff40554b04e" alt=""><figcaption></figcaption></figure>

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FLiWjUkH1MUkc7nDFL2D1%2Fimage.png?alt=media&#x26;token=b3371070-480f-44d0-8466-f27bcbb25b98" alt=""><figcaption></figcaption></figure>

### 6.2 L4 정책 생성

```
cat <<EOF | kubectl apply -f - 
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "rule1"
spec:
  description: "L3-L4 policy to restrict deathstar access to empire ships only"
  endpointSelector:
    matchLabels:
      org: empire
      class: deathstar
  ingress:
  - fromEndpoints:
    - matchLabels:
        org: empire
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
EOF

# 정책 확인
kubectl get cnp
kc describe cnp rule1
c0 policy get


# 파드 curl 접속 시도 시 파드 sh 접속 후 curl 시도하자!
# 데스스타 SVC(ClusterIP) 접속하여 웹 파드 연결 확인 >> Hubble UI 에서 drop 확인!
kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed

kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing

# 정책에 의해 PACKET DROP 확인
```

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FktfDSsqxwbxb10Hys5YZ%2Fimage.png?alt=media&#x26;token=917516dc-24a1-43e0-b915-ba10c2e48053" alt=""><figcaption></figcaption></figure>

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2Fb2zKc0qXsgTm7M7Wh1MQ%2Fimage.png?alt=media&#x26;token=818f986e-cf10-469e-a568-7c54b80fde2f" alt=""><figcaption></figcaption></figure>

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FL6pNljUfuAzN9ugfuVJk%2Fimage.png?alt=media&#x26;token=b3aa2b8e-f828-4c2f-b122-0709aaf7042f" alt=""><figcaption></figcaption></figure>

## 7. Bandwidth Manager

<figure><img src="https://579161259-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1gwifHg6wHBbDMYTGIsL%2Fuploads%2FVwyPT21F1RiixYVO1jX1%2Fimage.png?alt=media&#x26;token=e436a581-c0cf-4b49-83ed-cba1bb8eb3f9" alt=""><figcaption></figcaption></figure>

* 네트워크 트래픽 상한치 통제 장치 입니다.
* TCP 워크로드 최적화와 개별 파드에 대한 속도 제한을 위해 eBPF 를 사용합니다.

```
# 인터페이스 tc qdisc 확인
tc qdisc show dev ens5
qdisc mq 0: root 
qdisc fq_codel 0: parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
qdisc fq_codel 0: parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
qdisc fq_codel 0: parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
qdisc fq_codel 0: parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 

# 설정
helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values --set bandwidthManager.enabled=true

# 적용 확인
cilium config view | grep bandwidth
enable-bandwidth-manager                       true

# egress bandwidth limitation 동작하는 인터페이스 확인
c0 status | grep  BandwidthManager
BandwidthManager:        EDT with BPF [CUBIC] [ens5]

# 인터페이스 tc qdisc 확인 : 설정 전후 옵션값들이 상당히 추가된다
tc qdisc
tc qdisc show dev ens5
qdisc mq 8002: root 
qdisc fq 8005: parent 8002:2 limit 10000p flow_limit 100p buckets 32768 orphan_mask 1023 quantum 18030b initial_quantum 90150b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 2s horizon_drop 
qdisc fq 8003: parent 8002:4 limit 10000p flow_limit 100p buckets 32768 orphan_mask 1023 quantum 18030b initial_quantum 90150b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 2s horizon_drop 
qdisc fq 8004: parent 8002:3 limit 10000p flow_limit 100p buckets 32768 orphan_mask 1023 quantum 18030b initial_quantum 90150b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 2s horizon_drop 
qdisc fq 8006: parent 8002:1 limit 10000p flow_limit 100p buckets 32768 orphan_mask 1023 quantum 18030b initial_quantum 90150b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 2s horizon_drop 
```

```
# 테스트를 위한 트래픽 발생 서버/클라이언트 파드 생성
cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Pod
metadata:
  annotations:
    # Limits egress bandwidth to 10Mbit/s.
    kubernetes.io/egress-bandwidth: "10M"
  labels:
    # This pod will act as server.
    app.kubernetes.io/name: netperf-server
  name: netperf-server
spec:
  containers:
  - name: netperf
    image: cilium/netperf
    ports:
    - containerPort: 12865
---
apiVersion: v1
kind: Pod
metadata:
  # This Pod will act as client.
  name: netperf-client
spec:
  affinity:
    # Prevents the client from being scheduled to the
    # same node as the server.
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/name
            operator: In
            values:
            - netperf-server
        topologyKey: kubernetes.io/hostname
  containers:
  - name: netperf
    args:
    - sleep
    - infinity
    image: cilium/netperf
EOF

# egress BW 제한 정보 확인
kubectl describe pod netperf-server | grep Annotations:
Annotations:  kubernetes.io/egress-bandwidth: 10M

# egress BW 제한이 설정된 파드가 있는 cilium pod 에서 제한 정보 확인
c1 bpf bandwidth list
c2 bpf bandwidth list
IDENTITY   EGRESS BANDWIDTH (BitsPerSec)
904        10M

c1 endpoint list
c2 endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                 IPv6   IPv4           STATUS
           ENFORCEMENT        ENFORCEMENT
904        Disabled           Disabled          21565      k8s:app.kubernetes.io/name=netperf-server          172.16.2.153   ready

# 트래픽 발생 >> Hubble UI 에서 확인
# egress traffic of the netperf-server Pod has been limited to 10Mbit per second. 
NETPERF_SERVER_IP=$(kubectl get pod netperf-server -o jsonpath='{.status.podIP}')
kubectl exec netperf-client -- netperf -t TCP_MAERTS -H "${NETPERF_SERVER_IP}"
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec  
131072 16384   16384    10.00    9.54  # 10Mbps 제한 확인!

# 5M 제한 설정 후 테스트
kubectl get pod netperf-server -o json | sed -e 's|10M|5M|g' | kubectl apply -f -
c1 bpf bandwidth list
c2 bpf bandwidth list
kubectl exec netperf-client -- netperf -t TCP_MAERTS -H "${NETPERF_SERVER_IP}"
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec
131072 16384   16384    10.09    4.56  # 4.5Mbps 제한 확인!

# 20M 제한 설정 후 테스트
kubectl get pod netperf-server -o json | sed -e 's|5M|20M|g' | kubectl apply -f -
kubectl exec netperf-client -- netperf -t TCP_MAERTS -H "${NETPERF_SERVER_IP}"
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec
131072 16384   16384    10.00    18.95 # 19Mbps 제한 확인!

tc qdisc show dev ens5

# 삭제
kubectl delete pod netperf-client netperf-server
```
