L'échelleScale

kind (« Kubernetes in Docker ») fait tourner un vrai cluster dans des conteneurs Docker — parfait pour apprendre sans facture cloud. kubectl est la télécommande universelle de n'importe quel cluster Kubernetes. kind ("Kubernetes in Docker") runs a real cluster inside Docker containers — perfect for learning without a cloud bill. kubectl is the universal remote for any Kubernetes cluster.

# macOS
brew install kind kubectl

# Linux
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.23.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
curl -LO https://dl.k8s.io/release/v1.30.0/bin/linux/amd64/kubectl
chmod +x kubectl
sudo mv kubectl /usr/local/bin/kubectl

kind version
kubectl version --client

Une seule commande crée un nœud Kubernetes complet dans un conteneur et configure kubectl pour lui parler. One single command creates a complete Kubernetes node inside a container and configures kubectl to talk to it.

kind create cluster --name mlops
kubectl get nodes

Le cluster kind ne voit pas les images de votre Docker local : il faut les charger explicitement dans ses nœuds. L'oublier est LA cause n° 1 du fameux ImagePullBackOff. The kind cluster cannot see your local Docker images: you must load them into its nodes explicitly. Forgetting this is THE number 1 cause of the infamous ImagePullBackOff.

docker images fraud-api
kind load docker-image fraud-api:0.1 --name mlops

L'URL de MLflow ne vit pas dans l'image : elle vit dans une ConfigMap. Depuis un pod kind, la machine hôte s'appelle host.docker.internal (macOS / Windows) ; sur Linux, remplacez par http://172.17.0.1:5000 (l'IP du pont docker0). The MLflow URL does not live in the image: it lives in a ConfigMap. From a kind pod, the host machine is called host.docker.internal (macOS / Windows); on Linux, replace it with http://172.17.0.1:5000 (the docker0 bridge IP).

cd fraud-detection
mkdir -p k8s
cat > k8s/configmap.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: fraud-api-config
data:
  MLFLOW_TRACKING_URI: "http://host.docker.internal:5000"
EOF
kubectl apply -f k8s/configmap.yaml

Le contrat complet : 3 réplicas, ressources bornées, et les trois probes. La startupProbe est généreuse (30 × 2 s = 60 s de grâce) parce que le modèle est long à charger — sans elle, la liveness tue le pod avant la fin du chargement, en boucle. Gardez le serveur MLflow du bloc 5 allumé : les pods chargent le modèle au démarrage. The full contract: 3 replicas, bounded resources, and the three probes. The startupProbe is generous (30 × 2 s = 60 s of grace) because the model is slow to load — without it, liveness kills the pod before loading finishes, in a loop. Keep block 5's MLflow server on: pods load the model at startup.

cat > k8s/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fraud-api
  labels:
    app: fraud-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: fraud-api
      track: stable
  template:
    metadata:
      labels:
        app: fraud-api
        track: stable
    spec:
      containers:
        - name: fraud-api
          image: fraud-api:0.1
          imagePullPolicy: Never
          ports:
            - containerPort: 8000
          env:
            - name: MLFLOW_TRACKING_URI
              valueFrom:
                configMapKeyRef:
                  name: fraud-api-config
                  key: MLFLOW_TRACKING_URI
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
            failureThreshold: 3
EOF
kubectl apply -f k8s/deployment.yaml

Le Service sélectionne app: fraud-api sans regarder track — ce détail servira au canary. On observe les pods démarrer, puis on appelle l'API servie par le cluster. The Service selects app: fraud-api without looking at track — that detail will matter for the canary. Watch the pods start, then call the API served by the cluster.

cat > k8s/service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: fraud-api
spec:
  type: ClusterIP
  selector:
    app: fraud-api
  ports:
    - port: 80
      targetPort: 8000
EOF
kubectl apply -f k8s/service.yaml
kubectl get pods -w
# Ctrl-C once the 3 pods are Running and READY 1/1, then:
kubectl port-forward service/fraud-api 8000:80 &
sleep 2
curl -s http://localhost:8000/health

Tuez un pod : le Deployment constate l'écart avec l'état désiré (3 réplicas) et en recrée un, sans vous demander la permission. C'est l'idée déclarative en action. Kill a pod: the Deployment notices the gap with the desired state (3 replicas) and recreates one, without asking your permission. The declarative idea, in action.

kubectl get pods
kubectl delete pod $(kubectl get pods -l app=fraud-api -o jsonpath='{.items[0].metadata.name}')
kubectl get pods

Construisez la v0.2 (le contenu peut être identique — c'est le mécanisme qu'on observe), chargez-la dans kind, puis déployez-la en canary : mêmes labels app=fraud-api, mais track=canary. Le Service route vers les 4 pods → le canary reçoit ~25 % du trafic. Build v0.2 (the content can be identical — the mechanism is what we observe), load it into kind, then deploy it as a canary: same app=fraud-api labels, but track=canary. The Service routes to all 4 pods → the canary receives ~25% of the traffic.

docker build -t fraud-api:0.2 .
kind load docker-image fraud-api:0.2 --name mlops
cat > k8s/deployment-canary.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fraud-api-canary
  labels:
    app: fraud-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fraud-api
      track: canary
  template:
    metadata:
      labels:
        app: fraud-api
        track: canary
    spec:
      containers:
        - name: fraud-api
          image: fraud-api:0.2
          imagePullPolicy: Never
          ports:
            - containerPort: 8000
          env:
            - name: MLFLOW_TRACKING_URI
              valueFrom:
                configMapKeyRef:
                  name: fraud-api-config
                  key: MLFLOW_TRACKING_URI
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
            failureThreshold: 3
EOF
kubectl apply -f k8s/deployment-canary.yaml
kubectl get pods -l app=fraud-api

Envoyez 40 requêtes depuis l'intérieur du cluster (on vise /docs, que les probes ne touchent pas, pour un comptage propre), puis comptez ce qui est arrivé au canary dans ses logs. Send 40 requests from inside the cluster (we target /docs, which the probes never hit, for a clean count), then count what reached the canary in its logs.

kubectl run traffic --rm -it --restart=Never --image=busybox -- /bin/sh -c 'for i in $(seq 1 40); do wget -q -O /dev/null http://fraud-api/docs; done; echo done'
kubectl logs deployment/fraud-api-canary | grep -c 'GET /docs'

Les métriques tiennent ? Promotion : le Deployment stable passe en v0.2 par rolling update — un pod remplacé à la fois, jamais sous l'effectif — puis le canary est retiré. Si ça s'était mal passé : kubectl delete -f k8s/deployment-canary.yaml seul, et retour instantané à 100 % v0.1. Metrics hold? Promotion: the stable Deployment moves to v0.2 through a rolling update — one pod replaced at a time, never below strength — then the canary is withdrawn. Had it gone wrong: kubectl delete -f k8s/deployment-canary.yaml alone, and you are instantly back to 100% v0.1.

kubectl set image deployment/fraud-api fraud-api=fraud-api:0.2
kubectl rollout status deployment/fraud-api
kubectl delete -f k8s/deployment-canary.yaml
kubectl get deployment fraud-api -o wide

Le HPA a besoin de mesures CPU. Sur kind, metrics-server refuse les certificats des kubelets — le patch --kubelet-insecure-tls est le réglage standard pour un cluster local. The HPA needs CPU measurements. On kind, metrics-server rejects the kubelets' certificates — the --kubelet-insecure-tls patch is the standard tweak for a local cluster.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl patch -n kube-system deployment metrics-server --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl -n kube-system rollout status deployment/metrics-server
kubectl top nodes

Déclarez l'élastique (3 à 6 réplicas, cible 60 % de CPU), générez de la charge, et regardez. Si TARGETS ne décolle pas, lancez un second pod de charge (load2). La redescente, elle, prend ~5 minutes : l'HPA est prudent par conception. Declare the elastic (3 to 6 replicas, 60% CPU target), generate load, and watch. If TARGETS will not take off, start a second load pod (load2). Scaling down takes ~5 minutes: the HPA is cautious by design.

kubectl autoscale deployment fraud-api --cpu-percent=60 --min=3 --max=6
kubectl run load --image=busybox -- /bin/sh -c "while true; do wget -qO- http://fraud-api/health; done"
kubectl get hpa fraud-api -w
# Ctrl-C once REPLICAS goes above 3, then stop the load:
kubectl delete pod load

prometheus-fastapi-instrumentator instrumente toutes les routes et publie /metrics au format Prometheus. Une ligne dans requirements.txt, deux dans le code — c'est tout l'extrait. prometheus-fastapi-instrumentator instruments every route and publishes /metrics in Prometheus format. One line in requirements.txt, two in the code — that is the whole excerpt.

# requirements.txt: add this line
prometheus-fastapi-instrumentator

# src/serve/app.py: add these lines right after "app = FastAPI(...)"
from prometheus_fastapi_instrumentator import Instrumentator

Instrumentator().instrument(app).expose(app)

Le circuit devient un réflexe : build, kind load, set image, rollout. Si le port-forward de l'étape 6 est tombé pendant le rolling update (c'est normal), relancez kubectl port-forward service/fraud-api 8000:80 &. The circuit becomes a reflex: build, kind load, set image, rollout. If step 6's port-forward dropped during the rolling update (that is normal), rerun kubectl port-forward service/fraud-api 8000:80 &.

docker build -t fraud-api:0.3 .
kind load docker-image fraud-api:0.3 --name mlops
kubectl set image deployment/fraud-api fraud-api=fraud-api:0.3
kubectl rollout status deployment/fraud-api
curl -s http://localhost:8000/metrics | head -20

Trois objets dans un seul fichier : la ConfigMap de scrape (toutes les 15 s sur le Service fraud-api), le Deployment, le Service. Pas de Helm, pas de magie — juste du YAML lisible. Three objects in one file: the scrape ConfigMap (every 15 s against the fraud-api Service), the Deployment, the Service. No Helm, no magic — just readable YAML.

cat > k8s/prometheus.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: fraud-api
        metrics_path: /metrics
        static_configs:
          - targets: ["fraud-api:80"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.53.0
          args:
            - --config.file=/etc/prometheus/prometheus.yml
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
      volumes:
        - name: config
          configMap:
            name: prometheus-config
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  type: ClusterIP
  selector:
    app: prometheus
  ports:
    - port: 9090
      targetPort: 9090
EOF
kubectl apply -f k8s/prometheus.yaml
kubectl rollout status deployment/prometheus
kubectl port-forward service/prometheus 9090:9090 &

Le script compare la distribution des scores récents à une référence (test de Kolmogorov-Smirnov) et sort en code 1 si elle a bougé — exactement ce qu'un cron ou une CI sait transformer en alerte. On le teste avec une dérive fabriquée ; régénérez recent_scores avec beta(2, 20) pour voir le cas « OK ». The script compares the recent score distribution to a reference (Kolmogorov-Smirnov test) and exits with code 1 if it moved — exactly what a cron job or a CI can turn into an alert. We test it with a manufactured drift; regenerate recent_scores with beta(2, 20) to see the "OK" case.

pip install scipy numpy
mkdir -p scripts data
cat > scripts/check_drift.py <<'EOF'
"""Compare recent prediction scores against a reference distribution (KS test)."""
import json
import sys
from pathlib import Path

import numpy as np
from scipy import stats

REFERENCE = Path("data/reference_scores.json")
RECENT = Path("data/recent_scores.json")
P_THRESHOLD = 0.05


def load_scores(path):
    if not path.exists():
        print(f"missing file: {path}")
        sys.exit(2)
    return np.array(json.loads(path.read_text()))


def main():
    ref = load_scores(REFERENCE)
    recent = load_scores(RECENT)
    ks_stat, p_value = stats.ks_2samp(ref, recent)
    print(f"reference: n={len(ref)}  mean={ref.mean():.3f}  q95={np.quantile(ref, 0.95):.3f}")
    print(f"recent:    n={len(recent)}  mean={recent.mean():.3f}  q95={np.quantile(recent, 0.95):.3f}")
    print(f"KS statistic={ks_stat:.3f}  p-value={p_value:.4f}")
    if p_value < P_THRESHOLD:
        print("ALERT: score distribution has drifted")
        return 1
    print("OK: no significant drift detected")
    return 0


if __name__ == "__main__":
    sys.exit(main())
EOF
python - <<'EOF'
import json
import numpy as np
rng = np.random.default_rng(0)
json.dump(rng.beta(2, 20, 1000).tolist(), open("data/reference_scores.json", "w"))
json.dump(rng.beta(4, 10, 500).tolist(), open("data/recent_scores.json", "w"))
EOF
python scripts/check_drift.py
echo "exit code: $?"

Suspect n° 1 : le code. Kubernetes garde l'historique des rollouts — la marche arrière est une commande, à répéter à froid pour qu'elle soit un réflexe le jour J. Suspect number 1: the code. Kubernetes keeps the rollout history — reverse gear is one command, to be rehearsed in calm weather so it is a reflex on the day.

kubectl rollout history deployment/fraud-api
kubectl rollout undo deployment/fraud-api
kubectl rollout status deployment/fraud-api
kubectl get deployment fraud-api -o wide

Suspect n° 2 : le modèle. Le geste du bloc 5 : repointer l'alias @production du registry vers la version précédente — aucun rebuild, aucun manifest. Un rollout restart force les pods à recharger models:/fraud-detector@production. Suspect number 2: the model. Block 5's move: repoint the registry's @production alias to the previous version — no rebuild, no manifest. A rollout restart forces the pods to reload models:/fraud-detector@production.

python - <<'EOF'
from mlflow import MlflowClient

client = MlflowClient(tracking_uri="http://localhost:5000")
current = client.get_model_version_by_alias("fraud-detector", "production")
print("in production before:", current.version)
previous = str(int(current.version) - 1)
client.set_registered_model_alias("fraud-detector", "production", previous)
after = client.get_model_version_by_alias("fraud-detector", "production")
print("in production after:", after.version)
EOF
kubectl rollout restart deployment/fraud-api
kubectl rollout status deployment/fraud-api