第 10 章 企业内网集群部署方案

10.1 单机/多机分布式算力规划、成本测算

10.1.1 算力规划

DeepSeek-V3 671B 部署需求

配置项	最低配置	推荐配置
GPU 型号	A100 80GB	H100 80GB
GPU 数量	16	32
单卡显存	80GB	80GB
节点数量	2 (8卡/节点)	4 (8卡/节点)
网络带宽	100Gbps InfiniBand	400Gbps InfiniBand

不同模型规模部署方案

模型	单卡显存需求	推荐GPU	节点数	预估成本
DeepSeek-V3 671B	80GB+	H100	16	1200万
DeepSeek-V3 33B	40GB+	A100	4	300万
DeepSeek-V3 7B	24GB+	A10	1	50万
DeepSeek-R1 7B	24GB+	A10	1	50万

10.1.2 成本测算

硬件成本

class CostCalculator:
    def __init__(self):
        self.gpu_prices = {
            "H100": 750000,
            "A100": 300000,
            "A10": 50000
        }
        
        self.node_cost = 100000
        self.network_cost = 50000
        self.storage_cost_per_tb = 5000
    
    def calculate_hardware_cost(self, gpu_type, gpus_per_node, num_nodes):
        gpu_total = self.gpu_prices[gpu_type] * gpus_per_node * num_nodes
        node_total = self.node_cost * num_nodes
        network_total = self.network_cost * num_nodes
        
        return {
            "gpu_cost": f"{gpu_total / 10000:.2f}万",
            "node_cost": f"{node_total / 10000:.2f}万",
            "network_cost": f"{network_total / 10000:.2f}万",
            "total": f"{(gpu_total + node_total + network_total) / 10000:.2f}万"
        }
    
    def calculate_operational_cost(self, gpu_type, gpus_per_node, num_nodes, hours_per_day=24):
        power_consumption = {
            "H100": 700,
            "A100": 400,
            "A10": 150
        }
        
        gpu_power = power_consumption[gpu_type] * gpus_per_node * num_nodes
        node_power = 500 * num_nodes
        
        total_power_watts = gpu_power + node_power
        total_power_kw = total_power_watts / 1000
        
        electricity_cost_per_kwh = 0.8
        daily_cost = total_power_kw * hours_per_day * electricity_cost_per_kwh
        annual_cost = daily_cost * 365
        
        return {
            "daily_cost": f"{daily_cost:.2f}元",
            "annual_cost": f"{annual_cost / 10000:.2f}万"
        }

成本测算示例

calculator = CostCalculator()

hardware = calculator.calculate_hardware_cost("H100", 8, 2)
print("硬件成本:", hardware)

operational = calculator.calculate_operational_cost("H100", 8, 2)
print("运维成本:", operational)

10.1.3 存储需求

模型	权重大小 (FP16)	权重大小 (FP8)	KV Cache (2048 tokens)
671B	1.3TB	650GB	100GB/node
33B	66GB	33GB	10GB/node
7B	14GB	7GB	2GB/node

10.2 Docker容器化打包、K8s集群编排完整脚本

10.2.1 Dockerfile

FROM nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y     python3.10     python3.10-dev     python3-pip     git     wget     && rm -rf /var/lib/apt/lists/*

RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 1

RUN pip3 install --upgrade pip setuptools wheel

COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt

RUN pip3 install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121

WORKDIR /app

COPY . .

RUN mkdir -p /data/models /data/logs /data/cache

ENV PYTHONPATH=/app:$PYTHONPATH

CMD ["python3", "inference/generate.py"]

10.2.2 requirements.txt

transformers==4.35.2
torch==2.1.0
accelerate==0.24.1
datasets==2.14.6
sentencepiece==0.1.99
protobuf==4.24.4
numpy==1.24.3
scipy==1.11.3
tqdm==4.65.0

10.2.3 Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deepseek-v3
  namespace: ai
spec:
  replicas: 1
  selector:
    matchLabels:
      app: deepseek-v3
  template:
    metadata:
      labels:
        app: deepseek-v3
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: deepseek-v3
        image: deepseek-v3:latest
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 8
            memory: 256Gi
            cpu: 64
          requests:
            nvidia.com/gpu: 8
            memory: 256Gi
            cpu: 64
        volumeMounts:
        - name: model-storage
          mountPath: /data/models
        - name: log-storage
          mountPath: /data/logs
        env:
        - name: MODEL_PATH
          value: "/data/models/deepseek-v3-671b"
        - name: TENSOR_PARALLEL_SIZE
          value: "8"
        - name: PORT
          value: "8000"
      volumes:
      - name: model-storage
        persistentVolumeClaim:
          claimName: model-pvc
      - name: log-storage
        persistentVolumeClaim:
          claimName: log-pvc

10.2.4 Kubernetes Service

apiVersion: v1
kind: Service
metadata:
  name: deepseek-v3-service
  namespace: ai
spec:
  type: NodePort
  selector:
    app: deepseek-v3
  ports:
  - port: 8000
    targetPort: 8000
    nodePort: 30007

10.2.5 PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-pvc
  namespace: ai
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Ti

10.3 离线内网权重分发、无互联网环境部署

10.3.1 离线部署策略

1. 权重打包：在有网络环境下载并打包模型权重
2. 镜像离线导入：将 Docker 镜像导出为 tar 文件
3. 内网仓库部署：搭建内网 Docker 仓库
4. 配置同步：同步配置文件和脚本

10.3.2 离线部署脚本

import os
import subprocess
import tarfile

class OfflineDeployer:
    def __init__(self, model_path, target_hosts, target_path="/data/models"):
        self.model_path = model_path
        self.target_hosts = target_hosts
        self.target_path = target_path
    
    def package_model(self, output_path="model.tar.gz"):
        print(f"Packaging model from {self.model_path}...")
        
        with tarfile.open(output_path, "w:gz") as tar:
            tar.add(self.model_path, arcname=os.path.basename(self.model_path))
        
        print(f"Model packaged to {output_path}")
        return output_path
    
    def package_docker_image(self, image_name, output_path="image.tar"):
        print(f"Saving Docker image {image_name}...")
        
        subprocess.run([
            "docker", "save", "-o", output_path, image_name
        ], check=True)
        
        print(f"Image saved to {output_path}")
        return output_path
    
    def distribute_file(self, local_path, remote_path):
        for host in self.target_hosts:
            print(f"Copying {local_path} to {host}:{remote_path}...")
            
            subprocess.run([
                "scp", local_path, f"{host}:{remote_path}"
            ], check=True)
            
            print(f"Copied to {host}")
    
    def deploy_model(self):
        model_package = self.package_model()
        
        self.distribute_file(model_package, self.target_path)
        
        for host in self.target_hosts:
            subprocess.run([
                "ssh", host,
                f"mkdir -p {self.target_path} && tar -xzf {self.target_path}/model.tar.gz -C {self.target_path}"
            ], check=True)
    
    def deploy_docker_image(self, image_name, registry_url):
        image_package = self.package_docker_image(image_name)
        
        self.distribute_file(image_package, "/tmp/")
        
        for host in self.target_hosts:
            subprocess.run([
                "ssh", host,
                f"docker load -i /tmp/image.tar && docker tag {image_name} {registry_url}/{image_name} && docker push {registry_url}/{image_name}"
            ], check=True)

10.3.3 离线部署步骤

deployer = OfflineDeployer(
    model_path="/data/models/deepseek-v3-671b",
    target_hosts=["node1", "node2", "node3", "node4"]
)

deployer.deploy_model()

deployer.deploy_docker_image(
    image_name="deepseek-v3:latest",
    registry_url="10.0.0.100:5000"
)

10.4 动态扩缩容、弹性算力调度配置

10.4.1 Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: deepseek-v3-hpa
  namespace: ai
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deepseek-v3
  minReplicas: 1
  maxReplicas: 8
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

10.4.2 自定义指标扩缩容

from kubernetes import client, config
import time

class CustomScaler:
    def __init__(self):
        config.load_incluster_config()
        self.apps_api = client.AppsV1Api()
    
    def scale_deployment(self, name, namespace, replicas):
        deployment = self.apps_api.read_namespaced_deployment(name, namespace)
        deployment.spec.replicas = replicas
        self.apps_api.patch_namespaced_deployment(name, namespace, deployment)
        
        print(f"Scaled {name} to {replicas} replicas")
    
    def scale_based_on_load(self, name, namespace, current_qps, threshold_low=100, threshold_high=1000):
        deployment = self.apps_api.read_namespaced_deployment(name, namespace)
        current_replicas = deployment.spec.replicas
        
        if current_qps > threshold_high and current_replicas < 8:
            self.scale_deployment(name, namespace, current_replicas + 1)
        elif current_qps < threshold_low and current_replicas > 1:
            self.scale_deployment(name, namespace, current_replicas - 1)

10.4.3 弹性算力调度策略

class ElasticScheduler:
    def __init__(self):
        self.scaler = CustomScaler()
        self.min_replicas = 1
        self.max_replicas = 8
    
    def run(self, name, namespace):
        while True:
            current_qps = self.get_current_qps(name, namespace)
            
            self.scaler.scale_based_on_load(name, namespace, current_qps)
            
            time.sleep(60)
    
    def get_current_qps(self, name, namespace):
        response = requests.get(f"http://{name}-service.{namespace}.svc.cluster.local:8000/metrics")
        metrics = response.text
        
        for line in metrics.split("
"):
            if "http_requests_total" in line:
                return float(line.split()[-1])
        
        return 0

本章小结：

本章详细介绍了企业内网集群部署方案，包括算力规划、成本测算、Docker容器化打包、K8s集群编排、离线部署和动态扩缩容。这些技术为企业级大模型部署提供了完整的基础设施解决方案。
更多资讯:lxb20110121