前言

在腾讯云上囤了个算力平台，尝试本地化部署deepSeek

前期验证

首先验证以下是否支持NVIDIA框架，终端命令输入

nvidia-smi

输出如下

ubuntu@VM-0-17-ubuntu:~$ nvidia-smi
Fri Feb 28 12:03:57 2025       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:08.0 Off |                    0 |
| N/A   35C    P0    23W / 300W |      0MiB / 32768MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

检查是否安装cuda服务，一般云服务器都没有安装这项服务，终端输入

nvcc --version

返回如下：

Command 'nvcc' not found, but can be installed with:

sudo apt install nvidia-cuda-toolkit

安装NVIDIA CUDA Toolkit

终端输入以下命令

sudo apt update
sudo apt install nvidia-cuda-toolkit

接下来只需要等待即可，安装完成后输入

nvcc --version

看到输出为

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

说明安装成功

安装python

这里推荐安装高版本的python，我安装的是python3.10
终端输入

sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update

使用PPA进行安装，安装过程中需要根据提示输入y或者enter，按照提示做即可，完成上述步骤后，输入

sudo apt install python3.10

验证安装是否成功，输入

python3.10 --version

输出

Python 3.10.16

表示安装成功

下载deepseek模型

安装 ModelScope

ModelScope 是一个模型中心，用于下载和管理模型
终端输入

pip install modelscope

安装过程中可能会出现警告信息

ubuntu@VM-0-17-ubuntu:~$   pip install modelscope
Looking in indexes: http://mirrors.tencentyun.com/pypi/simple
Collecting modelscope
  Downloading http://mirrors.tencentyun.com/pypi/packages/42/50/f6f8fce132552f201ba2eb30524b2fd54327bded7afae2a0cd49ace230d3/modelscope-1.23.1-py3-none-any.whl (5.9 MB)
     |████████████████████████████████| 5.9 MB 12.4 MB/s 
Collecting urllib3>=1.26
  Downloading http://mirrors.tencentyun.com/pypi/packages/ce/d9/5f4c13cecde62396b0d3fe530a50ccea91e7dfc1ccf0e09c228841bb5ba8/urllib3-2.2.3-py3-none-any.whl (126 kB)
     |████████████████████████████████| 126 kB 12.9 MB/s 
Collecting tqdm>=4.64.0
  Downloading http://mirrors.tencentyun.com/pypi/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl (78 kB)
     |████████████████████████████████| 78 kB 14.1 MB/s 
Collecting requests>=2.25
  Downloading http://mirrors.tencentyun.com/pypi/packages/f9/9b/335f9764261e915ed497fcdeb11df5dfd6f7bf257d4a6a2a686d80da4d54/requests-2.32.3-py3-none-any.whl (64 kB)
     |████████████████████████████████| 64 kB 12.5 MB/s 
Collecting charset-normalizer<4,>=2
  Downloading http://mirrors.tencentyun.com/pypi/packages/1e/0d/38ef4ae41e9248d63fc4998d933cae22473b1b2ac4122cf908d0f5eb32aa/charset_normalizer-3.4.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)
     |████████████████████████████████| 144 kB 10.8 MB/s 
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.25->modelscope) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.25->modelscope) (2019.11.28)
Installing collected packages: urllib3, tqdm, charset-normalizer, requests, modelscope
  WARNING: The script tqdm is installed in '/home/ubuntu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script normalizer is installed in '/home/ubuntu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script modelscope is installed in '/home/ubuntu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed charset-normalizer-3.4.1 modelscope-1.23.1 requests-2.32.3 tqdm-4.67.1 urllib3-2.2.3

我们需要将路径添加到PATH中

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

接下来下载模型,将命令中的/path/to/save/model替换成你想保存的地方

  modelscope download --model deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --local_dir /path/to/save/model

接下来就会开始下载，下载比较慢，可以喝杯咖啡再回来
安装完成之后安转Vllm，这是一个高效的大模型语言推理库，支持快速本地化部署模型，首先我们需要创建一个虚拟环境，防止包冲突

python -m venv venv

如果提示

The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt install python3.8-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/home/ubuntu/venv/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']

则按提示进行安装

sudo apt install python3.8-venv

激活虚拟环境

source venv/bin/activate

激活后会有提示，前面会有venv字眼

(venv) ubuntu@VM-0-17-ubuntu:~$ 

接着安装VLLM

pip install vllm

这里我遇到了一个错误
ERROR: Command errored out with exit status 1:
command: /home/ubuntu/venv/bin/python /tmp/tmpsl0s1a8l prepare_metadata_for_build_wheel /tmp/tmp7ifh43sp
cwd: /tmp/pip-install-ixjsu8of/tokenizers
Complete output (6 lines):
Cargo, the Rust package manager, is not installed or is not on PATH.
This package requires Rust and Cargo to compile extensions. Install it through
the system’s package manager or via https://rustup.rs/
Checking for Rust toolchain…
----------------------------------------
ERROR: Command errored out with exit status 1: /home/ubuntu/venv/bin/python /tmp/tmpsl0s1a8l prepare_metadata_for_build_wheel /tmp/tmp7ifh43sp Check the logs for full command output.
这里是因为缺少缺少 Rust 编译器（Cargo），导致 tokenizers 依赖无法编译，直接安装
curl --proto ‘=https’ --tlsv1.2 -sSf https://sh.rustup.rs | sh
执行后关闭终端再打开
source $HOME/.cargo/env

安转完成后。使用

CUDA_VISIBLE_DEVICES=0 vllm serve /path/to/save/model --port 8102 --max-model-len 16384

就可以开始启动推理服务啦