Qwen2-VL deployment instances

dependents

git+https://git.kaiyuancloud.cn/yumoqing/apppublic git+https://git.kaiyuancloud.cn/yumoqing/ahserver

preinstallation

first, create a new python virtual env

python3 -m venv ~/vl

create two shell scripts named vlpy and vlpip:

#!/usr/bin/bash
~/vl/bin/python $*

and

#!/usr/bin/bash
~/vl/bin/pip $*

and copy them to the bin under you $HOME folder, and chmod +x to them

mv vlpip vlpy ~/bin
chmod +x ~/bin/vl*

isntallation

follow instuctions from Qwen2-VL, remember to change pip to vlpip

do the following

vlpip install git+https://github.com/huggingface/transformers@21fac7abba2a37fae86106f87fcf9974fd1e3830 accelerate
pip install qwen-vl-utils[decord]

git clone https://git.kaiyauncloud.cn/yumoqing/qwenvl
cd qwenvl/script
sudo isntall.sh

Change model or http port

there is a config.json file under qwenvl folder, change the "modelname" and "port" value to suite your requirements

model to use

• Qwen/Qwen2-VL-7B-Instruct-AWQ
• Qwen/Qwen2-VL-7B-Instruct
• Qwen/Qwen2-VL-7B-Instruct-GPTQ-Int4
• Qwen/Qwen2-VL-7B-Instruct-GPTQ-Int8
• Qwen/Qwen2-VL-72B-Instruct
• Qwen/Qwen2-VL-72B-Instruct-AWQ
• Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4
• Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int8
• Qwen/Qwen2-VL-2B-Instruct
• Qwen/Qwen2-VL-2B-Instruct-AWQ
• Qwen/Qwen2-VL-2B-Instruct-GPTQ-Int4
• Qwen/Qwen2-VL-2B-Instruct-GPTQ-Int8