GitHub - akx/ggify: Tool to download models from Huggingface Hub and convert them to GGML/GGUF for llama.cpp

akx / ggify Public

Tool to download models from Huggingface Hub and convert them to GGML/GGUF for llama.cpp

Notifications You must be signed in to change notification settings

Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 19 Commits
.github/workflows		.github/workflows
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
LICENSE		LICENSE
README.md		README.md
ggify.py		ggify.py
pyproject.toml		pyproject.toml

A small tool that downloads models from the Huggingface Hub and converts them into GGML for use with llama.cpp.

Download and compile llama.cpp.
Set up a virtualenv using the requirements from llama.cpp.
Install this package in that virtualenv (e.g. pip install -e .).
Run e.g. python ggify.py databricks/dolly-v2-12b (nb.: I haven't tried with that particular repo)
You'll end up with GGML models under models/....

You can set --llama-cpp-dir (or the LLAMA_CPP_DIR environment variable) to point to the directory where you've compiled llama.cpp.