Setting Up a Local LLM
This is a quick and dirty guide to setting up a local LLM. You’ll see how to run the Qwen2.5-Code-3B-Instruct model on your local machine using vllm. You’ll then setup the CodeCompanion plugin in NeoVim for interacting with the model directly from your editor. vllm Installation and Server Setup Step one is to install the vllm CLI utility: python -m venv local-llm source local-llm/bin/activate pip install vllm The vllm tool will download and standup a local server for the model. Take note of what hardware you have available (RAM, CPU, GPU/VRAM) and then browse models at hugginface.co. This example kicks off a Qwen2.5-Code-3B-Instruct model server: ...