Running Large Language Models locally used to feel like a pipe dream for those of us with modest hardware. I have an old PC and hardware specs are: Ryzen 5 3600 CPU, 16GB of system RAM, and a GTX 1650 Super with only 4GB of VRAM. I’ve always felt that these ar far too limited for running a serious AI model. Every guide online seemed to imply that if I didn’t have a high-end workstation card or a dual-GPU setup, I’m locked out of the local AI loop.

Then I came across Ollama. It is an open-source framework built specifically to democratise local AI, effectively splitting model weights between your graphics card and system memory.

But let’s navigate the VRAM Bottleneck first!

To understand why this works(running LLM models in low spec hardware I mean), you have to look at how LLMs handle memory. Normally, an AI model wants to sit entirely inside your GPU’s ultra-fast Video RAM (VRAM). A small 3-billion parameter model (like Llama 3.2 3B) takes up roughly 2.5GB of space, meaning it fits entirely within a 4GB frame buffer and runs at blazing-fast speeds.

But what happens when you want to step up to a much smarter 7B or 9B model?

This is where Ollama’s partial offloading architecture shines. Instead of throwing an “Out of Memory” error and crashing, Ollama intelligently loads as many layers of the model as it can fit into your 4GB of VRAM, and then distributes the remaining layers across your standard system RAM.

The obvious trade-off here is speed. Because your system RAM and CPU processing speeds are significantly slower than dedicated graphics hardware, shifting the bottleneck to system memory drops your tokens-per-second generation speed. You will notice a steady, deliberate typing pace rather than an instant explosion of text. However, because Linux manages system resources tightly, this hybrid split results in zero desktop lag. The rest of your operating system remains completely fluid and responsive.

And the the Secret Sauce: Quantisation

You might wonder how a 9-billion parameter model which would normally require nearly 18GB of uncompressed VRAM can fit into a machine with 16GB of total system RAM. The answer is quantisation.

By default, models are downloaded in a 4-bit quantised format (usually denoted as Q4_K_M). Quantisation compresses the numerical weights of the model from bulky 16-bit floating points down to lightweight 4-bit integers. This shrinks the footprint of a model like Google’s Gemma 2 9B down to roughly 5.5GB, allowing it to easily share your 16GB of system RAM alongside your web browser and text editor without choking your swap space.

Now let’s configure a local LLM model as a test.

First step: getting the environment ready (I’m using Ubuntu)

Setting this up on Ubuntu is incredibly clean. There are no complex Python dependency trees or broken environment paths to debug. Open your terminal and pull down the official installation script:

bash
curl -fsSL https://ollama.com | sh

Once the script completes, you need to ensure that Ollama can actually see your graphics hardware to utilise CUDA acceleration. Verify your NVIDIA driver state using the system management interface:

bash
nvidia-smi

If the command correctly prints out your driver version, GPU temperature, and your 4GB VRAM limit, Ollama will automatically hook into your graphics card on launch without any further configuration.

Deploying the Models

With the background daemon running, you can pull a high-quality model straight to your machine. For writing or researching text, Google’s Gemma 2 9B punches significantly above its weight class in vocabulary nuance. To fetch and run it, use:

bash
ollama run gemma2:9b

The tool will download the compressed layers, automatically calculate how many layers to allocate to your GTX 1650 Super, and open an interactive prompt indicated by a triple bracket (>>>)

Because we are intentionally pushing our hardware to its limit, keeping clean memory habits is vital. You cannot run standard Linux commands while actively chatting with the model. When you are finished experimenting, you must exit the chat environment first by typing:

bash
>>> /exit

Once you press Enter, you will be returned to your standard Ubuntu command line prompt (user@ubuntu:~$). Now, you can (and you should) explicitly flush the heavy model weights out of your active system memory to keep your desktop responsive:

bash
ollama stop gemma2:9b

If you ever want to verify that your system memory and GPU VRAM have been completely freed up, you can audit your active background processes with:

bash
ollama ps

If the command returns an empty list, the model has been successfully purged from your hardware resources, leaving your machine completely unencumbered for gaming, browsing, or compiling code.

And that’s it! Now you can follow those steps and run whatever AI model you’d like on your local machine for free! Well, not totally free of course, because you still need to pay your own electricity costs.

Please do your own research and explore what other pre-trained models are out there that you can harness for your work or studies.