Exploring Local AI Models and RAG: Our Journey with Ollama and LM Studio

By Peet van der Westhuizen

Over the past few months, our innovation team has been on a mission to explore the practical applications of local AI models. With AI evolving at breakneck speed, the question we kept asking ourselves was: how can we harness its power in a way that’s efficient, cost-effective, and adaptable to our existing tech stack? This led us down the path of working with Ollama and LM Studio, running smaller language models locally, and diving deep into the potential of Retrieval-Augmented Generation (RAG).

Why Local Models?

As a software development company, we value control, privacy, and performance. Cloud-based LLMs are powerful, but they come with dependencies on external services, ongoing costs, and potential latency issues. By running smaller models locally, we gain:

Data Privacy: Sensitive client data never leaves our environment.
Cost Efficiency: Reducing reliance on expensive API calls.
Performance: Faster response times with well-optimized local resources.

The Tools: Ollama and LM Studio

Ollama provided a streamlined way to run open-weight models on our local machines. It’s easy to set up, flexible, and supports a range of optimized models. LM Studio added a user-friendly interface for managing and experimenting with those models, making iteration and testing much faster.

We tested models like Deepseek-R1, various versions of Mistral (including 7b-instruct-v0.2 with different quantization levels), Mixtral:8x7b, Llama3 (ranging from 8b to 70b with varying quantizations), Qwen2 (7b and 32b), and even specialized models like Lt-Llama-2-13b-instruct-hf. The standout for our coding needs so far has been the Qwen2.5-7B-Instruct-1M-GGUF model, which we found on Hugging Face.

We evaluated them on performance, response quality, and resource usage. One of our key considerations was balancing model size and capability—smaller models often meant lower inference costs but required more fine-tuning to reach the quality we needed.

Applying RAG

RAG (Retrieval-Augmented Generation) turned out to be a game-changer. Instead of relying solely on a model’s training data, RAG enables the model to fetch real-time, contextually relevant information from external data sources. We applied RAG in scenarios like:

Knowledge Base Search: Providing up-to-date, accurate answers based on internal documentation.
Customer Support: Pulling from product manuals and support articles for quick, informed responses.
Governance Reporting: Accessing current compliance data to generate precise reports.

Challenges and Lessons Learned

One of our biggest hurdles was managing resource intensiveness. Running LLMs locally requires solid hardware—GPUs make a massive difference. We also had to optimize prompts and memory usage to prevent slowdowns and ensure models performed well under load.

Another lesson was around model choice. Smaller models often needed significant RAG support to provide accurate, nuanced answers. Fine-tuning was sometimes necessary, adding complexity but also increasing precision.

What’s Next?

As we continue to refine our approach, we’re excited about the potential of local AI in building powerful, privacy-focused applications. We’re exploring more advanced RAG implementations and fine-tuning techniques to push performance even further.

If you’re on a similar journey or have insights on running local LLMs, we’d love to hear from you. Let’s shape the future of AI together.

Peet van der Westhuizen