LLM Engineering & Open Source Resource Index

Welcome to my developer blog. This site serves as a quick reference collection of tools, libraries, serving frameworks, and notes for working with open-weights foundation models.

Note: Interactive compute environments and notebooks are accessible via the Workspace link in the sidebar.

Models & Foundation Hubs

A quick breakdown of open-weight foundation models and local execution runtimes:

• Hugging Face Hub Platform
  The primary distribution platform for open weights, datasets, and Space demos.
• Ollama CLI / Go
  Cross-platform runner to pull and execute models (Llama 3.1, Gemma 2, Mistral) locally on CPU/GPU.
• Llama 3 & 3.1 Foundation
  Meta's foundation architecture with 128k token context windows and multimodal capabilities.

Orchestration & RAG Frameworks

Libraries used to assemble multi-step reasoning agents, vector database retrieval, and structured generation:

• LangChain Python/TS — Modular abstraction layer for agentic memory, prompt templates, and tool-calling chains.
• LlamaIndex Python/TS — Data framework optimized for Retrieval-Augmented Generation (RAG) and index query pipelines.
• DSPy Python — Stanford's framework for compiling declarative language model modules instead of manual prompt engineering.

Serving Engines & Inference

High-performance backend runtimes designed for production throughput and memory-efficient batching:

• vLLM — High-throughput serving engine utilizing PagedAttention for non-contiguous KV cache management.
• llama.cpp — Lightweight C/C++ inference implementation supporting GGUF quantization on CPU and Metal/CUDA backends.
• PyTorch — Core deep learning library for custom model fine-tuning (LoRA/QLoRA) and execution.

Quick Reference Cheat Sheet

Common shell commands for local model testing and inference server startup:

# Quick Python example with vLLM OpenAI-compatible endpoint
import openai

client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="token")
response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Explain PagedAttention in two sentences."}]
)
print(response.choices[0].message.content)