{ "cells": [ { "cell_type": "markdown", "id": "9e7bd27c", "metadata": { }, "source": [ "# Free 32 GB-Class AI Server on Kaggle -- Ollama + Qwen3.8 27B + MTP\n", "\n", "**Created by Kanta Raj Luitel (Susant Luitel)**\n", "\n", "## Socials:\n", "[![Facebook](https://img.shields.io/badge/Facebook-%231877F2.svg?logo=Facebook\u0026logoColor=white)](https://facebook.com/Kantaraj.Luitel) [![Instagram](https://img.shields.io/badge/Instagram-%23E4405F.svg?logo=Instagram\u0026logoColor=white)](https://instagram.com/susantgamerz) [![LinkedIn](https://img.shields.io/badge/LinkedIn-%230077B5.svg?logo=linkedin\u0026logoColor=white)](https://linkedin.com/in/kantaraj-luitel) [![Pinterest](https://img.shields.io/badge/Pinterest-%23E60023.svg?logo=Pinterest\u0026logoColor=white)](https://pinterest.com/susantluitel) [![Reddit](https://img.shields.io/badge/Reddit-%23FF4500.svg?logo=Reddit\u0026logoColor=white)](https://reddit.com/user/Successful-Twist2608) [![TikTok](https://img.shields.io/badge/TikTok-%23000000.svg?logo=TikTok\u0026logoColor=white)](https://tiktok.com/@vortexeditz34) [![X](https://img.shields.io/badge/X-black.svg?logo=X\u0026logoColor=white)](https://x.com/Susantedit) [![Codepen](https://img.shields.io/badge/Codepen-000000?logo=codepen\u0026logoColor=white)](https://codepen.io/susant-gamerz) [![GitHub](https://img.shields.io/badge/GitHub-181717.svg?logo=github\u0026logoColor=white)](https://github.com/susantedit) [![WhatsApp](https://img.shields.io/badge/WhatsApp-25D366?logo=whatsapp\u0026logoColor=white)](https://wa.me/9779708838261) [![Email](https://img.shields.io/badge/Email-D14836?logo=gmail\u0026logoColor=white)](mailto:susantedit@gmail.com)\n", "\n", "---\n", "\n", "This notebook turns a free **Kaggle GPU session** into a temporary Ollama server for running a large GGUF model across **2x NVIDIA Tesla T4 GPUs**.\\n", "\\n", "### What this notebook sets up\\n", "\\n", "- Ollama on Kaggle\\n", "- Qwen3.8 27B Uncensored Q4_K_M from Hugging Face\\n", "- Model split across both T4 GPUs\\n", "- MTP (speculative decoding) configuration\\n", "- 64K context\\n", "- Optional Cloudflare Quick Tunnel for a temporary public API\\n", "- A simple local/public chat helper and benchmark\\n", "\\n", "### Before you start\\n", "\\n", "In Kaggle, open **Settings -\u003e Accelerator** and select **GPU T4 x2** when available. Also enable **Internet**.\\n", "\\n", "\u003e **VRAM note:** 2x T4 provides 30,720 MiB (about 30 GiB) of physical VRAM. This is a 32 GB-class setup; the notebook verifies the actual GPU memory with \nvidia-smi.\\n", "\\n", "\u003e **Storage note:** the Kaggle working disk is limited. This notebook downloads the model directly into Ollama\u0027s model store and avoids the manual GGUF -\u003e ollama create route, which would require a second full copy of the ~17 GB model.\\n" ] }, { "cell_type": "markdown", "id": "f1eda057", "metadata": { }, "source": [ "## 1. Check the GPU\n", "\n", "Run this first. You should see **two Tesla T4 GPUs**." ] }, { "cell_type": "code", "execution_count": null, "id": "15bc7735", "metadata": { }, "outputs": [ ], "source": [ "!nvidia-smi" ] }, { "cell_type": "markdown", "id": "823bef71", "metadata": { }, "source": [ "## 2. Install Ollama and the required system package\n", "\n", "Run this cell once per fresh Kaggle session." ] }, { "cell_type": "code", "execution_count": null, "id": "5e7984c8", "metadata": { }, "outputs": [ ], "source": [ "# Install Ollama\n", "!apt-get update -qq\n", "!apt-get install -y -qq zstd\n", "!curl -fsSL https://ollama.com/install.sh | sh\n", "!which ollama\n", "!ollama --version\n" ] }, { "cell_type": "markdown", "id": "58e31aa1", "metadata": { }, "source": [ "## 3. Start Ollama with a Kaggle-friendly configuration\n", "\n", "The model will be stored in `/kaggle/working/ollama-models` and the server will keep the model loaded after inference." ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "### 64K Context Configuration\n", "This demo uses a **65,536-token (64K) context window**. With 2× Tesla T4 GPUs, the model is distributed across both GPUs. The notebook will automatically install/locate Ollama if the executable is missing.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "e677e44c", "metadata": { }, "outputs": [ ], "source": [ "import os\n", "import shutil\n", "import subprocess\n", "import time\n", "import urllib.request\n", "from pathlib import Path\n", "\n", "OLLAMA_URL = \"http://127.0.0.1:11434\"\n", "OLLAMA_MODELS = Path(\"/kaggle/working/ollama-models\")\n", "SOURCE_MODEL = \"hf.co/JonathanColetti/Qwen3.8-27B-Uncensored-GGUF:Q4_K_M\"\n", "MTP_MODEL = \"qwen3.8-27b-uncensored-mtp\"\n", "\n", "# 64K context: larger context window for the live demo.\n", "NUM_CTX = 65536\n", "DRAFT_TOKENS = 2\n", "\n", "OLLAMA_MODELS.mkdir(parents=True, exist_ok=True)\n", "os.environ[\"OLLAMA_MODELS\"] = str(OLLAMA_MODELS)\n", "\n", "# Make the notebook resilient if Ollama is missing after a fresh/restarted Kaggle session.\n", "OLLAMA_BIN = shutil.which(\"ollama\")\n", "if not OLLAMA_BIN:\n", " print(\"Ollama executable not found. Installing it now...\")\n", " subprocess.run(\n", " [\"bash\", \"-lc\", \"curl -fsSL https://ollama.com/install.sh | sh\"],\n", " check=True,\n", " )\n", " OLLAMA_BIN = shutil.which(\"ollama\")\n", "\n", "if not OLLAMA_BIN:\n", " raise RuntimeError(\n", " \"Ollama installation completed but the executable could not be found.\"\n", " )\n", "\n", "OLLAMA_ENV = os.environ.copy()\n", "OLLAMA_ENV.update({\n", " \"OLLAMA_HOST\": \"127.0.0.1:11434\",\n", " \"OLLAMA_MODELS\": str(OLLAMA_MODELS),\n", " \"OLLAMA_KEEP_ALIVE\": \"-1\",\n", " \"OLLAMA_FLASH_ATTENTION\": \"1\",\n", " \"OLLAMA_KV_CACHE_TYPE\": \"q8_0\",\n", " \"OLLAMA_CONTEXT_LENGTH\": str(NUM_CTX),\n", " \"OLLAMA_NUM_PARALLEL\": \"1\",\n", " \"OLLAMA_MAX_LOADED_MODELS\": \"1\",\n", "})\n", "\n", "def ollama_ready():\n", " try:\n", " urllib.request.urlopen(f\"{OLLAMA_URL}/api/tags\", timeout=2).close()\n", " return True\n", " except Exception:\n", " return False\n", "\n", "if not ollama_ready():\n", " server_log = open(\"/tmp/ollama-server.log\", \"a\")\n", " ollama_server_process = subprocess.Popen(\n", " [OLLAMA_BIN, \"serve\"],\n", " env=OLLAMA_ENV,\n", " stdout=server_log,\n", " stderr=subprocess.STDOUT,\n", " )\n", "\n", " for _ in range(60):\n", " if ollama_ready():\n", " break\n", " time.sleep(1)\n", " else:\n", " server_log.close()\n", " raise RuntimeError(\n", " \"Ollama did not become ready. Check /tmp/ollama-server.log\"\n", " )\n", "\n", "print(\"✅ Ollama is ready:\", OLLAMA_URL)\n", "print(\"✅ Ollama binary:\", OLLAMA_BIN)\n", "print(\"✅ Model directory:\", OLLAMA_MODELS)\n", "print(\"✅ Context length:\", NUM_CTX)\n" ] }, { "cell_type": "markdown", "id": "8eaed122", "metadata": { }, "source": [ "## 4. Download the 27B GGUF directly into Ollama\n", "\n", "Do **not** manually download the GGUF and then run `ollama create` from the raw file. That route can require another full copy and exhaust Kaggle\u0027s working disk.\n", "\n", "The first pull is attempted normally. The `--insecure` fallback is included for Ollama builds that hit the Hugging Face/Xet error `blocked redirect to a different host`. Prefer a fixed Ollama release when your environment has one that resolves the redirect without this fallback." ] }, { "cell_type": "code", "execution_count": null, "id": "6a3c8ebe", "metadata": { }, "outputs": [ ], "source": [ "import subprocess\n", "\n", "# Ollama 0.34.x may hit a Hugging Face/Xet redirect error in this environment.\n", "# --insecure is used only as a practical workaround for that redirect.\n", "pull_args = [OLLAMA_BIN, \"pull\", SOURCE_MODEL]\n", "\n", "result = subprocess.run(pull_args, env=OLLAMA_ENV)\n", "if result.returncode != 0:\n", " print(\"Standard pull failed; retrying with --insecure...\")\n", " subprocess.run(\n", " [OLLAMA_BIN, \"pull\", \"--insecure\", SOURCE_MODEL],\n", " env=OLLAMA_ENV,\n", " check=True,\n", " )\n", "\n", "print(\"\\n✅ Model is available:\")\n", "subprocess.run([OLLAMA_BIN, \"list\"], env=OLLAMA_ENV, check=True)\n" ] }, { "cell_type": "markdown", "id": "82578b17", "metadata": { }, "source": [ "## 5. Confirm the model is installed" ] }, { "cell_type": "code", "execution_count": null, "id": "d6e60ce3", "metadata": { }, "outputs": [ ], "source": [ "!{OLLAMA_BIN} list" ] }, { "cell_type": "markdown", "id": "d1cda839", "metadata": { }, "source": [ "## 6. Load the base 27B model and verify GPU execution\n", "\n", "This request loads the model. Ollama should report `100% GPU` and both T4s should contain model memory." ] }, { "cell_type": "code", "execution_count": null, "id": "aa73f1fc", "metadata": { }, "outputs": [ ], "source": [ "!{OLLAMA_BIN} run \"hf.co/JonathanColetti/Qwen3.8-27B-Uncensored-GGUF:Q4_K_M\" \"Reply with exactly: Model is working.\"" ] }, { "cell_type": "code", "execution_count": null, "id": "68da8b38", "metadata": { }, "outputs": [ ], "source": [ "!{OLLAMA_BIN} ps" ] }, { "cell_type": "code", "execution_count": null, "id": "6019252d", "metadata": { }, "outputs": [ ], "source": [ "!nvidia-smi" ] }, { "cell_type": "markdown", "id": "b9e93c6c", "metadata": { }, "source": [ "## 7. Create the MTP model\n", "\n", "This creates an Ollama model definition that reuses the already-downloaded base model and sets the MTP parameters used by this notebook." ] }, { "cell_type": "code", "execution_count": null, "id": "8d3be034", "metadata": { }, "outputs": [ ], "source": [ "import json\n", "import urllib.request\n", "\n", "payload = {\n", " \"model\": MTP_MODEL,\n", " \"from\": SOURCE_MODEL,\n", " \"parameters\": {\n", " \"draft_num_predict\": DRAFT_TOKENS,\n", " \"num_ctx\": NUM_CTX,\n", " },\n", " \"stream\": False,\n", "}\n", "request = urllib.request.Request(\n", " f\"{OLLAMA_URL}/api/create\",\n", " data=json.dumps(payload).encode(\"utf-8\"),\n", " headers={\"Content-Type\": \"application/json\"},\n", " method=\"POST\",\n", ")\n", "with urllib.request.urlopen(request, timeout=300) as response:\n", " result = json.loads(response.read())\n", "print(result)" ] }, { "cell_type": "markdown", "id": "0e602d6c", "metadata": { }, "source": [ "## 8. Load and pin the MTP model" ] }, { "cell_type": "code", "execution_count": null, "id": "39c4ffdb", "metadata": { }, "outputs": [ ], "source": [ "import json\n", "import urllib.request\n", "\n", "payload = {\n", " \"model\": MTP_MODEL,\n", " \"prompt\": \"Reply with exactly: MTP is working.\",\n", " \"stream\": False,\n", " \"keep_alive\": -1,\n", " \"options\": {\n", " \"draft_num_predict\": DRAFT_TOKENS,\n", " \"num_ctx\": NUM_CTX,\n", " },\n", "}\n", "request = urllib.request.Request(\n", " f\"{OLLAMA_URL}/api/generate\",\n", " data=json.dumps(payload).encode(\"utf-8\"),\n", " headers={\"Content-Type\": \"application/json\"},\n", " method=\"POST\",\n", ")\n", "with urllib.request.urlopen(request, timeout=600) as response:\n", " result = json.loads(response.read())\n", "print(\"Response:\", result.get(\"response\", \"\").strip())\n", "print(\"MTP model loaded and pinned.\")" ] }, { "cell_type": "code", "execution_count": null, "id": "17c9e4bf", "metadata": { }, "outputs": [ ], "source": [ "!{OLLAMA_BIN} ps" ] }, { "cell_type": "code", "execution_count": null, "id": "a6d38fd7", "metadata": { }, "outputs": [ ], "source": [ "!nvidia-smi" ] }, { "cell_type": "markdown", "id": "ed4a3cd7", "metadata": { }, "source": [ "## 9. Benchmark generation speed\n", "\n", "This is a simple session-level measurement. Results can vary with prompt, model state, and Kaggle hardware conditions." ] }, { "cell_type": "code", "execution_count": null, "id": "d6e56c42", "metadata": { }, "outputs": [ ], "source": [ "import json\n", "import time\n", "import urllib.request\n", "\n", "prompt = \"\"\"Explain in about 200 words why GPUs are useful for running large language models. Cover memory bandwidth, parallel computation, and matrix multiplication.\"\"\"\n", "payload = {\n", " \"model\": MTP_MODEL,\n", " \"prompt\": prompt,\n", " \"think\": False,\n", " \"stream\": False,\n", " \"keep_alive\": -1,\n", " \"options\": {\n", " \"draft_num_predict\": DRAFT_TOKENS,\n", " \"num_ctx\": NUM_CTX,\n", " \"temperature\": 0,\n", " },\n", "}\n", "start = time.perf_counter()\n", "request = urllib.request.Request(\n", " f\"{OLLAMA_URL}/api/generate\",\n", " data=json.dumps(payload).encode(\"utf-8\"),\n", " headers={\"Content-Type\": \"application/json\"},\n", " method=\"POST\",\n", ")\n", "with urllib.request.urlopen(request, timeout=600) as response:\n", " result = json.loads(response.read())\n", "wall_time = time.perf_counter() - start\n", "eval_count = result.get(\"eval_count\", 0)\n", "eval_seconds = result.get(\"eval_duration\", 0) / 1e9\n", "speed = eval_count / eval_seconds if eval_seconds else 0\n", "print(result.get(\"response\", \"\").strip())\n", "print(\"\\n--- Benchmark ---\")\n", "print(f\"Generated tokens: {eval_count}\")\n", "print(f\"Generation time: {eval_seconds:.2f}s\")\n", "print(f\"Wall time: {wall_time:.2f}s\")\n", "print(f\"Generation speed: {speed:.2f} tokens/s\")" ] }, { "cell_type": "markdown", "id": "25640c55", "metadata": { }, "source": [ "# Optional: expose Ollama through a temporary public URL\n", "\n", "A Cloudflare Quick Tunnel gives you a temporary HTTPS endpoint such as `https://xxxxx.trycloudflare.com`.\n", "\n", "**Security:** the quick tunnel has no authentication. Anyone who gets the URL can send requests to your Ollama server while the Kaggle session and tunnel are alive. Do not use this for sensitive data and do not publish the URL in a video description or public post." ] }, { "cell_type": "markdown", "id": "3840303e", "metadata": { }, "source": [ "## 10. Install Cloudflare Tunnel\n", "\n", "Skip this section if you only need the local Ollama API inside Kaggle." ] }, { "cell_type": "code", "execution_count": null, "id": "27c28a79", "metadata": { }, "outputs": [ ], "source": [ "!curl -fL \"https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb\" -o /tmp/cloudflared-linux-amd64.deb\n", "!dpkg -i /tmp/cloudflared-linux-amd64.deb\n", "!cloudflared --version" ] }, { "cell_type": "markdown", "id": "01f73baf", "metadata": { }, "source": [ "## 11. Start the Cloudflare Quick Tunnel" ] }, { "cell_type": "code", "execution_count": null, "id": "3d8a1c39", "metadata": { }, "outputs": [ ], "source": [ "import queue\n", "import re\n", "import subprocess\n", "import threading\n", "import time\n", "\n", "def pipe_process_output(process, output_queue):\n", " for line in iter(process.stdout.readline, \"\"):\n", " output_queue.put(line)\n", "\n", "tunnel_is_running = (\n", " \"cloudflared_process\" in globals()\n", " and cloudflared_process.poll() is None\n", " and \"PUBLIC_OLLAMA_URL\" in globals()\n", " and PUBLIC_OLLAMA_URL\n", ")\n", "\n", "if not tunnel_is_running:\n", " cloudflared_process = subprocess.Popen(\n", " [\n", " \"cloudflared\", \"tunnel\",\n", " \"--no-autoupdate\",\n", " \"--protocol\", \"http2\",\n", " \"--edge-ip-version\", \"4\",\n", " \"--url\", OLLAMA_URL,\n", " \"--http-host-header\", \"localhost:11434\",\n", " ],\n", " stdout=subprocess.PIPE,\n", " stderr=subprocess.STDOUT,\n", " text=True,\n", " bufsize=1,\n", " )\n", " output_queue = queue.Queue()\n", " threading.Thread(\n", " target=pipe_process_output,\n", " args=(cloudflared_process, output_queue),\n", " daemon=True,\n", " ).start()\n", " deadline = time.monotonic() + 45\n", " PUBLIC_OLLAMA_URL = None\n", " while time.monotonic() \u003c deadline:\n", " if cloudflared_process.poll() is not None and output_queue.empty():\n", " break\n", " try:\n", " line = output_queue.get(timeout=1)\n", " except queue.Empty:\n", " continue\n", " match = re.search(r\"https://[a-z0-9-]+\\.trycloudflare\\.com\", line)\n", " if match:\n", " PUBLIC_OLLAMA_URL = match.group(0)\n", " break\n", " if not PUBLIC_OLLAMA_URL:\n", " raise RuntimeError(\"Cloudflare did not produce a public URL. Rerun this cell.\")\n", "\n", "print(\"Public Ollama API:\", PUBLIC_OLLAMA_URL)\n", "print(\"OpenAI-compatible base URL:\", f\"{PUBLIC_OLLAMA_URL}/v1\")\n", "print(\"Model name:\", MTP_MODEL)" ] }, { "cell_type": "markdown", "id": "2b7dc198", "metadata": { }, "source": [ "## 12. Test the Ollama API locally or through the tunnel" ] }, { "cell_type": "code", "execution_count": null, "id": "527c25b9", "metadata": { }, "outputs": [ ], "source": [ "import json\n", "import urllib.request\n", "\n", "def chat(prompt, base_url=OLLAMA_URL, system=None):\n", " messages = []\n", " if system:\n", " messages.append({\"role\": \"system\", \"content\": system})\n", " messages.append({\"role\": \"user\", \"content\": prompt})\n", " payload = {\n", " \"model\": MTP_MODEL,\n", " \"messages\": messages,\n", " \"stream\": False,\n", " \"keep_alive\": -1,\n", " \"options\": {\n", " \"draft_num_predict\": DRAFT_TOKENS,\n", " \"num_ctx\": NUM_CTX,\n", " },\n", " }\n", " request = urllib.request.Request(\n", " f\"{base_url.rstrip(\u0027/\u0027)}/api/chat\",\n", " data=json.dumps(payload).encode(\"utf-8\"),\n", " headers={\"Content-Type\": \"application/json\"},\n", " method=\"POST\",\n", " )\n", " with urllib.request.urlopen(request, timeout=600) as response:\n", " result = json.loads(response.read())\n", " return result[\"message\"][\"content\"]\n", "\n", "print(chat(\"Write a one-line Python hello-world program.\"))" ] }, { "cell_type": "markdown", "id": "3a2d14bc", "metadata": { }, "source": [ "### Test through the public tunnel\n", "\n", "Run this after the Cloudflare tunnel cell prints `PUBLIC_OLLAMA_URL`." ] }, { "cell_type": "code", "execution_count": null, "id": "6815e00f", "metadata": { }, "outputs": [ ], "source": [ "print(chat(\"Hello from the Cloudflare tunnel.\", base_url=PUBLIC_OLLAMA_URL))" ] }, { "cell_type": "markdown", "id": "2432482d", "metadata": { }, "source": [ "## Optional: OpenAI-compatible client settings\n", "\n", "Use the printed URL with `/v1` as the base URL in OpenAI-compatible tools.\n", "\n", "- **Base URL:** `https://YOUR-TUNNEL.trycloudflare.com/v1`\n", "- **Model:** `qwen3.8-27b-uncensored-mtp`\n", "- **API key:** a placeholder value accepted by your client\n", "\n", "The tunnel URL is temporary and normally disappears when the Kaggle session or tunnel stops." ] }, { "cell_type": "markdown", "id": "eeac1853", "metadata": { }, "source": [ "## Interactive Ollama CLI (optional)\n", "\n", "Run this only after all setup cells are complete." ] }, { "cell_type": "code", "execution_count": null, "id": "d1287900", "metadata": { }, "outputs": [ ], "source": [ "!{OLLAMA_BIN} run qwen3.8-27b-uncensored-mtp" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }