Welcome to localllm

Python package localllm provides functionalities to use a local LLM object alongside dspy

Download a model

First use localllm_download_model to download a local model. For the list of all models which you can quickly download, look to the help of localllm_download_model.

from localllm import localllm_download_model, localllm_list_models
path = localllm_download_model("localllm/gemma-3-270m-it-Q8_0")

path

'/home/runner/.localllm/models/google_gemma-3-270m-it-Q8_0.gguf'

Test the local model

from llama_cpp import Llama
import pprint
transformer = Llama(
    model_path = path, flash_attn = False, verbose = False,
    main_gpu = 0, n_gpu_layers = -1, n_ctx = 32768, n_threads = 1, swa_full = False, seed = 4321)
out = transformer("How much is 4x4")
pprint.pprint(out)

{'choices': [{'finish_reason': 'length',
              'index': 0,
              'logprobs': None,
              'text': '?\n\n4 * 4 = 16\n\nTherefore, 4x'}],
 'created': 1776292386,
 'id': 'cmpl-8045d355-2ae3-4f7d-bdcf-0330ffb242fd',
 'model': '/home/runner/.localllm/models/google_gemma-3-270m-it-Q8_0.gguf',
 'object': 'text_completion',
 'usage': {'completion_tokens': 16, 'prompt_tokens': 8, 'total_tokens': 24}}

Use the model directly alongside dspy

Use dspy alongside the local model

from localllm import LocalLLM, LocalChatAdapter
import dspy
dspy.configure(lm = LocalLLM(transformer), adapter = LocalChatAdapter())
class Go(dspy.Signature):
    sentence: str = dspy.InputField(desc = "A question")
    output:   str = dspy.OutputField(desc = "A city")

model = dspy.Predict(Go)
out = model(sentence="What is the capital of France")
pprint.pprint(out)

Prediction(
    output='Paris'
)

out["output"]

'Paris'

Another example with a translation model

from localllm import localllm_download_model, LocalLLM
from llama_cpp import Llama
path = localllm_download_model("localllm/translategemma-4b-it-q8_0")
transformer = Llama(
    model_path = path, flash_attn = False, verbose = False,
    n_ctx = 4096, swa_full = False, seed = 4321)

llm = LocalLLM(transformer)
messages = [
    {
        "role": "system",
        "content": "You are a professional Dutch (nl) to English (en) translator. Your goal is to accurately convey the meaning and nuances of the original Dutch text while adhering to English grammar, vocabulary, and cultural sensitivities. Produce only the English translation, without any additional explanations or commentary. Please translate the following Dutch text into English:",
    },
    {
        "role": "user",
        "content": "Wie heeft tijdens de laatste Olympische spelen de 100 meter spurt gewonnen?"
    },
]
llm(messages = messages)

['\nWho won the 100-meter sprint during the last Olympic Games?\n']

text = "\nPatient Profile:\n- Name: Ms. Suzy Rickie Hackett\n- Contact Number: 555-359-4515 (Home)\n- Date of Birth: 1941-09-09\n- Deceased on: 2021-08-03 at 01:47:39 (+02:00)\n- Address: 292 Buckridge Port, Massachusetts, 01532, US\n- Not a multiple birth\n- Language Spoken: English (United States)\n\nEncounter Details:\n1. Encounter Status: Finished\n   - Description: Well child visit procedure\n   - Attending Doctor: Dr. Valentine Leannon\n   - Visit START: 1941-09-09 at 20:43:56 (+02:00)\n   - Visit END: 1941-09-09 at 20:58:56 (+02:00)\n   - Relationship with patient: Primary performer\n\n2. Encounter Status: Unknown\n   - Description: Well child visit procedure\n   - Provider Organisation: SUNRISE HEALTHCARE LLC\n   - Visit START: 1950-09-12 at 19:43:56 (+01:00)\n   - Visit END: 1950-09-12 at 19:58:56 (+01:00)\n\n2 Procedures recorded for Ms. Suzy Rickie Hackett, both of unknown status.\n\nOrganization Information:\n- Name: SUNRISE HEALTHCARE LLC\n- Contact Number: 6176509959\n- Active: Yes\n- Type: Healthcare Provider\n\n\n"
messages = [
    {
        "role": "system",
        "content": "You are a professional English (en) to French (fr) translator. Your goal is to accurately convey the meaning and nuances of the original Dutch text while adhering to English grammar, vocabulary, and cultural sensitivities. Produce only the French translation, without any additional explanations or commentary. Please translate the following English text into French:",
    },
    {
        "role": "user",
        "content": text
    },
]
trans = llm(messages = messages)
trans

["\nProfil du patient\xa0:\n- Nom\xa0: Mme. Suzy Rickie Hackett\n- Numéro de téléphone\xa0: 555-359-4515 (Domicile)\n- Date de naissance\xa0: 09\xa0septembre 1941\n- Décédée le\xa0: 03\xa0août 2021 à 01\xa0h\xa047\xa039 (+02\xa0h)\n- Adresse\xa0: 292 Buckridge Port, Massachusetts, 01532, États-Unis\n- Pas de naissance multiple\n- Langue parlée\xa0: Anglais (États-Unis)\n\nDétails de la consultation\xa0:\n1. Statut de la consultation\xa0: Terminé\n   - Description\xa0: Consultation de routine pour un enfant en bonne santé\n   - Médecin traitant\xa0: Dr. Valentine Leannon\n   - Début de la consultation\xa0: 09\xa0septembre 1941 à 20\xa0h\xa043\xa056 (+02\xa0h)\n   - Fin de la consultation\xa0: 09\xa0septembre 1941 à 20\xa0h\xa058\xa056 (+02\xa0h)\n   - Relation avec le patient\xa0: Personne responsable\n\n2. Statut de la consultation\xa0: Inconnu\n   - Description\xa0: Consultation de routine pour un enfant en bonne santé\n   - Organisation de soins\xa0: SUNRISE HEALTHCARE LLC\n   - Début de la consultation\xa0: 12\xa0septembre 1950 à 19\xa0h\xa043\xa056 (+01\xa0h)\n   - Fin de la consultation\xa0: 12\xa0septembre 1950 à 19\xa0h\xa058\xa056 (+01\xa0h)\n\n2 Procédures enregistrées pour Mme. Suzy Rickie Hackett, dont le statut est inconnu.\n\nInformations sur l'organisation\xa0:\n- Nom\xa0: SUNRISE HEALTHCARE LLC\n- Numéro de téléphone\xa0: 6176509959\n- Actif\xa0: Oui\n- Type\xa0: Prestataire de soins de santé\n[/INST]"]

Directly use the model

• localllm_connect downloads the model if it does not exist
• Small LLM’s do not always adhere to provided prompt templates
  • LocalChatAdapter assumes json but falls back to parsing-based strategy if the output is malformed by the LLM
  • You can still inspect what the output from dspy is that the LLM gets

import dspy
from localllm import localllm_connect, LocalChatAdapter
model_name = "localllm/gemma-3-4b-it-Q4_K_M"
model_name = "localllm/Qwen3-4B-Instruct-Q4_K_M"
model_name = "localllm/LFM2.5-1.2B-Instruct-Q8_0"
llm = localllm_connect(model_name, 
                       model_kwargs = dict(
                            verbose = False, 
                            n_ctx = 32768, n_gpu_layers = -1, n_threads = 1, flash_attn = True, swa_full = True, seed = 4321),
                       adapter = LocalChatAdapter(trace = 2)) 

class Go(dspy.Signature):
    sentence: str = dspy.InputField(desc = "A question")
    value:    str = dspy.OutputField(desc = "A city")

predict = dspy.Predict(Go)
out = predict(sentence = "What is the capital of Hungary")

====================== Parsing LLM completion ======================
[[ ## value ##]
Budapest
[[ ## completed ##]]
====================== LocalChatAdapter LLM did not provide valid structured output, fallback to lenient parsing strategy ======================

out

Prediction(
    value='Budapest'
)

List of models

The following list of models can be easily downloaded. You can use them with localllm_connect("localllm/{model_name}")

from localllm import localllm_list_models
import pandas as pd
x = localllm_list_models()
x = pd.DataFrame.from_dict(x, orient = "index")
x[["source", "filename"]]

	source	filename
gemma-3-270m-it-qat-Q4_0	ggml-org	gemma-3-270m-it-qat-Q4_0.gguf
gemma-3-270m-it-Q8_0	bartowski	google_gemma-3-270m-it-Q8_0.gguf
gemma-3-1b-it-Q8_0	bartowski	google_gemma-3-1b-it-Q8_0.gguf
gemma-3-4b-it-qat-Q4_0	bartowski	google_gemma-3-4b-it-qat-Q4_0.gguf
gemma-3-4b-it-Q4_K_M	bartowski	google_gemma-3-4b-it-Q4_K_M.gguf
gemma-3-12b-it-qat-Q4_0	bartowski	google_gemma-3-12b-it-qat-Q4_0.gguf
GLM-4.6V-Flash-Q4_K_M	ggml-org	GLM-4.6V-Flash-Q4_K_M.gguf
translategemma-4b-it-q8_0	NikolayKozloff	translategemma-4b-it-q8_0.gguf
translategemma-12b-it-q4_k_m	NikolayKozloff	translategemma-12b-it-q4_k_m.gguf
LFM2.5-350M-Q8_0	LiquidAI	LFM2.5-350M-Q8_0.gguf
LFM2.5-1.2B-Instruct-Q4_K_M	LiquidAI	LFM2.5-1.2B-Instruct-Q4_K_M.gguf
LFM2.5-1.2B-Instruct-Q8_0	LiquidAI	LFM2.5-1.2B-Instruct-Q8_0.gguf
Qwen3-4B-Instruct-Q4_K_M	bartowski	Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf
Qwen3-8B-Q4_K_M	bartowski	Qwen_Qwen3-8B-Q4_K_M.gguf
Qwen3.5-0.8B-Q8_0	unsloth	Qwen3.5-0.8B-Q8_0.gguf
Qwen3.5-2B-Q4_K_M	unsloth	Qwen3.5-2B-Q4_K_M.gguf
Qwen3.5-4B-Q4_K_M	unsloth	Qwen3.5-4B-Q4_K_M.gguf
Qwen3.5-9B-Q4_K_M	unsloth	Qwen3.5-9B-Q4_K_M.gguf
gemma-4-E2B-it-Q8_0	ggml-org	gemma-4-E2B-it-Q8_0.gguf
gemma-4-E2B-it-Q4_K_M	unsloth	gemma-4-E2B-it-Q4_K_M.gguf
gemma-4-E4B-it-Q8_0	ggml-org	gemma-4-E4B-it-Q8_0.gguf
gemma-4-E4B-it-Q4_K_M	unsloth	gemma-4-E4B-it-Q4_K_M.gguf