Quickstart

Python package localllm provides functionalities to use a local LLM object alongside dspy. This looks like this.

import localllm
1llm = localllm.connect("localllm/gemma-4-E2B-it-Q8_0")

1

localllm_connect will download the model (if it is not downloaded yet) and loads the model

import dspy
class Go(dspy.Signature):
    sentence: str       = dspy.InputField(desc = "A question")
    value:    list[str] = dspy.OutputField(desc = "The answer")

2predict = dspy.ChainOfThought(Go)
out = predict(sentence = "What are the 10 most common first names in Japan")
out

2

Next, you can use dspy programs which will use your local model

Prediction(
    reasoning='Determining the absolute "10 most common" first names in Japan is complex as popularity can fluctuate and data sources vary. However, based on general demographic data and historical trends, certain names consistently rank high. Common names in Japan often follow traditional naming conventions or have meanings associated with nature or virtue. I will list some of the most frequently cited and traditional popular names.',
    value=['Sota', 'Yuki', 'Haruto', 'Ren', 'Riku', 'Minato', 'Hiroto', 'Kaito', 'Taro', 'Yuto']
)

class YourProgram(dspy.Signature):
    sentence:   str   = dspy.InputField(desc = "A question")
    value:      int   = dspy.OutputField(desc = "The answer")
    confidence: float = dspy.OutputField(desc = "Give a confidence score about your answer")

predict = dspy.Predict(YourProgram)
out = predict(sentence = "How many r's can you detect in the word strawberrrry? Please provide 1 number.")
out

Prediction(
    value=3,
    confidence=1.0
)