Quickstart

Python package localllm provides functionalities to use a local LLM object alongside dspy. This looks like this.

import localllm
1llm = localllm.connect("localllm/gemma-4-E2B-it-Q8_0")

1

localllm_connect will download the model (if it is not downloaded yet) and loads the model

import dspy 
class Go(dspy.Signature):
    sentence: str       = dspy.InputField(desc = "A question")
    value:    list[str] = dspy.OutputField(desc = "The answer")

2predict = dspy.ChainOfThought(Go)
out = predict(sentence = "What are the 10 most common first names in Japan")
out

2

Next, you can use dspy programs which will use your local model

Prediction(
    reasoning='Determining the absolute "10 most common" first names in Japan is complex as popularity can fluctuate and data sources vary. However, based on general demographic data and historical trends, certain names consistently rank high. Common Japanese names often follow traditional patterns or have meanings related to nature or virtue. I will list 10 names that are frequently cited as among the most popular or common in recent years.',
    value=['Sota', 'Yuki', 'Haruto', 'Ren', 'Kaito', 'Riku', 'Minato', 'Hiroto', 'Takumi', 'Asahi']
)

class YourProgram(dspy.Signature):
    sentence:   str   = dspy.InputField(desc = "A question")
    value:      int   = dspy.OutputField(desc = "The answer")
    confidence: float = dspy.OutputField(desc = "Give a confidence score about your answer")

predict = dspy.Predict(YourProgram)
out = predict(sentence = "How many r's can you detect in the word strawberrrry? Please provide 1 number.")
out

Prediction(
    value=3,
    confidence=1.0
)