Samuel Karp
@samuelkarp.com
254 followers 147 following 44 posts
Anti fascist. Chair of the Open Container Initiative TOB. Staff Software Engineer @ Google. Opinions = mine.
Anti fascist. Chair of the Open Container Initiative TOB. Staff Software Engineer @ Google. Opinions = mine.
![Mount Ollama binary
• Mount the Ollama binary from the host into the container
config. Mounts
append(config. Mounts, specs. Mount
Source:
"/usr/bin/ollama"
Destination:
"/usr/bin/ollama",
Type:
"bind".
Options:
[]string("rbind", "ro"},
})](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:e6d6ygltadw434kgrsfnjwql/bafkreifgp5livpbguqbhqp7otfuasyvlbaa377pxsggtoivtofdp3fj2eq)
![Mount OCI as a Modelfile
• Ollama needs to read the GGUF from a Modelfile file
modelPath: getTaskEnv ("OLLAMA_MODEL PATH", config.Process.Env)
modelfile: fmt.Sprintf("FROM %s", modelPath)
err = os.WriteFile(filepath.Join(tempDir, "Modelfile"), []byte (modelfile), 0644)
if err != nil {
return nil, err
}
config.Mounts append(config.Mounts, specs. Mount(
Source:
filepath.Join(tempDir, "Modelfile"),
Destination: "/tmp/Modelfile",
Type:
"bind",
Options: []string("rbind", "ro"},
})](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:e6d6ygltadw434kgrsfnjwql/bafkreifml3eg2dvhqdknyruiowdyyxwpd4hoy4yiaw6bojsuaohx7vsjie)
![Serve the Model
Set the container args to serve the model using Ollama
config.Process.Args = [] string(
/bin/bash",
"-c".
strings.Join([]string{
"ollama serve &",
"while ollama ps> /dev/null 2>&1; do sleep 1; done",
"ollama create modelName + --file/tmp/Modelfile",
"tail -f /dev/null",
}, "\n"),](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:e6d6ygltadw434kgrsfnjwql/bafkreigry7cnnqcywkk4tkhwpc3izqhjvuq2hm2xsvl7mjzjkxqitisuyu)
