starcoder : example for using scratch buffers to reduce memory usage - #176

Cool!
The sizes might need to be adjusted for the big starcoder model.
Do you have it handy to test it and see if that is the case?

Seems like it still doesnt work for starcoder :/

starcoder_model_load: loading model from '../models/bigcode/starcoder-ggml-q4_1.bin'
starcoder_model_load: n_vocab = 49152
starcoder_model_load: n_ctx   = 8192
starcoder_model_load: n_embd  = 6144
starcoder_model_load: n_head  = 48
starcoder_model_load: n_layer = 40
starcoder_model_load: ftype   = 2003
starcoder_model_load: qntvr   = 2
starcoder_model_load: ggml ctx size = 26724.47 MB
starcoder_model_load: memory size = 15360.00 MB, n_mem = 327680
starcoder_model_load: model size  = 11364.23 MB
starcoder_model_load: model size  = 11364.23 MB
main: prompt: '# a a a a a a a a a a a a a a a a a a a 
...
def fibonacci('
main: number of tokens in prompt = 8005
...
 a a a a a a a a a a a a a a a a a a a a a a a a a a a a a aggml_new_tensor_impl: not enough space in the scratch memory
Segmentation fault (core dumped)

Bumped the buffers to 256 MB. Still not sure if enough - need to figure out a better way to do this

Starcoder works now with 8k context! 🎉

$ ./bin/starcoder -m ../models/bigcode/starcoder-ggml-q4_1.bin -p "$(cat ../prompt.txt)" --top_k 0 --top_p 0.95 --temp 0.2 -n 200 -t 48
main: seed = 1684603021
starcoder_model_load: loading model from '../models/bigcode/starcoder-ggml-q4_1.bin'
starcoder_model_load: n_vocab = 49152
starcoder_model_load: n_ctx   = 8192
starcoder_model_load: n_embd  = 6144
starcoder_model_load: n_head  = 48
starcoder_model_load: n_layer = 40
starcoder_model_load: ftype   = 2003
starcoder_model_load: qntvr   = 2
starcoder_model_load: ggml ctx size = 26724.47 MB
starcoder_model_load: memory size = 15360.00 MB, n_mem = 327680
starcoder_model_load: model size  = 11364.23 MB

main: mem per token =   438584 bytes
main:     load time =  3832.32 ms
main:   sample time =     6.71 ms
main:  predict time = 1304651.88 ms / 161.87 ms per token
main:    total time = 1308736.00 ms

Btw, I saw you are using -t 48 - is that actually faster on your system?
I usually don't go over -t 8 even if the machine has many cores

I didn't compare 8 to 48 threads, but 48 was definitely faster than the default 4

Have fix for these in GGML, put in PRs but they are stuck in github flagged filters

segfault issue is mem_per_token not incrementing;
mpt main.cpp:

ctx flags added in separate PR. They work it seems with high token count, tested w/ 65k. Benefits from n_predict set to same as ctx size as the -1 value isn't supported seemingly in ggml.

Starcoder works now with 8k context! 🎉

$ ./bin/starcoder -m ../models/bigcode/starcoder-ggml-q4_1.bin -p "$(cat ../prompt.txt)" --top_k 0 --top_p 0.95 --temp 0.2 -n 200 -t 48
main: seed = 1684603021
starcoder_model_load: loading model from '../models/bigcode/starcoder-ggml-q4_1.bin'
starcoder_model_load: n_vocab = 49152
starcoder_model_load: n_ctx   = 8192
starcoder_model_load: n_embd  = 6144
starcoder_model_load: n_head  = 48
starcoder_model_load: n_layer = 40
starcoder_model_load: ftype   = 2003
starcoder_model_load: qntvr   = 2
starcoder_model_load: ggml ctx size = 26724.47 MB
starcoder_model_load: memory size = 15360.00 MB, n_mem = 327680
starcoder_model_load: model size  = 11364.23 MB

main: mem per token =   438584 bytes
main:     load time =  3832.32 ms
main:   sample time =     6.71 ms
main:  predict time = 1304651.88 ms / 161.87 ms per token
main:    total time = 1308736.00 ms

Hey I am facing the issue with Groovy 1.3 on GPT4all. I don't know why. Can you tell me about it ?