Maybe I’ll wind up as the only user of this rather abstruse Python module. But it seemed like this functionality was worth splitting out from inside of my personal Python testing framework, where it ...

I'm using llama-cpp-python==0.2.60, installed using this command CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python. I'm able to load a model using type_k=8 and type_v=8 (for q8_0 cache).