Running Dramatron locally using Koboldcpp + LLM
Experience an alternative user interface to writing with an LLM
There aren’t many user interfaces which break out of the now ubiquitous chat prompt for interating with an LLM.
Dramatron is a really interesting guided process and user interface. LLM interfaces really need more experimentations, if you’re interesetd in trying some other alternatives there are a few notable ones appearing in the open source LLM interfaces, https://github.com/FartyPants Playground, Twinbook and BlockWriter and https://novelai.net/ is also worth a look.
If you don’t have the capability to run Dramatron, you can take a look at the process in the video at the end of this article.
Why would you want to do this?
remove the guardrails; there are an incredible selection of characterful LLMs, which have been trained on specific styles or content, offering a much richer experience than some of the more corporate, business focused models.
data leakage; you keep control of your data your prompts and results won’t end up being fed back into any model without your knowledge
a lot of hosted LLMs have strict guardrails which depending on what type of dialog you’re authoring can quickly put a stop to your project.No fees using a local; you can experiment without racking up any costs
Installation and requirements
The process to get this up and running is a little complex, I’ll outline the main process below:
Use my Dramatron fork, it has a connector for koboldcpp’s API.
You’ll need to be running https://github.com/LostRuins/koboldcpp and have it running an appropriate model, a couple of recommendations would be
https://huggingface.co/TheBloke/dolphin-2.7-mixtral-8x7b-GGUF
https://huggingface.co/mlabonne/NeuralBeagle14-7B-GGUF
Clone my dramatron repository at https://github.com/makeplayhappy/dramatron
I've added a new Jupyter notebook in the colob folder (dramalocal-kobold.ipynb) which uses Koboldcpp API so you can run this using a local LLM server.
If you don’t have jupyter installed then install using pip
pip install jupyterlab
Locally you can run Jupyter notebooks with this command
jupyter notebook
or use the no-browser and ip flags if you want to run it on a local server (change 192.168.1.10 to whatever your local server's IP is)
jupyter notebook --no-browser --ip="192.168.1.10" --port=7080
then you can open up http://192.168.1.10:7080 in a browser connect to it, go into the collab folder and open up the dramalocal-kobold.ipynb notebook.
Then update the LLM_API_GENERATE_URL setting to use your local kobold api IP address (localhost or “http://192.168.1.10:5000” in my example)