I have always been bothered with the multinomial layer that picks the next token. It seems like you should be able to stick a classifier in there to detect if the probabilities align with a satisfactory answer and whether the model is unsure of the response. If that is the case, then it should branch out to a bigger model which would compute that single token that it is unsure about, and from that the rest of the response is obvious for the smaller model.
This way you could combine e.g. a 7B parameter model with a 70B parameter model, and get the quality of the larger model while most of the time you are only running the small model.
Edit: You could also store the full probabilities for each token, and then the classifier could detect if it had gone down a bad path, and then unwind the tokens and pick a different path.
That's reminiscent of a caching scheme, you could probably layer that down to even smaller and up to even larger models for increased performance and accuracy.
Better yet: have many smaller models that can when confused call upon larger models and the larger models could then pick the most appropriate smaller 'expert' model or, alternatively themselves escalate. Sort of a supervisor tree for language models.
Contextual loading of a bigger model would incur a lot of latency. In this case it might be a good idea to farm out the request to a non-local model. Could be a good avenue for a hybrid local & remote completion system.
I would be very surprised if (not at all)OpenAI didn't. I've noticed it with both gpt 3.5 and Gpt4 that if you ask it a difficult question the latency increases a lot, especially at the beginning of the answer. I wouldn't ve surprised if they did exactly what was described.
(Why Obsidian? Because then I don't have to write a text editor from scratch, and because one can then combine it with other plugins. Also because I intended for this implementation to work on mobile, but getting the UX right for that is annoying so it isn't supported right now.)
davinci-002 is a good publicly available model to start with. Weaving takes practice if you want something very specific.
This is the work of @repligate, I can't recommend them highly enough. A true creative in every sense of the word, a rare spirit with a fundamental grounding and understanding of the spirit of LLMs. I highly, highly recommended pursuing every inch of their writing that interests you.
I was actually in the process of developing something just like this.
Think of not wanting to pollute your main conversation when you come across something you don’t know in the LLMs response, so you create a side chat and then navigate back once you get the info you needed.
Can someone explain the primary use-case for this tool? Is it literally intended for writing a novel? Or just as sort of a plaything for exploring possibilities for generating text?
Wouldn't the limited context window constrain any attempt to write long form text with this model?
There is a 16k context window for GPT3.5. So for example, if you tried to put in a chapter of a novel then you would exceed the context window and the model would "forget" events from the start of the chapter.
Does it do or anyone knows some writing tool where the text can be anonymized before it is sent to GPT? I don't want to share sensitive data, like PII etc.
I could just manually specify words to be send to GPT as placeholders, "John Doe" -> "JD". Then on every response the placeholders would be resolved with the original text just for visual purposes "JD" -> "John Doe".
I know it can be made with a little programming and OpenAI API, but maybe no need to reinvent.
This is a tool intended for exploring base language models. The best publicly available base model on the OpenAI API is "davinci-002". Not davinci, but davinci-002.
i (biasedly) thought this was going to be a reddit-like infinitely nested threaded interface to chat with GPT (as opposed to the serial chat-like dialogue based one.
That's what I thought, too. Such an interface would be awesome to explore multiple trains of thought simultaneously and efficiently.
With the ability to seemlessly merge branches together using smart conflict resolution algorithms, this can be an invaluable tool for better decision-making.
link for curious: https://generative.ink/meta/block-multiverse/