I would like to use an alternate openai api address for sonic sage.
Specifically i want to use my selfhosted localai which is compatible with the openai api, but other people could use with azure gpt api.
Obviously the experience with localai would be unpredictable using any number of backends, but it would be fun to not have to use any paid services, since im using local ai for as much as i can
Do these local ones support the /v1/models endpoint and lie about GPT3/4? Our code currently has model-selection code which would be work to generalize vs just changing endpoint.
Im not exactly familiar with the azure gpt to say for absolute sure, but local is 1:1 openai api compatible including the models endpoint.
It would be cool to select any custom model, but i just put mixtral-8x7b named gpt-3.5-turbo in localai, and most apps just post to it not knowing any different, that is if they have support for an alternate api address.
In other words you can name the model what ever you want, and as long as the prompt templates are setup correctly by the user, localai will format it to llama-cpp and return the responce. It supports streaming as well
I just learned about openrouter.ai which will route openai api requests to the cheapest/most effective provider, weather that be gpt 3, 4, or something else based on their heuristics or user backend config
So while ill be content with simple url change, allowing to select from the models endpoint could still be cool.
Exactly the topic I came here to look for, same as @0xCBF, I am looking for a possibility of using my own, locally hosted, LLM.
Hello there,
i would love to use my own openAI endpoint, which is realy easy to set up now that out-of-the-box solutions are available. Maybe you could, as a first step, tell us what you expect the model to be named, and allow for a change of the API-Adress? As an experimental feature, obviously.
Many thanks, and keep up that great work 
Would love for this to be officially supported! I see it as a necessity to, at the very least, provide an option to configure a proxy, in the (officially supported) case someone uses Azure OpenAI Services.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.
