The Rest of the OpenAI API support?

[JohnGalt1717]

now that we have vllm support, it would be really nice if we had support for the rest of the functionality that the openAI api and vllm support enabled in docker.

https://docs.vllm.ai/en/stable/serving/openai_compatible_server/#supported-apis

Especially embeddings, transcribe, translate, and realtime api. (and image generation)

These are super useful while deving locally to make sure code works, run unit tests etc. without having to hit expensive cloud services you use in production.

(especially transcribe, and realtime for us)

[doringeman]

Hey @JohnGalt1717 , thanks for your feedback, I'll take a look on Monday on this.

I'll keep you posted, and hopefully even provide you a build ASAP.
Are you using the Model Runner bundled with Docker Desktop?

[JohnGalt1717]

Thanks!

I'm primarily using the vllm functionality because I'm on mac.

For your llama.cpp backend, and for vllm, we need multimodal chat completions that can accept binary per the spec so that you can pass in audio and get transcriptions out that way. Llama.cpp doesn't appear to currently support the transcription endpoint so it would have to return an error.

Vllm does, along with realtime, and this allows streaming audio in and getting text streamed back as an example which is far more efficient if that's all your doing and that's of high value. I suspect many would also find the translate endpoint of high value for the same reason.

[ericcurtin]

@JohnGalt1717 so we have a choice here. Implement the API at the DMR-level or in the core inference engine level (in the llama.cpp/vLLM layer). It depends on which and what part of the API it is, for example the Ollama API is implemented at the DMR-level. That's neat because it means you can use Ollama API against vLLM or llama.cpp. But that's not always the right thing to do, sometimes it makes more sense to implement the APIs in the core lower-level part vLLM/llama.cpp.

This feature is not prioritised right now but if you find someone to open PRs I'm sure it could be reviewed.

[doringeman]

FWIW something like this should work with llama.cpp:

{
    "model": "hf.co/ggml-org/ultravox-v0_5-llama-3_1-8b-gguf",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "transcribe this audio"
                },
                {
                    "type": "input_audio",
                    "input_audio": {
                        "data": "<data>",
                        "format": "mp3"
                    }
                }
            ]
        }
    ]
}

[JohnGalt1717]

FWIW something like this should work with llama.cpp:

{
    "model": "hf.co/ggml-org/ultravox-v0_5-llama-3_1-8b-gguf",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "transcribe this audio"
                },
                {
                    "type": "input_audio",
                    "input_audio": {
                        "data": "<data>",
                        "format": "mp3"
                    }
                }
            ]
        }
    ]
}

This as a baseline being able to do multi-modal would be a significant win.

I'd argue that your API surface area needs to match that of OpenAI, Ollama and any others. If the backend doesn't support it throw. You're limiting features by not doing this.

Alternatively, expose the backend's api directly. (i.e. I'd be fine if I could just use vllm's api directly because it's enabled and i wouldn't care about this. But because Docker has their own mapper in front of it, I can't.

[ericcurtin]

FWIW something like this should work with llama.cpp:

{
    "model": "hf.co/ggml-org/ultravox-v0_5-llama-3_1-8b-gguf",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "transcribe this audio"
                },
                {
                    "type": "input_audio",
                    "input_audio": {
                        "data": "<data>",
                        "format": "mp3"
                    }
                }
            ]
        }
    ]
}

This as a baseline being able to do multi-modal would be a significant win.

I'd argue that your API surface area needs to match that of OpenAI, Ollama and any others. If the backend doesn't support it throw. You're limiting features by not doing this.

Alternatively, expose the backend's api directly. (i.e. I'd be fine if I could just use vllm's api directly because it's enabled and i wouldn't care about this. But because Docker has their own mapper in front of it, I can't.

I agree, you don't need to argue! PRs welcome

[JohnGalt1717]

I'm not arguing it's a turn of phrase. I'm providing more information. You've made a choice on how to expose APIs that means that you've put yourselves on the hook for updating and maintaining and exposing the functionality of the model runners instead of doing it (which I don't think was wise). I'm trying to get you options as a result of that choice.

Given this isn't for the open source, and the company I work for pays and this is your business model, not sure why you'd ask for PRs to fix what you committed to doing when you decided not to just expose the model runner's api. You're way out of date and only have the most basic of functionality that AI supports from 3 years ago in your product and thus you're a bottle neck for me and everyone else that wants to use models in docker as a result.

Further, given that you can't run VLLM with acceleration on MAC on Docker Desktop any other way, but your free competitor does allow this (because GPU acceleration is exposed in their implementation), I would think it would be in high interest for you to make sure that you don't give people a reason on Mac to leave Docker Desktop, not pay and go to your free competitor.

Instead I got "Feel free to add functionality to our paid product for us."

I trust you'd see why this doesn't make sense and why someone would a. push back, and b. suggest possibly easier alternatives like just letting me get to VLLM's endpoints directly.

[ericcurtin]

I'm not arguing it's a turn of phrase. I'm providing more information. You've made a choice on how to expose APIs that means that you've put yourselves on the hook for updating and maintaining and exposing the functionality of the model runners instead of doing it (which I don't think was wise). I'm trying to get you options as a result of that choice.

Given this isn't for the open source, and the company I work for pays and this is your business model, not sure why you'd ask for PRs to fix what you committed to doing when you decided not to just expose the model runner's api. You're way out of date and only have the most basic of functionality that AI supports from 3 years ago in your product and thus you're a bottle neck for me and everyone else that wants to use models in docker as a result.

Just to be clear this is the open-source/community issues page. Paid support goes through:

https://www.docker.com/support/

and is prioritised.

Further, given that you can't run VLLM with acceleration on MAC on Docker Desktop any other way, but your free competitor does allow this (because GPU acceleration is exposed in their implementation), I would think it would be in high interest for you to make sure that you don't give people a reason on Mac to leave Docker Desktop, not pay and go to your free competitor.

We will always lag behind upstream vLLM (and especially on macOS) for 2 reasons. 1. We don't just pull from latest vLLM all the time, we carefully bump to ensure stability 2. vLLM on macOS goes through vllm-metal project, which can also lag behind the main vLLM project.

Instead I got "Feel free to add functionality to our paid product for us."

I trust you'd see why this doesn't make sense and why someone would a. push back, and b. suggest possibly easier alternatives like just letting me get to VLLM's endpoints directly.

[JohnGalt1717]

The issue is that you don't even support what's already there in your openai api. WIth the old version of vllm that you're using, any mlx model will work and there are lots of models for transcribe, audio, realtime etc. But you can't do it, because Docker hasn't implemented the apis, thus there is no way to pass them through. If you're not committed to keeping api pairity with OpenAI then Docker Models isn't viable for any developer because the entire point to using docker models is to expose the same api that you'll use in production.

Thus as it stands right now, the primary use case of docker models doesn't work for anything other than basic chat, and een then, it ignores the streaming flag and returns the entire thing which breaks all .NET Microsoft.Extensions.AI when you ask for a stream.

And again: while you're competitor has full hardware acceleration for GPUs, docker doesn't other than this singular approach, which means that you're losing (perspective) customers as a result.