Running with vLLM as a server
This is a strong reranker model. But it seems that the vLLM doesn’t support to run through a command line. Is there any one can ask vLLM support this model?
Yes. vLLM has supported this model since v0.20, included that PR. Great. Thank you all.
Is there a way to use it in a container with vllm already?
It dienst work with: vllm/vllm-openai:latest
Can this model be used in a chatbot that is provided free of charge? We do not monetize the chatbot in any way, and it is intended solely as an informational tool for citizens. Does the CC BY-NC 4.0 licence restrict the use of this model in such a non-commercial chatbot?
Hey @Schnappson , as per the license the model can be used for non commercial purpose, yes.
Thanks @gregco for your answer! However, I'm still unclear on what exactly "non-commercial" means in this context.
Could you clarify whether my use case qualifies as non-commercial? Specifically:
Does it count as non-commercial if it's only barely connected to monetization?
Or does making it available for public use automatically make it commercial, even if the product itself is free of charge?
Hi @Schnappson , "non-commercial" here means the use isn't primarily aimed at commercial advantage or making money (the use, not the user).
Being free or publicly available doesn't decide it either way, what matters is the purpose. A free citizen-information tool with no ads, upsells, or ties to a paid product or service is a clear non-commercial use, which is what you've described. However, if the chatbot is connected to revenue in some way, then its utilization does not fall under the non-commercial term of the license.
If it's an important deployment and you'd like written confirmation, feel free to reach out to us directly!