SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 4 days agoDo you host your own AI?message-squaremessage-square203linkfedilinkarrow-up1183arrow-down142file-text
arrow-up1141arrow-down1message-squareDo you host your own AI?SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 4 days agomessage-square203linkfedilinkfile-text
minus-squarefubarx@lemmy.worldlinkfedilinkEnglisharrow-up2·4 days agoFound vLLM to be the most efficient local runtime service. And “ray” as a good (but complicated) way to distribute the load: https://docs.ray.io/
Found vLLM to be the most efficient local runtime service. And “ray” as a good (but complicated) way to distribute the load: https://docs.ray.io/