Gentoo LLMs

[Vieri]

Hi,

I'm starting to look at the oh so many open-source software projects to run LLM models out there, and it's quite overwhelming.
There are some attempts of creating Gentoo ebuilds for some of them (eg. ollama in guru).
I'm particularly interested in the projects that allow to run models from a custom program or via REST APIs.
Not really looking for GUI or webUI.

Any suggestions here in the forum for worthy/updated software that installs from source (no binary packages) and can easily be set up with a custom ebuild?

Regards

[Navar]

https://github.com/rasbt/LLMs-from-scratch , a real Gentoo way.

For creating your own ebuild(s), https://wiki.gentoo.org/wiki/Basic_guide_to_write_Gentoo_Ebuilds

https://ollama.com/download , more "easy" mode. "install" and run local as your limited user (would suggest checking the script instead of blindly running and/or manual install instructions). I doubt Gentoo devs are large enough currently in resources and number, versus all other normal expected packages that may or may not need maintainers, testers, help, etc. to be chasing supporting any specific LLM being available out of the billion or so currently in existance due to crazed fad/demand, e.g., https://huggingface.co/models
_________________
(I don't speak for/represent my current employer)

[jlpoole]

I'm smitten by LLMs and am just paid $57 for the 2024 Manning press book Build a Large Language Model (From Scratch) featured at the https://github.com/rasbt/LLMs-from-scratch Navar mentioned.

I would be interested in collaborating with Gentoo enthusiasts on building AI type items. I installed LocalAI, and, of course, am hot on the trail to debug the manual install version (they have a docker version, I'm not a fan of docker, unless it's just to take a peek) where instantiating Piper is broken and it's a real devil of a time to unwrap the code and make sense of what is going on. My impression is that there is a subtle misalignment of libraries which may not have been verified during the install. Any rate, I was thinking it would be great to talk with other Gentoo users who are of like-mind to learn more about AI/models and deployment on your own.

Note: LocalAI developer who goes by this Gentoo handle: mudler [Ettore Di Giacinto], was formerly a Gentoo developer and moved onto Sabayon and now looks to be in the thick of his baby: LocalAI.

[Banana]

Moved from Portage & Programming to Gentoo Chat.
_________________
Forum Guidelines

PFL - Portage file list - find which package a file or command belongs to.
My delta-labs.org snippets do expire

[fedeliallalinea]

I use open-webui with ollama in a docker image.
The open-webui provide openai rest api because of that you can use shell_gpt or any other program with openai api support.
_________________
Questions are guaranteed in life; Answers aren't.

[lordof7]

llama.cpp may be worth a look. It appears to have an API, and it's CLI interface is nice and simple. I was able to get it working with Vulkan on an AMD RDNA2 card without too much grief (X11, mesa, vulkan-header ,vulkan-tools, vulkan-loader, and media-libs/shaderc to fill in for Vulkan-SDK.) I ended up just downloading the source and compiled with flags for vulkan. I run it locally.

This guide was helpful: https://blog.steelph0enix.dev/posts/llama-cpp-guide/

I've played with the smollm-1.7b model, and asked it about Gentoo. You may be amused by the answer:
---
> Ok, different subject. Do you know anything about Gentoo
Gentoo! Gentoo is a popular breed of horse known for its intelligence, athleticism, and striking color patterns. They are often referred to as "paint horses" due to their unique, swirling coat colors.
---

It does appear that there was a racehorse named Gentoo, but other than that I couldn't find any evidence.