ddh

Andy out here shooting straight through his foot and putting holes in his boat’s hull.

I’m still trying out combinations of hardware and models, but even my old Intel 8500T CPU will run around reading speed with a stock version of Meta’s Llama 3.2 3b (maybe the one you tried) with mostly good output—fine for rewriting content, answering questions about uploaded document stores etc.

There are thousands of models tuned for various purposes, so one of the key questions is your purpose. If you want to use your setup for something specific (e.g., coding SQL) you are going to be able to find a much more efficient model.

Ollama and Open WebUI, as far as I know, are just open source software projects created to run pre-trained models, and have the same business model as many other open source projects on Github.

The models themselves come from Google, Meta and others. Have a look at all the models available on Hugging Face. The models themselves are just binary files. They’ve been trained and there are no ongoing costs to use them apart from energy your computer uses to run them.

I run Ollama with Open WebUI at home.

A) the containers they run in by default can’t access the Internet, but they are provided access if we turn on web search or want to download new models. Ollama and Open WebUI are fairly popular products and I haven’t seen any evidence of nefarious activity so far.

B) they create a profile on me and my family members that use them, by design. We can add sensitive documents that the models can use.

C) they are restricted by what we type and the documents we provide.

I’m on a XR and have the option.

FWIW I run only very small databases e.g., sqlite ones shipped with applications, but haven’t had any problems in about a year now, and nothing that wasn’t recoverable from backup.

Correct, I run docker on a compute host that has no local storage. The host’s disks are on iSCSI LUNs.

What’s high availability to you? There’s an increasing cost.

Anyway, I have a similar setup. NAS for storage and OptiPlex Proxmox nodes for compute. But I went a different route and set up an iSCSI SAN and all the guests use block storage on the NAS. Guest backups by Proxmox go to file storage on the NAS and backed up to a second NAS.

“darktable is an open source photography workflow application and raw developer. A virtual lighttable and darkroom for photographers. It manages your digital negatives in a database, lets you view them through a zoomable lighttable and enables you to develop raw images and enhance them.”

Are you pirating shit? No? Guess what, use a VPN!

If you do set up a RAG store, please post the tech stack you use as I’m in a similar situation. The inbuilt document store management in ollama+openwebui is a bit clunky.

I’d be interested to see how it goes. I’ve deployed Ollama plus Open WebUI on a few hosts and small models like Llama3.2 run adequately (at least as fast as I can read) on even an old i5-8500T with no GPU. Oracle Cloud free tier might work OK.

Running an LLM can certainly be an on-demand service. Apart from training, which I don’t think we are discussing, GPU compute is only used while responding to prompts.

I’m tired, boss.

It’s a very alluring argument, to give up, to let them have it. But, not alluring enough.

Last Mozilla survey I filled out had a Mozilla domain. Is this actually Mozilla?

Cupboard + DiskStation + OptiPlex = Win

Tailscale is available as an official DSM package, so if it’s only you accessing it you could still block it from the Internet.

Jellyfin is also available as a native DSM package through SynoCommunity, FWIW.

https://thenophone.com/products/nophone-original

I’m interested in how you like Ceph.

My setup is similar, using a DS1522+ volume as shared block storage for an iSCSI SAN for three Proxmox nodes. Two nodes are micro PCs and the third is running on the 1522+. There’s a DS216j for backups.