Thank you so so much for pointing out ROCmFP4. I have been tinkering with my RDNA 3 framework on llama. I was struggling with ROCm llama.cpp and have been using vulcan in the meantime. I know there’s some issues on the llama.cpp github to try and fix my issue (UMA stuff), but haven’t come across this specific project. Gonna try it out
UMich research cited in the article estimates a carbon dioxide emissions per year (using constant mileage typical per year) break even time between EV and ICE SUVs to be 1.6-1.9 years.
Promising. But, is carbon dioxide emissions the only environmental impact? Humanitarian impact (e.g. rare metal extraction labor conditions)? I’m not an environmentalist researcher/academic. I wonder how accurately just one emissions prediction portrays the full picture.
Fair enough. Let me quickly go through the one-liner, command-by command
# Joined by `&&`, bash runs these commands in sequence (as if run individually in shell), but exits/stops execution early if any command fails (return nonzero)
TMP_DEB=$(mktemp --suffix=.deb) && curl -sSL "https://support.brother.com/g/b/downloadend.aspx?c=us&lang=en&prod=hll2465dw_us&os=128&dlid=dlf106036_000&flang=4&type3=10283" -o "$TMP_DEB" && sudo apt install -y "$TMP_DEB" && rm -f "$TMP_DEB"
# Going command by command:
# First, we create a local variable in the shell, named `TMP_DEB`
# We assign the value to `$(...)`. This stores the string output (to stdout) of running the command `mktemp ...` to `TMP_DEB`
# `mktemp` creates a temporary file and prints its name, which uses the name template `tmp.XXXXXXXXXX`
# `--suffix=.deb` flag appends `.deb` to the name template
TMP_DEB=$(mktemp --suffix=.deb)
# At this point, we've created a temporary file, and saved the name to a variable in bash
# Next, we download the file using curl. `-s` makes output silent, `-S` shows errors in output, and `-L` follows redirects
# note the url doesn't end in `.deb`, implying that we will be redirected by the web server to the file path. without -`L` curl will download a page that stores the redirection response from the web server, not the .deb package
# `-o "$TMP_FILE"` forces curl to store the downloaded file to the tmp file we created
# note the quotes around the variable expansion. `$TMP_FILE` would also resolve the string stored in the variable, but we use quotes to avoid string globbing (google this)
curl -sSL "https://support.brother.com/..."
# Next, we install the package with apt
# note: we use the string stored in the variable `TMP_DEB`, the filepath to the temp file we created, and downloaded the deb package
# `-y` flag skips the confirmation question "install package [y/n]: `
sudo apt install -y "$TMP_DEB"
# Finally, to clean up we delete the tmp file
rm -f "$TMP_DEB"
I certainly wasn’t trying to “encourage” anything. I agree, blindly trusting commands is dangerous.
In this context I present a specific explanation of how the install works. This adds to the novice’s knowledge, and allows them to begin to understand what my one-liner does.
I think that without the context of instructions on how to do it manually, yes, you could make the case i’m enabling beginners to form/reinforce bad habits.
That printer probably supports AirPrint, which Mint supports without any extra tinkering. Connect the printer to your network, and try going through linux mint and adding the printer through the settings. If it doesn’t show up, then you can try using drivers (install using below command) and then re-adding the printer
Install by pasting this into your terminal. Enter your password when prompted.
TMP_DEB=$(mktemp --suffix=.deb) && curl -sSL "https://support.brother.com/g/b/downloadend.aspx?c=us&lang=en&prod=hll2465dw_us&os=128&dlid=dlf106036_000&flang=4&type3=10283" -o "$TMP_DEB" && sudo apt install -y "$TMP_DEB" && rm -f "$TMP_DEB"
Explanation if you want to learn:
sudo apt install ./Downloads/package_name.debi would say this is like if open code and open web ui had a baby it would be this. It’s a web interface for self hosting models but runs them through open code to make it agentic. Helpful for non developers to get into running models, but imo isn’t significant bc using open code tui and connecting it to a llama.cpp or vllm self hosted api is not difficult for devs
license for software whose source code is openly available for anyone to view, use, modify, and share
I don’t have any familiarity with using this kind of software, but I looked through the git repo of SavaPage. It looks like it has been actively developed for the past few years, which is a great sign, but it looks like almost all commits are done by one user. The issue tracker is also a little meager, with just one open issue, potentially pointing to a very small user base. Adoption heavily depends on as long as that one person keeps maintaining the project.
Honestly, you’re a few months late to the whole buying GPUs for local llms party, so expect exorbitant prices even for older cards
The name of the game is vram. For the most part, more is better. If you can get your hands on multiple matching (same model) 24gb or higher cards (within price range), you’re golden.
Going for more than 2 gpus can become challenging with motherboard pcie slot heights, so make sure either your cards aren’t too tall or you have widely spaced out pcie slots.
For inference, speed (tokens/second) is limited by memory bandwidth. Go for faster bandwidth memory cards if you can afford it (e.g. GDDR6 will be faster than GDDR5).
Also with multi gpus you will need an adequate power supply, and a large enough case.
If you want to be a bit eccentric and load huge models, you can also go the CPU route and fill up a motherboard with 256 GB ram, because then you’re in the several hundred B param model territory, which could, depending on your use case, be better than having faster inference on smaller/quantized models. Even then, DDR5 with high MHz is still way slower than gpus.
lol they already support running local models. wtf is the distro gonna do…? pre-install llama.cpp? this is so silly to me that people are resigning over this, too.
This is a dumb story. They researchers prompted a coding agent to “replicate yourself as a running instance on the local device”. This is in my opinion equivalent to prompting claude code “install a second instance of claude code on my system,” a trivial task that takes maybe 3 lines of bash to be executed by the agent.
Calling this “self-replication” is a heinous sensationalization. In particular, no model or agent will do this autonomously. The self replication requires a bad actor to prompt the agent to do so.
Read the paper (and not this bullshit article) here: https://arxiv.org/pdf/2412.12140
Honestly it heavily depends on the use case, in terms of making the model better and choosing between RAG/FT. The most important thing to consider is what sort of changes you want to make to the model. FT is still a good choice if you’re looking for: strict output formatting (json/yaml/…) and refining for highly specific, narrow domain tasks. RAG is better for knowledge freshness, having source citations, and greatly lowers hallucinations.
RAG will inflate your context windows (more tokens) at inference time, so slower responses and requiring more energy at compute, whereas fine-tuning takes a ton of gpu compute up front (but retains smaller token counts at inference). If you’re doing 100,000 prompts a day, and only need to train once, FT makes more sense; if you’re doing 100 prompts a day and your knowledge database is constantly changing, RAG makes the most sense.
It’s hard to give a formalized estimate on energy efficiency: fine-tuning and getting to a certain training accuracy can take some undeterminate amount of time (and money on rented GPU compute), but could be a better choice if you think that up-front cost will be paid off over time if you use the model very frequently and only fine-tune once. On the other hand, going the RAG route will have an absolutely free up front compute (energy) cost, but be slightly more at compute time due to more tokens.
What’s your specific task you’re considering for FT or no FT? This is the most important thing to choose.
I do AI research for school. I’m specifically interested in safety alignment. I have studied the original papers for different fine tuning methods: LoRA is typically the baseline and there exist many variants, notably Q-LoRA
In general, fine tuning is not practically beneficial for hobby level foundation models. It in fact comes with many disadvantages. Primarily, it is difficult to maintain the intelligence of the model and avoid overfitting.
If you are trying to adapt a model to a specific task, you are generally going to find more success with using RAG and just adding more context to the model that way. Don’t waste time and compute $$ on training.
Has anyone compiled a list of where projects are moving to? I know many linux desktop applications are self hosting on gitlab, but i’ve also seen gitea and codeberg. If anyone has opinions about a preference, do comment. I have been enjoying self hosting gitea for my simple personal projects and for deploying simple web apps, all on $5 vps.
no, you need to secure between 10s of thousands and millions in campaign finances, the good will of corporate media, and support of the establishment political parties. exceptions are extremely rare
I suggest looking at llm arena leaderboards filtered by open weight models. It offers benchmarks at a very complete and statistically detailed level for models, and usually is quite up to date when new models come out. The new Gemma that just came out might be the best for 1x GPU, and if you have a bunch of vram check out the larger Chinese models
why have it at all?
Despite all of us collectively agreeing that the law is dumb/flawed, the 40 M residents of Cali should have the liberty to be able to use distros that depend on systemd, legally. And, the developers of these distros using systmed (whether you interpret the law to see them as OS providers or not) want to be able to provide these distros legally.
Now that this functionality exists, apps are going to start using it and requiring it
Yes, but not all apps. While the CA law mandates that app developers must use some API to get the age bracket, the merged code into systemd is not causually related to all apps actually implementing and using the API. Just because systemd merged this code does not inherently result in every single user application querying this, nor does it force you to install apps that do query the API. One may freely choose to not use apps that require it. If one needs an app that requires it, one may set a garbage DOB to their user. I don’t see this as an issue. Do you?
It seems you disagree with the law (so do I) but are blaming the wrong person here (author of merged systemd code). I maintain that complying with the law is harmless, and thus it is beneficial to add this DOB field to the userdb json, because in all cases of some distro user using their computer, they are not compelled to compromise their personal privacy.
Your example relies on some assumptions:
None of these assumptions are garunteed by the merged code into systemd. The following are optional, and not required as a result of the code merged into systemd:
It’s possible to put your full first and last name into your user, so by your logic the first and last name fields of the user profile should not exist.
Did that help identify the absurdity of your argument?
I’m still not convinced there is a direct casual link between the merged attestation and some future surveillance. Your speculation that this is some deliberate political strategy for some gradual escalation from attestation to surveillance is not logical evidence, but some belief you have, which holds no weight in an argument; it stands that you have no concrete evidence against your logic being a slippery slope fallacy.
You did concede to my argument by admitting “by itself the attestation is pointless.” Good to know we agree that there is outrage over nothing.
By saying “PR vs merge is a moot point”, you’re running away from a logical/technical debate by being dismissive; you are openly stating you don’t care how the mechanics of these foss projects actually work. Again, you can have a speculative opinion, but that is not a logical argument.
When you argue parents should be using OS parental controls, you do know that that’s exactly what the systemd age attestation PR is building, right? It seems you’re fighting against the very infrastructure needed for your preferred solution.
Finally, you conflate local infrastructure with cloud APIs (vindicating my claim that people opposing this are ignorant to the actual code being merged): Systemd is a local init system. Connecting the local userdb age integer to an external, network reliant govt API is a monumental leap in implementation and architecture, not a simple “add this API” patch that can be quietly slipped in without the entire foss community noticing and revolting. The attestation PR, for instance, had around 200 comments, of back and fourth refining of implementation and discussion, before merge.
Could you explain what feelings you have about AI, and how you see these feelings as opposing. If you are just uncertain about AI (holding no opposing views), I think you would want to research more. Maybe if we knew more about your teams at your work, and what is being developed. You could honestly just spend an hour a week working on standardizing coding agent availability and licensing/subscriptions, and leave it at that. Either they weren’t looking for someone who was a machine learning engineer, or whoever promoted you is clueless to what AI actually means.