this post was submitted on 02 Dec 2024
384 points (99.0% liked)
Technology
60052 readers
2857 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Actually AMD is pretty okay for running LLMs and other ML workloads. Many libraries now explicitly target rocm, you can just plop down vllm or the llama.cpp server and have it work with big models out of the box. There are some major issues (like flash-attention), but its quite usable.
Intel though? Their software is a mess. You have to jump throigh all sorts of hoops, use ancient builds of pytorch, use their own quantizations and such to get anything working, fix Python errors, and forget about batched enterprise backends like vllm. And this is just their IGPs and Arc, forget trying to use the vaunted NPUs for anything.
This could change if they actually had a cheap 48GB GPU (or a big APU) for AI devs to target... But they don't. And no one is renting Gaudi to build in support because its not even availible anywhere.
EDIT: oh, and one weird thing is the volume of Intel software support is high. Like they have all sorts of cool libraries, they make contributions to open projects... But its all disjointed and fragmented. Like theres no leadership or unified push, just random efforts flailing around.
Exactly.
Intel is shooting itself in the foot by going halfway. If they want to compete in the AI space, they need to go all-in w/ a solid software and hardware combo. But they don't.
They have the capability, they're just not focused. A good CEO should be able to provide that focus. Maybe they should hire Lisa Su. ๐
Speaking as an holder of AMD stock since ot was $8, and an all AMD CPU user, IMO Lisa Su is either an absolute idiot or colliding with her cousin, the CEO of Nvidia.
All they had to do was lift vram restrictions on consumer GPUs (so their OEMs could double the VRAM up) and sick like four engineers on bugs blocking the AI space, and they would be dominating the AI space and eating Nvidia's pie...
And they didn't. Like, its two phonecalls, thats it.
Intel had monumental problems it has to solve and struggles, but AMD has tiny ones they inexplicably ignore. Its mind boggling.
I work in CV and I have to agree that AMD is kind of OK-ish at best there. The core DL libraries like torch will play nice with ROCm, but you don't have to look far to find third party libraries explicitly designed around CUDA or NVIDIA hardware in general. Some examples are the super popular OpenMMLab/mmcv framework, tiny-cuda-nn and nerfstudio for NeRFs, and Gaussian splatting. You could probably get these to work on ROCm with HIP but it's a lot more of a hassle than configuring them on CUDA.