Local AI may be a practical/useful option

Join the discussion
Registration is free. Ask a follow-up in this thread, or start your own.
Sign up Log in
4 replies · 553 views · last reply
AlexB23
Messages
312
Reaction score
324
TL;DR
IMO, these videos experiment with local AI and give impressive results
AI is a love hate relationship for me. I am not a fan of AI slop that is being posted online. At the same time, AI can be used for some tasks. It has helped me find recipes for food, identify plants, summarize meetings, and more.

But when people think of AI, ChatGPT or Claude comes to mind. Massive models run on the cloud. Nowadays, headway has been made for people to run AI on their computer hardware offline.

YouTuber Gary Explains ran AI on his own computer using Unsloth Desktop, a free and open source interface for chatting and training small to large AI models. And no, Unsloth did not pay me. This is just a cool piece of tech to discuss. Is open source good for AI, or not?



Apache 2.0 (free and open source):
https://github.com/unslothai/unsloth
 
Physics news on Phys.org

Discussion

This video experiments with local AI running on a modest PC with no internet connection. Although is it hardware limited, the results are impressive.


This video shows that a graphics board can improve the speed of local AI tremendously, from 8 minutes to 0.36 seconds in the test he tried.


More local AI experiments to follow.
Can this undercut the need for so many massive AI data centers?
 
Last edited:
Reply
  • Like
Likes   Reactions: AlexB23 and Paul Colby
Serving your own llms on your own hardware has been a thing in the community for as long as llms have been around!

With the improvement of GPUs, you can run reasonably sized models at home. A fp8 quantized ~20B model (billion parameters) should be very doable. It would be ~20gb in memory for the weights, but don't forget kv cache and activation overheead. Maybe you could go even bigger if your hardware is better or you're more clever about it!

Bigger models with smaller quantization tend to be better than smaller models with less quantization. ;)
 
FactChecker said:
TL;DR: IMO, these videos experiment with local AI and give impressive results

This video experiments with local AI running on a modest PC with no internet connection. Although is it hardware limited, the results are impressive.


This video shows that a graphics board can improve the speed of local AI tremendously, from 8 minutes to 0.36 seconds.


More local AI experiments to follow.
Can this undercut the need for so many massive AI data centers?

Been running local LLMs since March 2024. They have come a long way since then. This ocean temperature graph was made using Gemma-4-26B-A4B, a local AI that can run on a graphics card, Mac, or CPU.

By the way, use Unsloth Desktop as the GUI for running local AI, cos that is free and open source. LM Studio is free, but closed source.

Local LLMs still need to be trained in datacenters, but inference can be done at home. Once the model is released to the public, it can be run on your own CPU or GPU.
 
Attachments
  • Ocean Temperature Graph made by Gemma-4-26B-A4B (20260716).webp
    Ocean Temperature Graph made by Gemma-4-26B-A4B (20260716).webp
    105 KB · Views: 1
Matterwave said:
Serving your own llms on your own hardware has been a thing in the community for as long as llms have been around!

With the improvement of GPUs, you can run reasonably sized models at home. A fp8 quantized ~20B model (billion parameters) should be very doable. It would be ~20gb in memory for the weights, but don't forget kv cache and activation overheead. Maybe you could go even bigger if your hardware is better or you're more clever about it!

Bigger models with smaller quantization tend to be better than smaller models with less quantization. ;)
Thanks! I tried a Physics Forums search for threads about local AI and didn't see any, but my search technique might be bad. Those videos I posted were news to me.