SmackerNews

Ask HN: (Why) Was the LLM breakthrough useful for images, audio, etc.?

5 points · 5 comments · 4 days ago · rogerrogerr

news.ycombinator.com/item?id=49588265

This has been bothering me for a while. I feel like I have a decent conceptual grasp of what LLMs are doing with written text. But it seems like they also unlocked a bunch of progress in understanding and generating images, audio, and video. I can’t twist my brain into understanding the connection.

Is the boom in generated non-text content also built on LLMs, or is it just correlated with it because a bunch of excitement drove investment into the industry? I’m hoping for an ELI-non-ai-but-cs-major, this has been bothering me for a while.

jerlendds3 days ago
VLM != LLM. Vision language models basically treat text tokens and image tokens the same. Post-training an LLM on images+text can improve its capabilities. Id recommend searching around the keyword VLM to find more resources on how multi-modal AI works.
- https://huggingface.co/blog/vlms
- https://en.wikipedia.org/wiki/Multimodal_learning
verdverm3 days ago
transformers were first an image understanding technique, the text processing came later, it's all about the training data and gradient descent, and allegedly attention
kypro1 day ago
While some neural network architectures can be designed to better fit certain tasks they are at their core general-purpose learning algorithms which can approximate any target function.
I feel like I have a decent conceptual grasp of what LLMs are doing with written text
Just think of it as input data. In theory it shouldn't matter what each token represents. They could be xbox controller buttons, image pixels, or text.
The model with enough training data will map those inputs to an expected output.

news.ycombinator.com/item?id=49588265

SmackerNews

A placeholder title for loading skeleton

example.com/path/to/article

000 points · username · 0 hours ago

example.com

SmackerNews

Ask HN: (Why) Was the LLM breakthrough useful for images, audio, etc.?

5 points · 5 comments · 4 days ago · rogerrogerr

news.ycombinator.com/item?id=49588265

jerlendds3 days ago
VLM != LLM. Vision language models basically treat text tokens and image tokens the same. Post-training an LLM on images+text can improve its capabilities. Id recommend searching around the keyword VLM to find more resources on how multi-modal AI works.
- https://huggingface.co/blog/vlms
- https://en.wikipedia.org/wiki/Multimodal_learning
verdverm3 days ago
transformers were first an image understanding technique, the text processing came later, it's all about the training data and gradient descent, and allegedly attention
kypro1 day ago
While some neural network architectures can be designed to better fit certain tasks they are at their core general-purpose learning algorithms which can approximate any target function.
I feel like I have a decent conceptual grasp of what LLMs are doing with written text
Just think of it as input data. In theory it shouldn't matter what each token represents. They could be xbox controller buttons, image pixels, or text.
The model with enough training data will map those inputs to an expected output.

news.ycombinator.com/item?id=49588265