Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?

It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of when "Just LLM it" will not be the right choice.

"Build an OS" wasn't a common university project because we were all expected to go out and work on Windows, but because understanding the bare-metal firmware for a computer helps you deeply understand how to intuit building for a whole class of problems.

I didn't do it because it was useful to me in a practical sense. It's because LLMs are fascinating and I want to know how they work. From that perspective it's been a great experience. I have afirm grasp of the basics. This makes it much easier to understand frontier concepts like compressed latent attention. I can follow the field and understand it.

Not sure I would have got as much out of it at seventeen. I have a lot of background and experience which made it much easier to learn. I wasn't struggling with the linear algebra or with python. I already knew pytorch and neural networks. That helped a lot and I covered these tutorials fast and could skip over large sections. A few evenings and the odd weekend day over a couple of months was enough for me.

For seventeen year olds the tutorials are good enough to make it possible to learn this but it would have taken a lot longer to understand. On the other hand I would have learned a lot more. I think I would have learned a lot of valuable stuff.

However I also think 17 year old me was studying for his A levels and probably this was right choice in terms of maximising future opportunities. I'm not sure I think learning about LLMs instead is sensible. Indeed it might be bad advice. But I can absolutely agree with the sentiment.I think 17 year old me would have wanted to do this too.

http://languagemodelbuilder.com teaches you (in a few hours to days) how to build an LLM from scratch. It's entirely free, without accounts, and without data collection.

Putting this in contrast with programming, I learned coding when I was 8, and it was incredibly stimulating to learn because you can quickly iterate and there were thousands of books and YouTube tutorials that dumb everything down and teach you fundamentals. All you needed was a $300 computer, and you can learn nearly anything you want, without being gatekept from this or that because you don't have enough vRAM / an sm_100 GPU.

There are other types of models like diffusion models right now that are showing more efficiency and have a higher ceiling for improvement. Understanding math and fundamentals are more important.

But of course, 10 years ago this wasn't obvious.

That said, I a good starting point for a 17yo is reading about perceptrons[0], then the basics of neural networks[1] (ex. 3-layer perceptron) then writing a program to train a 3-layer perceptron and classifying the MNIST dataset[2] - a dataset of characters.

This can anywhere between a day and a week and you will demystify the basics of neural networks and work your way forward with more advanced contemporary concepts.

Fun fact: any multi-layer perceptron neural net can be reduced to a 3-layer perceptron network.

[0] https://en.wikipedia.org/wiki/Perceptron [1] http://geeksforgeeks.org/deep-learning/neural-networks-a-beg... [2] https://www.kaggle.com/datasets/hojjatk/mnist-dataset/data

I’d probably say something like: do something you enjoy and seems like it might be useful, but accept that the pace of change may mean that whatever you study ends up being irrelevant.

Whatever solution there ends up being to this, it’s not going to be one that an individual 17 year old can implement. We’re past the point where individual good and bad choices matter that much to economic outcomes.

Being a super rich and an unhappy workaholic, or a super-impressive engineer who wakes up one day at 45 and realizes they regret wasting half their life (I ran into way too many of these) is a much worse fate than "not being rich from your startup" and working a relatively regular job while feeling fulfilled and happy by more than just work.

Especially in the US, which is uniquely bad at this and encourages people to work themselves to death, mental health and work life balance are much more valuable things for 17 year olds to focus on than finding good startup ideas.

In case you think i'm being a bit dramatic, let's look at the state of 17 year old mental health in the heart of Silicon Valley:

"The City of Palo Alto and the Palo Alto Unified School District approved a funded contract to place 24/7 human security guards and monitors at all four local Caltrain grade crossings, including the Churchill Avenue crossing directly adjacent to Palo Alto High School."

(in case it's not obvious, it's because of suicides by high school students)

The 17 year olds do not need advice on better startups, and this situation will never get better if we focus our advice on how to be better at work instead of how to be better at life. This will require redirecting the conversations.

I'd move to the middle of nowhere and work multiple jobs on a farm and in construction. Learn how to grow food, and build things. Meet the farmer's daughter, and marry her. Then, buy my own land, grow my own food, and build my own things.

Nothing wrong with learning the theory and understanding the papers. Getting to that point you’ll have to get your fundamentals down. Might be an interesting exercise.

But as a future? I guess we’ll see. I suspect the next financial apocalypse will determine if there is one. Another AI Winter that may outlast all others so far.

On an Amiga, I took various public domain text documents from cover disks and counted the probability of the next word given the previous word. Then spat out random sequences of words from it and printed them out. It was called "Splurge". Basically a very very simple single layer statistical language model.

Some of the sentences were randomly not bad sentences, which seemed amazing at the time!

That kind of thing (and Core Wars and Tierra etc) did lead me to getting a job at an artificial life startup at the end of the decade. But that was in turn about 10/15 years too early (no GPUs).

There's some lesson from this about timing, but honestly I've gained the most as a person when I did something that was fun, ethical and gained an audience. A tricky combination.

None of us know what the future of work, education, or AI is going to look like. But your best bet is to become a life-long learner. Be it LLMs, musical instruments, physics, or business.

Learning AI isn't like learning HTML in the 90s then expecting to get a job at a tech company building websites. You can't just "learn how to build LLMs" and expect a frontier lab to hire you so I'd argue this is rather bad advise.

Additionally, unlike web development in the 90s you cant really do anything interesting yourself... All of the interesting/useful stuff will require huge amounts of compute and data so there isn't even much point in learning to start your own thing either.

As someone whose built many of NNs from scratch (hand written code, long before the days of LLMs), it's more or less useless knowledge if I wanted to work in a frontier lab or do anything interesting in the field.

I also think anyone thinking about going into a field which is basically a crossover of CompSci and Maths is absolutely insane right now. Even if you think there is a place for CompSci and Maths post LLMs, there's almost no chance anything you learn today will be relevant to the skills required in say 5-10 years.

Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as this post implies?

(Edit: And learn how honest business works)

https://github.com/raiyanyahya/how-to-train-your-gpt

"Someone asked what I'd do if I were 17. I'd learn how to build LLMs from scratch..."

That's funny, Paul Graham, because if I were 17 again,

I'd learn how to program in LISP.

https://www.paulgraham.com/rootsoflisp.html

https://www.paulgraham.com/iflisp.html

https://www.paulgraham.com/hundred.html

(And/or other LISP derived languages... Clojure, Scheme, Racket, TinyScheme, etc.)

I guess "the grass is always greener..." as that old expression, that old "chestnut", goes... :-)

I'm much older and less wise now, but I still afforded myself the opportunity to follow karpathy's tutorials to build a LLM from scratch. Got to play with a few ideas. Seen similar ideas turn up in frontier model work, which is quite gratifying.

There are so many ideas to try.

Currently playing with autoencoders that takes A and B and produce latents A', B', and C'. Reconstruction of A is from A' and C', B is from B' and C'

The idea is if C' can be made to improve both outputs, it must store as much information as it can about what is common to both inputs.

Also many people/kids don't have access to proper "productive" systems anymore, since the whole computing and electronics industry shifted to make "consumer"-devices like smartphones or laptops made for netflix, gaming and spotify.

Breaking the barrier to build a custom system, install linux (or developer tools for Windows, MacOS) is already a complex AND costly task. It was just way simpler in the late 90s and 00s to get something working.

There is very little reason for humans to get all too engrossed in this type of work now, today, with the hope of being good enough at it to command a high salary in 3-5 years. AI can already do it incredibly well, and they can do it persistently and doggedly 24 hours a day.

And younger people I meet don't even own laptops. I had a genz/millennial cusp friend who wrote all her college papers on her iphone.

The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM company, and still then it's a struggle. Those few that do train, they spend most of their budget on compute and have relatively small teams.

Getting experience in this field requires having access to very expensive hardware to begin with. And the skills will be quite hard to convert into any real value for someone, leading to a decent income, unless you have a ton of funding from patient investors, or you have decent contacts in Bay Area networks to get hired at the right place.

With all due respect, paulg is in somewhat of a bubble, this is not congruent with the global situation.

Building an LLM from scratch has a hard split between a tutorial project you can complete in a weekend (that's useless for actual usage) and then a solid 1km high brick wall if you want to create anything actually useful from scratch.

Modified open models have a very active community around them, without the need to look much further than Hugging Face.

If you are really 17, my advice is to identify use-case for AI (ideally relevant for businesses) that work most of the time and find ways to make them work pretty much every time. AI reliability is the scarcity right now.

And it's basically a weekend project to put transformers together in a ML library and train it.

The follow up comment,train it to play a game also doesn't make sense? Llms Sony really play games and there are better ml approaches to do that?

Benjamin: Yes, sir.

Mr. McGuire: Are you listening?

Benjamin: Yes, I am.

Mr. McGuire: Plastics.

Benjamin: Exactly how do you mean?

Mr. McGuire: There's a great future in plastics. Think about it. Will you think about it?

---

I love this scene because it so perfectly captures what it's like to be young and given advice, however well-meaning, by an older generation living in a world that no longer exists for the young. And it's ambiguous and trite enough to be essentially useless even if the underlying idea isn't terrible.

Good times.

I wonder if you can dig that out of the historical reddit database. I'd like to see that again. I love how everything is recorded now.

For the AI part, I recently started working through this CMU course on Modern AI which teaches you how to build an LLM from scratch. It was posted on HN a few months ago and has been great: https://modernaicourse.org/

If anyone has other recommendations for beginners, please share them!

Long-term there's probably a lot more potential, and much bigger markets, in robotics than in LLMs, but it will really take the long term to get there.

I think this kind of mindset should be taught way more in school so that people really appreciate learning a subject deeply.

Notice that what I would not do is try to start a startup. Instead I'd build the foundation of knowledge to base a startup on later.

There's also this little-known concept called learning things for learning's sake and not always trying to capitalize on it.

As a platform engineer being based mainly out of Australia/Hong Kong, opportunities seem to be getting less unless targeting high frequency trading or banking.

It seems like building a startup with the help of some AI tools might be the best bet.

Second in line, build your own agent, that's more in our ballpark, then customize it to your needs, both virtual and physical

Previously they had a class where they'd build apps for other teachers. Like tracking when clubs are, etc. But now that's become easy for teachers to vibe code themselves.

I suggested they shouldn't prereq this class on CS. Heck invite anyone in interested in "building things" and they can get practice at building apps for other people / themselves. Maybe that becomes a gateway TO CS - people who want to learn how things work under the hood.

I suggested post CS class for the CS people should probably be building an LLM or something :)

Is it something that's going to be fundamental for future technologies? I always plan to learn, but end up in a death spiral feeling like I'll invest so much time and energy only for the puck to have moved somewhere completely different.

Would gladly accept any advice :)

If the goal is to understand LLMs deeply, one would be better served by either joining one of the big AI companies or doing a PhD. And to be honest, I think this journey should have been started 5 years ago, because right now there's too much competition.

Better to start working with harnesses, evals, statistical analysis, etc. - where you don't need the huge hardware for pre-training etc.

I mean first that is already what plenty of 17yo are actually doing, because that is what they do at school or in parascholar activities. There are already countless of such tutorials where you can do that in an afternoon.

The pointless part though is precisely why Amazon and others are hunting for rare books, all the low hanging fruits have been picked already so just training a bigger model will simply mean burning more energy and money. Sure training a small one for the basic principle is a great pedagogical thing, training another one, medium, then maybe a large one, is also good in term of learning the process and architecture, but one should not expect it to be useful out of that context.

Pure players are precisely doing everything they can to corner the market by making their own scale unreachable by others. Smaller players with access to lesser infrastructure are thus betting on different market, e.g. embedded systems.

17yos should definitely build their (L)LMs from scratch and whatever bigger model they can train for free, or for cheap, but they should not expect that to bring them any riches.

Whether that makes a good long-term investment is quite debatable. IMO, LeCun’s take (at the end) is far more forward-looking since it aims to gain more insight into what could come next based on what we know about the current state of the art.

All that being said, the extent to which these people capitalize on our tendency to be blinded by the halo effect is incredible. A constant stream of bite-sized aphorisms…

This is especially at a different level for early startup figures who happened to be in the right place at the right time and usually did little more than digitizing mundane, traditional day-to-day processes. Yet they’re treated as geniuses and prophets, with people hanging on their every word as though everything they say contains some deeper wisdom. PG and the like often strike me as broken clocks and they’re still profiting from having been very early players in the game, who had good instincts for commercialization and capitalism.

————

LeCun’s reply:

I would try to figure out why LLMs can write my essays but not clean my bedroom. Then I would study topics in college and grad school that could help solve that problem. I'll figure out a set of methods and architectures beyond LLMs that can quickly learn to perform physical tasks as efficiently as humans and animals. That last item is also what I would if I were 30, 40, 50, or 66 years old

I'm sure they'll get right on powering up their computer from the hamster wheel, Paul.

My heart goes to all the kids out there that didn't get the fair shake let alone fair access to tech that gets these condescending "learn to code/learn to LLM" bootstrappy talks from rich pricks that don't know what life really can be like for a lot of American kids out there.

The real value is to unlock meaningful insights and directions from existing data that is there inside companies.

I live far outside any tech city, so maybe I do not understand. But working with LLMs full-time, building for clients and tons of own experiments, I see no value in building on LLMs.

The big companies stole the data. The average person can’t do that

Someone asked what I'd do if I were 17. I'd learn how to build LLMs from scratch, and then train ones as powerful as I could with whatever hardware I could get access to.

He's seeing a future for models running on everyday hardware just capable enough to do what the use case requires.

I spent a days reviewing the lecture notes for CS336: Language Modeling from Scratch - and then trained a nanoGPT-esque model in PyTorch.

I'd recommend trying it for those who are curious. Computational bottlenecks become much more intuitive when you've looked at the overall process.

learning about llms is not useful so that you can make llms later, you want to learn about it so that you can work on next generation architectures. llms before long i imagine will be left in the dust by ebm / physics oriented models especially that can have an embodied understanding of the world. but a lot of things you learn about them are transferable by doing something like this

The core technoology is pretty basic, developing a rudimentary understanding for why the individual parts work as well as they do is tricky.

I don’t see why learning how LLMs work is a bad project for a 17 year old.

Optimizing your entire career and the next decade+ of your life on LLMs? Yeah, probably not ideal. It’s almost always a bad idea to make long term decisions based on current trendy things.

And since everyone is using this topic to give their ideal advice to 17 year olds, my advice as a mid-30s guy: seriously consider becoming highly skilled at a specific thing, and don’t be scared off by the idea that it’ll take 5-10-15 years to get there.

When you’re 17-25, the timescale of a decade seems infinite. But it’s really not, and a decade spent “exploring and keeping your options open” sometimes just ends up with you being pretty decent but not amazing at a lot of random things.

Sometimes I wish I had just become a carpenter, chef, electrician, etc. – a specific skill set that leads to mastery over time, rather than the endless exciting-new-thing hamster wheel of working in tech.

if i were 17, i'd do these things:

1) LC until mediums

2) Calculus & linear algebra even if i understood nothing i'd just stare at vectors and derivatives.

(Besides the obvious nano gpt)

Even worse when they ask themselves.

I have my opinion on this but I'd like to hear the HN opinion, I will just say one thing:

If you are starting with little knowledge, like a 17 year old would, letting an LLM explain it to you is a terrible idea.

Wealth inequality is a huge factor in our ability to excel.

"Why would you reinvent the wheel rather than making it better?"

"To learn how wheels are made"

Banger reply.

My first read was "if you could redo your life from age 17 what would you do". To which "figure out how to make an LLM" would be an insane answer.

I think his advice here is maybe... a year or two too late to be good advice. I can't foresee the job market for machine learning experts being better than it is now in 5-6 years time when said hypothetical 17 year old would be most ready to start career hunting. Either the bubble is gonna pop and the market will be flooded with laid off AI talent.

Even if I'm wrong about there being a bubble at all, I still think that in 5 years time the tech will just have matured to the point of diminishing returns on refining existing architectures. Plateauing until some PHD comes up with something as ground breaking as attention.

I’m sure I’ll get torn to pieces for this but it’s frustrating to continually witness people treat a single person’s prose as the Word of God.

Learn how to make language models from scratch, yes. But learn how to use them, in the context of other machine learning tools, on very small hardware.

When the bubble bursts (and I still tend towards thinking it could burst rather than be deflated in a manageable way), the focus will be on uses of AI that are not like the hyperscalars' products.

People will still be interested in useful AI being added to small things — assistive technologies, home security, garden monitoring, their phones and smartwatches, robotics.

Instead of reductive, reactive make-an-anthropic-competitor advice like this, what about advising 17 year olds to focus on broad, integrated, helpful AI — or on going back through eighty years of history to look at AI projects that failed and reassess them?

Wait no it’s not, that was always happening.

What’s crazy is that people still believe in it.

I will never understand this mentality, to let yourself be so dependent on something you don't understand at all is to live like a child. But it is very common. I doubt too many 17 year olds will bother even trying to understand what an LLM is, let alone build one from scratch.

If the two current bottlenecks, for this LLM madness that could very well be a bubble, are processing capacity and accuracy (a second processing problem) then what comes next? Isn’t that where young people should be looking or are we just giving up on innovation?

If you are 17, go be yourself, whatever that is, in whatever way you want that to be, but do it so authentically and fully. Be unapologetic about what you love and what motivates you, and pursue that with passion and commitment.

Owner of Golf Club Company says I should dedicate my life to golf lmfao.

He capitalizes on greed and hype but with a soft, sober and thoughtful voice so as to lull you with rationalism and now 20 years of his “disruption” has mostly ruined modern society and a whole generation of techies have been led astray into trying to “change the world” is the world of today (minus the magic technology really any better than 20 years ago?)

- Good for him and his Tech Bros, bad for the rest of society

the lessons have svg diagrams, ai chat inline (google docs), etc

apologies if this comes off as slop, but its been working great for me.

I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into mature-on-the-way-to-dying tech (LLMs) I'd rather focus on fundamentals - good ols linear models, regressions, stat etc.

But sure, make the kids even more depressed by telling them they need to learn how to build an LLM so they can get a job working themselves to death to make Paul and friends rich.

I don't think you can really call yourself a developer unless you at least have an idea how to build a more complex software project like a compiler, and maybe have built a toy one either at uni or for fun.

It's not clear how long this LLM age of AI will last (to be replaced by something better), but nowadays any developer should at least understand the basics of ANNs, and more than just the "hello world" of a cat vs dog CNN. An LLM/Transformer is maybe the equivalent of a compiler in that regard - something that we all use and is complex enough to present a bit of a challenge. You should at least understand the basics of how an LLM is built, and maybe building a toy LLM will/should become the new Comp. Sci. degree toy compiler replacement.