I spent bits of today watching Youtube videos about linear regression. One of the best ones I found was this StatQuest video on the topic. I enjoyed his style so much, I ordered his introductory book on Machine Learning. I’m planning a digital fast over a long weekend in the second quarter of the year, and I’ll take the book with me to continue learning offline.
Today I learnt about a new concept: Overfitting.
Key takeaways today:
Overfitting: This is when a machine learning model fits too strictly to the training data that it was trained on. So much so, that the model doesn’t generalize well to data outside of the training set (i.e. new data.)
Why does overfitting happen? A few drivers:
Small training data set — limited training examples might throw up patterns that don’t account for the true distribution of the data.
High model complexity — having too many parameters might throw up noise or patterns that are not useful;
Biased data — this could throw up patterns that are not generalizable across wider a population of data.
Today I’ve started a “100 days of AI” challenge. I’ve used LLMs across a few personal projects here, here, and here. But I’d like to understand the basics of AI and machine learning a bit better.
My motive isn’t to retrain as a ML engineer or data scientist. Instead, I want to challenge myself to go beyond a superficial understanding of one of the greatest technology advancements of our time, particularly given my work in technology investing.
Every day, I’ll aim to do anything from 10 to 90 mins of learning, experiments, and review. Here we go!
Day 1: Overview & Learnings
I’m working my way through an introductory machine learning course by IBM, which provides an AI Engineering Professional Certificate at the end.
I’m about 25% of the way through it, and already, we got to build a simple linear regression model that predicts CO2 emissions based on engine size. I’ve put the GitHub code for this here. Most of the action comes from the code snippet below.
Key takeaways today:
Artificial intelligence (AI) is a broad definition that covers computer system that performs intelligent human-like functions.
Machine learning is a subset of AI that uses statistical techniques to learn from data and infers patterns and makes predictions.
Deep learning is a subset of machine learning and it uses neural networks (inspired by the brain structure) to learn from data.
I’m going to be looking at mostly machine learning for now. This can be split across supervised learning (where you provide data that’s ‘labelled’) and unsupervised learning (where the data is ‘unlabelled’).
Popular techniques for ML models include regression and classification (in supervised learning), and clustering and association (in supervised learning).
There are a number of open source libraries that make it easier to prepare and build machine learning models. The key ones I’ll probably use a lot are NumPy, SciPy, Pandas, and Scikit learn.
Many concepts feel foreign to me at the moment, but as I spin up a couple of projects and work through tutorials, I should start to get the basics down!
After the OpenAI developer event yesterday investor Harry Stebbings tweeted, “Holy shit. Can you imagine a GPT where you could ask any question and it uses advice from 3,000 20VC episodes to answer your questions from the best VCs in the world.” That possibility is in fact already a reality. Here’s how I spun up a working prototype rapidly. Demo videos below.
Some background: I built something similar six months ago with a different dataset. However, that process took several hours over a few evenings. Today, you can make custom bots in minutes. Let’s walk through the broad steps I took for the 20VC bot.
First, I downloaded a sample of 60 episodes from the 20VC podcast. I then used AssemblyAI’s API to transcribe the MP3s in a big batch. You could also use Whisper, which is cheaper and perhaps even faster. I went for AssemblyAI instead because of familiarity and a need to prototype quickly.
The next step was to convert these transcripts into a database that GPT4 could use. For that, I used Retool — a platform that lets you drag and drop files into a database of embeddings that language models understand. Retool also provides chatbot interfaces you can use right off the shelf. And voila! I had a bot that could query 60 episodes of 20VC for knowledge and advice.
To create a full version of the 20VC bot you would need all 3,000 episodes and a reasonable budget for large language model services. This process will rack up a bill in the hundreds (maybe thousands) of dollars, but it’s small change for a media or investment business.
To Harry and his team, I hope this demo shows what’s possible. Even without the upcoming OpenAI feature that enables anyone to create their own GPT, you can build custom GPTs already and with impressive speed.
I’ve been doing a lot of coffee meetings recently and often, I need to find venues—mostly cafes—that are convenient for both parties. Using Google maps for this can be cumbersome, and so I thought, why not ask ChatGPT-4 to write a simple program that could solve this problem? The app could take two locations and automatically identify a list of coffee shops located roughly halfway between the two addresses.
To my surprise, and in less than 30 minutes of working with GPT and the Python programming language, I had working code. Here’s the output of that initial process.
Wrote this scrappy little app in < 30 mins after dinner thanks to GPT4 help . Now if I have a coffee plans with a friend and want to meet somewhere that's about equally distant for both parties, I punch in the post codes and it maps out the options for me. pic.twitter.com/gzAl91GP8V
While this code was enough for me, it wasn’t user-friendly for non-technical people. So I went back and forth with ChatGPT-4 through a bunch of queries until I cobbled together a web app that anyone could use. You can see how that turned out in this demo video.
Feel free to try out the app here, but only for a short while before my Google Maps API budget is depleted.
What did I learn from this?
Going through this process of iteration and collaboration with AI was fun, but it also drove home a point that most tech-savvy people are already familiar with: AI can write code that works, but it’s not a full-on substitute for a good software developer. (This means now is still a good time to learn to code!)
Deploying even the simplest of apps involves a maze of tools and systems. It’s not just about the code. In my case, I had to set up a Google developers’ account to be able to use their maps technology. (This involved going through their documentation when the GPT-written code turned out to be out of date!) I also had to research and debate the merits of various hosting providers for the app before deciding which one to use. Additionally, I had to buy a domain name and link it to my servers. And then of course, I couldn’t forget the basics, like setting up analytics and regularly backing up the app code on Github, among other steps.
Of course people who do this work daily find it trivial in a technical sense. However, even seasoned software developers grumble about how time-consuming it is to get all these tools and platforms working together for a public-facing app.
Simply put, you can’t fully automate the process of building things that will be used in the real world by real people. We’re not there yet. But what tools like GPT can do is speed up your prototyping process. Furthermore, if you have a touch of technical know-how, you can quickly automate a variety of personal tasks that don’t need to be public or require a full-fledged app. To me, that’s enough reason to be optimistic about how generative AI will meaningfully impact global productivity in the years to come.
Personal branding — assuming brand can even be applied to a complex human being in the same way it can to a commodity — is mostly a by-product of something else: making valuable and meaningful contributions in your areas of interest.
People who tirelessly and directly work on personal brand often do so at the expense of other activities that matter. And at the extreme end, there are some who go as far as fraud just to build a name for themselves. (See this list of fraudulent Forbes 30 under 30 candidates for example).
Paradoxically, and as this paper about the ‘Best-New-Artist Grammy Nomination Curse’ puts it, if you seek recognition directly, you probably won’t do your best work. That’s because people-pleasing antics and unrealistic versions of success are a distraction from what really matters. In contrast, if you care less about public opinion and awards, you may find yourself producing better and more original work.
For these reasons I’m not sold on the idea of spending lots of time on a “personal brand”, especially if it precedes any meaningful contributions from an individual. Attempts to establish a personal brand without genuine achievement are, at best, fruitless busy work and, at worst, delusional.
Only a select few can pull off a personal brand. It emerges in its strongest form after incredible achievement. Beyonce, Michael Jordan, Steve Jobs: each of these individuals has a great personal brand. But in all cases their mastery of craft preceded mastery of image.
Chances are, you won’t tread the same path as Bey, Mike, or Steve. That’s okay. It means you can skip the personal branding frenzy. Instead, focus on doing exceptional work and contribute meaningfully in your areas of interest. Your reputation will naturally grow and you won’t have to rely on a contrived personal image to open doors. Let the superstars have their personal branding. For the rest of us, there are more impactful ways to invest our time.
I’ve been reading Fred Wilson’s blog for almost a decade and his writing inspired my move into venture capital some years back. He’s a seasoned early-stage investor and a co-founder of Union Square Ventures, one of the best venture capital firms of all time. (They invested in the likes of Stripe, Twitter, and Coinbase).
I’m still reading through each and every one of Fred’s blog posts because there’s a ton of early-stage investing knowledge in it, and his writing is such a delight to read. But as a fan of his blog I also wanted to create an interactive way of traversing his knowledge base.
ChatVC: An AI bot that uses avc.com blog posts.
Enter GPT, Langchain, and Chromadb. These tools helped me quickly build a prototype chatbot that I could ask questions about early-stage investing. Below are some gif examples of the answers I can get from the bot. I also tweeted about it here. How did I build it?
How did I build it?
There were broadly three steps. First I used an open source Python library called BeautifulSoup to extract all the text from https://avc.com/. Second, I created an AI-native database of that text using Chroma. This part is really cool because Chroma stores the text in a highly multi-dimensional space where related concepts can be found easily. Finally, I used Langchain to connect to OpenAI’s GPT model for chat and language capabilities. (The user interface is powered by Gradio.)
All this sounds super technical for non-coders, but I’m not a software engineer by the way. I’ve completed online coding classes before and I know a few basic programming principles, but it’s not my area of expertise. I built this VC AI by using open source tools and asking ChatGPT for a lot of help!
Implications for Entrepreneurs
If I can build something like this over a few evenings, and the technology that enables people to do it is getting simpler and more accessible, we’re going to see a massive rise in the number of people who can create software tools and apps. And one of the things I’m excited about here is that entrepreneurs who previously weren’t technical enough to start a tech company can do so without the hurdle of having to find a technical co-founder. That partner could be an AI.
Implications for Investors
I’ve built a prototype investor chatbot for personal use and learning but I don’t believe it can displace any seasoned investors, yet. Coincidentally, just as I was finishing the prototype, Fred Wilson wrote about this same topic yesterday and he makes a number of points that I agree with.
The thing is, there are facts, there’s knowledge, and then there’s wisdom. Everyone has cheap access to facts. Wikipedia does a wonderful job of that. Many people also accumulate knowledge with time and some expense – i.e. the know-how and nuance of applying different facts and ideas in a specialist area. AI is now getting really good at this.
The final element is wisdom, and that’s hard-won. It’s very human. It involves living through an experience and internalising the whys and counterfactuals of what happened and what could have happened. I can’t just read a blog post from Fred Wilson or ask an AI about how to deal with a situation and all of a sudden become a wise person. That takes significant time, reflection, and tangible practical experience.
AI might become wise some day, but from what I’ve built and learnt about large language models so far, it will be exceptionally difficult to replicate that. I remain open to the idea that this could change quickly if some new innovation in AI emerges. For now though, I don’t think investor jobs are going to be displaced by bots. Investing will, however, be augmented by AI.
The excitement that surrounded the personal computer revolution in the 1980s and the advent of the world wide web in the 1990s has surely been eclipsed by what’s going on in artificial intelligence today.
Bill Gates – who was around for both eras – believes that the “development of AI is as fundamental as the creation of the microprocessor, the personal computer, the Internet, and the mobile phone.”
The Economist and others go further. They liken what’s coming with AI to the impact the printing press ignited 600 years ago, when a new general purpose technology led to an explosion of knowledge and productivity, as well as widespread upheavals and disruptive social change.
ChatGPT – the fastest-growing internet app ever – and the large language model that powers it are the fuel behind the excitement. The app is unlike anything we’ve ever seen. It’s so good at answering a wide variety of questions that it feels as if you’re chatting to the collective knowledge of humanity (or at least the publicly available text the AI was trained on.)
Note: ChatGPT is the fastest growing app of all time.
I’ve worked in tech as a “non-techy” person on the investment and operational side of things for several years now, and I get to meet technologists often. I also often seek out and synthesize non-mainstream content about technology from a diverse group of friends, Twitter accounts, forums, and niche blogs. Never in this time have I felt a greater sense of technology acceleration than now.
From AI researchers highlighting that “2023 has already seen more advances in AI than any other year,” to hearing from software developers about how thrilled they are to build on this new technology, and how some are terrified of it (the potential near term challenges and long run risks should not be ignored), the path ahead is going to be full of surprises.
Note: Computers that train AI models are doubling in power every 6 months. This means a 1000x increase in power every 5 years if the trajectory continues. [Chart by Sevilla et. al 2022 and adapted by Korinek 2023.]
How should we prepare? I’m not entirely sure since I’m also just coming to terms with what’s happening. However, I’ve adopted the technology early (I first spoke about GPT on a BBC show back in 2020); I’m exploring how I can invest in entrepreneurs who are building AI tools that will get us to the future more safely; and I want to learn and do more of what makes us positively and uniquely human.
My hope is that we end up in a future where AI helps us solve some of the world’s biggest problems, rather than make them worse. But for that to happen, a lot more of the public will need to engage the topic today.
University has never been more expensive, so is it still worth it if it will cost new students £100,000 in graduate debt and take 40 years to repay?
Recently we’ve seen the cost of everyday items skyrocket. From pasta prices to fuel and energy bills hitting record highs. But while the cost-of-living crisis is making headlines, a lesser-told story is the fact that the average graduate today ends up with almost three times as much student debt as they did 10 years ago.
In addition, this cost-of-education crisis is expected to balloon further. The upcoming changes to the student loan system could see graduates pay back £100,000 of debt and interest over their working lives.
With such a huge price tag, you would expect university to still be the superior option. However, the research I conducted for my book, Is Going to Uni Worth it?, showed that there are many cases where an alternative, such as an apprenticeship, could be a better option.
How can you determine what’s likely to work for you? There are five key areas detailed in the book, but answering the three questions below can act as a starting guide.
1. What do you want to do in the future?
If you know what you’d like to do in the future and that path requires a degree, university is an obvious choice. For example an aspiring astronomer or biologist must complete a degree. In contrast, you don’t need a degree to work as a journalist, accountant, or banker. These careers can be pursued via an apprenticeship.
2. What’s your learning preference?
University emphasises academics (i.e. lectures, reading assignments) while an apprenticeship focusses on the practical applications of knowledge. Practical learners are therefore better served by an apprenticeship, while conventional academia is better pursued at university.
3. What’s your affordability consideration?
Some university courses can be expensive. For example, just 4% of doctors come from a working-class background, and part of the reason for this is that many graduates of medicine accumulate £80,000 or more of student debt and often require additional financial support from their family to complete their training.
Thankfully, from 2023 it will be possible to train and qualify as a doctor by taking a degree apprenticeship, which doesn’t come with the student debt of a traditional path. So if you wished to be a doctor and the cost was out of reach, a degree apprenticeship would be a no-brainer.
These three questions are just the start when it comes to figuring out whether university is for you or not. But — as is the case with all interesting decisions — there’s no perfect answer, just one that’s good enough and hopefully worth it for you.
The world emits about 50 billion tonnes of greenhouse gases annually. In contrast, the average UK citizen contributes roughly 10 tonnes each year to the grand total. This represents one part in every five billion parts of global emissions. It’s one fifty-millionth of a percent (0.00000002%). In other words, it’s a rounding error and practically negligible, according to some people.
Given how tiny our personal carbon footprints are, it’s easy to dismiss individual efforts. It’s also common to feel despair about how much influence you can really have on climate change – after all, what meaningful difference can a ‘one fifty-millionth percenter’ make in the grand scheme of things?
In recent months I’ve come to appreciate that our personal action matters more than we realise. In fact if you combine a lower personal carbon footprint with climate activism, the impact you can have is hugely meaningful. Here’s how I see it.
Every tonne of CO2 matters
First, let’s bring to life what 1 tonne of CO2 really looks like. This is roughly the amount someone in the UK emits every 5 weeks. Below you’ll see how much volume you can fill up with just 1 tonne of CO2.
Now imagine dumping one of these cubes or balls outside your home every five weeks for 60+ years. Would it be something you’d ignore? Is it really that negligible? Or, does this volume of CO2 actually count and would you be compelled to do something about?
What you do is magnified negatively or positively
Our collective impact on the planet is unquestionable. We already haveirreversible climate damage and events such as the recent 1-in-a-1,000-year heatwave in Canada – which was 150 times less likely to happen without human influence – are a testament to this. But surely our individual action is so tiny that there’s no point trying, right? This statement is wrong on two accounts.
1.Negative Knock-on Effects
Although the world’s climate is a complex system and it’s impossible to know what specific bit of pollution could lead to catastrophe, all excessive emissions play a part in destabilising the system.
Consider a game of Jenga as a rough example. At first, you can remove several blocks without affecting the overall structure that much. But as you remove one block after the other, you eventually get to a point where a small change is followed by the entire system collapsing.
In the Jenga example, small changes appear harmless until they aren’t. This is also the case with climate. Someone’s pollution seems harmless but in the background, it’s linked to a collective instability that each one of us—to varying degrees of course—is responsible for. So although one person’s footprint will always appear negligible in isolation, its systematic influence – which accumulates as time goes on – isn’t. Tiny changes can drive harmful butterfly effects.
2. Positive Knock-on Effects
The things you do about climate can also have wider positive knock-on effects. For example if you decide to live an eco-friendly life, your friends will take notice and ask about it. This won’t necessarily turn them into climate activists, but in learning about your choices, your friends (and their friends, too) might consider doing something good for the planet.
You can also multiply your impact through activism. And when I say activism, I don’t just mean just going out on the streets and campaigning – although this is certainly a proven means of enacting change. You can be a climate activist in other ways.
For instance, you can amplify your impact when you vote for and support politicians who are serious about climate change; you can influence sustainable investments with your wallet when you spend money with climate-positive businesses; and finally, you may also choose to put your talents and time to work at an employer that’s aligned with a greener future, just like I did when I joined tickr.
~
The challenges of climate change are undeniably epic. Governments and corporations will have to work hard together at a global level to avert disaster. However, what’s clear to me now is that there’s no reason to feel despair about how much you can do on personal level.
All our carbon footprints count. Who we vote for and where we spend our money and time all matters. And although our individual actions might not count for much in isolation, the reality is that they often go further than we realise – certainly much further than our ‘one fifty-millionth of a percent’ would suggest.
How will governments and society respond to a global climate disaster? Kim Stanley Robinson paints a vivid picture of what’s possible in a brilliant work of fiction that straddles science, economics, and politics. In The Ministry for the Future, Robinson takes the reader to a near future and lays out in impressive detail the impact of climate change at a global and personal level.
Though the book is a work of fiction, it’s grounded in sufficient reality to bring key issues close to home. Take the geopolitical challenges of climate change as an example. We are familiar with how difficult international relations can be in times of crisis. But with climate change, the dynamics will be incredibly harder. Here’s one area in the book where we see tensions flare up, with an Indian government official expressing anger at what’s happened to her country:
“What you need to know now is that we are scared here, and angry too. It was Europe and America and China who caused this heat wave, not us. I know we have burned a lot of coal in the last few decades, but it’s nothing compared to the West. And yet we signed the Agreement to do our part. Which we have done. But no one else is fulfilling commitments, no one is paying the developing nations, and now we have this heat wave.”
Another area of realism was the frantic attempts at geo-engineering. Messing with climate in this fashion could backfire in all sorts of ways. However, it’s realistic to imagine that it may be necessary if we don’t cut emissions and have no other choice but to try anything with a non-zero chance of success. With such bleak prospects the characters in the book can’t help but occasionally turn to humour. One passing thought near the end of the book exemplifies this well:
“Geoengineering? Yes. Ugly? Very much so. Dangerous? Possibly. Necessary? Yes. Or put it this way; the international community had decided through their international treaty system to do it. Yet another intervention, yet another experiment in managing the Earth system, in finessing Gaia. Geobegging.”
I can’t share more from the book without spoiling the story but I can leave you with the last thought I had after reading it. And that thought is this: The fossil fuel-driven economic growth of the last century may end up being the greatest and most harmful speculative bubble of all time. Which is to say, we’re enjoying an inflated sense of progress today that’s inconsistent with the unimaginable costs that could be due in the future. However, if the quasi-fictional planet in Robinson’s book is anything to go by, taking drastic measures today gives us a chance. But delayed action dooms us to failure.