Michael Tefula

Decision Trees. I went through the intro to ML material on decision trees this morning.

I created some data in Excel and a simple decision tree machine learning model that predicts whether someone works in tech (value 1) or not (value 0). The data is synthetic, and I shaped it so that the ‘salary’ feature was the key predictor instead of age, sex, and region. Here’s the output tree.

My model had an accuracy of 77%. The confusion matrix below provides more on performance. For example, the prediction ‘precision’ of whether someone works in tech roles was quite good, at 85.7% (12 true positives and 2 false positives out of 14 positive predictions).

However, the model was less good at identifying people who don’t work in tech, with a true negative rate of 68.8% (11 true negatives out of 16 negative predictions, including 5 cases where non-tech job cases were mistakenly identified as tech).

Key takeaways:

• Decision trees are a supervised machine learning technique for classifying data (via classification trees) or predicting numeric values (via regression trees).
• As you can see from the first chart, decision trees are good because they are more easily interpretable, and you can follow the steps to know how a classification was made. Cases where this is useful include:
  • Finance situations, e.g. Loan application decisions, investment decisions
  • Healthcare situations, e.g. Diagnosis by going through symptoms and other features
  • Marketing, e.g. Customer segmentation via some attributes, churn prediction.
• How you create decision trees?
  • The easiest thing to do is to use a Python library that does this for you. Here are some simple examples I did in Python.
  • Otherwise, the general process revolves mainly around feature selection. This is as follows:
    • Start at the root node.
    • Find the best feature that splits the data according to a metric, such as ‘information gain’ or ‘gini impurity’. You do this by going through all the features and splitting your data according to each one in isolation to see how well the data is split. Once you find the best feature, you build a branch with that feature.
    • You then repeat the above process, but below the previous branch and with a subset of data.
    • You keep going until you’re happy with the depth (note that if you go too deep, you might have issues with overfitting).

Yesterday, I skimmed through k-fold cross-validation. I wanted to return to it today because I still wasn’t sure about what exactly we are assessing with it, and why.

Here‘s the clearest Youtube video I found on the topic. The screenshot below from that video provides an easy summary.

Key takeaways:

• K-fold cross validation is a method of evaluating the performance of a specific model configuration.
  • Model configuration includes the choice of an algorithm (e.g. linear regressions, decision trees, KNNs, neural networks), feature selection (e.g. the independent variables used to predict the dependent variable), and hyperparamters (e.g. the number of neurons in a neural network.)
• To perform K-fold validation you,
  • (1) split the data into ‘k’ equal parts (the “folds”);
  • (2) you train k-number of models using your chosen ‘model configuration’ across the k-number of folds; for example in the image above, model 1 is trained on folds 2-5 and tested on fold 1; model 2 is trained on folds 1 and 3-5, and tested on fold 2 and so forth;
  • (3) you then assess the performance of each model across the k-folds and aggregate this performance measure, usually using an average (or median, depending on the data type);
  • (4) the final accuracy tells you how good your model configuration is.
• If your aggregated accuracy is strong, you can then train your model on the whole data set.
• K-fold cross validation is useful in situations where you have a small amount of data and a small test set. In those instances, the opportunity to test your model is limited, and k-fold can boost that, allowing you to test a model configuration across a larger number of scenarios.

Today I learnt about cross validation (scikit-learn has Python helper functions for this.) This is where you split your training and testing data into different sets, and then you iteratively train and test against different combinations to assess how well a model performs on unseen data.

Two common methods of cross validation are k-fold validation and leave-one-out cross validation.

K-fold involves splitting the data into equal parts and rotating across them in terms of testing and training.

With leave-one-out cross validation, you select one data point for testing, and use the rest for training. You then move to another data point for testing, and then use the remainder for training, and so forth, until every data point has been used for testing.

I’ll return to these concepts when I write some more code next week.

Ps. One thing I’m realising as I dig into the basics of machine learning is that it’s a mix of art and science in terms of choosing techniques that may produce the best models. There’s a bunch of trial and error, even though it’s a deeply rigorous and mathematical field.

Today, I went through a classifier lab on the intro ML course. There are several bits I didn’t quite understand but GPT helped get me over the basics. For example, I will need to review my notes on the Jaccard Index and F1-score (evaluation metrics for classifier models), and the concept of normalisation, where you transform your data without changing its distribution. This makes it easier to calculate distances between points, a critical bit when trying to make classification predictions.

On the latter point, I’ve included some charting code in the github repo here (see image below), which helped me understand the normalisation concept. The charting code was written by GPT, with some minor tweaks from me.

Key takeaways:

• Classification is a supervised machine learning approach.
• It makes a prediction about what discrete class some item should fall into.
• Classifiers can be used for spam detection, document classification, speech recognition, or even to predict if a certain customer will churn, based on a variety of characteristics.
• Classification algorithms include k-nearest neighbour (which I’ve put on github here), decision trees, and logistic regression (which instead of putting an item into a class, gives you a probability that it will fit a particular bucket.)
• The K-nearest neighbours algorithm was fun to learn about, and the intuition for it is simpler than I expected. The basic notion is as follows: for a given item to predict on, look at a select number of neigbhours (the k-number), and predict the outcome based on the most popular category that those neighbours are in (or the neighbours’ mean or median of the values you’re trying to predict for e.g. house price based on location, square foot size etc.)
• Classification algorithms can be evaluated with a number of accuracy measures, such as the Jaccard Index, a F1-score, or Log Loss. I didn’t cover these in detail but I did enough to get the very basics.

I spent bits of today watching Youtube videos about linear regression. One of the best ones I found was this StatQuest video on the topic. I enjoyed his style so much, I ordered his introductory book on Machine Learning. I’m planning a digital fast over a long weekend in the second quarter of the year, and I’ll take the book with me to continue learning offline.

Today I learnt about a new concept: Overfitting.

Key takeaways today:

• Overfitting: This is when a machine learning model fits too strictly to the training data that it was trained on. So much so, that the model doesn’t generalize well to data outside of the training set (i.e. new data.)
• Why does overfitting happen? A few drivers:
  • Small training data set — limited training examples might throw up patterns that don’t account for the true distribution of the data.
  • High model complexity — having too many parameters might throw up noise or patterns that are not useful;
  • Biased data — this could throw up patterns that are not generalizable across wider a population of data.

Today I’ve started a “100 days of AI” challenge. I’ve used LLMs across a few personal projects here, here, and here. But I’d like to understand the basics of AI and machine learning a bit better.

My motive isn’t to retrain as a ML engineer or data scientist. Instead, I want to challenge myself to go beyond a superficial understanding of one of the greatest technology advancements of our time, particularly given my work in technology investing.

Every day, I’ll aim to do anything from 10 to 90 mins of learning, experiments, and review. Here we go!

Day 1: Overview & Learnings

I’m working my way through an introductory machine learning course by IBM, which provides an AI Engineering Professional Certificate at the end.

I’m about 25% of the way through it, and already, we got to build a simple linear regression model that predicts CO2 emissions based on engine size. I’ve put the GitHub code for this here. Most of the action comes from the code snippet below.

Key takeaways today:

• Artificial intelligence (AI) is a broad definition that covers computer system that performs intelligent human-like functions.
  • Machine learning is a subset of AI that uses statistical techniques to learn from data and infers patterns and makes predictions.
  • Deep learning is a subset of machine learning and it uses neural networks (inspired by the brain structure) to learn from data.
  • There’s a good explainer of all this here.

• I’m going to be looking at mostly machine learning for now. This can be split across supervised learning (where you provide data that’s ‘labelled’) and unsupervised learning (where the data is ‘unlabelled’).
• Popular techniques for ML models include regression and classification (in supervised learning), and clustering and association (in supervised learning).
• There are a number of open source libraries that make it easier to prepare and build machine learning models. The key ones I’ll probably use a lot are NumPy, SciPy, Pandas, and Scikit learn.

Many concepts feel foreign to me at the moment, but as I spin up a couple of projects and work through tutorials, I should start to get the basics down!