Overcome your fear for Thai food with Foody Dudy. {Creating an image classification model with fastai}

14 min readMay 30, 2021

Link to Github Repo: https://github.com/GemmyTheGeek/FoodyDudy

Foody Dudy

Have you ever encountered a problem with Thai food’s spiciness? Before you start eating your food, you think: “Hey, this is not too spicy,” however, when you start eating, your mouth gets set on fire? If you did, you won’t anymore because Foody Dudy, the Dude of classifying Thai food, has got your back!

Enjoy Thailand : Enjoy Thai Food

This model’s main reason came to life is because most tourists upon arrival in Thailand don’t know Thai food. They would have to try to guess which dish was spicy and what are the ingredients. However, with Foody Dudy, tourists will no longer have to struggle with Thai Food. Instead, they happily dine, knowing how spicy their dish is.

A lot of People Question about Thai Food’s Spiciness

They can also get rid of the anxiety that the dish might contain something they’re allergic to, and all they have to do is take a photo of their food and upload it to Foody Dudy. As we all know, the food you eat while traveling will affect the way you look at that country.

If you want to try out Foody Dudy, then go to this link:https://foodydudy.herokuapp.com/.

Screenshot of Foody Dudy after you submit your Thai food image

Data Set

The primary target audience for my model is tourists coming to Thailand, which meant the source for the database needs to be from a traveling/tourist website. That’s where CNN Travel comes into play; this website was like a gold mine, filled with the names of the 50 popular Thai food. At first, the database’s menu consisted of the 50 Thai menus from CNN Travel, but when I was gathering the images for each menu, a problem occurred. Some of the menus couldn’t supply enough images, so those menus were discarded. I also added some menus that I thought were popular and should’ve been on the list, like fried eggs and omelets.

I decided that each menu would have 1000 images because my hypothesis was “the more images, the more accurate it will be!” If you do the math: 1000 images * 48 menus = 48,000 images in total. That’s a lot of images! There was no way I was going to download every single image one by one manually.

A couple of research later (This is a joke, I know my grammar’s off): https://pypi.org/project/jmd-imagescraper/ is a link to an image scraping program in PyPI. Image scraping is having a program search and download images for you. My image scraper, in particular, can set the number of images downloaded, set the keyword, and create a file for the images.

However, a problem appeared right away. The images that the program downloaded were primarily inaccurate and unusable. This problem forced me to clean every single image in the folder. The program had downloaded around 1000 images, but only 150 to 200 images were left after cleaning them. I then understood the limits of my image scraper and decided to lower the expectations from 1000 images to 300.
I decided to change 1,000 images into only 300 because it seemed like a reasonable number, and the image scraper could supply most of the pictures. Even though the expectations had sunk tremendously, the image scraper still couldn’t find enough images.
It meant I had to handpick the rest. My three primary sources were: Google, Bing, and Youtube. Once I manually downloaded the rest of the images for each menu, I had another problem. Since there were three sources to download images from, there was a chance that some of the images were overlapping each other.
I googled my problem and came across this video: https://www.youtube.com/watchv=658U3o7hlbI&ab_channel=JieJenn. This video was about creating a program for finding similar images inside of a folder. Just what I needed! I used this code with all 48 menus and came across some images that overlap. Here is the chart:

After I found the duplicate images, I had to replace them, which meant I had to download more images from my three sources. However, to make sure that the images I downloaded weren’t going to overlap with another image already in the folder, I decided to search with a different keyword. My keyword before was either in Thai or Thai Karaoke, so I changed it to English. (EX: แกงเขียวหวานไก่/gaeng keaw hwaan ==> Chicken Green Curry) To make sure that there are no more overlapping images, I ran it through my program once more. NOTE: It’s important not to have the same images in different sets because it will cause Data Leakage, which will ruin our model.

Link to Cleaning Codes: https://github.com/GemmyTheGeek/CleaningCodeFoodyDudy/tree/main

Finally, I split each menu into three sets: train, valid, and test. These classes are necessary for training our model. I decided to use an 80% train, 10% valid, and 10% test formula. This meant that the train set has 240 images per one menu, valid has 30 images per one menu, and test has 30 images per one menu. That’s how I prepared my dataset.
Train Set: The set which is used to train our model.
Valid Set: The set in which is used for showing us the best version of our model.
Test Set: Used to test our model’s accuracy.

Image Augmentation

Since we have our data set, I wanted our model to use the images with the best accuracy, so I experimented with augmentation. Augmentation changes an image or modifying it, like making it brighter, tilt it, enlarge it, or even twist it. I played with augmentation and got these results:

The best augmentation (from my data) was enlarging my image to 512px + default fastai augmentation (Error Rate = 2%), and the second-best augmentation was ZoomX2 (Error Rate = 4%). So I decided to combine both of them, thinking that the results would be better than 512px + default fastai augmentation. However, when it finished training, the results were surprising:

The Error rate was better than ZoomX2 but worse than 512px + default fastai augmentation. I kept on experimenting and found a combination I think would work, ZoomX1.5, 512px, and LightX0.5. I got this result:

It was worse than ZoomX2 and 512px, so I decided to stick with 512px. I ran 512 px + default fastai augmentation again, and the results were even better! Instead of having an error rate of 2.9%, it decreased to 2.7%!

Toploss:

However, something strange happened while scrolling through the batch. The images were distorted, turned, and even warped! It turned out that even if I don’t use warp =, rotate =, or bright =; it will still perform the default augmentation. Instead, I only wanted to resize my image to 512 px, which is not what is happening here. That’s when it occurred that I could use item_trans instead of aug_trans, and that’s exactly what I did. The results were not too great:

So I went back to only using aug_trans = 512 px.

Squish, Pad, Crop

In item_trans, there are multiple ways for you to make your image a specific size. You can squish your image into that size, which means that your image might be distorted or stretched. You can pad your image to shrink your image until its X or Y axis fits and then fill in the empty slots with darkness. Or you could crop the image into that size, which is pretty straightforward. I tried out all of them, and here are the results:

Squish:

Code: item_tfms=Resize(512, method=ResizeMethod.Squish)
Result:

Batch:

Pad:

Code: item_tfms=Resize(512, method=ResizeMethod.Pad)
Result:

Batch:

Crop:

Code: item_tfms=Resize(512, method=ResizeMethod.Crop)
Result:

Batch:

If you were to compare all of the error rates, squish has got the best one. I then decided to fine-tune squish ten times, and here is the result:

The error rate was 3.9% which was not too great, so I decided not to use squish with my aug_trans size = 512.

Test Set

If we want to test the accuracy of our model, we can’t rely on the results we’re getting now. That’s where the test set is used, I first fine-tuned my model ten times with Resnet-34 as the architecture, and then I saved our model to test. I then ran the code to test our model with the test set:

The results were not too great:

However, I had a theory that if my test set’s images’ resolution was equal to my train set’s images’ resolution, which was 512 px due to the aug_trans, then it might work better. I decided to try out that theory and added an aug_trans to the code, and now the code looks something like this:

I ran this code, and the confusion matrix was way better than last time!

I also decided to run the classification report on this. Here is the result:

Precision: How many of our predictions have we guessed correctly.
Recall: How many of our predictions were correct out of all the same class photos.
F1 Score: (Precision * Recall) / (Precision + Recall)

Want to know more about these three marks? Go check out this video: https://www.youtube.com/watch?v=2osIZ-dSPGE&ab_channel=codebasics

If you are still confused, don’t worry; these results are pretty good in simple language.

These are some of the top losses of 512px + default fastai augmentation on the test set:

Most of the images that my model guessed incorrectly might’ve fooled me too because they look so similar! If I were to predict each image I would’ve probably made the same mistake.

Let me explain to you why my AI got this wrong.

The reason the first one got it wrong was that the image was a close-up of the sauce. This tricks my model to think that the sauce is the main point of the dish. So instead of being pork, my model predicts it to be stew.
The second image’s problem was that the image was taken of a pork knuckle but you can’t really see the pork, instead, you see an egg, so my model thinks that this dish is egg in sweet brown sauce.
My model thinks that the last one is fried rice but it’s actually shrimp fried rice. This is because you can’t really see the shrimp because it is concealed underneath vegetables.

WebApp Implementation

Once my code was ready alongside my dataset, I decided it was time for deployment. There are many ways to deploy your model, but I went along with fastai and deployed my model onto mybinder.org. However, before I am able to deploy my model, I had to upload my files into Github. When I tried to upload my files pass through their website, it gave me the following error message:

That meant that I had to find out another way to upload my files onto Github. I knew that sooner or later, I would find a way because there were other repos that had files way bigger than mines. I was fumbling over documents and videos when I came across the answer, LFS, Large File Storage. I then installed the program, and with some help from Youtube videos and tutorials, I was finally able to upload my files onto Github. Here’s a little cartoon of this scenario:

Here is the link to my Github repository: https://github.com/GemmyTheGeek/FoodyDudy

After that, I was able to go and deploy it onto Mybinder, and all I had to do was follow fastai’s tutorial! I was pleased about it until something became evident, whenever I opened my web app, Mybinder would take a long time to load.

I got advice from my mentor to deploy the model on Heroku with Streamlit instead of Mybinder since it was much faster. I also thought this was a good idea and started watching tutorials and doing some research on this topic. When I was confident enough, I tried to deploy my app, but it didn’t work.
So then my mentor asked me if I needed a little guiding, and of course, I said yes. After the meet-up with my mentor, I felt a bit more confident, even though we weren’t able to deploy foody dudy. I did more research and found out that the reason I wasn’t able to deploy my model was that the slug size was too big.

I was happy that I found out the error and with a bit more research I found this website: https://stackoverflow.com/questions/62532673/how-to-deploy-git-lfs-on-heroku. I followed it and fixed my error by pushing it with a specific command “git push heroku master --no-verify”! Now I tried to deploy it again and it worked!

Here is the link (Heroku): https://foodydudy.herokuapp.com/

Here is the link (Mybinder): https://mybinder.org/v2/gh/GemmyTheGeek/DoFoodyDudy/main?urlpath=%2Fvoila%2Frender%2Finferencer.ipynb

I noticed right away that Heroku could load much faster than Mybinder can! If you don’t believe me run these two links side by side and see how much quicker Heroku is than Mybinder.

Limitations

Even though this model is great at classifying Thai food, if somebody were to give it a random image (person, things, place, etc.) to test, it still predicts Thai food.

The Most Challenging Part

I think that throughout the entire journey of making my model, there were two hardest parts:

One of the hardest parts for me in making this model is cleaning the data set. Some people might think that the only thing that will get trained is the model, but they would be wrong. We have to train ourselves too. When cleaning our data set, we need to know what images to delete and what images to keep. Some of the menus I didn’t even know existed, so I had to do research and try to find information on what was unique about the menu.
Another tricky part I had is trying to eliminate all of the duplicate images. As I said earlier, I had trouble with identical images because I had to download from 3 different sources manually:

Youtube
Google
Bing

I tried to fix this problem by coding a program to find duplicate images, and once the program found some, I replaced the image with another image. However, I had to run the program multiple times on the same file because the images that replaced the duplicate images might be the same image as another image in the file.

I have to search in different keywords and try my best to find good-quality images that weren’t already in the file. Some of the menus were really popular, and I didn’t have too much of a hassle replacing the duplicate images. However, some not-so-popular menus have a remarkable lack of images and are difficult to find images for.

Next Step

The main goal for Foody Dudy, from the start to the end, is for tourists to dine happily in Thailand. Furthermore, I am sure every bit of effort I put into this model will be worth it in the end because this data set (14,400 images of Thai food) can help the community by saving them from the torture of finding images for a good and clean data set.

Feelings

During this program, I had a lot of mixed emotions and feelings.

It was my first time making an AI model, so I encountered many problems, but each problem gives me more experience, so a problem was knowledge in disguise.
This AI Builder Project gave me more understanding of what AI is actually like. However, it wouldn’t be possible if it weren’t for Central Group, VISTEC, and AIReaserch Thailand. Thank you very much for such a fantastic project to help Thai kids prepare for the future world of AI.
I also would like to thank my mentors, P’ Charin, P’ Joy, Uncle Engineer, and P’ A, for helping me throughout this journey.

Sharing Data

I also put Foody Dudy into Kaggle which I think is a great way to share data with other people. Here is the link to my Kaggle: https://www.kaggle.com/somboonthamgemmy/foodydudy

I also uploaded the notebook into Kaggle, that way other people can clone it and edit it to their liking.

Biography

Hello! I am Phacharaphol (Gemmy) Somboontham. I am 12 years, three months, and three days old, as of right now (Date: June 3, 2021). I like to do programming on games because I like to play them. I also do other stuff like playing with my cat and soccer. Over the last two months, I have been committed to Foody Dudy, an Image Classification Model. I began this model because of AI Builder, a program for kids who want to build good AI, organized by Central Group, VISTEC, and AIReaserch Thailand.

Support Me: https://www.youtube.com/c/codingforkids?sub_confirmation=1

Credits

Mentors:

P’ Charin (Github: https://github.com/cstorm125)
P’ Joy (LinkedIn: https://th.linkedin.com/in/nattakarnphaphoom)
Uncle Engineer (Facebook: https://www.facebook.com/UncleEngineer)
P’ A (Facebook: https://www.facebook.com/kanes.sumetpipat/)
Central Group, VISTEC, and AIReaserch Thailand: https://www.facebook.com/aibuildersx

Overcome your fear for Thai food with Foody Dudy. {Creating an image classification model with fastai}

Foody Dudy

Enjoy Thailand : Enjoy Thai Food

Data Set

Image Augmentation

Squish, Pad, Crop

Test Set

WebApp Implementation

Limitations

The Most Challenging Part

Next Step

Feelings

Sharing Data

Biography

Credits

Reference

Written by GemmyTheGeek