Troubleshooting

Question 1

Why is your model worst then the authers?

Answer 1

Implementation bug
Hyper parameters choices - model could be extremely sensitive
Data model fit - different data from the paper
Dataset construction - most time in industry is spent on datasets and not models

Question 2

בגדול מה צריכה להיות האסטרטגיה?

Answer 2

פסימיסם,

בגלל שקשה לעשות דיבאג - אז להתחיל ממש בדברים הפשוטים ואז לעלות את המורכבות

Question 3

בגדול מה התהליך של בניית מודל

Answer 3

להתחיל בפשוט - לבחור מודל קל ודאטה קל

ליצור ולדבג את המודל

להעריך את התוצאות

לשפר את ההיפרפרמטר

לשפר את הדאטה/המודל

Question 4

When starting simple - what architecture to choose?

Answer 4

For images start with LeNet like architecture then move to resnet

For sequences start with Transformer/attention model than move to wavenet like model

For other start with simple fully connected NN

For multiple input - say a picture with a phrase - start with making each input into a lower dimensional feature space for example use convNet and flatten the results, same with sequence use LSTM and keep the final vector. Then concaténate all together and pass the output through a fully connected layer

Question 5

Optimizer defualts

Answer 5

Adam with learning rate of 3e-4
Relu for cnn tanh for lstm
Regularización - none
Normalization - none (like batch normalization - not the one on the input)
Both are none because they are a source of bugs

Question 6

Should i normalize the input data?

Answer 6

Yes!
Make aure to do it!
And that its not done automaticaly

Question 7

How to simplify the problem so we can start easy

Answer 7

Small training set (less then 10,000)
Less classes/objects/smaller pictuers etc..
Create a simple synthetic training set

Question 8

3 General advice for implementing the model

Answer 8

1. Lightweight implementation
   פחות מ200 שורות קוד…
2. Of the shelf components
3. Build complicated pipelines later