I'm not really convinced this would work in practice. The trick seems to depend on the fact that the dataset is a sequence of frames of the same object shot from slightly different angles. But that's a terrible dataset - it might work for training a toy proof-of-concept, but if you actually wanted to do calorie estimation in the wild, you'd need a much more varied (and larger) training set. And once you have that, you lose the properties that made this labelling approach viable in the first place.
Hi Imnimo, I wrote the article and definitely understand your concerns. The point is not the specific steps I took working in general for most datasets, but more the overall idea of using a more data science-y approach to labelling rather than just blindly throwing your data at a workforce.
A more varied dataset will require additional strategies. We have done this type of thing with various datasets and what normally works is a combination of some vertical models, heuristics specific to the dataset, classical computer vision techniques, and some human label seeding/correction.
I guess the way I look at it is that if you can automatically label your training set, you either have solved the problem you set out to learn (just use your labeler as your classifier/detector/whatever), or you're exploiting some limitation of the training set. Given a human-annotated test set, I'd want to see a comparison between three outputs:
-The outputs of the auto-labeler. If this is strong, you've learned that you didn't need the training set after all - you managed to solve the problem without it!
-The outputs of a model trained on auto-labeled data. If this is strong but the above test was not, then this pipeline makes sense.
-The outputs of a model trained on human-labeled data. If this is strong but the above tests were not, we're in trouble.
If none of the three are strong, then the training data was lacking (assuming we've done our best on tuning the model we're trying to train), and so no real value was gained by annotating it.
I see it a bit differently. I see it as two separate(but correlated) tasks. There is labelling the data and building a robust model. There is a nuanced gap between the two. The labelling task and the model task live a different constraint space.
When you are labelling data, you have access to strategies and means that might not be available to your downstream model. In our experience this includes a human in the loop component, building non-robust ensemble models(we call these micro-models), and some "guess work" functions on the data. All of this together can make an "auto labeller" that does pretty well getting labels made, but really the sum of these strategies is very different from some well trained neural network that will be running on edge or whatever.
The point of a model is not to label the data, it's to generate some value in some out of sample task, quite different from strategies that you can run in a sandboxed environment with your training data.
Sure, but you need to demonstrate that the auto-labeled training data is valuable by showing that a model trained on it performs as well (or close to as well) as the same model trained on human-labeled data. Without that, we're just eyeballing the auto-labels and saying "looks good I guess!"
Obviously we should expect that the auto-labeler fails on the test set, because we assume we're exploiting some convenience that won't be available at test time. But we should still try - it might reveal that our task is too easy to need the model we were planning to train, or it might reveal that our test set is not actually representative.
Yea, so that's more of a comment on the accuracy of the auto generated labels, because this approach doesn't assume a different representative set of data than with human labelled data, just that less of the data is human labelled.
So it comes down to how good the auto generated labels are(from a human perspective), which is a fair point that I didn't address much in the article, but in general comes down to a good QA process(which is applied to both human labels and machine labels equally because humans also make mistakes in this stuff).
In the article the dataset was small enough and the labels simple enough that I could run very quick visual inspection over the results, but for more complicated tasks we have a more rigorous human review process for evaluating label accuracy(again to both human and algorithm produced labels). The auto generated labels may not be more efficient overall if they require a lot of correction after review, but for this case, and a lot of other ones, they just are empirically are.
This reminds me of Snorkel (though unclear from the article if they’re using Snorkel’s trick of aggregating many weak heuristics). It can be made to work even in the real world. The rub is that coming up with these programmatic labelers is easier said than done especially for complex data.
It works well if a domain expert can say something without “cheating” and looking at the data like “put a box around round red objects because those are always apples”. But in practice people tend to cheat and look at the data first, and you end up with humans trying to emulate ML, poorly.
Definitely easier said than done, but the process at least makes labelling interesting. Sometimes you run into roadblocks where you can't get past just having a human doing some element of the labelling, but once you do have a few algorithmic strategies that work reasonably well on a representative sample of your data, you can usually scale them pretty effectively to the rest of your data
A common mistake in applied computer vision is to use a classical method (e.g. distanced-based watershed) to buoy your training set. You’ll end up with a computationally expensive method (e.g. a region-based convolutional neural network) that’s a poor replication of the classical method. The major advantage of learning-based methods is to go _beyond_ classical performance and make inferences comparable to the manually annotated image. There’s no shortcut.
Hi Sriku, with regards to your first point, not necessarily. I mentioned in another comment, but the model you are using the labels to build and the labelling process are related but not the same, they have different fundamental constraints and rely on different techniques. You can't have a human in the loop to help you out in a live model as an example.
You are right there are a bunch of difficult problems this technique isn't perfect for, but it actually can still help improve the efficiency of labelling a lot and when I do it I get the added bonus of understanding the dataset a lot better.
I still wouldn't push this beyond a limited number of cases where the average human can identify patterns and explain them without much doubt. For example if you want to apply this to radiology images, where the intention behind using a DL technique is to discover patterns we may not be able to notice and exploit them, the approach would probably be as labour intensive as labeling datasets.
Otherwise I overall agree with you that we should consider this where we can .. as evidenced by my snorkel link.
I hear you, but I don't even think the labor intensiveness is a lost cause here. Labor intensiveness in mining insights from a dataset is worth way more than labor intensiveness in manual labelling.
I think we are lazily giving up our intellectual power to models hoping that they will just discover patterns by magic, where it is actually very worth to go through the data science process starting with labelling because we actually learn as humans. The thesis is that this will also make our DL models better in the long run. We would never have come up with cool algorithms if we just always outsourced this work to models.
You could first make a simple labeler for these contrived problems, then you use your trained algorithm and correct it on problems of increasing complexity.
Feature extraction from a pre-trained model? Sure, again depends how you use it. We have used feature extraction + clustering for some of these labelling tasks successfully in the past.
