Doing the dev is the only way, I would recommend though reading the Hinton dropout paper and trying to implement a non full featured Neural Network framework to implement those ideas. Then pick papers to implement in your toy framework. Once you have done this a few times, you will get a really good idea how all this stuff works and ideas to extend things. It also REALLY helps if you have a very strong linear algebra background, so if your weak there do lots of reading and practice getting those skills up to snuff. I may not be the best example, because I worked in computer graphics heavily over my career. My suggestion though is to try and apply DNN's and ML techniques in somewhere you have domain experience. It really will help a lot with the intuition part since you have valuable domain experience to give you ideas to try.

This is a path I took around 4 years ago, and I have built some seriously valuable stuff in that time.

Also I would personally say to not discount the value of backend stuff, doing scalable ML processing is not off the shelf and lots of value can be created by making scalable reusable systems that can run machine learning models. Most of those ML PHd's can't (or don't want to) build scalable distributed systems. If you can train experiments faster, or actually do the work to make an end to end system work outside the lab, you can make real measurable contributions.