Schema depicting an architecture with an embedded solver.
In this work, we have addressed the problem of disparity between two worlds: deep learning and combinatorial optimization. Ideally, we would like to have the best of both worlds, having rich feature representations through deep neural networks and efficient algorithm implementations that enable combinatorial generalization. We have shown that such algorithms indeed can be used as neural network building blocks, with a small tweak to the backward pass to provide an informative gradient. This means that we can take efficient implementations of algorithms and use them right away, without modifying the algorithms themselves. More information is available in our blog post on the subject.
We presented this work at ICLR 2020. Since it was a virtual conference, also a short video describing the work is available online accompanied by a poster.
We are maintaining a codebase with PyTorch wrappers for some standard solvers, available here.
In search for practical application of the blackbox-differetiation theory, we turn to computer vision. Concretely, we show that applying blackbox-backprop to computer vision benchmarks in recall and Average Precision for retrieval and detection tasks consistently improves the underlying architectures’ performance.
The main component that enables this is the blackbox formulation of the argsort operation used for ranking making the use of blackbox-differentiation theory possible. We made a blog post describing the method, which we call RaMBO (Rank Metric Blaxkbox Optimization). Further information about the paper (including a short and long oral presented at CVPR 2020) can be found here.