{"code":1,"total":3,"0":{"title":" Characterizing Out-of-Distribution Error via Optimal Transport<\/strong>
\r\n Yuzhe Lu*, Yilong Qin*, Runtian Zhai<\/strong>, Andrew Shen, Ketong Chen, Zhenlin Wang, Soheil Kolouri, Simon Stepputtis, Joseph Campbell, Katia Sycara<\/em>
\r\n In NeurIPS 2023<\/em>
\r\n Paper<\/a>   Code<\/a>","text":"Predicting a model's performance on OOD data without labels is an important task for machine learning safety, and has aroused lots of interest. While a number of methods have been proposed by prior work, they often underestimate the actual error, sometimes by a large margin. In this work, we identify pseudo-label shift, or the difference between the predicted and true OOD label distributions, as a key indicator to this underestimation.<\/strong><\/em> We then propose COT and COTT, which address this issue by making the extra natural assumption that the train and test label distributions do not differ too much. Our extensive experiments on synthetic and real datasets show that the proposed methods outperform existing ones by large margins.","img":"https:\/\/runtianzhai.com\/img\/cot_neurips.png","imgalt":"COT"},"1":{"title":" Responsible AI (RAI) Games and Ensembles<\/strong>
\r\n Yash Gupta, Runtian Zhai<\/strong>, Arun Suggala, Pradeep Ravikumar<\/em>
\r\n In NeurIPS 2023<\/em>
\r\n Paper<\/a>","text":"Several recent works have studied the societal effects of AI; these include issues such as fairness, robustness, and safety. In many of these objectives, a learner seeks to minimize its worst-case loss over a set of predefined distributions (known as uncertainty sets), with usual examples being perturbed versions of the empirical distribution. In other words, aforementioned problems can be written as min-max problems over these uncertainty sets. In this work, we provide a general framework for studying these problems, which we refer to as Responsible AI (RAI) games. We provide two classes of algorithms for solving these games: (a) game-play based algorithms, and (b) greedy stagewise estimation algorithms. The former class is motivated by online learning and game theory, whereas the latter class is motivated by the classical statistical literature on boosting, and regression. We empirically demonstrate the applicability and competitive performance of our techniques for solving several RAI problems, particularly around subpopulation shift.","img":"https:\/\/runtianzhai.com\/img\/rai.png","imgalt":"RAI"},"2":{"title":" Predicting Out-of-Distribution Error with Confidence Optimal Transport<\/strong>
\r\n Yuzhe Lu, Zhenlin Wang, Runtian Zhai<\/strong>, Soheil Kolouri, Joseph Campbell, Katia P. Sycara<\/em>
\r\n In ICLR 2023 Trustworthy ML Workshop<\/em>
\r\n Paper<\/a>   Code<\/a>","text":" We present a simple yet effective method to predict a model's OOD performance<\/em><\/strong> on an unknown distribution without any additional annotation. Our approach is rooted in the Optimal Transport theory, viewing test samples' output softmax scores from deep neural networks as empirical samples from an unknown distribution. We show that our method, Confidence Optimal Transport (COT), provides robust estimates of a model's performance on a target domain. Despite its simplicity, our method achieves state-of-the-art results on three benchmark datasets and outperforms existing methods by a large margin. ","img":"https:\/\/runtianzhai.com\/img\/cot_workshop.png","imgalt":"COT"}}