Large-scale Study of Substitutability in the Presence of Effects

By: Islam Md Johirul, Hoan Nguyen, and Hridesh Rajan

Abstract

Modern software systems are increasingly including machine learning (ML) as an integral component. However, we do not yet understand the difficulties faced by software developers when learning about ML libraries and using them within their systems. To that end, this work reports on a detailed (manual) examination of 3,280 highly-rated posts related to ten ML libraries, namely Tensorflow, Keras, Scikit-learn, Weka, Caffe, Theano, MLLib, Torch, Apache Mahout, and H2O, on Stack Overflow, a popular online technical Q and A forum. We classify these questions into seven typical stages of an ML pipeline to understand the correlation between the library and the stage. We also perform inter- and intra-library analyses to understand broad trends. Our findings reveal the urgent need for software engineering (SE) research in this area. Both static and dynamic analyses are mostly absent and badly needed to help developers find errors earlier. While there has been some early research on debugging, much more work is needed. API misuses are prevalent and API design improvements are sorely needed. Enabling reuse of trained models across libraries needs attention. Last and somewhat surprisingly, a tug of war between providing higher levels of abstractions and the need to understand the behavior of the trained model is prevalent. These findings suggest new paths for SE researchers to help improve the engineering of software that includes ML components.

ACM Reference

Md Johirul, I. et al. 2018. What Do Developers Ask About ML Libraries? A Large-scale Study Using Stack Overflow. Technical Report (Nov. 2018).

BibTeX Reference

@inproceedings{Islam18,
  author = {Md Johirul, Islam, and Nguyen, Hoan and Rajan, Hridesh},
  title = {What Do Developers Ask About ML Libraries? A Large-scale Study Using Stack Overflow},
  booktitle = {Technical Report},
  series = {Technical Report},
  month = {November},
  year = {2018},
  entrysubtype = {Technical Report},
  abstract = {
    Modern software systems are increasingly including machine learning (ML) as 
    an integral component. However, we do not yet understand the difficulties 
    faced by software developers when learning about ML libraries and using them 
    within their systems. To that end, this work reports on a detailed (manual) 
    examination of 3,280 highly-rated posts related to ten ML libraries, 
    namely Tensorflow, Keras, Scikit-learn, Weka, Caffe, Theano, MLLib, Torch, 
    Apache Mahout, and H2O, on Stack Overflow, a popular online technical Q and A forum. We classify 
    these questions into seven typical stages of an ML pipeline to understand the 
    correlation between the library and the stage. We also perform inter- and 
    intra-library analyses to understand broad trends. Our findings reveal the urgent 
    need for software engineering (SE) research in this area. Both static and dynamic 
    analyses are mostly absent and badly needed to help developers find errors 
    earlier. While there has been some early research on debugging, much more work is 
    needed. API misuses are prevalent and API design improvements are sorely 
    needed. Enabling reuse of trained models across libraries needs attention. 
    Last and somewhat surprisingly, a tug of war between providing higher levels 
    of abstractions and the need to understand the behavior of the trained model 
    is prevalent. These findings suggest new paths for SE researchers to help 
    improve the engineering of software that includes ML components.
  }
}