Boa Meets Python: A Boa Dataset of Data Science Software in Python Language

By: Sumon Biswas, Md Johirul Islam, Yijia Huang, and Hridesh Rajan

Abstract

The popularity of Python programming language has surged in recent years due to its increasing usage in Data Science. The availability of Python repositories in Github presents an opportunity for mining software repository research, e.g., suggesting the best practices in developing Data Science applications, identifying bug-patterns, recommending code enhancements, etc. To enable this research, we have created a new dataset that includes 1,558 mature Github projects that develop Python software for Data Science tasks. By analyzing the metadata and code, we have included the projects in our dataset which use a diverse set of machine learning libraries and managed by a variety of users and organizations. The dataset is made publicly available through Boa infrastructure both as a collection of raw projects as well as in a processed form that could be used for performing large scale analysis using Boa language. We also present two initial applications to demonstrate the potential of the dataset that could be leveraged by the community.

ACM Reference

Biswas, S. et al. 2019. Boa Meets Python: A Boa Dataset of Data Science Software in Python Language. MSR’19: 16th International Conference on Mining Software Repositories (May 2019).

BibTeX Reference

@inproceedings{sumon2019pydata,
  author = {Sumon Biswas and Md Johirul Islam and Yijia Huang and Hridesh Rajan},
  title = {Boa Meets Python: A Boa Dataset of Data Science Software in Python Language},
  booktitle = {MSR'19: 16th International Conference on Mining Software Repositories},
  location = {Montreal, Canada},
  month = {May},
  year = {2019},
  entrysubtype = {conference},
  abstract = {
    The popularity of Python programming language has surged in recent years due to
    its increasing usage in Data Science. The availability of Python repositories in
    Github presents an opportunity for mining software repository research, e.g.,
    suggesting the best practices in developing Data Science applications, identifying
    bug-patterns, recommending code enhancements, etc. To enable this research, we have
    created a new dataset that includes 1,558 mature Github projects that develop Python
    software for Data Science tasks. By analyzing the metadata and code, we have included
    the projects in our dataset which use a diverse set of machine learning libraries and
    managed by a variety of users and organizations. The dataset is made publicly available
    through Boa infrastructure both as a collection of raw projects as well as in a
    processed form that could be used for performing large scale analysis using Boa language.
    We also present two initial applications to demonstrate the potential of the dataset
    that could be leveraged by the community.
  }
}