GitHub IssueA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. Resolution Time Prediction System Analysis

Data Acquisition and Storage:

• Data Collection: Approximately 2 million GitHub issuesA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. were downloaded. These were sourced from the top 100 repositories across diverse and popular programming languages, including Python, Go, and Java, to ensure a broad and representative dataset.
• Data Management: The collected issueA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. data was then systematically upserted into a PostgreSQL database. This relational database was chosen for its robustness and efficiency in handling large volumes of structured and semi-structured data, facilitating complex queries and data retrieval for model training.

Feature Engineering:

A crucial step was the engineering of a rich set of features from the raw issueA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. data. This process aimed to extract meaningful signals that could predict resolution times. Key feature categories include:

• Text Embeddings: Sophisticated text embeddings were generated for the title and body of each issueA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics.. These embeddings convert textual information into dense vector representations, capturing the semantic meaning and context, which are vital for understanding the issue'sA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. content.
• Numerical Features: A variety of numerical features were derived, such as:
  • The number of issuesA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. concurrently open within the same repository at the time of an issue'sA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. creation (reflecting repository load).
  • The total number of unique contributors to the repository (indicating community size and activity).
• Additional features, as detailed in the Feature Analysis section (e.g., time-based features, content metadata), were also engineered to provide a comprehensive view of each issueA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics..

• AccuracyThe proportion of all predictions (both positive and negative) that were correct. While our model achieves high accuracy (96.5%), this metric can be misleading with imbalanced datasets because always predicting the majority class (valid) would still yield high accuracy.: 0.9653 (96.5%)
• PrecisionThe proportion of positive identifications (won't-fix predictions) that were actually correct. Our high precision (95.5%) means that when the model predicts an issue as won't-fix, it's usually right. However, this doesn't tell us how many won't-fix issues we're missing.: 0.9552 (95.5%)
• RecallThe proportion of actual positives (true won't-fix issues) that were correctly identified. Our low recall (10.4%) reveals a critical weakness: the model is missing about 90% of the issues that actually won't be fixed, likely due to the severe class imbalance in the training data.: 0.1041 (10.4%)
• F1 ScoreThe harmonic mean of precision and recall, providing a single metric that balances both concerns. Our low F1 score (0.1878) indicates poor overall performance despite high accuracy, because the model is failing to identify most won't-fix issues.: 0.1878
• AUCArea Under the Receiver Operating Characteristic curve - measures the model's ability to distinguish between classes across various threshold settings. Our relatively high AUC (0.8542) suggests that the model has good discriminative power, but we're not leveraging it effectively due to the imbalanced data problem.: 0.8542

Analysis: The significant importance of classification uncertainty and text embeddings suggests that the semantic contentThe underlying meaning and context expressed in the issue text, beyond just keywords or surface-level statistics. This includes concepts like the problem domain, the complexity of the issue description, technical specificity, and how the issue is framed - all of which can influence how quickly it might be resolved. of issuesA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. is valuable for predicting resolution time. The system effectively leverages this through the two-layer approach.

Understanding which features drive prediction performance provides valuable insights into what factors affect GitHub issueA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. resolution times. This section expands on the visualizations above with detailed comparisons between models and feature sets.

Model Comparison Insights

The striking differences between XGBoost and the ensemble model's feature importance reveal fundamental differences in how these models learn:

• XGBoost's Focus: XGBoost places extraordinary importance (85.20%) on classification_uncertainty when embeddings are not used, suggesting it relies heavily on this single feature for predictions.
• Ensemble's Balanced Approach: The ensemble model distributes importance more evenly across features, suggesting it leverages multiple signals more effectively.
• Embedding Effect: Adding embeddings reduces XGBoost's reliance on classification_uncertainty from 85.20% to 38.18%, while the ensemble model sees a more dramatic shift from 25.58% to 11.86%.
• Feature Selection: XGBoost identifies different secondary features as important compared to the ensemble—for example, has_error_message and author_is_core_contributor rank higher in XGBoost.

Bug Identification Features

Features related to bug identification show surprising patterns:

• has_error_message: Ranked 3rd (1.40%) by XGBoost but only 14th (1.05%) by the ensemble
• has_reproduction_steps: Generally low importance across all models despite being considered valuable in bug reports
• has_code: Very low importance (typically below 0.6%) compared to code_to_text_ratio (3.62% in ensemble)
• This suggests that the quality and proportion of code in an issueA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. is more important than its mere presence
• XGBoost appears more sensitive to specific technical indicators in the issueA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. content

HuggingFace Multilingual E5-Large-Instruct

The system uses intfloat/multilingual-e5-large-instructA state-of-the-art text embedding model from HuggingFace that converts text into high-dimensional vector representations. It's based on the E5 architecture (which stands for "Embeddings from bidirectional Encoder representations and contrastive predictive coding") and has been fine-tuned with instructions to better capture task-specific semantics. With 560M parameters, it offers a balance between quality and computational efficiency. for generating embeddings.

Advantages:

• Speed: Relatively fast inference compared to larger models
• Multilingual: Handles issuesA GitHub Issue is a way to track tasks, enhancements, bugs, or other items of work within a GitHub repository. It serves as a discussion forum and progress tracker for specific topics. in different languages
• Instruction-tuned: Better at capturing task-specific semantics
• Reasonable Size: 560M parameters (vs. billions in larger models)

Limitations:

• Quality Tradeoff: Not as high-quality as larger embedding models
• Technical Content: May not fully capture programming-specific semantics
• Context Length: Limited context window compared to newer models

Trade-offs and Considerations

When migrating to the cloud, several trade-offs need to be considered:

• Cost vs. Performance: More powerful instances cost more but reduce processing time
• Embedding Quality vs. Speed: Consider testing larger embedding models despite increased cost
• Data Volume vs. Detail: Balance between processing more repositories or deeper analysis
• Real-time vs. Batch Processing: Determine if predictions need to be real-time or batched