{"id":"arxiv:2507.00014","kind":"preprint","title":"SWE-Bench-CL: Continual Learning for Coding Agents","authors":["Thomas Joshi","Shayan Chowdhury","Fatih Uysal"],"author_count":3,"year":2025,"published":"2025-06-13","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2507.00014","nct_id":null,"source_url":"https://arxiv.org/abs/2507.00014","archive_url":"https://space-kitty.com/papers/arxiv:2507.00014","updated":"2025-06-13","pmid":null,"pmcid":null,"openalex_id":"W4416843709","pdf_url":"https://arxiv.org/pdf/2507.00014","abstract":"Large Language Models (LLMs) have achieved impressive results on static code-generation benchmarks, but real-world software development unfolds as a continuous stream of evolving issues, fixes, and feature requests. We introduce SWE-Bench-CL, a novel continual learning benchmark built on the human-verified SWE-Bench Verified dataset introduced by OpenAI and Princeton-NLP in 2024. By organizing GitHub issues into chronologically ordered sequences that reflect natural repository evolution, SWE-Bench-CL enables direct evaluation of an agent's ability to accumulate experience, transfer knowledge across tasks, and resist catastrophic forgetting. We complement the dataset with (i) a preliminary analysis of inter-task structural similarity and contextual sensitivity, (ii) an interactive LangGraph-based evaluation framework augmented with a FAISS-backed semantic memory module, and (iii) a suite of specialized continual learning metrics -- including average accuracy, forgetting, forward/backward transfer, tool-use efficiency, and a generalized Composite Continual Learning Score and CL-F-beta score -- to capture the stability-plasticity trade-off. We outline a rigorous experimental protocol comparing memory-enabled and memory-disabled agents across diverse Python repositories. All code and data are publicly available at https://github.com/thomasjoshi/agents-never-forget, providing the community with a reproducible platform for developing more adaptive and robust AI agents in software engineering.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.AI","cs.SE"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"ai-agents","name":"LLM Agents & Tool Use","domain_slug":"ai"}],"cited_in_dossiers":[]}