Yehoshua 15:61-63 יהושע ט״ו:ס״א-ס״ג

לא יכלו בני יהודה להורישם. שנינו בספרי: רבי יהושע בן קרחה אומר יכולין היו, אלא שלא היו רשאין מחמת השבועה שנשבע אברהם לאבימלך, ויבוסי זה לא על שם האומה היה, אלא מגדל דוד שהיה בירושלים ושמו יבוס, ובני אותו מחוז מפלשתים היו, וכשכבשו בני יהודה את ירושלים לא הורישו את בני אותו מחוז:

The descendants of Yehudah were unable to expel them. We learned in Sifri: “Rabbi Yehoshua be Korcho says, ‘They were really able [to expel them] but they were not permitted to do so because of the oath that Avrohom had sworn to Avimelech.³¹See Bereishis 21:23. These Yevusim were descendants of Avimelech, whom Avrohom had sworn not to harm. Avimelech witnessed all the miracles that God performed on Avrohom’s behalf, and he asked him to swear that he would not harm him or his descendants for three generations and Avrohom agreed. These Yevusim Were not part of the Yevusi nation,³²Why then were they called Yevusim? Rashi goes on to explain that they were actually of the Philishtim and thus protected by Avrohom’s oath. but the Tower of David which was in Yerushalayim was named Yevus, and those who lived in that vicinity were of the Philishtim. Thus when the descendants of Yehudah conquered Yerushalayim, they did not expel the inhabitants of that vicinity.