Yiannopoulos thanked Breitbart for letting him transmit conservative and libertarian ideas to an audience that otherwise would never have heard them, but said he did not want his "poor choice of words" to detract from his colleagues' important work.
"This is my decision alone," he told a news conference in New York city. "When your friends have done right by you, you do right by them. For me, now, that means stepping aside."
Yiannopoulos was banned from Twitter last year after making remarks on race, religion and sex that incited racial attacks on an African-American actress. On Feb. 1, violent protesters forced the cancellation of a speech he was due to give at the University of California at Berkeley.
The latest controversy stemmed from a video more than a year old in which Yiannopoulos said he thought that when it came to relations between men and boys, "there are certainly people who are capable of giving consent at a younger age" than the law allows.
On Monday, after the video resurfaced, organizers of the Conservative Political Action Conference, or CPAC, rescinded an invite for him to speak at the annual event.
The same day, publisher Simon & Schuster said it canceled the publication of Yiannopoulos' book "Dangerous," which was due out on June 13.
The 33-year-old Briton told reporters he was sexually abused between the ages of 13 and 16 by two men, but did not realize at the time it was abuse.
"I can look back now and see that it was," Yiannopoulos said. "My experience as a victim led me to believe that I could say anything on the subject. ... I don't believe that sex with 13 year olds is OK. ...I am horrified by pedophelia."
Yiannopoulos said some of the remarks were taken out of context and that he was at times speaking about his specific experience of being abused.
Despite the most recent backlash, which he described as the worst of his career, Yiannopoulos said he planned to continue to try to grow an audience.
In the months ahead, Yiannopoulos said he would launch a new media website. He said he would focus more on entertainment and campus tours in his professional future instead of Breitbart journalism.