To that end, he sent a letter to the board this weekend outlining his case that he was wrongly asked to step down as chairman, that his comments had been “mischaracterized” in press reports, and that the whole affair began when he refused to work with Kanye West as a co-spokesman for the company.

Schnatter resigned last week as chairman of Papa John’s after Forbes said that he used the n-word and spoke in graphic terms about lynchings during a conference call in May. Papa John’s also pulled Schnatter’s image from its logo, TV ads and other marketing to distance itself from its founder.

The comments came during what Schnatter called in his letter a “diversity media training” session with Laundry Service, a marketing agency. “During and after that meeting, the Laundry Service leadership strongly urged that our company retain Kayne West as my co-spokesman in the television spots and other promotions,” Schnatter wrote.

“I told them that would not work because he uses the ‘N’ word in his lyrics,” Schnatter said in the letter, which was first reported by CNBC. “During this diversity media training, which covered a wide number of topics, I was asked whether I was racist. I, of course said no—which is a truthful statement as those of you who know me well will attest.”

After members of the ad agency asked Schnatter about controversial comments he made regarding NFL owners—which led to his stepping down as CEO of the company last year—Schnatter explained that he never uses the n-word. But in the process of defending himself, he apparently used that very word.

“I then said something on the order of, Colonel Sanders used the word ‘N,’ (I actually used the word,) that I would never use that word and Papa John’s doesn’t use that word,” Schnatter wrote. But, he claims, he never meant it as a racial epithet.

Schnatter also claimed that Laundry Service asked Papa John’s for $6 million in payment—more than four times what Schnatter says they were owed—because some of its staff had been offended by his comments. In the end, the company “gave into to this extortion attempt” and paid them $2.5 million.

Papa John’s and Laundry Service did not respond to requests from Fortune for comment.

Schnatter has also hired as his attorney Patricia Glaser, a high-profile business trial attorney who has represented a number of companies and celebrities, including Harvey Weinstein in his legal battle against the company he founded, Weinstein & Co.

Glaser, also sent a letter to Papa John’s board arguing that Schnatter had been improperly removed as chairman last week. Glaser claimed that Delaware law requires directors to be removed following a vote by shareholders, which Papa John’s did not hold before dismissing him.

Glaser also chided the board for not conducting an independent investigation of what happened on the conference call, as Schnatter had urged. “In other words, Mr. Schnatter’s conduct and recommendations were strongly in the best interests of the Company. The board’s handling of these same events, however, is not,” Glaser writes.

Yesterday, Papa John’s board of directors terminated an office leasing agreement, effectively kicking Schnatter out of his office at the company’s headquarters. A number of sports teams and other organizations have also moved to suspend marketing campaigns they had with the company.