I grew up in Arlington, Va. My mother worked a night shift at a drugstore, and my father was a cab dispatcher for National Airport. In high school I had a great athletic director and wanted to be just like him. So I went to George Washington University, where I received a BS and master’s degree in education. I was hired as GW’s intramural director, then promoted to assistant athletic director in 1975.

When GW built its 5,000-seat Charles E. Smith Center, I hired an advertising firm to do marketing and met an executive named Harry Rhoads Jr. One day he sent me a Fortune article about the Harry Walker Agency. In it, Henry Kissinger asked why he should sign with Walker instead of a competitor. Walker’s response was, “I don’t have any competitors.” Harry [Rhoads] taped a note to the page with the words, “No competitors?”

It was 1980, and my wife, Paula, saw it as an opportunity. Paula knew that no matter how happy I was as an athletic director, I’d never be happy working for someone else. She persuaded me to end my teaching career and start my own business. She quit her job as a program manager for the University of Oklahoma in D.C. and convinced Harry to quit his job.

We set out to start a lecture agency. We had no savings, a 1-year-old baby, and a mortgage. So we got a second mortgage for $25,000, and Harry chipped in $25,000. We lived on that $50,000 for almost two years.

I called a friend who had a large stationery closet in his office. He let us put two desks and two phones in it. In 1980 there was no Internet. We’d think of famous names and write to them. We’d get a lawyer’s letter back, saying not to interfere with an existing agreement. That’s how we learned there were actually four or five big agencies and some mom-and-pops on the East Coast. Sitting in our closet late at night, I would close my eyes, shake my head, and ask myself, “What have we done?”

We weren’t booking anyone and were within two to three months of having to give up when I got a call from Steve Bell, then anchor for ABC’s Good Morning America. I had gotten him access to the GW pool for a story earlier. Steve had left his agency and called to see if we’d represent him. The tradition then was to sign a contract for a year or two. We were so excited, we sealed the deal with a quick handshake and no paperwork.

In D.C., everyone knows everyone. Word of our handshake spread. Knowing they could walk out on us at any time attracted others. Knowing we could lose them at any time made us work harder. It had taken 10 months to get our first speaker, and that first year we earned $700.

In those days our savings were always running out. Our Friday nights were spent watching Dallas, putting brochures in envelopes, licking stamps, and driving to the post office. If I had a romantic dinner with Paula, at some point I’d say, “What would you think about this person as a client?” When you start a company knowing nothing, all you do is obsess about it. We never took a vacation.

After Steve, we signed more journalists: Peter Jennings, David Brinkley, George Will, and others. Then we had a huge growth period, adding Alex Haley, Oliver North, Terry Bradshaw, and others.

The big turning point came in 1988, when we got an interview to represent President Ronald Reagan. All the big agencies were considered. In February 1989, after he had left office, we got a call that the President and First Lady had selected us. I was stunned.

I’m superstitious, so I never asked why we were chosen, on the off chance it was a mistake. Years later I learned that the President selected us because we were the only startup, and he wanted to give us a chance.

After Margaret Thatcher decided to retire, President Reagan called her to recommend us. Gen. Norman Schwarzkopf called us from a bunker near the end of the Gulf War to ask us to represent him. We’ve represented three out of the last four U.S. Presidents, the last four Prime Ministers of England, countless world leaders, journalists, authors, and many sports legends.

I earned people’s trust by staying at my desk and doing my job. I never took things for granted. Every person I’ve represented I consider a good friend. That came from putting their interest in front of mine. If they had a travel problem at 1 a.m., I was there to help. I’d hire a private plane for them at my own expense. We changed the industry by doing all the travel arrangements ourselves so the client could rest easy, knowing the speaker was coming. Even today, everyone is signed with a handshake.

All the people I’ve represented got where they are because of a passion and a turning point in their lives. I wanted to share their stories to inspire others. They show that if you have a passion about something, you can make it happen, no matter your background. 

How to Give a Great Speech

Bernie Swan, Cofounder and chairman of Washington Speakers Bureau

Love your content and focus on your own experiences. The most compelling speeches deal with adversity and how you overcame it. You can tell those stories with emotion. The greatest lesson you can share is, “I survived, and you can too.”

Step away from the podium and practice. Think of giving a good speech like a concert pianist preparing for a performance. Practice your speech in front of several people who don’t know you well.

Scout the room where you’ll be speaking. Walk around it while it’s empty so that you know the best places to stand. You want to feel as comfortable as if you were walking around your living room.

A version of this article appears in the September 1, 2016 issue of Fortune with the headline “The Man Behind the Orator.”