Note: Text has been lightly edited for clarity and does not match audio exactly.

Yvonne Durant: Listeners, this is Editor Yvonne Durant at Audible. Today, we're with the historian, legal scholar, professor of history at Johns Hopkins, and author of several books, Martha S. Jones. We're going to talk about her memoir, The Trouble of Color: An American Family Memoir. Martha, I must say, this is the first memoir that I listened to that felt like I was listening to a thriller. It was very twisty and surprising, and I'm thinking, was it for you?

MJ: Well, that's absolutely right. I really wanted to take folks along this journey with me and to experience not only the twists and the turns, the dead ends, the mysteries that persist and more, in part because I think that's part of the story of our families, that we can't tie everything up with a neat bow. We live with the mysteries, with the surprises, and even with the, I don't know, would you say silence? I think that's part of the story.

YD: Right. Listeners, for your information, Martha does the narration. I want to ask you, how did you find that experience? Was it difficult at points? Because you're telling your story and it wasn't always hahaha. Tell me about it.

MJ: Yeah, I don't think it's easy to read your own words out loud as a writer. You have to resist the urge to edit one more time, because we all want a chance to edit one more time. But I had an amazing producer, Josie, and a terrific engineer, and that meant that I was in great hands. That is huge.

In an interesting way, it surprised me. There's a tag at the end of the audiobook about copyright and the rights of authors. I found that very moving. We live in such a strange time. To say out loud what it means for authors to write, to put their work into the world, and to have it be protected and honored, that really touched me in a way I hadn't expected. So, it was full of surprises, actually, reading it aloud.

YD: You gave us a lot of information. Highly personal information. It's out there now. Wanna take anything back?

MJ: Oof. No. Absolutely not. I think this kind of conversation is affirming for me that the things that were hard to put out there were important to put out there, because I'm hearing how they connect to other people's stories. That is one of the reasons we write our books, is to make space for other folks. I think I have felt affirmed by early readers and listeners to the book who see parts of themselves in the book.

YD: Right. Just an aside from me, I was very impressed that you were able to find so much. My family have so little documentation, so that was just terrific. I wanted to ask you, you were about to look at a family tree, I think maybe you were in a library someplace, and you said, "Before I allowed myself to look at this…" Why the word allowed? Were you apprehensive? You had had enough for the day? Can you explain why you used the word allowed?

MJ: Well, I think for me, it's not unique that I am in the early chapters of this book in the time of slavery. There is very little that survives in my family, like in most families, that come directly to us from our ancestors who were enslaved. So, I hesitated for a while about, for example, going into the archives created by slaveholders. They exist, they are there for us, but it's a fraught and maybe even we'd say a toxic kind of space that I felt reluctant to enter. I had to exhaust everything I could that kept me out of those kinds of materials.

"Part of what this book is intended to do in its own way is to help folks read and hear stories not unlike their own... and to make those stories part of our American story and not a sidebar."

Then I let myself. I had to trust myself that I could go in and, in a way, not be bamboozled by the way in which a slaveholder tells the story, because that's not the story of my family. I need to be able to read between the lines carefully, rather than accept his story at face value. And that's tough when you have so little on the other side.

YD: Yeah, I see your point. My family is similar to yours, and our history shows on our skin. I have cousins with hazel eyes and dirty-blonde hair, and three of the sisters came out light, two of the brothers came out dark. And it's just so funny when I read about how enslaved women had no agency. We rarely use the word rape. We don't call it like it is a lot. We'll say, “take advantage of.” We stay away from that word. We all know that there're Blacks who look like you, like my cousins. That's the history. We're wearing it on our skins. Do you think we romanticize that sometimes? I know rape is a hard word. And then I'm going to give you another zinger here. How is it that Black men kind of, quote, are "blamed" for owning rape? How did that happen?

MJ: So, I think on your second point, the invention, the fiction of Black men as the perpetrators of sexual violence is a very deliberate one that is created, as I understand it, it's created at the end of the 19th century with the rise of Jim Crow, with the rise of segregation and disenfranchisement and lynching. And the great Ida B. Wells, the suffragist, journalist, and more, Wells documents the falsehoods that undergird the lynching of Black men. So that is very important. It is a cover, as Wells says. That's a cover for a long history of rape perpetrated by white men on Black women. Wells says, "You don't really want me to talk about that, do you?" to the men who are perpetuating these fictions about Black men to cover up their own violence. So, that's important.

Historians talk a lot about the language we use, and rape is now part of our vocabulary. I don't always use that word in this book. And the reason for that is that part of what I'm trying to do as I recount family stories as best I can, is to rely on and to honor the kind of language that women in the past used themselves. I'm with you. From a 21st-century perspective, rape is the right word. Maybe it's the only word. But my great-great-grandmother, though she complains formally about the men—

YD: Was that Moosie?

MJ: This is Isabella. Great-great-grandmother. She'd been enslaved. She bore nine children by the man who owned her. When she complains about him, as she does as a free woman, because he's not supporting her and her children, she refers to him as their father and doesn't give a label to her relationship to him. I have to sit with that. I don't have to agree with her exactly, but I think I have to respect that she's telling the story in the way she needs to tell it. And we need to hear that, too. Even as we have our own ideas about what that was and what that meant for her, it was important that he was a father, a father who she believed was obliged to support those children, even if they had been born of force or rape, as we would say.

YD: Right. I think she was protecting herself, protecting her children and soothing her pain as much as she could. I mean, she had these children, so to take care of them, right?

MJ: Absolutely. This is what I mean, that she tells the story she needs, the story that she needs to persist. She's going to be abandoned by this man. I'm not giving anything away in the book. I don't think that will surprise readers. But she has to persist, and that means she has to construct a narrative about who they are and where they come from, why they look the way they do, why they carry the names that they carry. Today, in the 21st century, I think we can say, I think you might agree, there's no shame that our foremothers endured what they endured. On their part, there's no shame. The shame is the shame of the men who perpetrated that violence.

YD: Sure.

MJ: Instead, we admire them, and precisely for what they endured and how they endured, how they made us possible by what they endured. But that's a 21st-century sensibility that hasn't always been true in our history. Both of those things can be true, right? That she might want to mute the violence. That was her way of coping. I can be more forthcoming about the violence. That's my way of coping. But both of those things, one is as true as the other.

YD: Right. By the way, I smiled at the mention of Adrian Piper, the artist. I know her work. I went to her retrospective at MoMA, Museum of Modern Art.

MJ: Me too.

YD: She had those cards. I remember, years before, reading a piece in Mademoiselle magazine, to show you how long ago, and she used to give out this card. She has very fair skin, and people would say dumb stuff, racist stuff, not realizing she was Black. So she would give out these cards, do you remember, “Excuse me, what you just said, maybe you don't know this, but what you just said, it was racist…" Something like that. And she'd hand people that card, which I thought was fabulous.

MJ: Piper is brilliant. I think for a lot of my young adult years, Piper is a beacon. Because she's stepping fully into her, what I would refer to as her ambiguity. The way in which she confounds the eye when it comes to race. And the cards are one manifestation of that. She has another piece, a self-portrait, it's something like “Me Exaggerating My Negroid Features.” I love that piece too. She was one of the few folks who were really public and unapologetic and, of course, sophisticated in thinking about the sort of dilemmas that visit us when folks can get us wrong, if you will, by the eye.

"I don't think our work or our purpose as Black folks is to police one another. I don't. I think it's to understand and to care for one another."

I was at that MoMA retrospective, and a lot of folks I know were moved and amused and, in a sense, seeing a part of our stories writ large. That's part of what this book is intended to do in its own way, is to help folks read and hear stories not unlike their own, represented in the big world. And to make those stories part of our American story and not a sidebar, and not an exception, but much more at the center of who we are than maybe we thought.

YD: I was happy to see her story within your story, that you included her, that was terrific. So, you're in North Carolina one day doing some research. You get hungry, go into a diner, a waitress comes up to you. And what does she ask?

MJ: Remarkably, she asked me, out of nowhere, "What race are you?" What?! Now, I've heard that question many times. That won't surprise you. You're looking at me. Folks can't see me. They can Google me and see how I appear to the eye. But I'm alone. I've made a pilgrimage back to North Carolina. This is where my people are from, but now most of them are gone, and I've come back and I'm terrified. I'm terrified. I don't know what answer to give, and I don't know what the repercussions of the answer are, is the truth. For a long moment, the alarm bells go off before I realize my waitress is just curious. If there is such a thing in America. She's just curious.

YD: Meanwhile, she was a woman of color, Filipino.

MJ: I remember the thing that took three more beats to sink into me in the moment, which is that she wasn't somebody who was necessarily coming to me by way of a longstanding American South, Black-white binary perspective. I think she really was curious, and maybe the American South baffled her, as it should baffle all of us when it comes to race and color. But she moved on and went to the next customer after I gave her some answer or another. I can't even tell you what answer I gave.

YD: Yeah, and that's interesting. First of all, that she probably encountered the same question. I don't know how many Filipinos there are in that town of North Carolina, so she probably said, "Well, people ask me, I guess it's okay." Who knows what informed her reasons for asking you. But you don't have an idea of how you answered or what you could have said?

MJ: No, I think that my friends would say that's my PTSD kicking in. They're being funny, but it's to say, I think there are a number of moments in the book when I can't quite recall what I said, and I think that is because the alarm bell goes off and the adrenaline goes up and you're in that fight-or-flight kind of state of mind. I know what I did, but I don't remember what I said.

YD: I'm looking at you, I see a light-skinned Black woman. White people will look at you and they won't have a clue. Like, we can tell, I could tell in a room of white people who's Black. If I saw you, your skin talks to me. That's because you could be my cousin, I guess. But some white people don't have a clue. Why do you think that is?

MJ: Well, one answer is because we live with the legacy of Jim Crow and we live lives that are much more segregated and much more separate than the state of the law might suggest. Which is to say, when you meet me, you go through your family archive, your family Rolodex, to read me. But if you're someone who hasn't been part of or exposed to or engaged with the fullness of Blackness, maybe you can't see me.

But I think the other answer to the question is context. I think that I certainly live a professional life where it would never occur to folks sometimes that a Black woman would be in the room, at the table, at the podium, at the microphone. And so now it's about class and status that shapes people's expectations about who's in the room. I'll say, in my experience, I think Black folks can be as much a victim to that as white folks, right?

YD: Oh, yes.

MJ: We read people by the context in which they're in. I lived for years on the Lower East Side of Manhattan, and everybody thought I was Puerto Rican. I went to Hong Kong and people thought I was South Asian. And that's about context. That is a lesson, right? In the ways in which the room we're in figuratively really matters, can matter for how we're read. Tack onto that, degrees and professional accomplishment, class, and I think you further confound things. Then I'm going to say a thing, I hope it's okay to say it, and then I go to Martha's Vineyard in the summertime and everybody knows who you are. Without knowing who you are. And that's context once again.

YD: Exactly. So, here's another little thing that made me chuckle. I lived two blocks away from the Lenox Hill Neighborhood House.

MJ: Oh, my goodness. That is too much.

YD: When did your father work there? Late ’50s, ’60s?

MJ: Mid ’50s.

YD: I would've never guessed a Black man worked in there. They knew he was Black, didn't they?

MJ: I believe so.

YD: That's where he met your mother, right?

MJ: That's where he met my mother. And that tells you something about him, doesn't it? You can appreciate it, that he had been born in North Carolina and raised in Black Greensboro, knew who he was, but had been sent away to school and had been educated in predominantly white settings, such that I think he knew how to maneuver in a space like Lenox Hill, even if he was the only one, and I suspect, as you're saying, I suspect he was the only one, but I don't know for certain. In fact, part of his pride was that he knew how to maneuver the world in that way. He played bridge with the ladies at Lenox Hill and met my mother and married her. He really made a life there. It wasn't merely a performance. That's part of who he was.

YD: Yeah, I found him fascinating, and back then, no one was talking about dyslexia, right? Yet even with this issue, he went on to several schools. He always got in, right?

MJ: Yeah, he did. I think this is true in his generation for many people with learning disabilities, he was regarded as lazy, not studious enough. But I share your sense, it's remarkable what he accomplished, given that no one, despite a really fine education and educators who were very committed to him, he accomplished as much as he did. I have his letters. This was back in the day before the telephone call was a thing you did except in an emergency. And so he writes to his parents every week for years and years and years, and I can see literally on the page how his ability to write transforms by his sheer will. By his sheer will. Because no one has diagnosed him or taught him, as we do today, how to overcome or move around the challenges of dyslexia. He does that on his own, and I admire him so much for that.

"We come from pain, we come from injury. It is worth reminding one another that that's not the only way we need to come to each other."

YD: That’s amazing. Do you think there were times in his life, not that he passed, but he just didn't correct people?

MJ: I'm sure that by the end of his life, I think partly out of fatigue, and I understand that. How many times a day can you actively or proactively correct people who are misreading you? But I think also out of a sense, you know, my father was of the civil rights generation, and he wasn't a naive man or a romantic, but he certainly had come of age in a world where he did want to just be Paul. I think many people shared that aspiration, and I think in his own modest way, he arrived at that place. He and I made very different choices about the kind of worlds we live in and how we wear our Blackness, if you will. But I understand, and that was part of what was important to me in this book, was not to judge him, or even to draw a sharp contrast between him and me, but to understand him and to appreciate that he was a happy person, especially later in his life. And that's remarkable.

YD: Well, he did it his way.

MJ: He did. And isn't that part of the story too, I think? There are many aspects to our families and to our culture, including the policing of all kinds of boundaries. I came to appreciate in this book how imposing and how ill-fitting those boundaries can be in our lives, and that I don't think our work or our purpose as Black folks is to police one another. I don't. I think it's to understand and to care for one another. And that's always been true. That is certainly true here in 2025. And the book helped me, I guess I would say, develop that muscle, which is how to come to people with a mind toward understanding rather than with a mind toward policing.

YD: Yeah. And not assuming they're out to hurt you. People just don't know. There's a lack of exposure. I'm so glad my parents, we went everywhere, did all kinds of things, went to integrated public schools, so important. But your family is so impressive. You have doctors, lawyers, educators. Listeners, Martha's family, was it your uncle who founded Bennett?

MJ: My grandfather was the first president of Bennett when it reorganized as a women's college in 1926.

YD: Yes. A very important historically Black university, college.

MJ: His mother, my great-grandmother, Jenny, is part of the church community in Greensboro, North Carolina, at St. Matthew's Methodist Church. Bennett is one of those schools that is founded after the war by former slaves in a church basement. And my great-grandmother is there, I think, having no idea that in 1926, her baby, her last child, would come back to Greensboro to be president. I don't think her ambitions went quite that far, at least not in 1873. But it's to say he had a long and important connection to Bennett, so that when he came back, it was a homecoming for him.

YD: I learned a lot listening to your book. I didn't know Alice Walker brought us the word “colorism.”

MJ: Thank you to Alice Walker for giving us that language to interrogate the problem of color, even among and between us.

YD: Yeah, we're damaged too. I mean, for what we've gone through hundreds of years, I always feel like unless every other day someone's tapping me on a shoulder, gently, reminding me as if I don't know who I am or what I am. Sometimes it's subtle. Sometimes it's not. I don't get upset anymore. I do say, "Is this person worth schooling?"

MJ: I hear you, and I feel you. What I sometimes say to people is Blackness is not a card. Nobody gets to revoke your card because they don't appreciate how you come to Blackness. That's the first thing. Now, I'm a woman of mature years, and so I say I've been at this too long. You also don't get to send me home. I'm not going to go anywhere. But I also want to say to folks, a bit maybe what I hear you saying, which is that we come from pain, we come from injury. It is worth reminding one another that that's not the only way we need to come to each other. For me, maybe that's one of the strong lessons of the book.

YD: Well, all I can say, you must be so proud of you, of yourself, when you look at all the work you've done. You're relentless. I know that there was pain, there was surprise. Like I said, it reads like a mystery. I'm not even going to give away one of the things I'm wondering about, but it doesn't matter. Listeners, you have to listen. Martha, thank you for saying yes to this interview. I so appreciate it. It's nice getting to know you. And listeners, you can find The Trouble of Color on Audible.

MJ: Thank you.