JAFFREY — After nearly four decades of complex problem-solving, late-night phone calls and often punishing hours, Dr. James D. Potter is set to retire from practicing medicine.

Despite the challenges of his job, Potter said he looks back on his career with gratitude and relishes what he described as the detective work inherent in the profession.

“There’s the occasional challenging case where I feel I was instrumental in finding some unusual diagnosis and steering someone in the right direction so they got the care they needed in a timely enough way,” he said.

June 30 will mark Potter’s final day at Jaffrey Family Medicine, which is owned by Monadnock Community Hospital in Peterborough. He has worked there for 22 years.

Dr. Lisa Ramey of Peterborough worked alongside Potter for much of that time. She described herself as a great fan of his.

“I quote him frequently,” Ramey said. “He always said that the important thing is to make sure that everybody knows that they matter. That should be key in any profession. Jim has always made that clear, so that we’re really listening and making sure we know what people need.”

Ramey commended Potter for his energy and ability to juggle multiple positions, including stints as medical director at Compassus Hospice and Palliative Care in Bedford, Good Shepherd Rehabilitation and Nursing Center in Jaffrey and RiverMead in Peterborough.

“I’ve always been amazed at how many hats he was willing to wear,” Ramey said.

Born in 1953, Potter grew up in Santa Ana, Calif. His father, Jack, was a general surgeon, while his mother, Margo, was a homemaker with a talent for painting.

Early on, Potter had an interest in science and knew he wanted to pursue medicine — he just didn’t know which kind. He went through a phase in middle school where he thought he’d like to be a veterinarian but was swiftly disabused of that notion after shadowing a vet performing surgery on a dog with a broken leg.

“It just seemed kind of impersonal,” Potter said. “It was almost barbaric. I didn’t really feel like that was for me.”

By his sophomore year of high school, he knew for certain he wanted to go to medical school.

Growing up in Santa Ana, Potter was a Boy Scout and played several instruments — trombone, glockenspiel and clarinet, among them. During his junior year, his high school marching band performed in the famous Rose Parade that precedes the Rose Bowl NCAA football game in Pasadena.

Looking back on that day, he recalled lugging around the glockenspiel in the unrelenting Southern California sun. The xylophone-like instrument weighs more than 40 pounds, he said.

“It’s a very long parade,” Potter added. “And we had these big wool uniforms with this metallic-inlay vest over the top and these big furry hats. You’re just sweating underneath all that.”

After high school, Potter studied chemistry at Harvey Mudd College, a small liberal arts school in Claremont, Calif. When he graduated with a degree in chemistry in 1975, his class size was 65 people.

Potter said he enjoyed the rigorous academic environment at Harvey Mudd and had opportunities to pursue various research projects. He and some other students received an $8,000 grant from the National Science Foundation to research the environmental impact of a proposed expansion project at Ontario International Airport in Ontario, Calif.

“Basically, what it showed was that the pollution was so bad that, yes, there would be an increase, but people wouldn’t even notice,” Potter said.

He attended the University of Rochester School of Medicine and Dentistry in Rochester, N.Y., from 1975-79. His experience conducting research at Harvey Mudd led him to believe he wanted to do a combination M.D.-Ph.D. program and study brain chemistry.

“But it didn’t take me very long after I got there to realize that working in a research lab wasn’t for me,” Potter said.

He recalled visiting a laboratory in Rochester and watching researchers running experiments on rats. The researchers removed the brains of those rats, he said, and tossed them in a blender to test for chemical balances.

“That’s what you do?” he said in mock disbelief, his eyes going wide.

After that, Potter’s focus quickly shifted to primary care. He found the interpersonal aspect of the work appealing, far more so than liquefying rat brains.

“I very quickly switched to the idea that I wanted to be working with people one-on-one,” he said.

Since the beginning of his career as a primary care doctor, Potter has been drawn to rural communities. In fact, he describes the through line as a progression to smaller and smaller towns.

“Growing up in a more urban environment, I’ve always liked the countryside and open spaces,” Potter said.

In Santa Ana, he joked, the homes were so close together that if you weren’t careful, you’d bump your elbow against the neighbors.

After completing his residency at the former Somerset Medical Center in Somerville, N.J., Potter started looking at communities around northern New England. He wrote letters to medical professionals in Maine, New Hampshire and Vermont, asking whether they could identify communities in need of a family doctor.

The lists they sent back were overwhelming, Potter said. He decided to travel around the region and see some places for himself, visiting about a dozen communities and meeting with professionals at local hospitals. One destination jumped out at him.

“When I drove into Peterborough, I knew it was the kind of place I was looking for,” Potter said. “It just felt right.

Potter started his own practice in Peterborough in 1982, renting office space in a building owned by Monadnock Community Hospital. He took out a loan to renovate the space, purchase office equipment and place ads in the paper. Within minutes of the phone getting connected, prospective patients started calling.

“I think I saw eight or 10 people in the office that first day while we were still trying to unpack,” Potter said. “It took off like a shot.”

Potter remembers those first couple of years on his own as demanding and hectic. The job required him to be on-call 24/7. But early on, he discovered family medicine was his great passion, with continuity of care as his guiding principle.

“That’s the key,” Potter said. “You’re getting to know people really well over a number of years where you are their family doctor. Ideally, you’re taking care of multiple members of the same family and even multiple generations.”

Dr. Richard P. Frechette, a physician at Monadnock Family Care in Peterborough, said he’s known Potter for 31 years. The two became well acquainted volunteering as Boy Scout leaders in Peterborough, Frechette said.

“Jim was a pretty valuable leader,” he added, mentioning cycling trips to Nantucket and canoe trips in the North Country.

Frechette described Potter as a very good doctor — diligent, quiet and steady.

“He’s leaving a very big gap,” Frechette said. “A lot of patients really liked him. There are very few talented family physicians who want to come work in the country.”

Ramey agreed.

“I think a number of patients are still in denial,” she said of Potter’s retirement. “It’ll be very, very tough to fill his shoes.”

During his early years in Peterborough, Potter was picking up shifts in the emergency department and the intensive care unit at Monadnock Community Hospital. There, he met his wife, Patti, in 1983.

An Alstead native, Patti was working in the radiology department. After getting to know each other on breaks in the hospital cafeteria, they began dating and got married in October 1984.

“We were a hospital romance,” Potter said.

After four years of solo practice, Potter took a job as medical director at the former Beech Hill Hospital, a substance-abuse treatment center in Dublin. He held that position for about 10 years.

Substance-abuse treatment was a dramatic change from family medicine, Potter said, and provided a tremendous learning opportunity.

“It took me into a whole other area of medicine that I never thought I’d have any interest in,” Potter said. “But it was remarkably rewarding.”

In addition to helping people struggling with addiction, Potter said he enjoyed being part of an interdisciplinary team and working alongside a wide range of professionals — psychiatrists, psychologists, counselors, nutritionists and nurses.

And there was another perk to the position: regular hours.

“For the first time, I had two out of every three weekends off,” Potter said. “I could come home at night and actually have dinner with my wife.”

But over the years Potter said he found himself spending more and more time on the phone arguing with insurance companies, trying to get one more day of coverage for patients fighting addiction. He grew frustrated with other aspects of the hospital, he said, and decided to return to primary care in 1996.

“That’s where my roots were, and that’s what I really love.”

Potter got in touch with some associates at Monadnock Community Hospital and learned there was an opportunity opening at Jaffrey Family Medicine. He took the job without hesitation.

Potter said he’s excited about his retirement but is finding it difficult to say goodbye to patients he’s known for decades. He’s been saying those goodbyes on a one-to-one basis for the past several months.

“That whole continuity of care concept that drew me to primary care in the first place also makes it hard when you have to say goodbye,” he said. “You get to know these people really well, and it’s almost like they’re extended family.”

Contemplating the free time ahead of him, Potter is looking forward to focusing on one of his favorite hobbies — woodworking. He’s especially enthusiastic about a niche area called segmented turning, a type of lathe-based woodworking where the initial work piece is composed of multiple parts that have been glued together.

“I’m kind of a novice at it,” Potter said, showing off some of the ornate bowls and vases he’s fashioned over the years. “I’m very much on the steep part of the learning curve, but I’m getting better.”

About six years ago, Potter and his wife moved from Peterborough to a home on Thorndike Pond in Jaffrey. As an avid hiker and cross-country skier, he can now take full advantage of those pristine, rural surroundings.

“It feels right,” Potter said. “I don’t want to wait so long that I can’t do the things I really want to do. It’s a good time to retire.”