In a rebuke of a potent New Jersey political machine, Amy Kennedy, a former schoolteacher whose husband is a nephew of President John F. Kennedy, won Tuesday’s primary battle to take on Jeff Van Drew, a freshman congressman who defected from the Democrats with a pledge of loyalty to President Trump.

Ms. Kennedy’s main opponent, Brigid Callahan Harrison — who was backed by both of New Jersey’s United States senators and two of its longtime political power brokers — conceded the race in a YouTube video soon after the polls closed at 8 p.m. An hour later, The Associated Press declared Ms. Kennedy the winner.

New Jersey’s primary election was conducted mainly using mail-in ballots, and the official results were not expected to be known for at least a week as ballots, which needed to be postmarked by Tuesday to be counted, trickled in.

It was the first broad test of voting by mail in New Jersey, making Election Day as much a referendum on the voting method as on the candidates who were running.

Ms. Kennedy, a mother of five, will now challenge Mr. Van Drew, who left the Democratic Party in December and was endorsed by Mr. Trump.

“People here in South Jersey are ready for change,” Ms. Kennedy said in a victory speech that aired live on Facebook a little after 10 p.m. “We need leaders who will feel when we hurt, listen when we speak, provide direction, guidance, and show up when it matters.”

Ms. Kennedy, who was introduced by Gov. Philip D. Murphy, focused on Mr. Van Drew, saying, “We’ve had enough of you, and Donald Trump.”

In a taped concession speech, Dr. Harrison, a professor of political science at Montclair State University, urged Democrats to unite behind Ms. Kennedy. “Each of us has the responsibility to get involved and help her any way we can,” she said.

Ms. Kennedy, 41, was backed by Mr. Murphy, and the race was seen as a proxy war between the governor and his two main Democratic rivals, George E. Norcross III, an insurance executive who wields significant power in Trenton, and Stephen M. Sweeney, the Senate president.

Progressive groups that supported Ms. Kennedy said the results demonstrated that the once-invincible clout Democratic Party county leaders have wielded in New Jersey no longer offered an ironclad path to victory.

“For decades, the South Jersey Democratic machine has been a barrier to progressive change,” said Sue Altman, the leader of the left-leaning Working Families Alliance. “This is an absolutely huge win for progressives.”

Mr. Norcross released a statement praising Ms. Kennedy’s campaign.

“Congratulations to Amy Kennedy, who has won a strong victory in today’s primary,” said Mr. Norcross, a member of the Democratic National Committee whose brother, Donald Norcross, represents a neighboring district in Congress. “As I said months ago, I look forward to supporting the Democratic nominee in the general election.”

The contest for the Second Congressional District, which stretches from Atlantic City west to the Pennsylvania border, now represents a race between Mr. Trump, who held a raucous rally for Mr. Van Drew in Wildwood, N.J., and the Kennedy political dynasty. Mr. Trump beat Hillary Clinton by 5 percentage points in the district, which for 24 years before Mr. Van Drew’s election was represented by a Republican.

Credit…Doug Mills/The New York Times

With help from Mr. Norcross, Dr. Harrison notched the support of six of the district’s eight county chairs, but Ms. Kennedy won the coveted Democratic Party line in Atlantic County, where more than a third of the district’s voters live.

Will Cunningham, 34, a lawyer and a former investigator for the House Committee on Oversight and Reform who campaigned as a progressive, also mounted a lively primary challenge that effectively leveraged social media during a time when most traditional campaigning was halted by the coronavirus pandemic.

New Jersey voters also were deciding dozens of other primary races, including a Republican primary to take on Representative Andy Kim, a Democrat in a swing district.

Only about half of the state’s polling sites were open for the filing of provisional ballots, and voters could also deliver ballots in person to drop boxes set up in each county.

Mr. Murphy has said he was watching to see how the voting went before determining how broadly to rely on mailed ballots for the presidential election in November, where the stakes will be much higher.

“We’re going to be watching very closely for any shenanigans that we hear about,” Mr. Murphy, a Democrat, said on Monday.

In states like Washington that have long embraced voting by mail, there has been no evidence that it leads to widespread fraud. That has not dulled sharp attacks by Mr. Trump, who has made dozens of false claims about the process and had predicted that it would result in “the most corrupt election in the history of our country.”

In the midst of the pandemic, there was nothing normal about how candidates campaigned. And Election Day itself veered sharply from the typical script.

There were no perfunctory photographs of candidates casting votes with their spouses and no insider buzz around victory parties or early poll results. Get-out-the-vote efforts began weeks ago, as campaigns were able to check on who had returned ballots and who had not.

Those who had not were targeted by telephone and text.

Hector Oseguera, 32, a lawyer who lost his primary challenge in northern New Jersey against Representative Albio Sires, said his campaign had planned to send 500,000 text messages to voters in the two weeks before Election Day, with a goal of contacting each voter four or five times.

Employees at election boards in New Jersey’s 21 counties were also in uncharted territory, having to confirm signatures on ballots while also tallying votes.

“This can serve as a test run for November,” said John Froonjian, the executive director of the William J. Hughes Center for Public Policy at Stockton University. “New Jersey has to be prepared to learn from what happens today and the next few days.”

Professor Froonjian moderated an online debate last month between three of the Democrats competing to run against Mr. Van Drew, who was elected as a Democrat and joined the Republican Party in December after voting against Mr. Trump’s impeachment.

Mr. Van Drew’s campaign to win re-election will be one of the most-watched congressional races this year, and the Democratic primary to choose a candidate to try to unseat him quickly turned toxic.

Ms. Kennedy, an advocate for mental health reform, was far better funded than her opponents, in part because of $500,000 in loans she made to her campaign.

In the final weeklong sprint to Election Day, a super PAC aligned with Mr. Norcross spent nearly half a million dollars on television advertising on behalf of Dr. Harrison. But in a race where most voters had sent in ballots weeks earlier, it proved to be too late.

Ms. Kennedy will be on the ballot in November alongside a voter referendum question about whether to legalize marijuana in New Jersey, a proposal she opposes but Mr. Murphy has championed. Her husband, who has spoken frequently about his struggle with addiction, is ardently against legalizing marijuana.

The final results of a Republican primary in a nearby South Jersey congressional district that also had its share of intrigue remained unclear. A former business executive, David Richter, 54, who had initially planned to challenge Mr. Van Drew in a general election as a Republican, switched districts after Mr. Van Drew switched parties.

He was facing a former elected county official, Kate Gibbs, who is 20 years younger. As a woman, Ms. Gibbs could appeal to female voters — a coveted group for Republicans hoping to expand their reach in New Jersey.

The winner will take on Mr. Kim, who narrowly won election in 2018 after a drawn-out campaign against a key ally of Mr. Trump’s.