Robert Hesse was expecting an imminent promotion to manager of Sub Zero Ice Cream, a nitrogen ice cream shop in Ventura, Calif., when it shut down in March because of the pandemic.

“I like to work,” said Mr. Hesse, a college graduate who turns 26 on Tuesday. “Otherwise I feel like I’m useless.” But he has been reluctant to seek a new job because he lives with his parents, who are not yet vaccinated, and is afraid of bringing the virus home to them.

“It’s just health concerns — I don’t really want to be around the general public yet,” he said.

Mr. Hesse represents what economists say is one of the most striking features of the pandemic-driven economic downturn: the tide of workers who, as the government counts things, have left the labor force.

In the year since the pandemic upended the economy, more than four million people have quit the labor force, leaving a gaping hole in the job market that cuts across age and circumstances. An exceptionally high number have been sidelined because of child care and other family responsibilities or health concerns. Others gave up looking for work because they were discouraged by the lack of opportunities. And some older workers have called it quits earlier than they had planned.

These labor-force dropouts are not counted in the most commonly cited unemployment rate, which stood at 6.2 percent in February, making the group something of a hidden casualty of the pandemic.

Now, as the labor market begins to emerge from the pandemic’s vise, whether those who have left the labor force return to work — and if so, how quickly — is one of the big questions about the shape of the recovery.

“There are a lot of dimensions related to the pandemic that I think are driving this phenomenon,” said Eliza Forsythe, a labor economist at the University of Illinois. “We don’t really know what the long-term consequences are going to be because it is different from the past.”

There is some reason for optimism. Economists expect that many who have left the labor force in the last year will return to work once health concerns and child care issues are alleviated. And they are optimistic that as the labor market heats up, it will draw in workers who grew disenchanted with the job search.

Mr. Hesse, for instance, said he planned to look for a new job in earnest once he is vaccinated and hoped to go back to work this year.

Moreover, after the last recession, many economists said those who left the labor force were unlikely to come back, whether because of disabilities, the opioid crisis, a loss of skills or other reasons. Yet labor force participation, adjusted for demographic shifts, eventually returned to its previous level.

But the speed with which the pandemic has driven workers from the labor force has had devastating effects that could leave lasting damage.

The labor force participation rate among those 16 or older has dropped to about 61 percent from 63 percent in February 2020. Among prime age workers — those 25 to 54 — it has declined to 81 percent from 83 percent.

Women in their prime working years have quit the labor force at nearly twice the rate of men, according to research by Wells Fargo, partly because more women work in industries like leisure and hospitality that are less suited to social distancing and partly because women are more likely to bear the burden of child care. The share of Black women who have left the labor force is more than twice the share of white men.

Then there are the many people who may be seeking a job but who are unavailable to take one because of health concerns, illness or caretaking obligations, putting them in what economists say is something of a gray area — between being unemployed and not in the labor force — that has become more common during the pandemic.

A single mother, Frankie Wiley, 29, worked as a housekeeper at a resort in Bloomington, Minn., until she was laid off last March. She would like a paid job, but she has to stay home with her 11-year-old daughter, who is attending school remotely.

“I take care of her, so I’m her only support,” she said. She said she plans to return to work once her daughter can go back to school safely.

Older workers have exited the work force in droves, including those who left out of health concerns or illness or who took the opportunity to retire early. Among those 55 or above, labor force participation has fallen to 38 percent from 40 percent in the last year.

A study from the research firm Oxford Economics estimates that around two million workers have left the labor force to retire since the start of the pandemic, more than twice the level in 2019.

That was the case for Ed Hoag, a public librarian for 35 years, who decided on an early retirement last summer out of concerns for his health. He and his wife have no children, and he was worried that if either of them got sick, there would be no one to take care of them.

Now 60, he spends his days reading at his home in Lambertville, N.J., where he moved a few years ago in anticipation of a retirement that had once seemed much further off.

“I do miss working,” he said. “I miss my colleagues and I miss the activity of the library, the people that would come in, the jobs we did. I do miss all that interaction. But I think that for myself and my wife, it was the right decision to make.”

For the legion of older workers who hope to return to work after the pandemic, a challenging path may lie ahead. Studies show that older people who leave the work force will have a more difficult time re-entering it because of age discrimination and other reasons. If that reality holds during the recovery, the number of older workers who have left the labor force — either because they could not find a job or because they retired early — could be one of the pandemic’s enduring consequences.

One prevailing question is whether employers, as in the past, will look askance at those who have been out of the labor force for a significant time.

Even in a tight labor market, long-term unemployed workers faced a stigma, said Maria Heidkamp, the director of the New Start Career Network, which helps older job seekers in New Jersey.

“In addition to any age, race or gender discrimination that they may already encounter, there’s a lot of evidence that it is easier to get a job if you already have a job,” she said. Though employers may overlook any pandemic résumé gap, she said, “there’s no reason to think that that is going to be different for these people, who are on the sidelines right now who want to come back.”

Still, because of the pandemic’s unique economic impact, many economists believe that the extraordinary number of people who have left the labor force will be more of a temporary blip than emblematic of a deeper structural issue.

“I don’t think overall the U.S. labor force participation rate is going to get stuck at a lower rate,” said Betsey Stevenson, a professor of economics and public policy at the University of Michigan, who was a member of President Barack Obama’s Council of Economic Advisers.

Already there is evidence that people who left the labor force are returning to work.

Labor participation among young people, which tumbled in the early stages of the pandemic, has rebounded significantly as service industries bounce back.

And as the vaccination rate continues to rise and restrictions on activity lift across the country, many more people who have left the work force are beginning to plot their returns.

Since Heather Kilpatrick lost her job in private-event sales last March, she has spent her days at home in East Boston caring for her daughter, now 3.

Without her additional income, she and her husband, co-owner of a restaurant, could no longer afford day care at the local Y.M.C.A. So although Ms. Kilpatrick, 36, ached to go back to work, she felt as if she were trying to solve a chicken-or-egg dilemma.

“No disrespect to women who want to stay home, but that’s never been me,” she said.

Recently, she finally accepted a part-time job working from home for a restaurant group.

Her job began last week.

Ben Casselman and Jeanna Smialek contributed reporting.