WASHINGTON — During his last regulatory stint in Washington, Gary Gensler focused on reining in big Wall Street players that he believed were manipulating markets and assuming huge financial positions to the detriment of other investors.

If confirmed to lead the Securities and Exchange Commission, Mr. Gensler will have to confront an entirely new spin on that same game: Thousands of small investors who have banded together to amass giant stakes in GameStop and other companies with the aim of toppling big Wall Street players.

The frenzy around GameStop, whose stock has soared 1,700 percent in the last month, presents a huge challenge for Mr. Gensler and the S.E.C., which will have to reckon with a fundamental shift in the capital markets as a new breed of investor begins trading stocks in unconventional ways and for unconventional reasons.

Rather than snapping up a company’s shares because of a belief in that firm’s growth potential, the investors who piled into GameStop, AMC, BlackBerry and others did so largely to see how far they could drive up the price. Their motivation in many cases had less to do with making money than with causing steep financial losses for big hedge funds that were on the other side of that trade and had bet that the price of GameStop and other firms would fall.

Their ability to cause such wild market volatility was enabled by new financial apps — like Robinhood — that encourage investors to trade frequently and allow them to buy risky financial products like options as easily as they purchase a latte. Options — which are essentially contracts that give the investor the option to buy a stock at a certain price in the future — were what helped put the “short squeeze” on the hedge funds that had shorted the company’s stock.

“What’s going on with GameStop has almost nothing to do with GameStop as a company,” said Barbara Roper, director of investor protection for the Consumer Federation of America. “When you see the markets essentially turned into a video game or turned into a casino, that actually has some pretty serious repercussions for the way we use the markets to fund our economy.”

The question for Mr. Gensler and the S.E.C. will be what they can — or should — do about it.

In a statement on Friday, the S.E.C. said that it was “closely monitoring” the situation and that it would “act to protect retail investors when the facts demonstrate abusive or manipulative trading activity that is prohibited by the federal securities laws.”

But unlike the typical type of fraud or manipulation that regulators like Mr. Gensler are used to pursuing, the current frenzy involves investors who have publicly acknowledged the risks they are taking and even boasted about losses. Forums like Reddit’s WallStreetBets have entire threads devoted to “loss porn,” where traders post screenshots showing their portfolios in the red, to applause from other investors. (“I’m proud of you” and “Respect” are among the typical responses.)

That dynamic poses a challenge for an agency whose primary mission is to protect investors by ensuring they have enough information when deciding whether to trade and to enforce securities laws that were written before many of the GameStop investors were even born.

“The S.E.C. has for years worried about hedge funds coordinating their positions and coordinating bear raids and otherwise engaging in activities to move around a stock,” said Tyler Gellasch, a former S.E.C. lawyer who heads the Healthy Markets Association, an investor group. “There are reporting requirements around that. But we’ve never really thought about that being done en masse and in public. The S.E.C.’s rules haven’t thought about what happens when it’s 100,000 people coordinating via Reddit versus three people coordinating via email.”

Those who know Mr. Gensler say his first move will probably be determining what actually caused the momentum and who benefited. While many big hedge funds got crushed by the trades, there is speculation among market participants and securities lawyers that other big funds may have been fueling — and making money off — some of the volatility.

“First of all, the S.E.C. has got to figure out what the hell was going on,” said James Cox, a securities professor at Duke University School of Law. “The first question is going to be an empirical one — how much of this momentum was created by the hedge funds having to cover their short position and how much of the rest was the impact of the options trading — either buying the options or just executing on the options.”

A bigger issue for Mr. Gensler will be figuring out corrective actions. While the stock market has always been something of a game, Mr. Cox and others say the recent events have perverted their original purpose, which is to provide a place for companies to raise capital by giving investors the information they need to determine where to put their money.

“When you see what’s happening with GameStop, you ask yourself, is this manipulation, is this mass psychosis or is there something wrong in our market structure that is causing this to happen,” said James Angel, a finance professor at Georgetown University’s McDonough School of Business. “This does illustrate some of the imperfections in our market structure and the real question is what, if anything, should be done about it.”

Mr. Gensler has spent the past several years teaching at the Massachusetts Institute of Technology, focusing on financial technology, cryptocurrencies and blockchain technology. His classes have addressed some of the knotty issues he will have to face if confirmed to the S.E.C., including the rise of new financial technology companies like Robinhood and the so-called roboadviser Betterment.

In a 2019 discussion at M.I.T., Mr. Gensler said it would be “best to show some flexibility” when considering whether to regulate fintech companies since heavy-handed rules could snuff out innovation. Mr. Gensler declined to be interviewed for this article.

If confirmed to the job, Mr. Gensler will have to tread carefully. The motivation behind the GameStop squeeze has been embraced by lawmakers and others, who see the trades as a welcome rebellion against the power of big Wall Street players and persistent inequities in the economy. Last week, Representative Alexandria Ocasio-Cortez of New York, a progressive Democrat, and Senator Ted Cruz, a conservative Texas Republican, both condemned efforts by Robinhood to restrict trading in GameStop and other companies, saying the firm was putting the interests of hedge funds above small investors.

Other lawmakers are warning against overreacting with more regulation. “When examining this episode, regulators and Congress should tread with extreme caution and avoid needlessly inserting themselves into equity markets,” Senator Patrick J. Toomey, Republican of Pennsylvania, said in a statement.

Representative Ro Khanna, a California Democrat, said in an interview that simply blocking retail investors from certain stocks was the wrong decision and that Mr. Gensler should look to the bigger fish — namely lightly regulated hedge funds — when looking for areas to regulate.

“We probably need to increase the capital requirements on short-selling for hedge funds, to make it more difficult,” Mr. Khanna said.

That is an area that will be more familiar to Mr. Gensler, who spent his tenure as chairman of the Commodity Futures Trading Commission trying to stop big Wall Street firms from manipulating markets. That included bringing dozens of enforcement cases against big banks, which were accused of manipulating key rates that help determine certain prices across the financial system, including benchmark interest rates and foreign exchange rates.

Dan Berkovitz, a C.F.T.C. commissioner who served as general counsel under Mr. Gensler, said breaking up “the old boy network” was a major focus during their time at the agency.

“He wanted to break that whole culture up and introduce a culture of competition instead of a cozy coexistence,” Mr. Berkovitz said. “That was his philosophy, and coming from Goldman he saw from the inside how that worked.”

Mr. Gensler, whose confirmation hearing has not yet been scheduled, will face pressure to bring a similar focus to the S.E.C. On Sunday, Senator Elizabeth Warren, Democrat of Massachusetts, said the GameStop episode underlined that S.E.C. regulators needed to “get off their duffs” and work to make the market more transparent. Among other things, she said new regulations should halt company stock buybacks for the purpose of pushing up share prices.

“In the long run, if we have a market that is transparent, that’s level, that helps individual investors come into that market and, frankly, helps make that market more efficient,” she said on CNN’s State of the Union. “The hedge funds, many of the giant corporations, they love the fact that the markets are not efficient.”

“GameStop is just the latest ringing of the bell that we have a real problem on Wall Street,” Ms. Warren said. “It’s time to fix it.”

Jeanna Smialek contributed reporting.