J.P. Morgan's entry charge to the club was the $1.7 billion it paid to buy trading house Sempra and its infrastructure to store and ship oil and metals.
Today, the five banks control 70 percent of the commodities trading pot with the rest split between mid-sized players such as Credit Suisse and Bank of America/Merrill Lynch, the latter having fallen out of the top division since the financial crisis.
The halving of revenue is largely due to a regulatory crackdown on proprietary trading, when deals are done by banks for themselves rather than on behalf of clients. Regulators say banks should focus on serving the clients and helping the economy with credit.
Also taking a toll is the reduction of risk appetite for capital intensive businesses such as commodities because of stricter capital rules.
As a result, commodities business at banks is increasingly dominated by hedging of producers and consumers from sharp volatility, as well as sales of commodity-related indices to investors.
The proportion of physical trading is shrinking - especially dealing for the bank's own books - which irks regulators. That is estimated to constitute one fifth of total flows today as opposed to four fifths five years ago, say traders.
A further drop would be hard for banks to swallow. They say they need the scale and depth in the fray of daily business to know markets and serve clients properly.
"In today's volatile markets, knowledge and understanding of both the financial flows and the physical market fundamentals are vital," Credit Suisse says about its commodities team.
Understanding physical markets without trading them is close to impossible, said Silbert, a U.S. gas trader in the 1990s, who joined Deutsche from Merrill Lynch in 2007.
"In commodities you don't have a client business if you can't allocate risk capital. As soon as we stop taking a point of view and stop participating in the market in that way, then we restrict the ability to help our clients," he said.
Banks argue they benefit commodities markets by improving transparency and helping producers and consumers to lower risk.
"In the 1980s, banks were barely present in commodities markets activities and it was a pretty shadowy place dominated by physical merchants and cartels," says Henrik Wareborn, head of commodities trading at Natixis, who also worked for Goldman Sachs and BP.
"To a large extent those markets were opened up and transformed thanks to the entry of regulated European universal banks and U.S. investment banks," said Wareborn, who built a sizeable commodities desk at Lehman Brothers before the bank collapsed in 2008.
Critics accuse banks of adding to speculative froth in markets and regulators say they want to fix this.
Under the Basel 2.5 banking regulations, requirements to set aside capital for trading nearly tripled this year.
In the United States, in addition to the limits on proprietary trading - the Volcker rule - banks face the Dodd-Frank regulation, which limits positions they can take in derivatives.
The U.S. Federal Reserve is also locked in a battle with banks over whether to allow them to continue holding physical assets. If the banks lose that verdict they could be forced to sell assets such as oil storage and metals warehouses. Bankers bridle at the thought.
"The desire to vilify commodities markets activities at big banks is misplaced. With rare exceptions, commodities markets have worked impeccably in the past decade," adds Wareborn.
The banks won a small battle when a U.S. judge ruled last month to send the position limits rule back to the Commodity Futures Exchange Commission (CFTC) two weeks before it was to take effect, saying the regulator had to prove it was necessary.
The rule, already partly applied by the exchanges, would have drastically cut the trade of derivatives.
Sources at one large bank said it was forced to cut sales of commodities-linked indices after exceeding proposed limits on positions in the grains market.
In Europe, however, regulators are pressing ahead with position limits rules. Leading players say that Europe could end up with tougher regulations than the United States, potentially opening up a regulation arbitrage.
"The U.S. banks are coming to Europe to take bigger market share. They are better capitalised and actually less regulated than the European banks are likely to be," one said.
In Europe, regulations cannot be overthrown by a court ruling as in the United States. Much will also depend on the outcome of the U.S. presidential election and whether CFTC's current chairman Gary Gensler is replaced with a figure who might push for less regulations, bankers say.
Regulation will squeeze banks' commodity trading units along with other capital market businesses, according to McKinsey & Company, which predicts return on equity (ROE) in commodity units shrinking to 8 percent from 20 percent.
"Some of the worst-hit businesses with ROEs below the cost of capital may have to be disposed of, especially at banks with weak franchises," McKinsey said.
Insiders say the current ROE in commodities is about 12-16 percent, still far better than overall average banking returns.
Over the past year, banks have lost an army of commodities traders to better-paying merchants and hedge funds, with even heavyweights like Goldman losing out.
Banks argue that today their risk levels come nowhere near those of trading houses.
"Our commodities business is not about betting on commodity prices... Our business is a client-driven business where we execute on behalf of clients to achieve their financial and risk management objectives," J.P. Morgan's commodities chief Blythe Masters told CNBC earlier this year.
The arguments are falling on deaf ears at the regulators following the Libor interest rate manipulation and other scandals which have rocked the banking industry.
Some of the banks' activities, such as storing metals in warehouses, have also generated criticism.
"No one can afford growing because of regulations... We have to reinvent ourselves," said a senior executive.
One noticeable change this year is discussion among U.S. banks on how to transform desks.
Morgan Stanley set the ball rolling by considering selling parts of its commodities business to Qatar or allowing the unit's current management to buy it out, possibly together with a private equity firm.
Selling the capital-intensive business would raise funds while allowing the divested unit to maintain ownership of physical assets and resume proprietary trading.
Goldman, which traditionally had large proprietary trading, had also discussed a spinoff, the Wall Street Journal reported. Goldman said it never seriously considered the move.
J.P. Morgan said it is not considering any spin out for its commodities unit. The bank is better capitalised than its peers.
Regulations aside, the egos of the main players will play a role in a business dominated by big personalities such as Goldman's chief Lloyd Blankfein. "I find it hard to believe that Blankfein can say goodbye that easily to his baby," said a former Goldman trader.
Blankfein started his career as a precious metals salesman for Goldman's commodities arm J. Aron and was co-head of commodities in the 1990s. "Trading is in Goldman's blood," said the head of a major trading house.
Today, Goldman's top commodities job is held by Isabel Ealet, a former trader at French oil major Total who is also global co-head of securities.
J.P. Morgan's Masters, who helped to develop credit-default swaps in the 1990s and built its commodities business from scratch in just five years, will also soon be running regulatory affairs for JPM's investment banking, according to an internal memo seen by Reuters.
Like Ealet, Morgan Stanley's commodities boss Colin Bryce is an oil trading veteran, in his case from the long defunct Britoil. It was the commodities team at Morgan Stanley which suggested to management the spin off, sources said.
The bank would need to reduce its stake to below 25 percent in the unit in order to avoid Dodd-Frank oversight, sources said, something Morgan's leadership may hesitate to do since the unit has made such good revenue for many years.
At Deutsche, Silbert says he plans to expand revenue and market share after several years of speedy growth. The strategy, though, is a challenge amid a sharp retreat to core activities as Deutsche slashes jobs and costs.
The swift rise of Barclays up the ranks in commodities was dealt two severe blows with losses in metals markets last year and the recent departure of its commodities boss Roger Jones for trading house Mercuria after a decade in charge.
The bank decided against replacing him in the same role and asked its foreign exchange boss Mike Bagguley to look after the unit instead.
However, Barclays says its commodities business remains strong. It surprised the market earlier this year with a landmark supply deal with an oil refiner, the first in Europe.
The head of a major trading house said he feels much less pressured from banks these days than several years ago. "I don't see Goldman or Morgan leaving the trading space. With European banks, I'm not so sure," he says.
(Removes extraneous word from first paragraph)
(Additional reporting by Eric Onstad, editing by Richard Mably and David Stamp)