NEW YORK – As a historic oil and gas boom transforms the U.S. energy sector, Wall Street is losing the battle to remain the partner of choice for energy producers and major consumers seeking to protect themselves against volatile prices.

In the thriving Texas Permian oil patch and beyond, banks are being edged out by a handful of the world’s biggest corporations including BP Plc, Cargill and Koch Industries.

With Wall Street hamstrung by growing regulatory restrictions, a recently finalized ban on proprietary trading and increased capital requirements, these corporate behemoths are leveraging their robust balance sheets and savvy global trading desks to capture as much as a quarter of the global multibillion-dollar market for hedging commodity prices.

New risks have arisen this year that could tilt the scales further, as the Federal Reserve considers limiting banks’ ability to trade in real physical markets, the kind of deals that are increasingly important for many of the smaller and mid-sized companies at the fore of the U.S. energy renaissance.

Just ask Alan Barksdale, president and chief executive of Red Mountain Resources, a conventional driller in the Permian Basin. Early this year his company was shopping for a counterparty to execute derivative trades that would protect some of its near 900 barrels of daily production from a possible price drop. Barksdale, a former investment banker, was looking to lock in “costless collars”, a type of specialized options trade.

After reviewing a number of offers, including some from Wall Street firms, he chose BP Energy Corp, a unit of the oil and gas major’s trading division. In part that was because BP was already working with the firm’s lenders. But Barksdale was also interested in a partner who could one day take physical delivery of his crude, potentially netting Red Mountain an extra dollar or more per barrel.

“As you grow as a company, you’d like some flexibility to get some physical delivery,” Barksdale said. “When you’re dealing with somebody who is long a commodity, you get better service.”

Ten years ago, only a handful of banks would have likely handled such a trade. Over the past decade, however, more than a dozen rushed into the commodity trading business, acting as lenders, counterparties and risk managers.

Now some of the biggest are beating a hasty retreat. Deutsche Bank became the largest victim last week, announcing plans to exit most trading under mounting regulatory pressure and diminished profitability.

Others are like JPMorgan Chase & Co and Morgan Stanley are poised to carve out their large physical trading operations – things like oil storage tanks, gasoline cargoes and power plants – but will still compete fiercely on derivatives deals, trades they can combine with financing or other activities. Goldman Sachs has been resolute that the bank will continue trade both cash and paper commodities.

While most banks have blamed regulations and lower market volatility for the sharp slump in commodity earnings, at least a portion of the decline appears to stem from the corporate giants quietly stealing banks’ core business: serving clients.

Commodity revenues at the world’s top 10 investment banks has fallen from a peak of more than $14 billion in 2008 to just $5.5 billion last year, according to consultants Coalition. One senior executive at a top 10 commodity bank said corporations had taken as much as a quarter of the global hedge book away from banks, including deals with airlines and utilities.

“When you look at the market overall, the commercial firms are making in-roads into the hedging business,” says Andy Awad of Greenwich Associates, which conducts an annual survey of hundreds of companies that hedge commodity prices. “I would imagine the pace of change is going to increase.”

While the big non-bank companies have not yet cracked the top tier, four of them made it into the top 20 U.S. energy hedgers this year, he said.

RETURN OF THE CORPORATES

As banks withdraw, the conventional wisdom has been that foreign, privately owned commodity merchants like Vitol and Trafigura – which typically trade only for themselves – would fill the market void, particularly in the costly, complex realm of physical trading.

While they may help bolster liquidity, most of those firms are loathe to take on the onerous regulatory burden now required to become a major derivatives trader.

Yet this year, units of BP, Royal Dutch Shell and Cargill all formally entered the big leagues of derivative dealing, registering as “swap dealers” alongside dozens of the world’s biggest banks. As the most heavily regulated type of derivatives trader under the Dodd-Frank law’s financial reforms, they face onerous record-keeping and trade reporting rules, but also have the latitude to hedge with far more clients, and to trade in excess of $8 billion in swaps a year.

To be sure, banks retain many advantages in the business. As the leading lenders to the world’s industries, they can offer bundled services and leverage existing lines of credit; the derivatives operations of the biggest players, even those selling some parts such as JPMorgan, remain competitive on pricing.

Yet they are suffering set-backs across multiple fronts.

Some of their most valuable traders are now being hired away by private merchants who can offer higher salaries and bigger bonuses. Tougher capital requirements under the Basel III international accord are raising banks’ funding costs and narrowing profit margins.

“We have a strong balance sheet and an ability to manage these price risks,” said Cody Moore, head of North American Gas and Power at EDF Trading, a unit of France’s government-backed utility, EDF.

The group was formed in 1998 and expanded its international reach ten years later with the purchase of Lehman Brothers’ physical trading unit Eagle Energy during the financial crisis. Its revenue has surged 60 percent since 2008; pre-tax profits at the firm, one of the few to separate its financial performance from that of the parent group, reached nearly 500 million euros in 2012.

With some 350 people in its Houston office alone, EDF Trading is now the leading energy management provider for power generators in the United States. Last year it hired a small team to expand into oil market logistics.

Corporations have another advantage – unlike banks, they are not banned from trading with their own money.

Under the Volcker Rule, which was formally approved by regulators this month, banks can no longer engage in proprietary derivatives trading – giving them less incentive to chase customers simply for the benefit of valuable insight into a particular trend they may be able to trade themselves.

“In the past, the information was worth something to a bank if you had a proprietary desk,” says Eric Melvin, a former trader at an investment bank who now runs boutique Houston-based risk-advisory firm Mobius Risk Group.

He estimates that investment banks now account for only about half of the U.S. oil and gas-hedging business, with corporate merchants accounting for some 40 percent, up from almost nothing just a few years ago.

PHYSICAL STRENGTH

For most of these companies, one of their biggest selling points is the ability to manage the risk of some of the most esoteric or niche energy markets in the world – typically because they already trade those commodities for themselves.

“We’re willing to stand in as a provider of risk-management where many or most others won’t,” says Steve Provenzano, BP Energy’s Chief Commercial Officer for client hedging in the Americas. “Obviously our involvement in the physical business gives us credibility.”

Long the largest U.S. natural gas trader and a major global oil operator, BP also has 20 people who help arrange customer derivatives trades in North American alone, and more than 3,000 wholesale customer worldwide, he said.

While Wall Street awaits the completion of a Federal Reserve review of commodities trading – the results of which are expected early next year – corporations that hedge energy prices are placing a greater importance on the ability of a counterparty to trade in physical markets, according to Greenwich Associates’ latest survey.

“I think what we’re seeing is that people recognize you can’t divorce the financial and physical, they’re linked,” says Awad.

Meanwhile competitors are stealing a march.

Minneapolis-based Cargill, better known for its prowess in agricultural markets, has recently moved its Houston trading group to a larger office with room for over 100 traders, online industry publication SparkSpread.com reported this month. Cargill employs more than 1,000 people in its Geneva-based Energy, Transportation and Metals business, and executives have said they are looking to expand as others divest.

A spokesman for Cargill declined to comment on the business.

Koch Supply & Trading, a unit of the $115 billion a year conglomerate owned by Charles and David Koch, is famed for having traded the first oil swap over 25 years ago, and says it now has nearly 500 traders, marketers and energy and metal markets professionals worldwide. It expanded its European natural gas team last year, and minces no words in promoting itself as a more constant alternative to Wall Street.

“While some financial institutions’ market coverage varies with global market cycles, KS&T companies take a longer term view,” it says in a recent online brochure. Koch offers market liquidity “at times when others pull back.”

– REUTERS