It has come up before on this forum, but a good example of the cabinet crossing party lines is the book "Team of Rivals" describing Lincoln's cabinet (I am certain many here have read it). Lincoln incorporated his political enemies into his cabinet, and then used a combination of leadership, friendship, cajoling, and appealing to patriotism to get them to function effectively. It would sure be nice to have the consensus such a cabinet would (hopefully!) bring. It would definitely require a lot of maturity and leadership, though... otherwise you just end up helping a bunch of folks get juicy tell-all book deals down the road.
Another example would be FDR's recruitment of Frank Knox as
Secretary of the Navy 1940-1944 as in:

A Politician is Born
In the fall of 1934, Knox found himself raising money for the Republican Party and was such a success that he was invited to speak at meetings throughout the Midwest. Before long, so many people were saying Knox would make a good Republican presidential candidate that it seemed a likely possibility. When the delegates assembled at the 1936 Republican National Convention, however, it was clear that Governor Alfred Landon of Kansas was the overwhelming favorite for the presidential nomination. Rather than foster a split in the party, Knox ordered his name withdrawn from the race. The next day he became the governor’s unanimously approved running mate.

The Navy Years
Although FDR swept the 1936 election, at the end of that term he defied partisan biases and appointed Knox Secretary of the Navy. The appointment was a testament to Knox’s impressive capabilities and reputation, especially given his previously outspoken opposition to the New Deal. Yet for all his antagonism toward Roosevelt’s domestic reforms, Knox was an enthusiastic supporter of the president’s foreign policy. Since 1936, he had been watching with increasing alarm the political developments around the world. After Hitler annexed Austria in 1938, Knox pressed for approval of the president’s billion-dollar naval expansion plan. When war finally erupted in Europe in 1939, Knox wrote front-page editorials calling for widespread support of Roosevelt’s foreign policy decisions, the repeal of the neutrality laws, and a bipartisan cabinet.

Impressed by Knox’s opinions and aware of the leveraging potential of a bipartisan cabinet, Roosevelt offered Knox the position of secretary twice before he finally accepted. In response to those who questioned her husband’s motives, Mrs. Knox explained that for him, “patriotism was a living fire of unquestioned belief and purpose.”

The other example would be Donovan as CIO and head of the OSS, at the recommendation of Frank Knox.