I'll take a crack at this from the level of national security policy, doctrinal military strategy and the legal framework. In short, what were the "rules of engagement" for the expansion of massive American settlement from the Atlantic to the Pacific (Manifest Destiny, the "Indian Wars" and forced assimilation from the Indians' standpoint) ?

First off, one cannot find a presentation of comprehensive strategic doctrine for the "Indian Wars". For example, John Bigelow's The Principles of Strategy: Illustrated Mainly from American Campaigns (1891; free from Google) focuses on the Civil War and barely mentions the Indian campaigns in the period 1860-1890. See, John M. Gates, The U.S. Army and Irregular Warfare - chap. 2, Indians and "Insurrectos", for the paucity of doctrine at the policy and strategic levels:

Much of the army's work on the frontier was that of a federal constabulary. It served eviction notices on Indians and then forcibly removed them when required. lf "imprisoned" Indians "broke out" of the reservations, the army found them and coerced them back. Failing in the latter, it would attempt the equivalent of an arrest, an armed attack to force the Indians to surrender. Bands that raided white settlers, peaceful reservation Indians, or army posts engaged in criminal activity, in white eyes at least, and the army's task was that of the police officer, to track down the guilty parties and bring them back for punishment. Because of the numbers involved those activities sometimes looked like war, and in a few instances, when entire tribes fought against the intrusion of the white, it was. Most of the time, however, it was routine though difficult police work.
...
In his excellent study of the army in the West, Robert Wooster found neither a significant connection between the army's Civil War experience and its approach to Indian warfare nor the development of a doctrine of irregular warfare out of its Indian fighting experience. Officers often disagreed over such fundamentals as the timing of offensives, the optimum composition of forces, and the use of Indian auxiliaries. As Wooster observed, "military success against Indians was thus not attributable to a national strategic doctrine understood and practiced by officers in the field. It was instead the result of a commander's personal experiences in the West, his perceptions of Indians and the natural environment, the abilities of his subordinates, and simple good fortune."[11]

[11] Wooster, The Military and United States Indian Policy, 1865-1903 (1988) p.213.
Weigley and Linn are to the same effect. No "Small Wars Manual" exists for the "Indian Wars".

I see three basic "rules of engagement" policies:

1. Law Enforcement ("constabulary") Rules

These are noted by Gates and Wooster (Utley is also cited by Gates), where Indians were tried for various offenses under civil laws in civilian courts. The status of those Indians was roughly that of resident aliens (an ironic analogy, indeed). Law enforcement rules were applied even when actual armed conflict occurred with regular Federal troops - as in 1898 Minnesota (when members of the 3rd Infantry were killed). See, Matsen, The Battle of Sugar Point - a Re-Examination (1987) ("As for Bug-O-Nay-Ge-Shig himself, he remained true to his conviction regarding the white judicial system and once again avoided arrest. None of the Ojibway who fought at Sugar Point ever surrendered, and it is not clear what they could have been charged with if they had."); and Roddis, The Last Indian Uprising in the United States (1919):

Troops were poured into the Indian country, not only for the sake of actual protection in case of an extensive uprising, but also to impress the Indians with the fact that recourse to arms was hopeless and that the government was determined to suppress any armed resistance to its authority. At the same time a thorough investigation of the Indians' complaints in regard to the disposal of the dead timber on their land was promised.

Influenced by the tact of the Indian commissioner, persuaded by the chiefs and leading men of the tribe, which has always been conspicuously friendly to the whites, and also, probably, impressed by the military force brought to the scene, the Bear Islanders gradually acceded to the demands of the marshals and by the middle of October practically all the men for whom warrants had been issued were in the hands of the authorities. They were transferred to Duluth for trial.

When their cases came up before Judge Lochren on October 21, all were found guilty and were given sentences varying from sixty days imprisonment and a fine of twenty-five dollars to ten months and one hundred dollars. On December 13, the Indian office recommended that the term of imprisonment be commuted to two months and that the fines be remitted, and finally on June 3, 1899, the pardons were granted.
2. U.S. Insurrection (cf., Lieber Code) Rules

In those cases, Indians were treated as rebels and tried before military tribunals, rather than in civilian courts. One could fairly analogize them to the Mexican War precedents (applicable to Mexican nationals resisting in occupied territory) or the Civil War precedents (applicable to Confederate irregulars). Two examples are the Dakota War of 1862:



and U.S. Army hangs four Modoc Indians for the murder of a Civil War hero.

3. Traditional Laws of War

Since the Indian "First Nations" were not recognized as sovereign states by the US, this was not a popular theory given the 19th century's state-centric concept underlying application of the Laws of War. Today, an Indian tribe could be much more easily found to be a group engaged in an armed conflict, to which the Laws of War apply. In any event, a brave (and/or politically astute) Federal Circuit judge (brother of a Supreme Court justice) held that the Laws of War applied. See, Di Silvestro, In The Shadow of Wounded Knee: The Untold Final Chapter of the Indian Wars (2005) (HT to Polarbear1605 for recommending the book to me):

On December 29, 1890, the U.S. Seventh Cavalry killed more than 150 Lakota men, women and children at Wounded Knee, S.Dak. Was it a battle or a massacre? That became the key point of dispute when a Brul Lakota warrior named Plenty Horses was brought to trial for the murder of Lt. Edward Casey, whom he had killed a week after the slaughter. If the U.S. was not at war with the Lakota, reasoning went, then the Lakota were murdered; but if a state of war did exist, then Plenty Horses's "fatal bullet through the back of Casey's skull" was also an act of war, not murder. Complicating the juridical conundrum was a simpler case: shortly after Casey's death, the "infamous" Culbertson brothers attacked a peaceful Indian encampment. Would an Indian hang for killing a white officer? Could two white men be convicted for killing a settlement of Indians?
The answer in both cases was "No".

Regards

Mike