Your question combines a multiplicity of questions. Lawyers like neat, little boxes to start with (no matter how much we bitch about them), since we can then test for possible solutions - and exceptions. Anyway, here is a hypothetical, which covers your facts and a bit more.

Flatland - legislature enacts - blasphemy (defined as X, but NOT Y, in statute) is a crime committed by any person anywhere in the world, and subject to the "universal jurisdiction" of Flatland's courts.

Curvedland - legislature enacts - blasphemy (defined as Y, but NOT X, in statute) is a crime committed by any person anywhere in the world, and subject to the "universal jurisdiction" of Curvedland's courts.

Topsyland - constitution (which supercedes all other laws, national and international) provides - blasphemy (whether defined as X or Y) is NOT a crime; no Topsyland national shall be detained, charged, tried or convicted for blasphemy anywhere in the world; and any such action by any other nation shall be considered an act of war against Topsyland.
------------------------------
Now, recalling that jurisdiction to prescribe (legislate) is separate from jurisdiction to detain, we have to start with jurisdiction to prescribe (legislate). From a domestic law standpoint, each nation will consider its law to be correct. Thus, a stalemate between Flatland and Curvedland; and Topsyland's ICBMs will be aimed at both of the other nations. Can I Law provide an answer ? Maybe.

First we look to substance and then to procedure. As to substance, the basic rule is that Universal Jurisdiction applies to crimes made "heinous" by a vast majority of nations (no set number or percentage, since there is no International Constitution). Assuming the three nations are representative of the world, blasphemy cannot be a crime of universal jurisdiction because there is nothing approaching a common rule. Hence, only Topsyland has it right on that point.

Note that this scenaero could turn on its head if, in fact, most nations regarded X as blasphemy, Y as NOT, and that Topsyland should be regarded as an atheistic pariah nation.

-----------------------------
The procedural devil is in the details - and the detail here is how does one enforce our hypothetical I Law principle (blasphemy is not a crime of universal jurisdiction). Unless the nations have in one way or another agreed to submit the issue to an international body with enforcement powers, that principle will not be enforced except by nations, individually or collectively.

Extraterritorial enforcement by nations, individually or collectively, of what it or they believe to be the correct I Law principle is problematic because it clashes with the general I Law principle that "Thou shall not infringe on the domestic sovereignty of any nation" - and there are also a zillion limitations on the use of armed force, etc. So, in a world where there are shifting alliances, etc., I Law is an ideal and not a reality.

Now, let us take Flatland which wants to enforce its law extraterritorially against a national of Plateauland, who has blasphemed by saying X. Now, there are few imaginary situations we can posit for Plateauland:

1. Plateauland has the same law as Flattland. In that case, Plateauland probably will prosecute anyway; and everyone in Flatland and Plateauland will be happy - except the blasphemers.

2. Plateauland has no law on blasphemy, but believes X is blasphemy because of its religious beliefs. Now, we have a split again depending on the facts. If the blasphemer is a Plateauland national, Plateauland would probably not be willing to hand him over to Flatland by extradition (formal). It is possible that a segment in Plateauland (say, its security police) would be willing to do that - so, we might have rendition (informal). If the blasphemer were a national of Curvedland or Topsyland, extradition might well occur.

3. Plateauland has no law or religious position on blasphemy (assume it is multi-cultural), but has not gone as far as Topsyland. Plateauland is not likely to either extradite or render what it considers an alleged blasphemer to Flatland. Which brings us to your question about extraordinary rendition.
---------------------------------
Extraordinary rendition, since it does not have the formal (extradition) or informal (rendition) consent of the nation in which it occurs, is an intrusion into the territorial sovereignty of that nation. Whether it is an act of war or not depends on the extent of the intrusion. For example, the British detention of seaman on US registered ships was regarded as an act of war in 1812 - whether the detentions or the declaration of war would be valid under present I Law is another question.

Note that traditionally the remedy for the intrusion is left to the nation whose territory has been violated - it may not care. Also, extraordinary rendition may or may not be a defense for the person rendered, since a domestic court may not care about the means used to bring the criminal before it.

Finally, we might have some nation decide that extraordinary rendition is a "heinous" crime subject to universal jurisdiction - in which case, around and around we go again.

-------------------------------------
In a slightly different situation, could a military or para-military unit operating in your place of residence during an "undeclared war" (i.e. state vs. non-state actor) validly seize you and both detain and charge you?
Yes, depending on the facts, of course. BTW: throw out the term "undeclared war" - it is not material to I Law which deals with "armed conflict". It may or may not have relevance in domestic law.

PS: General hypothetical questions are not particularly useful here. It would be better to set out the facts of an actual case and deal with them.