I'm just swapping tailgates and apart from the actual metalwork and glass, most other bits on my old tailgate are in better condition so they'll need swapping across too. Most of it seems pretty straightforward, but the tailgate has a rubber surround that goes all round the glass and at the top becomes much chunkier, almost part of the structure of the tailgate.

Will I irreparably break the window seal if I pull this away? Is there a secret to getting it off and replacing it?

If its anything like the rear 1/4 side windows then I would imagine the rear window is sealed in.
I used a specific window tool remover from the inside of the car to cut the seal. Once that was done the window pushes out easily.
The cutting process is a pain & the depending on how good you cut the seal the rubber surround can be re-used.

Unlike my first attempt cutting out the 1/4 rear window. The knife/tool blade was to short. I was unable to cut the seal complete in the corner area. I was getting impatient & pushed on the glass & it shattered.

Ticky

LOL so in other words, leave it alone unless absolutely necessary!