But I can carry such reactor into Oort cloud and beyond. I can power my ship in a shadow of a planet or asteroid and most importantly I am not dependent on one megastructure, which can fail, be destroyed or taken over by hostile forces.

The dyson sphere can power a huge factory, capable of making thousands if not millions of warships per second. It can also power a star-sized laser capable of vaporizing pretty much any incoming army, or even bigger objects like small planets. Your dyson sphere will be safe to everything but civilizations capable of playing marbles with stars and black holes.

Also you can use your star-sized factory to make starships with fusion reactors. Not a problem.

But if you want to get beyond Jupiter, then whole Dyson swarm is useless to you, because you don't have source of energy. And again when somebody will get control over such swarm, then such person has control over whole civilization around that star which was foolish enough to build it and rely on it at a first place.

What stops you from using a bit of the sun's output that you are capturing to materialize a starship with it's own reactor and thrusters, and then open a window in the sphere to let that ship leave?

I mean, it looks like you want to _contain_ all the energy of a sun for the sake of containing it, not to do cool stuff with it.