Do you have a link to your stuff?

> have you considered using a server/daemon process that runs separately and therefore more controllably outside a DAW

I'm slowly coming to the same conclusion, for audio plugins on GPUs.