The problem ended up being objects that were outside of the work area but were marked hidden and not as output. Apparently even if they are not set to output they are still downloaded to the controller? Or at least affect the output sent to the controller in some way. After deleting those objects everything was A-OK. I think moving them inside the workspace would have been fine too.