I have a Laravel app where users upload large documents, and a background job sends the text to an LLM for summarization. Some of these jobs take 2-3 minutes to finish because of the API latency and the sheer volume of tokens.
I’m using Laravel Horizon to manage my queues. Every time I deploy via my GitHub Action, I run php artisan horizon:terminate to make sure the workers pick up the new code.
The problem: When I run horizon:terminate, it sends a signal to the workers to finish their current job and then die. However, my deployment script then swaps the symlink and moves on. I’m seeing two issues:
1. If the job takes 3 minutes, but my deployment script times out or the server reboots, the job just disappears or fails.
2. Sometimes the “old” worker is still finishing an AI job using old code/logic, while the “new” workers are already starting jobs with the new code – causing data consistency issues in my DB.
How do I “gracefully” wait for these long AI jobs to finish before completing the deployment? What’s the best way to handle “heavy” jobs during a code swap?