I have a Laravel app where users upload large documents, and a background job sends the text to an LLM for summarization. Some of these jobs take 2-3 minutes to finish because of the API latency and the sheer volume of tokens.
I’m using Laravel Horizon to manage my queues. Every time I deploy via my GitHub Action, I run php artisan horizon:terminate to make sure the workers pick up the new code.
The problem: When I run horizon:terminate, it sends a signal to the workers to finish their current job and then die. However, my deployment script then swaps the symlink and moves on. I’m seeing two issues:
1. If the job takes 3 minutes, but my deployment script times out or the server reboots, the job just disappears or fails.
2. Sometimes the “old” worker is still finishing an AI job using old code/logic, while the “new” workers are already starting jobs with the new code – causing data consistency issues in my DB.
How do I “gracefully” wait for these long AI jobs to finish before completing the deployment? What’s the best way to handle “heavy” jobs during a code swap?
This is a classic “Long-Running Job” dilemma. In a standard web app, jobs take 500ms and horizon:terminate feels instant. In an AI app, your “grace period” needs to be massive.
1. Increase the timeout and balance_wait
In your config/horizon.php, you need to ensure that Horizon knows these jobs are marathons, not sprints. If your timeout is 60 (default) but the AI takes 120 seconds, Horizon will kill the worker before it’s done.
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default', 'ai-processing'],
'balance' => 'auto',
'processes' => 10,
'tries' => 1,
'timeout' => 300, // 5 minutes for AI jobs
],
],
],
2. The Deployment Script: Don’t just “Terminate and Go”
If you run horizon:terminate and then immediately swap your symlink, you have a “Ghost Worker” problem. The old worker is still alive in memory, running the old code, even though the current symlink points elsewhere.
The Fix: You need to implement a “Signal and Wait” or use Job Batching.
If you use Job Batching, you can check if a batch is still active before allowing the deployment script to proceed:
# In your deploy.sh
php artisan horizon:terminate
# Optional: Wait until the specific AI queue is empty before swapping symlinks
# This is "Slow" but "Safe"
while [ $(php artisan queue:size ai-processing) -gt 0 ]; do
echo "Waiting for AI jobs to clear..."
sleep 5
done
ln -sfn $new_release_dir $current_dir
3. Use ShouldBeUnique for AI Jobs
To prevent the “Old vs New” logic conflict, implement the ShouldBeUnique interface on your AI jobs. This ensures that if an “old” version of a job is still grinding away, a “new” version won’t start for the same resource (e.g., the same Document ID) until the first one finishes.
4. The “Separate Server” Strategy (The Pro Move)
For high-scale AI apps, the best practice is to move your AI workers to a dedicated worker server.
– Web Server: Deploys instantly via symlink swap.
– Worker Server: Deploys using a “Blue-Green” strategy where you spin up a whole new set of workers and only shut down the old ones once their specific queue is drained.
Summary:
– Bump your timeout in Horizon config.
– Avoid horizon:terminate if you can’t afford a job restart; instead, design jobs to be idempotent (safe to run twice).
– Use X-Accel-Buffering: no if you are also streaming the AI response to the UI while the job runs.