← All posts

Bulk-importing movies from TMDB with Laravel queues

June 30, 20262 min read

The feature that took the most work in Film Arşivim is the one least visible on screen: bulk import. The user pastes hundreds of movie titles, each line is searched on TMDB in the background, matches enter the archive and misses land in a report. This post covers the design decisions behind that pipeline.

Why synchronous import dies

The obvious route — looping over TMDB inside a single request — breaks in three places: the request timeout, one bad line taking down the whole batch, and the user staring at a blank screen for minutes. The fix is classic but the details matter: every line becomes its own queue job.

class ProcessImportItemJob implements ShouldQueue
{
    public int $tries = 3;
    public int $timeout = 45;

    public function __construct(public int $importItemId)
    {
        $this->onQueue('imports');
    }
}

The job receives the model's ID, not the model itself — the serialized payload stays small and the job always reads the current record when it runs.

A two-model design: batch + item

ImportBatch holds the aggregate state (how many items, how many finished, cancelled or not); ImportItem holds each line's own lifecycle (pending → processing → done/not_found/error). The split buys three things for free: live progress (batch counters), a per-item error report, and cancellation — once the batch is marked cancelled, queued jobs exit before doing any work.

Idempotency is non-negotiable

In queue land every job runs at least once — sometimes twice. That's why the job opens with defense:

if (! in_array($item->status, ['pending', 'error'], true)) {
    return; // already processed — a second run is harmless
}

The same defense exists for duplicates: if the user's archive already has that tmdb_id, the item closes as duplicate and no second copy is written.

Binding free text to TMDB

Users type "Esaretin Bedeli 1994" and they type "shawshank". The raw query never goes straight to search; a smart search with year extraction and normalization picks the best candidate. When nothing matches, the item doesn't vanish silently — it lands in the report as not_found. In bulk processing, the worst behavior is silent data loss.

Lessons

  • Give the job an ID, not a model: small payload plus always-fresh data.
  • Draw the state machine on paper first: writing code before the legal transitions are clear means batches stuck halfway in processing.
  • Perceived progress is half the feature: the same duration feels like "working" with a live counter and "frozen" with a blank screen.
0%▌