Queues

This document describes how the package uses Laravel queues for background processing.

Overview

The package uses queues for four categories of work: AI content generation via OpenAI, asset processing after upload, analysis tasks (SEO and accessibility), and admin-triggered asset maintenance reports. All jobs implement ShouldQueue and use the Dispatchable, InteractsWithQueue, Queueable, and SerializesModels traits.

Queue usage is opt-in for most features and is controlled per-feature via configuration flags. When queuing is disabled, most features fall back to synchronous processing inline.

Configuration

All settings live under the textstemapp key (published from config/textstem.php).

OpenAI

'openai' => [
    'enabled'              => env('OPENAI_ENABLED', true),
    'api_key'              => env('OPENAI_API_KEY', ''),
    'use_queue'            => env('OPENAI_USE_QUEUE', true),
    'model'                => env('OPENAI_MODEL', 'gpt-4o-mini'),
    'generate_description' => false,
    'generate_keywords'    => false,
    'generate_tags'        => false,
    'generate_teaser'      => false,
],

Accessibility

'accessibility' => [
    'enabled'                => env('ACCESSIBILITY_ENABLED', true),
    'use_queue'              => env('ACCESSIBILITY_USE_QUEUE', true),
    'store_reports'          => env('ACCESSIBILITY_STORE_REPORTS', true),
    'report_retention_days'  => env('ACCESSIBILITY_REPORT_RETENTION_DAYS', 30),
],

SEO

'seo' => [
    'enabled'               => env('SEO_ENABLED', true),
    'use_queue'             => env('SEO_USE_QUEUE', true),
    'use_ai'                => env('SEO_USE_AI', true),
    'store_reports'         => env('SEO_STORE_REPORTS', true),
    'report_retention_days' => env('SEO_REPORT_RETENTION_DAYS', 30),
],

Asset upload processing

'uploads' => [
    'use_queue'            => env('TEXTSTEM_UPLOADS_USE_QUEUE', false),
    'enable_fingerprint'   => env('TEXTSTEM_UPLOADS_ENABLE_FINGERPRINT', true),
    'generate_thumbnails'  => env('TEXTSTEM_UPLOADS_GENERATE_THUMBNAILS', true),
    'extract_metadata'     => env('TEXTSTEM_UPLOADS_EXTRACT_METADATA', true),
    'queue_connection'     => env('TEXTSTEM_UPLOADS_QUEUE_CONNECTION'),
],
Key Default Description
use_queue false Enable queued post-upload processing. When false, fingerprinting and metadata run synchronously in the request.
enable_fingerprint true Compute a perceptual fingerprint on each uploaded asset for duplicate detection.
generate_thumbnails true Generate a JPG thumbnail from the first page of uploaded PDFs.
extract_metadata true Extract EXIF and file metadata after upload.
queue_connection null Override the queue connection used for upload jobs. Defaults to the application default when null.

Job Reference

AI content generation

All OpenAI jobs use an HTTP client with a 300-second timeout. Model and token settings can be overridden per-call through the dispatching action.

Job File Dispatched by Result
OpenAiModelRequest src/Jobs/OpenAiModelRequest.php GenerateModelDescription Updates description on a WranglerPage or other model; fires ModelDescriptionUpdated
OpenAiPostRequest src/Jobs/OpenAiPostRequest.php GeneratePostDescription Updates description on a Post; 20-word summary
OpenAiArticleRequest src/Jobs/OpenAiArticleRequest.php GenerateArticle Updates content on a Post; fires ArticleGenerated
OpenAiKeywordsRequest src/Jobs/OpenAiKeywordsRequest.php GenerateModelKeywords Updates keywords on any model; fires ModelKeywordsUpdated
OpenAiTagsRequest src/Jobs/OpenAiTagsRequest.php GenerateTags Calls an optional $callback with the generated tag string
OpenAiTeaserRequest src/Jobs/OpenAiTeaserRequest.php GeneratePostTeaser Updates teaser on a Post; 1024 max tokens
GenerateAltTextJob src/Jobs/GenerateAltTextJob.php Asset::created model hook Resizes image to 1200px, encodes as WebP, sends to OpenAI vision, saves alt text to Asset.description; fires ModelDescriptionUpdated

Asset processing

These three jobs are dispatched together by UploadManager after a file is stored. They run independently (not chained). Each reports success or failure via ProcessingTracker and fires an AssetProcessed event.

Job File Purpose
UpdateAssetFingerprint src/Jobs/UpdateAssetFingerprint.php Computes a perceptual fingerprint for duplicate detection
GenerateAssetThumbnails src/Jobs/GenerateAssetThumbnails.php Generates a JPG thumbnail from the first page of a PDF
ExtractAssetMetadata src/Jobs/ExtractAssetMetadata.php Extracts EXIF and file metadata via ExtractMeta action

Asset maintenance reports

Dispatched from the Maintenance dropdown on the asset list screen (textstem::tables.assets-maintenance). Both iterate the entire assets table, so the menu entries are disabled unless queue.default is a real (non-sync) connection. Each writes a status (running → done / failed), last_result and up to 500 matched IDs per bucket to storage/app/textstem/asset-maintenance.json via PackageSettings::mutate() (an exclusive-lock read-modify-write, so the two jobs finishing at once cannot clobber each other's key). While status is running the menu shows a spinner and polls (wire:poll.3s); a run still running 30 minutes after its started_at is treated as dead (worker killed before failed() ran) so the spinner stops and the report can be re-run. timeout is 900s. Writes to assets use saveQuietly() so a bulk reconcile does not fire a per-asset ModelUpdated storm.

Job File Dispatched by Result
ReportUnusedAssetsJob src/Jobs/ReportUnusedAssetsJob.php Assets → Maintenance → Report: Unused Assets Runs SearchAssetReferences per asset; stores the IDs referenced in no post / page / component / global-option body under unused_assets
AuditAssetStorageJob src/Jobs/AuditAssetStorageJob.php Assets → Maintenance → Report: Storage Audit Per asset: stamps meta.summary = AssetFileInfo::missingSummary() for a missing file (and clears a stale one when the file is back), flags broken thumbnails, and recomputes width/height/meta.filesize from disk. A disk that throws is recorded as unverified and never stamped. Stores missing, unverified, broken_thumbnails and recomputed ID lists under storage_audit

Analysis

Job File Dispatched by Result
SEOAnalysisJob src/Jobs/SEOAnalysisJob.php SEOService::analyze() Runs basic HTML checks plus optional OpenAI analysis; stores report in seo_reports table
AccessibilityAnalysisJob src/Jobs/AccessibilityAnalysisJob.php AccessibilityService::analyze() Runs WCAG checks plus OpenAI analysis; stores report in accessibility_reports table

Action to Job Mapping

Each AI action checks textstemapp.openai.use_queue and either dispatches the job or runs the prompt class synchronously.

Action Job (queued path) Sync path
GenerateModelDescription OpenAiModelRequest DescribeArticle prompt class
GeneratePostDescription OpenAiPostRequest Inline HTTP call
GenerateArticle OpenAiArticleRequest Inline HTTP call
GenerateModelKeywords OpenAiKeywordsRequest GenerateKeywords prompt class
GenerateTags OpenAiTagsRequest SuggestTags prompt class
GeneratePostTeaser OpenAiTeaserRequest Inline HTTP call

Asset Upload Pipeline

UploadManager::handle() (called from the asset upload flow) orchestrates the post-upload pipeline:

  1. Stores the file on the configured disk.
  2. Creates the Asset record via AssetCreator.
  3. Fires AssetUploaded.
  4. If use_queue is true, dispatches UpdateAssetFingerprint, GenerateAssetThumbnails, and ExtractAssetMetadata individually (each respects the optional queue_connection override).
  5. If use_queue is false, runs fingerprinting and metadata extraction synchronously.

ProcessingTracker tracks per-stage state (processing, done, failed) on the Asset record and fires AssetProcessed events that can be used to notify the UI of progress.

Other queued work

These also run on the queue, so they need a worker too:

  • Content approval emails to publishers and authors (ReviewRequestedMail, ReviewDecidedMail, IdleDraftMail).
  • Form Data emails to staff and senders, and webhooks, when FORMS_ENABLED=true.
  • Live page screenshots, when TEXTSTEM_PAGE_SCREENSHOTS=true (textstem:page-screenshots --queue).

Scheduled publishing itself doesn't use the queue: textstem:publish-due runs from the scheduler every minute. See Scheduled Tasks.
esc