Skip to main content

Immediate Jobs

Overview​

The Immediate Jobs panel provides comprehensive management of one-time pipeline jobs in the Shakudo Dashboard. It enables users to create, monitor, and control data pipeline jobs that execute immediately upon creation, without a recurring schedule. Users can manage jobs across both the primary cluster and satellite clusters, with full filtering, monitoring, and action capabilities.

Access & Location​

  • Route: ?panel=jobs
  • Navigation: Workloads → Jobs
  • Access Requirements: None (public access, role-based filtering applies)
  • Feature Flags: None

Key Capabilities​

Create Immediate Jobs​

Create and execute one-time pipeline jobs with comprehensive configuration options including environment settings, git integration, resource allocation, notification setup, and custom parameters. Jobs can be created for the primary cluster or satellite clusters.

Monitor Job Execution​

View real-time status of running jobs and historical records of completed jobs. Track job progress, duration, resource utilization, and execution logs through integrated Grafana dashboards and event monitoring.

Manage Job Lifecycle​

Control job execution with actions including:

  • Clone existing jobs with configuration preservation
  • Convert immediate jobs to scheduled jobs
  • Terminate running or pending jobs
  • Pin important jobs for quick access
  • Publish jobs for team visibility
  • Clone with customized Pod YAML specifications

Apply advanced filtering across multiple dimensions including status, job name, environment configuration, parameters, timeframes, duration, owner, billing project, and custom search across names, paths, IDs, and authors.

Multi-Cluster Support​

Manage jobs across the primary Kubernetes cluster and satellite clusters with dedicated views and cluster-specific configurations.

User Interface​

Main View​

The panel presents a dual-tab interface for cluster selection (Primary Cluster / Satellite Clusters) and job type (Running / History). Each view includes:

  • Create Immediate Job button (top-right)
  • Filter chips for active filters with reset option
  • Parameter filtering dialog for complex parameter queries
  • Pagination controls (20 jobs per page)
  • Real-time data refresh capabilities

Dialogs & Modals​

  1. Create Immediate Job Dialog

    • Purpose: Configure and launch a new immediate pipeline job
    • Fields: Job name, environment config, pipeline path, working directory, git settings (branch/commit), timeout settings, retry configuration, parameters, notifications, secrets, billing project, service account, cloud SQL proxy settings
    • Actions: Create job, clone from existing, cancel

  2. Job Details Dialog

    • Purpose: View comprehensive information about a specific job
    • Fields: All job metadata, execution logs, events, parameters, resource allocations, custom configurations
    • Actions: Access via clicking job name in table

  3. Events and Logs Dialog

    • Purpose: Real-time monitoring of job execution logs and Kubernetes events
    • Fields: Pod events, container logs, timestamps
    • Actions: Available for running jobs in the "Logs" column

  4. Cancel Job Dialog

    • Purpose: Terminate a running or pending job
    • Actions: Confirm termination (available via Actions menu)

  5. Parameters Filter Dialog

    • Purpose: Filter jobs by specific parameter key-value pairs
    • Fields: Multiple parameter pairs (key/value combinations)
    • Actions: Apply filters, accessible via info chip in filter bar

Tables & Data Grids​

  1. Running Jobs Table (Current Immediate Jobs)

    • Columns (visible): Actions, Name (with pin), Status, Logs (events/Grafana), Env Config, ID, Pipeline Path, Created, Duration
    • Columns (hidden): Custom Image URL, Output Notebooks Path, Billing Project ID, Service Account ID, Status Reason, Schedule, Owner, Branch, Commit ID, Active Timeout, Max Tries
    • Actions: View details, pin/unpin, actions menu (clone, convert to scheduled, terminate)
    • Filtering: Quick search, advanced filters via toolbar, status auto-filtered to 'in progress' and 'pending'
    • Features: Real-time updates, copy-to-clipboard for IDs and paths, published job indicators

  2. History Jobs Table (Past Immediate Jobs)

    • Columns (visible): Actions, Name (with pin), Status (with failure reason tooltips), Logs (Grafana), Env Config, ID, Pipeline Path, Created, End, Duration
    • Columns (hidden): Same as Running Jobs table
    • Actions: View details, pin/unpin, clone, convert to scheduled, publish/unpublish
    • Filtering: Quick search, advanced filters via toolbar, status auto-filtered to exclude 'in progress' and 'pending'
    • Features: Status reason tooltips for failed jobs, historical job analysis, duration tracking

Both tables support:

  • Column visibility management (show/hide columns)
  • Pinned columns (Actions pinned to right)
  • Row pinning (pinned jobs appear at top)
  • Pagination with page count display
  • Loading states with circular progress indicators
  • Error handling with user-friendly messages

Technical Details​

GraphQL Operations​

Queries:

  • getImmediateJobs - Retrieves paginated list of immediate jobs with comprehensive filtering, ordered by pinned status and start time
  • countJobs - Returns total count of jobs matching filter criteria for pagination
  • getImmediateSatelliteJobs - Retrieves immediate jobs from a specific satellite cluster (polls every 5 seconds)
  • countJobsOnSatelliteCluster - Returns count of satellite cluster jobs for pagination
  • getSatelliteClusters - Retrieves satellite cluster information for job operations

Mutations:

  • createPipelineJobWithAlerting - Creates a new immediate job with full configuration including alerting setup
  • cancelPipelineJob - Terminates a running or pending immediate job
  • updateOnePipelineJob (pinJob) - Toggles pin status for a job
  • createSatellitePipelineJobWithAlerting - Creates immediate job on satellite cluster

Subscriptions:

  • None (uses polling for satellite cluster jobs at 5-second intervals)

Component Structure​

  • Main Component: /components/Jobs/Immediate/ImmediateJobsPanel.tsx
  • Tables:
    • /components/Jobs/Immediate/Tables/CurrentImmediateJobsTable.tsx
    • /components/Jobs/Immediate/Tables/PastImmediateJobsTable.tsx
  • Main Table Component: /components/Jobs/Immediate/ImmediateJobsTables.tsx
  • Dialogs:
    • /components/Jobs/Immediate/ImmediateJobDialog.tsx (create/clone)
    • /components/Jobs/Immediate/satelliteCluster/ImmediateSatelliteJobDialog.tsx (satellite jobs)
    • /components/Jobs/shared/Details/satelliteCluster/SatelliteJobsDetailsDialog.tsx
    • /components/Jobs/shared/EventsAndLogsDialog.tsx
    • /components/Jobs/shared/SatelliteEventsAndLogsDialog.tsx
  • Actions: /components/Jobs/Immediate/ImmediateJobActionsMenu.tsx
  • Loader: /components/Jobs/shared/JobsLoader.tsx

State Management​

Uses Jotai atoms for global state:

  • JobsPanelSectionAtom - Controls current view (table/create/details/clone)
  • CurrentImmediateJobsAtom / PastImmediateJobsAtom - Job data storage
  • CurrentJobFiltersAtom / PastJobFiltersAtom - Active filter states
  • CurrentImmediateJobsTablePageAtom / PastImmediateJobsTablePageAtom - Pagination state
  • CurrentJobsDataLoadingAtom / PastJobsDataLoadingAtom - Loading states
  • immediateJobsClusterTypeTabAtom - Cluster tab selection (0=primary, 1=satellite)
  • immediateJobsTableTypeTabAtom - Job type tab selection (0=running, 1=history)
  • selectedSatelliteClusterAtom / selectedSatelliteClusterIdAtom - Satellite cluster selection

Filter System​

Advanced filtering supports:

  • Quick Search: Searches across job name, pipeline path, ID, owner, and user email
  • Status: Filters by job status (automatically applied per table)
  • Timeframe: Filter by recent time ranges (last hour, day, week, etc.)
  • Timeline: Filter by specific start/completion date ranges
  • Duration: Filter by execution duration (more than / less than)
  • Parameters: Filter by custom parameter key-value pairs
  • Job Type: Filter by environment configuration
  • Owner/Group: Filter by job ownership
  • Billing Project: Filter by billing project ID
  • User: Role-based filtering (non-admins see only their own jobs unless published)

Filters persist in URL query parameters for shareable links and browser navigation.

Role-Based Access​

  • Email Filter: Users without WorkloadRoles.filters.email permission automatically see only their own jobs and published jobs
  • Public Jobs: Jobs marked as "published" are visible to all users
  • Filter enforcement happens at GraphQL query level with user filter parameter

Common Workflows​

Create and Run a New Job​

  1. Click "Create Immediate Job" button in top-right
  2. Configure job settings (name, environment, git, parameters)
  3. Set optional notifications and resource allocations
  4. Click "Create" to launch the job
  5. Job appears in "Running" tab with "in progress" or "pending" status
  6. Monitor logs via "Logs" column (events dialog or Grafana link)
  7. Upon completion, job moves to "History" tab

Clone an Existing Job​

  1. Navigate to desired job in Running or History table
  2. Click "Actions" menu on the job row
  3. Select "Clone" from dropdown
  4. Modify configuration in pre-populated dialog
  5. Create the cloned job
  6. New job starts with same configuration as original

Monitor Job Progress​

  1. Find job in "Running" tab
  2. Click job name to view detailed information
  3. Use "Logs" column to access real-time events
  4. Click Grafana icon to view comprehensive logs in Grafana dashboard
  5. Monitor status, duration, and resource utilization
  6. Terminate job if needed via Actions menu

Filter Jobs by Parameters​

  1. Apply filters via toolbar filter options
  2. Click on parameter chip (if parameters already filtered)
  3. Add/edit parameter key-value pairs in dialog
  4. Apply filters to see matching jobs
  5. Clear filters using "Reset Filters" button

Convert to Scheduled Job​

  1. Locate immediate job in table
  2. Click "Actions" menu
  3. Select "Convert to Scheduled"
  4. Configure schedule settings (cron expression, timezone)
  5. Save to create scheduled version of job

Publish a Job for Team Access​

  1. Find your job in the History or Running table
  2. Click "Actions" menu
  3. Select "Publish" to make job visible to all team members
  4. Published jobs show "published" chip indicator
  5. Other users can now view and clone your job

Notes & Tips​

Performance Optimization​

  • Tables load 20 jobs per page to balance performance and usability
  • Satellite cluster jobs poll every 5 seconds for status updates
  • Primary cluster jobs use standard GraphQL queries without polling
  • Filter operations include email-based scoping to reduce query load

Job Status Lifecycle​

  • pending: Job created, waiting for resources
  • in progress: Job actively running
  • done: Job completed successfully
  • failed: Job encountered errors (check Status Reason tooltip)
  • cancelled: Job manually terminated

Status Reason Interpretation​

  • "Timed out": BackoffLimitExceeded - Job exceeded retry limit
  • "All retries failed": DeadlineExceeded - Job exceeded active timeout
  • Custom reasons displayed as-is from Kubernetes

Best Practices​

  • Pin Important Jobs: Use pin feature to keep critical jobs at top of list
  • Use Descriptive Names: Job names appear in all views and should be meaningful
  • Set Appropriate Timeouts: Configure both timeout (total) and activeTimeout (per retry)
  • Leverage Parameters: Use parameters for flexible, reusable job configurations
  • Monitor Billing: Use Billing Project ID to track costs per project
  • Enable Notifications: Configure alerts for long-running or critical jobs
  • Publish for Collaboration: Share successful job configurations with team via publish feature

Limitations​

  • Jobs cannot be edited after creation (clone and modify instead)
  • Maximum 20 jobs displayed per page
  • Status updates for primary cluster jobs not real-time (refresh manually)
  • Satellite cluster jobs limited to available satellite clusters
  • Filter persistence tied to URL (clearing browser history loses filters)

Grafana Integration​

  • Grafana links automatically generated for jobs with logs
  • Links include time range scoping based on job start/completion times
  • Satellite cluster Grafana uses cluster-specific domain
  • Jobs must have completed or be running to generate valid links

Satellite Cluster Considerations​

  • Must select a satellite cluster before creating jobs
  • Satellite jobs have separate configuration dialog
  • Pod specs and resources validated against satellite cluster capabilities
  • Clone operations preserve satellite cluster context
  • Satellite jobs cannot be converted to primary cluster jobs
Last updated on by chyuen-devsentient