Skip to main content

Scheduled Jobs

Overview​

The Scheduled Jobs panel provides comprehensive management capabilities for recurring pipeline jobs that execute on a defined schedule using cron expressions. This feature allows users to create, monitor, and manage automated data pipelines, batch processing tasks, and other recurring workloads that run at specified intervals across both the primary Kubernetes cluster and satellite clusters.

Access & Location​

  • Route: ?panel=scheduled-jobs
  • Navigation: Workloads → Scheduled Jobs
  • Access Requirements: None (standard user access)
  • Feature Flags: None

Key Capabilities​

Create Scheduled Jobs​

Users can create new scheduled jobs with full control over execution timing, resource allocation, and pipeline configuration. Jobs can be configured to run shell scripts or multi-step YAML pipelines on a recurring schedule with support for timezone-specific execution, parallel runs, and automatic retry mechanisms.

Monitor Active and Past Executions​

The panel provides separate views for currently scheduled jobs (pending, in progress, or scheduled status) and historical job executions (completed, failed, or cancelled). Users can track job status, execution duration, and view detailed logs and metrics for troubleshooting.

Clone and Modify Jobs​

Existing scheduled jobs can be cloned to create new jobs with similar configurations, streamlining the process of creating variations or testing different parameters. Jobs can also be cloned from their customized pod YAML specifications for advanced users.

Suspend and Resume Jobs​

Scheduled jobs can be suspended to temporarily stop them from running on their schedule without deleting the configuration. Suspended jobs can be resumed at any time to continue their scheduled execution.

Trigger Manual Executions​

Users can manually trigger immediate execution of a scheduled job outside of its normal schedule, useful for testing or running jobs on-demand while maintaining the scheduled configuration.

Satellite Cluster Support​

The feature supports managing scheduled jobs across both the primary cluster and remote satellite clusters, providing a unified interface for distributed workload management.

User Interface​

Main View​

The panel uses a tabbed interface with two primary sections:

  1. Primary Cluster Tab: Manages scheduled jobs running on the main Kubernetes cluster
  2. Satellite Clusters Tab: Manages scheduled jobs on connected satellite clusters (requires satellite cluster selection)

Each cluster view contains two sub-tabs:

  • Scheduled: Shows active scheduled jobs (status: pending, in progress, or scheduled)
  • History: Shows past job executions (status: done, failed, cancelled)

Dialogs & Modals​

  1. Create/Clone Scheduled Job Dialog

    • Purpose: Full-screen dialog for creating new scheduled jobs or cloning existing ones
    • Fields:
      • General Tab:
        • Name (alphanumeric with dashes, dots, underscores; max 63 chars)
        • Environment Config (pod specification)
        • Pipeline Type (Shell or Multi-step YAML)
        • Pipeline Path (relative path to shell script or YAML file)
        • Schedule (cron expression)
        • Timezone
        • Run Parallel (allow concurrent executions)
      • Advanced Tab:
        • Use Default Command
        • Active Timeout (seconds, -1 to 86400)
        • Max Retries (0-10)
        • Working Directory
        • Git Repository (server, branch, commit)
        • Failure Alerts (enable/disable with alert targets)
        • Cloud SQL Proxies (sidecar for GCP database connections)
        • External Dependencies (secrets, service accounts, billing projects)
      • Parameters Tab:
        • Key-value pairs for job parameters
    • Actions: Create Scheduled Job, View GraphQL Mutation, Customize Pod YAML

  2. Job Details Dialog

    • Purpose: Displays comprehensive details about a specific scheduled job
    • Includes: Job configuration, execution history, logs, and metrics
    • Actions: Clone, Trigger Now, Suspend/Resume, Delete

  3. Create from Pod YAML Dialog

    • Purpose: Advanced creation dialog for customizing the Kubernetes pod specification
    • Allows direct editing of pod YAML before job creation

Tables & Data Grids​

  1. Current Scheduled Jobs Table

    • Columns:
      • Actions (dropdown menu)
      • Name (clickable link to details, with pin button)
      • Status (visual indicator)
      • Schedule (human-readable cron description)
      • Environment Config
      • Custom Image URL (hidden by default)
      • ID (copy-to-clipboard)
      • Pipeline Path (copy-to-clipboard)
      • Created (timestamp)
      • Output Notebooks Path (hidden by default)
      • Billing Project ID (hidden by default)
      • Service Account ID (hidden by default)
      • Status Reason (hidden by default)
      • Owner (hidden by default)
      • Branch (hidden by default)
      • Commit ID (hidden by default)
      • Active Timeout (hidden by default)
      • Max Tries (hidden by default)
    • Actions: Clone, Clone from YAML, Toggle Publish, Trigger Now, Suspend/Resume, Delete
    • Filtering: Quick search (name, pipeline path, ID, author), advanced filters (status, type, group, billing project, parameters)
    • Features: Pin jobs to top, pagination (20 per page), real-time status updates

  2. Past Scheduled Jobs Table

    • Columns: Same as Current Scheduled Jobs Table
    • Actions: Clone, Clone from YAML (Delete not available for completed jobs)
    • Filtering: Same as Current Scheduled Jobs Table with history-specific statuses

Technical Details​

GraphQL Operations​

Queries:

  • getScheduledJobs - Retrieves scheduled jobs from the primary cluster with filtering and pagination

    • Parameters: offset, limit, whereClause (supports complex filtering)
    • Returns: Job details including configuration, status, timing, and metadata
    • Includes: countJobs for pagination

  • getScheduledSatelliteJobs - Retrieves scheduled jobs from satellite clusters

    • Parameters: satelliteClusterName, offset, limit, whereClause
    • Returns: Same job details as primary cluster query

  • GetPipelineJobPodSpecJson - Generates Kubernetes pod specification from job parameters

    • Used during job creation to validate and preview the pod configuration

Mutations:

  • cancelScheduledJob - Deletes a scheduled job and cancels any running executions

    • Parameters: id (job ID)
    • Returns: Confirmation with deleted job ID

  • suspendOrResumeScheduledJob - Toggles the suspended state of a scheduled job

    • Parameters: id (job ID), suspend (boolean)
    • Returns: Updated job with new suspended status

  • createPipelineJobWithAlerting - Creates a new scheduled job with all configuration options

    • Parameters: jobName, jobType, schedule, timezone, timeout, activeTimeout, maxRetries, pipelineYamlPath, workingDir, gitServerName, branchName, commitId, parameters, notificationsEnabled, notificationTargetIds, and many more
    • Returns: Created job details

  • PinJob - Pins or unpins a job to keep it at the top of the table

    • Parameters: id (job ID), pin (boolean)
    • Returns: Updated pin state

Subscriptions:

  • None (uses polling with 5-second interval for satellite clusters)

Component Structure​

  • Main Component: /root/gitrepos/monorepo/apps/hyperplane-dashboard/components/Jobs/Scheduled/ScheduledJobsPanel.tsx
  • Tables:
    • /root/gitrepos/monorepo/apps/hyperplane-dashboard/components/Jobs/Scheduled/Tables/CurrentScheduledJobsTable.tsx
    • /root/gitrepos/monorepo/apps/hyperplane-dashboard/components/Jobs/Scheduled/Tables/PastScheduledJobsTable.tsx
  • Dialogs:
    • /root/gitrepos/monorepo/apps/hyperplane-dashboard/components/Jobs/Scheduled/ScheduledJobDialog.tsx
    • /root/gitrepos/monorepo/apps/hyperplane-dashboard/components/Jobs/Scheduled/satelliteCluster/ScheduledSatelliteJobDialog.tsx
  • Actions Menu: /root/gitrepos/monorepo/apps/hyperplane-dashboard/components/Jobs/Scheduled/ScheduledJobsActionsMenu.tsx

Common Workflows​

Creating a Basic Scheduled Job​

  1. Click "Create Scheduled Job" button in the top-right
  2. Enter a unique name (alphanumeric with dashes, dots, underscores)
  3. Select an Environment Config (pod specification)
  4. Choose Pipeline Type (Shell or Multi-step YAML)
  5. Enter the relative path to your script or YAML file
  6. Configure the schedule using a cron expression
  7. Select the timezone for execution
  8. (Optional) Configure advanced settings: timeouts, retries, git repository
  9. (Optional) Add parameters as key-value pairs
  10. Review the Job Summary panel on the right
  11. Click "Create Scheduled Job"

Cloning an Existing Job​

  1. Locate the job you want to clone in either the Scheduled or History table
  2. Click the "Actions" dropdown menu for that job
  3. Select "Clone" from the menu
  4. The create dialog opens pre-populated with the job's configuration
  5. Modify any settings (name, schedule, parameters, etc.)
  6. Click "Create Scheduled Job" to create the clone

Suspending and Resuming a Job​

  1. Find the scheduled job in the Scheduled tab
  2. Click the "Actions" dropdown menu
  3. Select "Suspend" to pause scheduled executions
  4. To resume, click "Actions" → "Resume" on the suspended job
  5. Suspended jobs remain in the Scheduled tab with a "suspended" status indicator

Triggering a Manual Execution​

  1. Navigate to the Scheduled tab
  2. Find the job you want to run immediately
  3. Click "Actions" → "Trigger Now"
  4. The job creates a new immediate execution while maintaining its schedule
  5. View the execution in the Immediate Jobs panel

Managing Jobs on Satellite Clusters​

  1. Click the Satellite Clusters tab (satellite icon)
  2. Select a satellite cluster from the dropdown
  3. View, create, clone, or manage jobs specific to that cluster
  4. Jobs on satellite clusters have the same functionality as primary cluster jobs
  5. Note: Satellite cluster jobs cannot be suspended (only available on primary cluster)

Filtering and Searching Jobs​

  1. Use the quick search bar to find jobs by name, pipeline path, ID, or author
  2. Apply advanced filters using the table toolbar:
    • Status (pending, in progress, scheduled, done, failed, cancelled)
    • Environment Config type
    • Group
    • Billing Project ID
    • User (filtered by email for non-admin users)
    • Parameters (key-value pairs)
  3. View active filters as chips above the tables
  4. Clear individual filters by clicking the X on a chip
  5. Click "Reset Filters" to clear all filters

Publishing Jobs for Team Access​

  1. Locate the job in the Scheduled or History tab
  2. Click "Actions" → "Publish" on a private job
  3. Published jobs display a "published" chip next to the name
  4. Published jobs are visible to all users in the workspace
  5. To unpublish, click "Actions" → "Unpublish"
  • Jobs - One-time job executions
  • Services - Long-running microservices with exposed ports
  • Environment Configs - Pod specifications and resource allocations
  • Git Repositories - Source control integration for pipeline code
  • Billing Projects - Cost tracking and allocation for job executions
  • Alert Targets - Notification channels for job failures

Notes & Tips​

Scheduling Best Practices​

  • Use crontab.guru to validate and understand cron expressions
  • Consider timezone settings carefully, especially for jobs that need to run at specific business hours
  • The "Run Parallel" option should be disabled for jobs that cannot run concurrently (e.g., jobs that write to the same database table)
  • Use the human-readable schedule description in the table to verify your cron expression

Resource Management​

  • Active Timeout (-1 for unlimited, or 1-86400 seconds) determines how long a job can run before being terminated
  • Max Retries (0-10) controls automatic retry attempts for failed job starts
  • Default values: 86400s timeout, 86400s active timeout, 2 max retries

Git Integration​

  • Default Environment Configs require git repository integration (cannot be disabled)
  • Custom Environment Configs can optionally use git repositories or rely on files baked into the image
  • Specifying a commit ID requires also specifying the branch name
  • Working directory defaults to /tmp/git/monorepo/ after git sync

Parameters​

  • Parameters are passed to the pipeline as environment variables
  • Use parameters for dynamic configuration without modifying code
  • Parameter keys and values must both be non-empty strings
  • Filter jobs by parameter values in the advanced filters

Pinning Jobs​

  • Pin frequently accessed jobs to keep them at the top of the table
  • Pinned jobs appear first, followed by other jobs sorted by start time (descending)
  • Pin status is preserved across page refreshes

Performance Considerations​

  • Tables are paginated at 20 jobs per page for optimal performance
  • Satellite cluster jobs poll every 5 seconds for status updates
  • Use filters to narrow down large job lists
  • Hidden columns can be shown via the column menu (three-dot icon in column headers)

URL State Management​

  • Filters are persisted in URL parameters (currentFilters, pastFilters)
  • Job details can be accessed directly via URL with ?panel=scheduled-jobs&jobId={id}
  • Satellite cluster job details require both jobId and satelliteClusterName parameters

Limitations​

  • Maximum job name length: 63 characters
  • Job names must start and end with alphanumeric characters
  • Job names cannot contain spaces
  • Maximum active timeout: 86400 seconds (24 hours)
  • Maximum retries: 10
  • Only jobs on the primary cluster can be suspended/resumed
  • Publishing/unpublishing is only available for primary cluster jobs
Last updated on by chyuen-devsentient