Skip to main content

Alert Targets

Overview​

Alert Targets is a notification management system that allows users to configure and manage external notification channels for receiving alerts about workload failures and cluster events in the Shakudo platform. Users can set up integrations with popular communication platforms (Slack, PagerDuty, Mattermost) or email groups to receive real-time notifications when their jobs, microservices, or scheduled jobs encounter errors or when critical cluster events occur (e.g., stack component failures or PVC capacity issues).

Access & Location​

  • Route: ?panel=alert-targets
  • Navigation: Shakudo Objects → Alert Targets
  • Access Requirements:
    • View: All authenticated users can view their own alert targets
    • Create: Dashboard Admin or Dashboard Maintainer roles
    • Delete: Dashboard Admin role (users can delete their own targets)
    • View All: Dashboard Admin or Dashboard Maintainer roles can view all alert targets across users
  • Feature Flags: None

Key Capabilities​

Create Alert Targets​

Configure notification channels to receive alerts about workload failures. Supports multiple integration types:

  • Slack: Via webhook URL or bot token with channel specification
  • PagerDuty: Via integration key for incident management
  • Email Groups: Comma-separated list of email addresses
  • Mattermost: Via incoming webhook URL

Manage Alert Subscriptions​

Subscribe alert targets to receive cluster-wide notifications about critical infrastructure events, including stack component failures and persistent volume capacity alerts. This is controlled via the "Receive Cluster Alerts" toggle on each target.

Associate with Workloads​

Alert targets can be attached to immediate jobs, scheduled jobs, and microservices during workload creation or editing. When a workload fails, all associated alert targets receive notifications.

Monitor Target Usage​

View which workloads are using each alert target through detailed dialog that shows all associated immediate jobs, scheduled jobs, and microservices with filtering and search capabilities.

User Interface​

Main View​

The main panel displays a data grid showing all alert targets with the following information:

  • Name: Clickable link to view detailed information about the target
  • Type: Visual indicator of the integration type (Slack, PagerDuty, Email Groups, Mattermost)
  • Created On: Timestamp of when the target was created
  • Receive Cluster Alerts: Toggle switch to enable/disable cluster-wide alert subscriptions
  • Actions: Delete button (visible only for targets owned by the user or for admins)

The grid supports:

  • Column visibility management
  • Sorting by various fields
  • Manual refresh via toolbar button
  • Pagination (20 items per page)

Dialogs & Modals​

  1. Create Alert Target Dialog

    • Purpose: Configure a new notification channel
    • Fields:
      • Target Channel: Dropdown to select integration type (Slack, PagerDuty, Email Groups, Mattermost)
      • Alert Target Name: User-friendly name for the target
      • Type-specific fields:
        • Slack: Webhook URL OR (Bot Token + Channel Name)
        • PagerDuty: Integration Key
        • Email Groups: Comma-separated email addresses
        • Mattermost: Incoming webhook URL
    • Actions: Cancel or Create
    • Validation: Email format validation for email groups, required field validation

  2. Detail Alert Target Dialog

    • Purpose: View comprehensive information about an alert target and its associated workloads
    • Sections:
      • Target Information: Shows integration type with icon and specific details (email addresses for email targets, channel name for Slack)
      • Workload Type Selector: Dropdown to filter by workload type (Immediate Jobs, Scheduled Jobs, Microservices)
      • Associated Workloads Table: Data grid showing all workloads using this target
    • Actions: Close button
    • Table Features: Column management, quick search, pagination (10 items per page)

  3. Delete Alert Target Dialog

    • Purpose: Confirm deletion of an alert target
    • Prompt: Confirmation message with target name/ID
    • Actions: Cancel or Delete
    • Note: System-created targets cannot be deleted

Tables & Data Grids​

  1. Main Alert Targets Table

    • Columns: Actions, Name, ID (hidden), Type, Created On, Receive Cluster Alerts
    • Actions: Click name to view details, toggle cluster alerts, delete target
    • Filtering: By cluster alert subscription status
    • Features: Column customization, sorting by most fields

  2. Associated Workloads Table (in Detail Dialog)

    • Columns: Name, Logs, Status, Env Config, ID (hidden), Pipeline Path (hidden), Start
    • Actions:
      • Click name to navigate to workload panel with filter applied
      • View logs in Grafana (for running workloads)
      • View events and logs (for in-progress workloads)
    • Filtering: By workload type (immediate jobs, scheduled jobs, microservices)
    • Features: Quick search, column customization, manual refresh

Technical Details​

GraphQL Operations​

Queries:

  • getTargetNotifications - Retrieves all alert targets for the current user (or all targets for admin users). Includes target details like name, type, email/channel info, creation date, and cluster alert subscription status.

Mutations:

  • createJobNotificationTarget - Creates a new alert target with specified integration type and configuration. Sensitive data (API keys, tokens, webhook URLs) is base64-encoded before storage.
  • deleteJobNotificationTarget - Permanently deletes an alert target by ID.
  • setNotificationGlobal - Updates the cluster alert subscription status (isGlobalReceiver field) for a specific alert target.

Subscriptions:

  • None

Component Structure​

  • Main Component: /components/AlertTargets/AlertTargetsPanel.tsx
  • Dialogs:
    • /components/AlertTargets/CreateAlertTargetDialog.tsx
    • /components/AlertTargets/DetailAlertTargetDialog.tsx
    • /components/AlertTargets/DeleteAlertTargetDialog.tsx
  • Supporting Components:
    • /components/AlertTargets/DetailAlertTargetTableButton.tsx - Name button that opens detail dialog
    • /components/AlertTargets/AlertTargetAutocomplete.tsx - Autocomplete component for selecting alert targets
    • /components/AlertTargets/utils.tsx - Helper functions for mapping target types to display names and icons

Integration Types​

The feature supports four notification target types defined in the Prisma schema:

  • SLACK - Slack workspace integration
  • PAGERDUTY - PagerDuty incident management
  • EMAIL - Email distribution lists
  • MATTERMOST - Mattermost team chat

Security Considerations​

  • Sensitive authentication data (webhook URLs, API tokens, integration keys) is base64-encoded before being sent to the backend
  • Users can only view and delete their own alert targets by default
  • Admin users (Dashboard Admin, Dashboard Maintainer) can view all alert targets across the organization
  • Only Dashboard Admin users can delete any alert target

Common Workflows​

Create a Slack Alert Target​

  1. Click "Create Alert Target" button in the main panel
  2. Select "Slack" from the Target Channel dropdown
  3. Enter a descriptive name for the alert target
  4. Choose integration method:
    • Option A: Paste Slack Webhook URL
    • Option B: Enter Slack Bot Token (xoxb-...) and Channel Name
  5. Click "Create" to save the configuration

Subscribe Target to Cluster Alerts​

  1. Locate the alert target in the main table
  2. Find the "Receive Cluster Alerts" column
  3. Toggle the switch to "on" (blue) to enable cluster alert notifications
  4. The target will now receive alerts for stack component failures and PVC capacity issues

View Workloads Using an Alert Target​

  1. Click on an alert target name in the main table
  2. In the detail dialog, use the "Workload Type" dropdown to filter by:
    • Immediate Jobs
    • Scheduled Jobs
    • Microservices
  3. View the list of workloads with status, environment config, and logs
  4. Click on a workload name to navigate to its details panel

Attach Alert Target to a Workload​

  1. When creating or editing a job/service, find the "Alert Targets" section
  2. Use the autocomplete field to search and select one or more alert targets
  3. Save the workload configuration
  4. The selected targets will receive notifications when the workload fails
  • Immediate Jobs - Can be configured to use alert targets for failure notifications
  • Scheduled Jobs - Can be configured to use alert targets for failure notifications
  • Microservices - Can be configured to use alert targets for failure notifications
  • Stack Components - Failures trigger notifications to targets with cluster alerts enabled

Notes & Tips​

Best Practices​

  • Use descriptive names for alert targets that indicate their purpose (e.g., "Engineering Team Slack", "Critical Alerts - PagerDuty")
  • For production workloads, consider using multiple alert targets (e.g., both Slack and email) for redundancy
  • Enable cluster alerts only for targets that should receive infrastructure-related notifications

Email Group Configuration​

  • Provide comma-separated email addresses without spaces: user1@example.com,user2@example.com
  • Email validation ensures proper format before saving
  • No configuration of SMTP server is required - the platform handles email delivery

Slack Integration Options​

  • Webhook URL: Simpler setup, posts to a single predetermined channel
  • Bot Token + Channel: More flexible, allows specifying the channel in the dashboard and can be reused across multiple alert targets

PagerDuty Integration​

  • Requires a PagerDuty integration key from your PagerDuty service
  • Each alert target can route to a different PagerDuty service/escalation policy
  • Alerts are sent as incidents with appropriate severity levels

Limitations​

  • System-created alert targets (createdBySystem flag) cannot be deleted through the UI
  • Alert targets created by other users are only visible to Dashboard Admin and Dashboard Maintainer roles
  • Cluster alert subscription requires create permissions (Dashboard Admin or Maintainer roles)

Troubleshooting​

  • If notifications are not being received, verify the integration credentials are correct
  • For Slack bot tokens, ensure the bot has been added to the target channel
  • For email targets, check spam/junk folders if notifications are missing
  • System-wide alert target issues should be reported to platform administrators
Last updated on by chyuen-devsentient