Skip to main content

Data Stack Graph

Overview​

The Data Stack Graph is a real-time network visualization feature that displays the data flow and connections between stack components (databases, message queues, etc.), Shakudo processes (jobs, sessions, microservices), and Kubernetes namespaces. It provides a visual representation of active network traffic and communication patterns within the platform, helping users understand how different components interact with each other.

Access & Location​

  • Route: ?panel=data-stack-graph
  • Navigation: Monitoring → Data Stack Graph
  • Access Requirements: None
  • Feature Flags: None

Key Capabilities​

Real-time Network Monitoring​

Visualizes active connections between stack components and Shakudo workloads with automatic updates every 5 seconds. The graph shows data flow direction with animated particles moving along connection lines to represent active traffic.

Connection Discovery​

Automatically detects and displays network connections between:

  • Stack Components (Platform Apps like databases, message brokers, monitoring tools)
  • Microservices (Deployed Shakudo services)
  • Jobs (Pipeline workloads)
  • Sessions (Development environments)
  • Kubernetes Namespaces (Container orchestration layers)

Standalone Component Detection​

Identifies and lists components that have no active network connections, helping users understand which resources are isolated or potentially underutilized.

User Interface​

Main View​

The panel consists of a two-column layout:

  1. Left Panel (Standalone Components)

    • Lists components with no active connections detected
    • Displays component icons and names
    • Groups multiple jobs/sessions with count indicators (e.g., "Jobs (5)", "Sessions (3)")
    • Shows "All components have active connections" when no isolated components exist

  2. Right Panel (Interactive Network Graph)

    • Force-directed graph visualization of connected components
    • Nodes represent different types of resources with distinctive icons:
      • Platform Apps: Component-specific logos
      • Microservices: Shakudo microservice icon
      • Jobs: Pipeline workload icon
      • Sessions: Development environment icon
      • Namespaces: Generic component icon
    • Edges show directional data flow between nodes
    • Animated particles flow along edges representing active traffic
    • Node highlights: Components with recent traffic show an orange glow effect
    • Auto-zoom functionality to fit all nodes within view

Visual Elements​

Node Types:

  • Platform Apps & Microservices: White background with component logo
  • Jobs: Pipeline icon on gray background
  • Sessions: Development environment icon on gray background
  • Namespaces: Generic icon on gray background

Edge Styling:

  • Active connections: Gray lines (#808080) with animated particles
  • Inactive connections: Light gray lines (#D9D9D9) with no animation
  • Traffic indicators: Orange animated particles (#CF602980) flowing from source to target

Status Indicators:

  • Recent Activity: Orange glow/shadow around nodes that have had recent network traffic
  • Last Update: Timestamp displayed at bottom right showing when data was last refreshed

Dialogs & Modals​

This feature does not include any dialogs or modals. All interactions occur within the main visualization.

Tables & Data Grids​

No tables are present. The feature uses a list view for standalone components and a force-graph visualization for connected components.

Technical Details​

GraphQL Operations​

Queries:

  • getNodesAndLinks - Retrieves visual node and edge data for the network graph
    • Returns getVisualNodesAndEdges: Array of stack components and their connection information
    • Returns getVisualPodInfo: Array of pod/workload information including jobs, sessions, and microservices
  • getActiveTrafficStatus - Checks for active network traffic status (defined but not actively used in current implementation)

Mutations: None

Subscriptions: None

Polling:

  • The getNodesAndLinks query polls every 5 seconds (5000ms) to maintain real-time updates
  • Polling starts when component mounts and stops on unmount

Component Structure​

  • Main Component: /components/DataStackGraph/DataStackGraphPanel.tsx
  • Visualization: /components/DataStackGraph/DataStackForceGraph.tsx
  • State Management: /atoms/DataStackGraphAtoms.tsx
  • GraphQL Queries: /graphql/data-stack-graph/

Technical Implementation Details​

Force-Directed Graph Configuration:

  • Link Distance: 200 pixels between nodes
  • Charge Strength: -1000 (repulsion force between nodes)
  • Distance Max: 500 pixels maximum interaction distance
  • Center Strength: 0 (no centering force)
  • Warmup Ticks: 250 iterations before rendering
  • Cooldown Ticks: 1000 iterations for stabilization
  • Cooldown Time: 1000ms

Node Rendering:

  • Node size: 64 pixels (IMAGE_SIZE constant)
  • Platform Apps and Microservices render on white background
  • Other node types render on gray (#D9D9D9) background
  • Recent traffic indicated by orange shadow/glow (#CF6029)
  • Custom canvas rendering for each node type

Link Rendering:

  • Directional particles represent request flow
  • Particle speed: 0.005 (slow animation for visibility)
  • Particle width: 7 pixels
  • Link width: 2 pixels
  • Number of particles is randomized (3-7 particles per active link)

State Management:

  • Uses Jotai atom (DataStackGraphLastUpdateDateAtom) to track last update timestamp
  • Update timestamp refreshed every 5 seconds via interval
  • Graph engine re-renders only when node list changes to optimize performance

Data Processing​

Node Filtering:

  • Only nodes with active connections (present in link IDs) are displayed in the graph
  • Standalone nodes are moved to the left panel list
  • Sessions and Jobs are grouped with counters when isolated
  • Namespace nodes are excluded from standalone list

Node Mapping:

  • Kubernetes namespaces renamed for clarity:
    • hyperplane-pipelines → "Workloads"
    • hyperplane-jhub → "Development"

Common Workflows​

Understanding Component Connectivity​

  1. Open the Data Stack Graph panel from Monitoring navigation
  2. Review the network graph to see how components communicate
  3. Observe animated particles to understand data flow direction
  4. Check the "Standalone" list to identify isolated components

Monitoring Active Traffic​

  1. Look for nodes with orange glow indicating recent network activity
  2. Observe particle animation speed and density for traffic volume indicators
  3. Check the last update timestamp to confirm data freshness
  4. Allow graph to auto-refresh every 5 seconds for real-time monitoring

Identifying Component Relationships​

  1. Locate a specific component by its icon or hover over nodes to see names
  2. Follow edge lines to trace data flow paths
  3. Identify which components serve as data sources vs. consumers
  4. Understand dependencies between microservices and backing services
  • Distributed Workloads: Monitor job execution and resource usage
  • Stack Components: Manage deployment and configuration of platform applications
  • Microservices: Deploy and manage custom services that appear in the graph
  • Sessions: Development environments that may connect to stack components

Notes & Tips​

Performance Considerations:

  • The graph automatically zooms to fit all nodes when first rendered or when node topology changes
  • Graph stabilization takes approximately 1 second after data updates
  • Polling occurs every 5 seconds but component only re-renders when node list changes

Visual Interpretation:

  • Thicker particle streams indicate more active connections (randomized 3-7 particles per link)
  • Gray edges without particles indicate configured but inactive connections
  • Nodes are automatically positioned by force-directed algorithm for optimal spacing
  • Graph layout may shift slightly as new connections are discovered or removed

Limitations:

  • Particle count is randomized and does not represent actual request volume
  • The feature displays network connectivity but not performance metrics
  • Historical data is not stored; only current connections are shown
  • No drill-down capability to see detailed request logs or metrics

Best Practices:

  • Use this feature to validate microservice deployment connectivity
  • Check for unexpected connections that might indicate misconfiguration
  • Monitor the standalone list for resources that should be connected but aren't
  • Combine with other monitoring panels for comprehensive system health assessment

Browser Requirements:

  • Requires HTML5 Canvas support for graph rendering
  • Best viewed on desktop browsers with adequate screen space
  • Responsive layout adjusts for mobile devices but desktop recommended for clarity
Last updated on by chyuen-devsentient