Camunda Platform 8: BPMN and Process Orchestration

Process orchestration is essential for modern enterprises managing complex workflows across distributed systems. Camunda Platform 8 represents a fundamental shift from traditional BPM engines to cloud-native process orchestration. Here's what I've learned implementing Camunda across enterprise projects.

Why Process Orchestration Matters

In microservices architectures, coordinating business processes across multiple services becomes increasingly complex. Without proper orchestration:

Visibility gaps: No clear view of process state across services
Error handling complexity: Each service handles failures differently
Audit challenges: Reconstructing process history from distributed logs
Coordination overhead: Manual tracking of multi-step operations

Camunda Platform 8 Architecture

Zeebe: The Process Engine

Camunda 8 is built on Zeebe, a horizontally scalable workflow engine:

Camunda Platform 8 Architecture:

┌─────────────────────────────────────────────────────────────┐
│                      Camunda Cloud                          │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │   Zeebe     │  │   Zeebe     │  │   Zeebe     │        │
│  │  Broker 1   │  │  Broker 2   │  │  Broker 3   │        │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘        │
│         │                │                │                │
│         └────────────────┼────────────────┘                │
│                          │                                 │
│  ┌───────────────────────▼───────────────────────┐        │
│  │              Zeebe Gateway                     │        │
│  │         (gRPC API / REST API)                 │        │
│  └───────────────────────┬───────────────────────┘        │
│                          │                                 │
├──────────────────────────┼──────────────────────────────────┤
│                          │                                 │
│  ┌───────────┐  ┌───────▼───────┐  ┌─────────────┐        │
│  │  Operate  │  │   Tasklist    │  │  Optimize   │        │
│  │(Monitoring)│  │(Human Tasks) │  │ (Analytics) │        │
│  └───────────┘  └───────────────┘  └─────────────┘        │
│                                                             │
└─────────────────────────────────────────────────────────────┘
                          │
         ┌────────────────┼────────────────┐
         │                │                │
    ┌────▼────┐     ┌────▼────┐     ┌────▼────┐
    │ Worker  │     │ Worker  │     │ Worker  │
    │Service 1│     │Service 2│     │Service 3│
    └─────────┘     └─────────┘     └─────────┘

Key Architectural Differences from Camunda 7

Aspect	Camunda 7	Camunda 8
Engine	Embedded Java engine	Distributed Zeebe brokers
Scaling	Vertical (larger JVM)	Horizontal (add brokers)
Database	RDBMS (PostgreSQL, etc.)	Append-only log + Elasticsearch
Communication	REST/Java API	gRPC (primary), REST (gateway)
Execution	Synchronous	Asynchronous job workers
Deployment	Self-managed	Cloud-native / Self-managed

BPMN 2.0 Fundamentals

Core BPMN Elements

BPMN Element Categories:

Events (Circles):
  ○ Start Event      - Process entry point
  ◎ Intermediate     - Mid-process events
  ● End Event        - Process completion

Activities (Rounded Rectangles):
  ┌─────────┐
  │  Task   │        - Single unit of work
  └─────────┘
  ┌─────────┐
  │ ═══════ │        - Sub-process (embedded)
  │  Task   │
  └─────────┘

Gateways (Diamonds):
  ◇ Exclusive (XOR)  - One path based on condition
  ◆ Parallel (AND)   - All paths execute
  ○◇ Inclusive (OR)  - One or more paths
  ◇→ Event-based     - Wait for first event

Sequence Flows:
  ────────→          - Normal flow
  ─ ─ ─ ─→          - Conditional flow

Practical BPMN Example: Order Processing

<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL"
                  xmlns:zeebe="http://camunda.org/schema/zeebe/1.0">

  <bpmn:process id="order-processing" isExecutable="true">

    <!-- Start Event -->
    <bpmn:startEvent id="order-received" name="Order Received">
      <bpmn:outgoing>flow1</bpmn:outgoing>
    </bpmn:startEvent>

    <!-- Validate Order -->
    <bpmn:serviceTask id="validate-order" name="Validate Order">
      <bpmn:extensionElements>
        <zeebe:taskDefinition type="validate-order" />
      </bpmn:extensionElements>
      <bpmn:incoming>flow1</bpmn:incoming>
      <bpmn:outgoing>flow2</bpmn:outgoing>
    </bpmn:serviceTask>

    <!-- Decision Gateway -->
    <bpmn:exclusiveGateway id="validation-result" name="Valid?">
      <bpmn:incoming>flow2</bpmn:incoming>
      <bpmn:outgoing>flow-valid</bpmn:outgoing>
      <bpmn:outgoing>flow-invalid</bpmn:outgoing>
    </bpmn:exclusiveGateway>

    <!-- Parallel Gateway - Process Order -->
    <bpmn:parallelGateway id="parallel-start">
      <bpmn:incoming>flow-valid</bpmn:incoming>
      <bpmn:outgoing>flow-payment</bpmn:outgoing>
      <bpmn:outgoing>flow-inventory</bpmn:outgoing>
    </bpmn:parallelGateway>

    <!-- Process Payment -->
    <bpmn:serviceTask id="process-payment" name="Process Payment">
      <bpmn:extensionElements>
        <zeebe:taskDefinition type="process-payment" retries="3" />
      </bpmn:extensionElements>
      <bpmn:incoming>flow-payment</bpmn:incoming>
      <bpmn:outgoing>flow-payment-done</bpmn:outgoing>
    </bpmn:serviceTask>

    <!-- Reserve Inventory -->
    <bpmn:serviceTask id="reserve-inventory" name="Reserve Inventory">
      <bpmn:extensionElements>
        <zeebe:taskDefinition type="reserve-inventory" retries="3" />
      </bpmn:extensionElements>
      <bpmn:incoming>flow-inventory</bpmn:incoming>
      <bpmn:outgoing>flow-inventory-done</bpmn:outgoing>
    </bpmn:serviceTask>

    <!-- Join Parallel -->
    <bpmn:parallelGateway id="parallel-join">
      <bpmn:incoming>flow-payment-done</bpmn:incoming>
      <bpmn:incoming>flow-inventory-done</bpmn:incoming>
      <bpmn:outgoing>flow-to-ship</bpmn:outgoing>
    </bpmn:parallelGateway>

    <!-- Ship Order -->
    <bpmn:serviceTask id="ship-order" name="Ship Order">
      <bpmn:extensionElements>
        <zeebe:taskDefinition type="ship-order" />
      </bpmn:extensionElements>
      <bpmn:incoming>flow-to-ship</bpmn:incoming>
      <bpmn:outgoing>flow-to-end</bpmn:outgoing>
    </bpmn:serviceTask>

    <!-- End Events -->
    <bpmn:endEvent id="order-completed" name="Order Completed">
      <bpmn:incoming>flow-to-end</bpmn:incoming>
    </bpmn:endEvent>

    <bpmn:endEvent id="order-rejected" name="Order Rejected">
      <bpmn:incoming>flow-invalid</bpmn:incoming>
    </bpmn:endEvent>

  </bpmn:process>
</bpmn:definitions>

Building Job Workers

TypeScript Worker Implementation

import { ZBClient } from 'zeebe-node';

interface OrderVariables {
  orderId: string;
  customerId: string;
  items: Array<{ sku: string; quantity: number; price: number }>;
  totalAmount: number;
}

interface ValidationResult {
  isValid: boolean;
  validationErrors: string[];
}

// Create Zeebe client
const zbc = new ZBClient({
  camundaCloud: {
    clusterId: process.env.ZEEBE_CLUSTER_ID!,
    clientId: process.env.ZEEBE_CLIENT_ID!,
    clientSecret: process.env.ZEEBE_CLIENT_SECRET!,
  },
});

// Validate Order Worker
zbc.createWorker<OrderVariables, ValidationResult>({
  taskType: 'validate-order',
  taskHandler: async (job) => {
    const { orderId, items, totalAmount } = job.variables;

    console.log(`Validating order ${orderId}`);

    const errors: string[] = [];

    // Validate items
    if (!items || items.length === 0) {
      errors.push('Order must contain at least one item');
    }

    // Validate amounts
    const calculatedTotal = items?.reduce(
      (sum, item) => sum + item.price * item.quantity,
      0
    ) || 0;

    if (Math.abs(calculatedTotal - totalAmount) > 0.01) {
      errors.push('Order total does not match item prices');
    }

    // Validate inventory availability (external call)
    for (const item of items || []) {
      const available = await checkInventory(item.sku, item.quantity);
      if (!available) {
        errors.push(`Item ${item.sku} not available in requested quantity`);
      }
    }

    return job.complete({
      isValid: errors.length === 0,
      validationErrors: errors,
    });
  },
});

// Process Payment Worker
zbc.createWorker<OrderVariables & { paymentMethod: string }>({
  taskType: 'process-payment',
  taskHandler: async (job) => {
    const { orderId, totalAmount, customerId, paymentMethod } = job.variables;

    console.log(`Processing payment for order ${orderId}`);

    try {
      const paymentResult = await paymentService.charge({
        customerId,
        amount: totalAmount,
        currency: 'GBP',
        method: paymentMethod,
        orderId,
      });

      return job.complete({
        paymentId: paymentResult.transactionId,
        paymentStatus: 'completed',
      });
    } catch (error) {
      // Fail the job - Zeebe will retry based on retries config
      return job.fail(`Payment failed: ${error.message}`);
    }
  },
});

// Reserve Inventory Worker
zbc.createWorker<OrderVariables>({
  taskType: 'reserve-inventory',
  taskHandler: async (job) => {
    const { orderId, items } = job.variables;

    console.log(`Reserving inventory for order ${orderId}`);

    const reservations = await Promise.all(
      items.map(item =>
        inventoryService.reserve({
          sku: item.sku,
          quantity: item.quantity,
          orderId,
        })
      )
    );

    return job.complete({
      reservationIds: reservations.map(r => r.id),
    });
  },
});

// Ship Order Worker
zbc.createWorker<OrderVariables & { reservationIds: string[] }>({
  taskType: 'ship-order',
  taskHandler: async (job) => {
    const { orderId, customerId } = job.variables;

    console.log(`Creating shipment for order ${orderId}`);

    const shipment = await shippingService.createShipment({
      orderId,
      customerId,
      items: job.variables.items,
    });

    // Send notification
    await notificationService.send({
      type: 'ORDER_SHIPPED',
      customerId,
      data: {
        orderId,
        trackingNumber: shipment.trackingNumber,
        estimatedDelivery: shipment.estimatedDelivery,
      },
    });

    return job.complete({
      shipmentId: shipment.id,
      trackingNumber: shipment.trackingNumber,
    });
  },
});

Worker Best Practices

// Idempotent worker design
zbc.createWorker({
  taskType: 'process-payment',
  taskHandler: async (job) => {
    const { orderId } = job.variables;

    // Check if already processed (idempotency)
    const existing = await paymentRepo.findByOrderId(orderId);
    if (existing) {
      console.log(`Payment already processed for order ${orderId}`);
      return job.complete({
        paymentId: existing.id,
        paymentStatus: existing.status,
      });
    }

    // Process new payment
    // ...
  },
  // Worker configuration
  maxJobsToActivate: 32,      // Batch size
  timeout: Duration.seconds.of(30),  // Job timeout
  pollInterval: Duration.milliseconds.of(300),
});

Starting Process Instances

Via REST API

// Start process instance
const processInstance = await zbc.createProcessInstance({
  bpmnProcessId: 'order-processing',
  variables: {
    orderId: 'ORD-12345',
    customerId: 'CUST-789',
    items: [
      { sku: 'WIDGET-001', quantity: 2, price: 29.99 },
      { sku: 'GADGET-002', quantity: 1, price: 49.99 },
    ],
    totalAmount: 109.97,
    paymentMethod: 'card',
  },
});

console.log(`Started process instance: ${processInstance.processInstanceKey}`);

With Message Correlation

// Start via message (event-driven)
await zbc.publishMessage({
  name: 'order-received',
  correlationKey: 'ORD-12345',
  variables: {
    orderId: 'ORD-12345',
    // ... order data
  },
  timeToLive: Duration.seconds.of(60),
});

// Correlate message to running instance
await zbc.publishMessage({
  name: 'payment-callback',
  correlationKey: 'ORD-12345',  // Matches running instance
  variables: {
    paymentStatus: 'confirmed',
    transactionId: 'TXN-ABC123',
  },
});

Error Handling and Compensation

BPMN Error Events

// Throw BPMN error for business exceptions
zbc.createWorker({
  taskType: 'process-payment',
  taskHandler: async (job) => {
    try {
      const result = await paymentService.charge(/* ... */);
      return job.complete({ paymentId: result.id });
    } catch (error) {
      if (error.code === 'INSUFFICIENT_FUNDS') {
        // BPMN error - caught by error boundary event
        return job.error('PAYMENT_FAILED', 'Insufficient funds');
      }
      if (error.code === 'CARD_DECLINED') {
        return job.error('PAYMENT_DECLINED', 'Card was declined');
      }
      // Technical failure - retry
      return job.fail(`Payment service error: ${error.message}`);
    }
  },
});

Compensation Pattern

Saga Pattern with Compensation:

    ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
    │   Reserve    │────▶│   Charge     │────▶│    Ship      │
    │  Inventory   │     │   Payment    │     │    Order     │
    └──────────────┘     └──────────────┘     └──────────────┘
           │                    │                    │
           │ Compensate         │ Compensate         │
           ▼                    ▼                    ▼
    ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
    │   Release    │◀────│   Refund     │◀────│   Cancel     │
    │  Inventory   │     │   Payment    │     │   Shipment   │
    └──────────────┘     └──────────────┘     └──────────────┘

Monitoring with Operate

Key Metrics to Track

Metric	Description	Alert Threshold
Active Instances	Currently running processes	> 10,000
Incident Rate	Failed jobs / Total jobs	> 5%
Cycle Time	Start to end duration	> SLA
Job Latency	Time in queue before pickup	> 30s
Worker Throughput	Jobs completed per minute	< baseline

Operate Dashboard Integration

// Custom metrics for Prometheus
import { Counter, Histogram } from 'prom-client';

const processStarted = new Counter({
  name: 'camunda_process_started_total',
  help: 'Total processes started',
  labelNames: ['process_id'],
});

const taskDuration = new Histogram({
  name: 'camunda_task_duration_seconds',
  help: 'Task execution duration',
  labelNames: ['task_type'],
  buckets: [0.1, 0.5, 1, 2, 5, 10, 30],
});

// Instrument workers
zbc.createWorker({
  taskType: 'validate-order',
  taskHandler: async (job) => {
    const timer = taskDuration.startTimer({ task_type: 'validate-order' });
    try {
      // ... task logic
      return job.complete(result);
    } finally {
      timer();
    }
  },
});

Key Takeaways

Zeebe is fundamentally different: Async job workers replace synchronous execution—design accordingly
Horizontal scaling: Add brokers and workers independently based on load
Idempotency is critical: Workers may receive the same job multiple times—design for replay
Error handling strategy: Distinguish between technical failures (retry) and business errors (BPMN error events)
Compensation over rollback: Implement saga patterns for distributed transactions
Observability first: Instrument workers from day one—Operate provides visibility but custom metrics add depth

Camunda Platform 8 represents a paradigm shift in process orchestration. The move to cloud-native architecture unlocks horizontal scalability but requires rethinking how you design and implement process automation. \