Ansible Playbooks: Detailed Guide

OmniCRM products are provisioned using Ansible, allowing for automated service management based on the specific requirements of each product and its associated inventory.

See also: SIM Card Provisioning <concepts_sim_provisioning> for a complete example of Ansible-based provisioning for mobile services, including physical SIMs and eSIMs.

How Playbooks and Products Work Together

Critical Concept: Playbooks are what actually create services in OmniCRM. When you assign a playbook to a product, you're defining what happens when that product is provisioned - but that can mean different things for different products.

Products Trigger Playbooks

When a product is provisioned in OmniCRM:

The product definition specifies which playbook to run (via provisioning_play field)
The product passes variables to the playbook (via provisioning_json_vars and inventory selections)
The playbook executes and does whatever it's programmed to do
The playbook determines what gets created (if anything)

What Playbooks Can Do

A single provisioning playbook can:

Create Multiple Services
A bundled product playbook might create:

A main internet service record
An IPTV addon service record
A VoIP service record
All with one product provision action

Create No Services
Some playbooks don't create service records at all:

A playbook that just configures CPE equipment
A playbook that sends configuration to network equipment
A playbook that updates external systems

Create One Service
The most common pattern:

Create a single service record for the customer
Link inventory to that service
Set up billing for that service

Modify Existing Services
Topup and addon playbooks:

Don't create new services
Update existing service records (add data, extend expiry, etc.)
Add balances to existing billing accounts

Perform Actions Without Service Records
Some playbooks are purely operational:

Reset account balances
Swap inventory items between customers
Generate reports or configurations

Example: Different Playbook Behaviors

# Product 1: Mobile SIM Service (creates 1 service)
# provisioning_play: play_simple_service
- Creates service record in CRM
- Creates billing account in OCS
- Assigns SIM card and phone number inventory
- Sends welcome email

# Product 2: Internet Bundle (creates 3 services)
# provisioning_play: play_bundle_internet_tv_voice
- Creates internet service record
- Creates IPTV service record
- Creates VoIP service record
- Links all to same customer
- Single billing account for the bundle

# Product 3: Data Topup (creates 0 services)
# provisioning_play: play_topup_no_charge
- Finds existing service by service_id
- Adds data balance to existing OCS account
- Updates service expiry date
- NO new service created

# Product 4: CPE Configuration (creates 0 services)
# provisioning_play: play_prov_cpe_mikrotik
- Generates router configuration
- Updates inventory record with config
- Emails config to support team
- NO service created (just equipment setup)

The key point: The playbook defines the behavior, the product is just a trigger.

Plays vs Tasks

Understanding the distinction between Plays and Tasks is fundamental to working with OmniCRM playbooks.

Play (Playbook)
A complete provisioning workflow that orchestrates multiple tasks to achieve a business objective. Plays are the top-level playbooks stored in OmniCRM-API/Provisioners/plays/ and are referenced in product definitions.

Examples:

play_simple_service.yaml - Provision a basic service
play_topup_no_charge.yaml - Apply a free topup to a service
play_prov_cpe_mikrotik.yaml - Configure customer premises equipment

Task (Reusable Component)
A self-contained, reusable set of operations that can be included by multiple plays. Tasks are prefixed with task_ and live in the same directory.

Examples:

task_welcome_email.yaml - Send a welcome email to a customer
task_activate_olt.yaml - Activate OLT equipment
task_notify_ocs.yaml - Send notifications to the billing system

The relationship between them:

# play_simple_service.yaml (A Play)
- name: Simple Provisioning Play
  hosts: localhost
  tasks:
    - name: Main provisioning block
      block:
        - name: Create service
          uri: ...

        - name: Configure billing
          uri: ...

        # Include reusable task
        - include_tasks: task_welcome_email.yaml

        # Include post-provisioning tasks
        - include_tasks: post_provisioning_tasks.yaml

Playbook Structure and Anatomy

All OmniCRM playbooks follow a consistent structure. Understanding this structure is essential for creating and maintaining playbooks.

Basic Structure

Every playbook starts with these standard headers:

- name: Descriptive Name of the Playbook
  hosts: localhost           # Always localhost for OmniCRM
  gather_facts: no           # Disabled for performance
  become: False              # Don't escalate privileges

  tasks:
    - name: Main block
      block:
        # Provisioning tasks go here

      rescue:
        # Rollback/cleanup tasks go here

Header Explanation

name
Descriptive name shown in provisioning logs and UI. This appears as playbook_description in the provision record.

hosts: localhost
All OmniCRM playbooks run on localhost since they interact with remote systems via APIs, not SSH.

gather_facts: no
Ansible's fact gathering is disabled because:

We don't need system information
It adds unnecessary overhead
Can crash browsers if displayed in debug output

become: False
No privilege escalation is needed since we're making API calls, not modifying system files.

Configuration Loading

Every playbook must load the central configuration file:

tasks:
  - name: Include vars of crm_config
    ansible.builtin.include_vars:
      file: "../../crm_config.yaml"
      name: crm_config

This makes the configuration available as crm_config.ocs.cgrates, crm_config.crm.base_url, etc.

The crm_config.yaml typically contains:

ocs:
  cgrates: "10.0.1.100:2080"
  ocsTenant: "default_tenant"
crm:
  base_url: "https://crm.example.com"

Variable Access Patterns

Variables can come from several sources:

From the Product Definition:

- name: Access product_id passed by OmniCRM
  debug:
    msg: "Provisioning product {{ product_id }}"

From Inventory Selection:

- name: Get inventory ID for SIM Card
  set_fact:
    sim_card_id: "{{ hostvars[inventory_hostname]['SIM Card'] | int }}"
  when: "'SIM Card' in hostvars[inventory_hostname]"

From API Responses:

- name: Get Product information from CRM API
  uri:
    url: "http://localhost:5000/crm/product/product_id/{{ product_id }}"
    method: GET
    headers:
      Authorization: "Bearer {{ access_token }}"
    return_content: yes
  register: api_response_product

- name: Use the product name
  debug:
    msg: "Product name is {{ api_response_product.json.product_name }}"

Common Playbook Patterns

Service Provisioning Pattern

This is the most common pattern for creating new services.

- name: Service Provisioning Playbook
  hosts: localhost
  gather_facts: no
  become: False

  tasks:
    - name: Main block
      block:

      # 1. Load configuration
      - name: Include vars of crm_config
        ansible.builtin.include_vars:
          file: "../../crm_config.yaml"
          name: crm_config

      # 2. Get product information
      - name: Get Product information from CRM API
        uri:
          url: "http://localhost:5000/crm/product/product_id/{{ product_id }}"
          method: GET
          headers:
            Authorization: "Bearer {{ access_token }}"
          return_content: yes
          validate_certs: no
        register: api_response_product

      # 3. Get customer information
      - name: Get Customer information from CRM API
        uri:
          url: "http://localhost:5000/crm/customer/customer_id/{{ customer_id }}"
          method: GET
          headers:
            Authorization: "Bearer {{ access_token }}"
          return_content: yes
        register: api_response_customer

      # 4. Set facts from retrieved data
      - name: Set package facts
        set_fact:
          package_name: "{{ api_response_product.json.product_name }}"
          package_comment: "{{ api_response_product.json.comment }}"
          setup_cost: "{{ api_response_product.json.retail_setup_cost }}"
          monthly_cost: "{{ api_response_product.json.retail_cost }}"

      # 5. Generate unique identifiers
      - name: Generate UUID
        set_fact:
          uuid: "{{ 99999999 | random | to_uuid }}"

      - name: Generate Service UUID
        set_fact:
          service_uuid: "Service_{{ uuid[0:8] }}"

      # 6. Create account in billing system
      - name: Create account in OCS/CGRateS
        uri:
          url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
          method: POST
          body_format: json
          headers:
            Content-Type: "application/json"
          body:
            {
              "method": "ApierV2.SetAccount",
              "params": [{
                  "Tenant": "{{ crm_config.ocs.ocsTenant }}",
                  "Account": "{{ service_uuid }}",
                  "ActionPlanIds": [],
                  "ActionPlansOverwrite": true,
                  "ExtraOptions": {
                    "AllowNegative": false,
                    "Disabled": false
                  },
                  "ReloadScheduler": true
              }]
            }
          status_code: 200
        register: ocs_response

      - name: Verify OCS account creation
        assert:
          that:
            - ocs_response.status == 200
            - ocs_response.json.result == "OK"

      # 7. Add initial balance
      - name: Add 0 Monetary Balance
        uri:
          url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
          method: POST
          body_format: json
          body:
            {
              "method": "ApierV1.AddBalance",
              "params": [{
                  "Tenant": "{{ crm_config.ocs.ocsTenant }}",
                  "Account": "{{ service_uuid }}",
                  "BalanceType": "*monetary",
                  "Categories": "*any",
                  "Balance": {
                      "ID": "Initial Balance",
                      "Value": 0,
                      "ExpiryTime": "+4320h",
                      "Weight": 1,
                      "Blocker": true
                  }
              }]
            }
          status_code: 200
        register: balance_response

      # 8. Create service record in CRM
      - name: Get current date and time in ISO 8601 format
        command: date --utc +%Y-%m-%dT%H:%M:%S%z
        register: current_date_time

      - name: Add Service via API
        uri:
          url: "http://localhost:5000/crm/service/"
          method: PUT
          body_format: json
          headers:
            Content-Type: "application/json"
            Authorization: "Bearer {{ access_token }}"
          body:
            {
              "customer_id": "{{ customer_id }}",
              "product_id": "{{ product_id }}",
              "service_name": "{{ package_name }} - {{ service_uuid }}",
              "service_type": "generic",
              "service_uuid": "{{ service_uuid }}",
              "service_billed": true,
              "service_taxable": true,
              "service_provisioned_date": "{{ current_date_time.stdout }}",
              "service_status": "Active",
              "wholesale_cost": "{{ api_response_product.json.wholesale_cost | float }}",
              "retail_cost": "{{ monthly_cost | float }}"
            }
          status_code: 200
        register: service_creation_response

      # 9. Add setup cost transaction
      - name: Add Setup Cost Transaction via API
        uri:
          url: "http://localhost:5000/crm/transaction/"
          method: PUT
          headers:
            Content-Type: "application/json"
            Authorization: "Bearer {{ access_token }}"
          body_format: json
          body:
            {
              "customer_id": {{ customer_id | int }},
              "service_id": {{ service_creation_response.json.service_id | int }},
              "title": "{{ package_name }} - Setup Costs",
              "description": "Setup costs for {{ package_comment }}",
              "invoice_id": null,
              "retail_cost": "{{ setup_cost | float }}"
            }
          return_content: yes
        register: transaction_response

      # 10. Include post-provisioning tasks
      - include_tasks: post_provisioning_tasks.yaml

      rescue:

      # Rollback/cleanup section
      - name: Print all vars for debugging
        debug:
          var: hostvars[inventory_hostname]

      - name: Remove account in OCS
        uri:
          url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
          method: POST
          body_format: json
          body:
            {
              "method": "ApierV2.RemoveAccount",
              "params": [{
                  "Tenant": "{{ crm_config.ocs.ocsTenant }}",
                  "Account": "{{ service_uuid }}",
                  "ReloadScheduler": true
              }]
            }
          status_code: 200
        ignore_errors: True
        when: service_uuid is defined

      - name: Delete Service from CRM if it was created
        uri:
          url: "http://localhost:5000/crm/service/service_id/{{ service_creation_response.json.service_id }}"
          method: DELETE
          headers:
            Authorization: "Bearer {{ access_token }}"
          status_code: 200
        ignore_errors: True
        when: service_creation_response is defined

      - name: Fail if not intentional deprovision
        assert:
          that:
            - action == "deprovision"

Topup/Recharge Pattern

Used for adding credits, data, or time to existing services.

- name: Service Topup Playbook
  hosts: localhost
  gather_facts: no
  become: False

  tasks:
    - name: Include vars of crm_config
      ansible.builtin.include_vars:
        file: "../../crm_config.yaml"
        name: crm_config

    # 1. Get service information
    - name: Get Service information from CRM API
      uri:
        url: "http://localhost:5000/crm/service/service_id/{{ service_id }}"
        method: GET
        headers:
          Authorization: "Bearer {{ access_token }}"
        return_content: yes
      register: api_response_service

    # 2. Get product information (what to topup)
    - name: Get Product information from CRM API
      uri:
        url: "http://localhost:5000/crm/product/product_id/{{ product_id }}"
        method: GET
        headers:
          Authorization: "Bearer {{ access_token }}"
        return_content: yes
      register: api_response_product

    # 3. Extract service details
    - name: Set service facts
      set_fact:
        service_uuid: "{{ api_response_service.json.service_uuid }}"
        customer_id: "{{ api_response_service.json.customer_id }}"
        package_name: "{{ api_response_product.json.product_name }}"
        topup_value: "{{ api_response_product.json.retail_cost }}"

    # 4. Execute action in billing system (free topup)
    - name: Execute Action to add credits
      uri:
        url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
        method: POST
        body_format: json
        body:
          {
            "method": "APIerSv1.ExecuteAction",
            "params": [{
                "Tenant": "{{ crm_config.ocs.ocsTenant }}",
                "Account": "{{ service_uuid }}",
                "ActionsId": "Action_Topup_Standard"
            }]
          }
        status_code: 200
      register: action_response

    - name: Verify action executed successfully
      assert:
        that:
          - action_response.status == 200
          - action_response.json.result == "OK"

    # 5. Reset any triggered limits
    - name: Reset ActionTriggers
      uri:
        url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
        method: POST
        body_format: json
        body:
          {
            "method": "APIerSv1.ResetAccountActionTriggers",
            "params": [{
                "Tenant": "{{ crm_config.ocs.ocsTenant }}",
                "Account": "{{ service_uuid }}",
                "Executed": false
            }]
          }
        status_code: 200

    # 6. Update service dates
    - name: Calculate new expiry date
      command: "date --utc +%Y-%m-%dT%H:%M:%S%z -d '+30 days'"
      register: new_expiry_date

    - name: Update Service with new expiry
      uri:
        url: "http://localhost:5000/crm/service/{{ service_id }}"
        method: PATCH
        headers:
          Authorization: "Bearer {{ access_token }}"
          Content-Type: "application/json"
        body_format: json
        body:
          {
            "service_deactivate_date": "{{ new_expiry_date.stdout }}",
            "service_status": "Active"
          }

    # 7. Optional: Send notification
    - name: Send Notification SMS
      uri:
        url: "http://sms-gateway/api/send"
        method: POST
        body_format: json
        body:
          {
            "source": "CompanyName",
            "destination": "{{ customer_phone }}",
            "message": "Your service has been topped up. New expiry: {{ new_expiry_date.stdout }}"
          }
        status_code: 201
      ignore_errors: True

CPE Provisioning Pattern

Used for configuring customer premises equipment (routers, modems, ONTs).

- name: CPE Provisioning Playbook
  hosts: localhost
  gather_facts: no
  become: False

  tasks:
    - name: Include vars of crm_config
      ansible.builtin.include_vars:
        file: "../../crm_config.yaml"
        name: crm_config

    # 1. Get inventory item for CPE
    - name: Set CPE inventory ID from hostvars
      set_fact:
        cpe_inventory_id: "{{ hostvars[inventory_hostname]['WiFi Router CPE'] | int }}"
      when: "'WiFi Router CPE' in hostvars[inventory_hostname]"

    # 2. Get CPE details from inventory
    - name: Get Inventory data for CPE
      uri:
        url: "{{ crm_config.crm.base_url }}/crm/inventory/inventory_id/{{ cpe_inventory_id }}"
        method: GET
        headers:
          Authorization: "Bearer {{ access_token }}"
        return_content: yes
      register: api_response_cpe

    # 3. Get customer site information
    - name: Get Site info from API
      uri:
        url: "{{ crm_config.crm.base_url }}/crm/site/customer_id/{{ customer_id }}"
        method: GET
        headers:
          Authorization: "Bearer {{ access_token }}"
        return_content: yes
      register: api_response_site

    # 4. Update CPE inventory with location
    - name: Patch CPE inventory item with location
      uri:
        url: "{{ crm_config.crm.base_url }}/crm/inventory/inventory_id/{{ cpe_inventory_id }}"
        method: PATCH
        body_format: json
        headers:
          Authorization: "Bearer {{ access_token }}"
        body:
          {
            "address_line_1": "{{ api_response_site.json.0.address_line_1 }}",
            "city": "{{ api_response_site.json.0.city }}",
            "state": "{{ api_response_site.json.0.state }}",
            "latitude": "{{ api_response_site.json.0.latitude }}",
            "longitude": "{{ api_response_site.json.0.longitude }}"
          }
        status_code: 200

    # 5. Generate credentials
    - name: Set CPE hostname
      set_fact:
        cpe_hostname: "CPE_{{ cpe_inventory_id }}"
        cpe_username: "admin_{{ cpe_inventory_id }}"

    - name: Generate random password
      set_fact:
        cpe_password: "{{ lookup('pipe', 'cat /dev/urandom | tr -dc a-zA-Z0-9 | head -c 16') }}"

    # 6. Generate WiFi credentials
    - name: Set WiFi SSID
      set_fact:
        wifi_ssid: "Network_{{ cpe_inventory_id }}"

    - name: Generate WiFi password
      set_fact:
        word_list:
          - apple
          - cloud
          - river
          - mountain
          - ocean

    - name: Create WiFi PSK
      set_fact:
        random_word: "{{ word_list | random }}"
        random_number: "{{ 99999 | random(start=10000) }}"

    - name: Combine WiFi PSK
      set_fact:
        wifi_psk: "{{ random_word }}{{ random_number }}"

    # 7. Generate configuration file
    - name: Set config filename
      set_fact:
        config_name: "{{ cpe_hostname }}_{{ lookup('pipe', 'date +%Y%m%d%H%M%S') }}.cfg"
        config_dest: "/tmp/{{ cpe_hostname }}_{{ lookup('pipe', 'date +%Y%m%d%H%M%S') }}.cfg"

    - name: Create config from template
      template:
        src: "templates/cpe_router_config.j2"
        dest: "{{ config_dest }}"

    # 8. Read generated config
    - name: Read config file
      ansible.builtin.slurp:
        src: "{{ config_dest }}"
      register: config_content

    # 9. Update inventory with provisioning info
    - name: Patch CPE inventory with config
      uri:
        url: "{{ crm_config.crm.base_url }}/crm/inventory/inventory_id/{{ cpe_inventory_id }}"
        method: PATCH
        body_format: json
        headers:
          Authorization: "Bearer {{ access_token }}"
        body:
          {
            "itemtext3": "{{ wifi_ssid }}",
            "itemtext4": "{{ wifi_psk }}",
            "management_url": "{{ cpe_hostname }}",
            "management_username": "{{ cpe_username }}",
            "management_password": "{{ cpe_password }}",
            "config_content": "{{ config_content.content | b64decode }}",
            "inventory_notes": "Provisioned: {{ lookup('pipe', 'date +%Y-%m-%d') }}"
          }
        status_code: 200

    # 10. Send config to support team
    - name: Email configuration to support
      uri:
        url: "https://api.mailjet.com/v3.1/send"
        method: POST
        body_format: json
        headers:
          Content-Type: "application/json"
        body:
          {
            "Messages": [{
              "From": {
                "Email": "provisioning@example.com",
                "Name": "Provisioning System"
              },
              "To": [{
                "Email": "support@example.com",
                "Name": "Support Team"
              }],
              "Subject": "CPE Config - {{ cpe_hostname }}",
              "Attachments": [{
                "ContentType": "text/plain",
                "Filename": "{{ config_name }}",
                "Base64Content": "{{ config_content.content }}"
              }]
            }]
          }
        user: "{{ mailjet_api_key }}"
        password: "{{ mailjet_api_secret }}"
        force_basic_auth: true
        status_code: 200

Auto-Renewal Pattern

Configure automatic recurring charges or renewals using CGRateS ActionPlans.

# Part of a topup playbook that sets up auto-renewal

# 1. Normalize auto_renew parameter
- name: Normalize auto_renew to boolean
  set_fact:
    auto_renew_bool: "{{ (auto_renew | string | lower) in ['true', '1', 'yes'] }}"

# 2. Create action for auto-renewal
- name: Create Action for AutoRenew
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "ApierV1.SetActions",
        "params": [{
          "ActionsId": "Action_AutoTopup_{{ service_uuid }}_{{ product_id }}",
          "Overwrite": true,
          "Actions": [
            {
              "Identifier": "*http_post",
              "ExtraParameters": "{{ crm_config.crm.base_url }}/crm/provision/simple_provision_addon/service_id/{{ service_id }}/product_id/{{ product_id }}"
            },
            {
              "Identifier": "*cdrlog",
              "BalanceType": "*generic",
              "ExtraParameters": "{\"Category\":\"^activation\",\"Destination\":\"Auto Renewal\"}"
            }
          ]
        }]
      }
    status_code: 200
  register: action_response
  when: auto_renew_bool

# 3. Create monthly ActionPlan
- name: Create ActionPlan for Monthly Renewal
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "ApierV1.SetActionPlan",
        "params": [{
          "Id": "ActionPlan_Monthly_{{ service_uuid }}_{{ product_id }}",
          "Tenant": "{{ crm_config.ocs.ocsTenant }}",
          "ActionPlan": [{
            "ActionsId": "Action_AutoTopup_{{ service_uuid }}_{{ product_id }}",
            "Years": "*any",
            "Months": "*any",
            "MonthDays": "*any",
            "WeekDays": "*any",
            "Time": "*monthly",
            "StartTime": "*now",
            "Weight": 10
          }],
          "Overwrite": true,
          "ReloadScheduler": true
        }]
      }
    status_code: 200
  when: auto_renew_bool

# 4. Assign ActionPlan to account
- name: Assign ActionPlan to account
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "ApierV2.SetAccount",
        "params": [{
          "Tenant": "{{ crm_config.ocs.ocsTenant }}",
          "Account": "{{ service_uuid }}",
          "ActionPlanIds": ["ActionPlan_Monthly_{{ service_uuid }}_{{ product_id }}"],
          "ActionPlansOverwrite": true,
          "ReloadScheduler": true
        }]
      }
    status_code: 200
  when: auto_renew_bool

# 5. Remove ActionPlan if auto-renew disabled
- name: Remove ActionPlan from account
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "ApierV1.RemoveActionPlan",
        "params": [{
          "Tenant": "{{ crm_config.ocs.ocsTenant }}",
          "Id": "ActionPlan_Monthly_{{ service_uuid }}_{{ product_id }}"
        }]
      }
    status_code: 200
  ignore_errors: true
  when: not auto_renew_bool

Reusable Tasks

Reusable tasks are small, self-contained playbooks that can be included by multiple plays. They promote code reuse and consistency.

Welcome Email Task

task_welcome_email.yaml - Sends a welcome email to new customers.

# This task expects these variables to be set by the parent play:
# - api_response_customer (customer details)
# - package_name (product name)
# - monthly_cost (recurring cost)
# - setup_cost (one-time cost)

- name: Set email configuration
  set_fact:
    mailjet_api_key: "{{ lookup('env', 'MAILJET_API_KEY') }}"
    mailjet_api_secret: "{{ lookup('env', 'MAILJET_SECRET') }}"
    email_from: "noreply@example.com"
    recipients: []

- name: Set email subject and sender name
  set_fact:
    email_subject: "Welcome to our service!"
    email_from_name: "Customer Service Team"

- name: Prepare list of recipients from customer contacts
  loop: "{{ api_response_customer.json.contacts }}"
  set_fact:
    recipients: "{{ recipients + [{'Email': item.contact_email, 'Name': item.contact_firstname ~ ' ' ~ item.contact_lastname}] }}"

- name: Get first contact name
  set_fact:
    first_contact: "{{ api_response_customer.json.contacts[0].contact_firstname }}"

- name: Send welcome email
  uri:
    url: "https://api.mailjet.com/v3.1/send"
    method: POST
    body_format: json
    headers:
      Content-Type: "application/json"
    body:
      {
        "Messages": [{
          "From": {
            "Email": "{{ email_from }}",
            "Name": "{{ email_from_name }}"
          },
          "To": "{{ recipients }}",
          "Subject": "{{ email_subject }}",
          "TextPart": "Dear {{ first_contact }}, welcome! Your service is ready.",
          "HTMLPart": "Dear {{ first_contact }},<br/><h3>Welcome!</h3><br/>Your {{ package_name }} service is now active.<br/>Monthly cost: ${{ monthly_cost }}<br/>Setup fee: ${{ setup_cost }}<br/>If you have any questions, contact support@example.com"
        }]
      }
    user: "{{ mailjet_api_key }}"
    password: "{{ mailjet_api_secret }}"
    force_basic_auth: true
    status_code: 200
  register: email_response

Post Provisioning Tasks

post_provisioning_tasks.yaml - Standard cleanup and notifications run after every provision.

# This file is included at the end of most provisioning playbooks
# It handles common post-provisioning operations

- include_tasks: task_notify_ocs.yaml

The task_notify_ocs.yaml might contain:

- name: Notify OCS of provisioning completion
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "APIerSv1.ReloadCache",
        "params": [{
          "ArgsCache": "*all"
        }]
      }
    status_code: 200
  ignore_errors: true

Common Operations

Working with Inventory

Retrieving inventory details:

- name: Get SIM Card inventory ID
  set_fact:
    sim_inventory_id: "{{ hostvars[inventory_hostname]['SIM Card'] | int }}"
  when: "'SIM Card' in hostvars[inventory_hostname]"

- name: Get SIM Card details
  uri:
    url: "{{ crm_config.crm.base_url }}/crm/inventory/inventory_id/{{ sim_inventory_id }}"
    method: GET
    headers:
      Authorization: "Bearer {{ access_token }}"
    return_content: yes
  register: sim_response

- name: Extract SIM details
  set_fact:
    iccid: "{{ sim_response.json.iccid }}"
    imsi: "{{ sim_response.json.imsi }}"
    ki: "{{ sim_response.json.ki }}"

Assigning inventory to customer:

- name: Assign SIM to customer
  uri:
    url: "{{ crm_config.crm.base_url }}/crm/inventory/inventory_id/{{ sim_inventory_id }}"
    method: PATCH
    headers:
      Authorization: "Bearer {{ access_token }}"
    body_format: json
    body:
      {
        "customer_id": {{ customer_id }},
        "service_id": {{ service_id }},
        "item_state": "Assigned"
      }
    status_code: 200

Date and Time Operations

Getting current date/time:

- name: Get current date and time in ISO 8601 format
  command: date --utc +%Y-%m-%dT%H:%M:%S%z
  register: current_date_time

- name: Get today's date only
  set_fact:
    today: "{{ lookup('pipe', 'date +%Y-%m-%d') }}"

Calculating future dates:

- name: Calculate expiry date 30 days from now
  command: "date --utc +%Y-%m-%dT%H:%M:%S%z -d '+30 days'"
  register: expiry_date

- name: Calculate date 90 days in future
  command: "date --utc +%Y-%m-%d -d '+{{ days }} days'"
  register: future_date
  vars:
    days: 90

Generating Random Values

UUIDs and identifiers:

- name: Generate UUID
  set_fact:
    uuid: "{{ 99999999 | random | to_uuid }}"

- name: Generate service identifier
  set_fact:
    service_uuid: "SVC_{{ uuid[0:8] }}"

Random passwords:

- name: Generate secure password
  set_fact:
    password: "{{ lookup('pipe', 'cat /dev/urandom | tr -dc a-zA-Z0-9 | head -c 16') }}"

Memorable passphrases:

- name: Set word list
  set_fact:
    words:
      - alpha
      - bravo
      - charlie
      - delta
      - echo

- name: Generate passphrase
  set_fact:
    word: "{{ words | random }}"
    number: "{{ 99999 | random(start=10000) }}"

- name: Combine into passphrase
  set_fact:
    passphrase: "{{ word }}{{ number }}"

Working with CGRateS/OCS

Creating accounts:

- name: Create billing account
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "ApierV2.SetAccount",
        "params": [{
          "Tenant": "{{ crm_config.ocs.ocsTenant }}",
          "Account": "{{ service_uuid }}",
          "ActionPlanIds": [],
          "ActionPlansOverwrite": true,
          "ExtraOptions": {
            "AllowNegative": false,
            "Disabled": false
          },
          "ReloadScheduler": true
        }]
      }
    status_code: 200
  register: account_response

Adding balances:

- name: Add data balance
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "ApierV1.AddBalance",
        "params": [{
          "Tenant": "{{ crm_config.ocs.ocsTenant }}",
          "Account": "{{ service_uuid }}",
          "BalanceType": "*data",
          "Categories": "*any",
          "Balance": {
            "ID": "Data Package",
            "Value": 10737418240,
            "ExpiryTime": "+720h",
            "Weight": 10
          }
        }]
      }
    status_code: 200

Executing actions:

- name: Execute charging action
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "APIerSv1.ExecuteAction",
        "params": [{
          "Tenant": "{{ crm_config.ocs.ocsTenant }}",
          "Account": "{{ service_uuid }}",
          "ActionsId": "Action_Standard_Charge"
        }]
      }
    status_code: 200

Getting account information:

- name: Get account details
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "ApierV2.GetAccount",
        "params": [{
          "Tenant": "{{ crm_config.ocs.ocsTenant }}",
          "Account": "{{ service_uuid }}"
        }]
      }
    status_code: 200
  register: account_info

Working with Attribute Profiles:

- name: Get AttributeProfile
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "APIerSv1.GetAttributeProfile",
        "params": [{
          "Tenant": "{{ crm_config.ocs.ocsTenant }}",
          "ID": "ATTR_{{ service_uuid }}"
        }]
      }
    return_content: yes
    status_code: 200
  register: attr_response
  ignore_errors: true

- name: Extract attribute value
  set_fact:
    phone_number: "{{ attr_response.json.result.Attributes | json_query(\"[?Path=='*req.PhoneNumber'].Value[0].Rules\") | first }}"
  when: attr_response is defined

Conditional Logic

Checking if variables exist:

- name: Use custom value or default
  set_fact:
    monthly_cost: "{{ custom_cost | default(50.00) }}"

- name: Only run if variable is defined
  debug:
    msg: "Service UUID is {{ service_uuid }}"
  when: service_uuid is defined

Boolean conditions:

- name: Provision equipment
  include_tasks: configure_cpe.yaml
  when: provision_cpe | default(false) | bool

- name: Skip if deprovision
  assert:
    that:
      - action != "deprovision"
  when: action is defined

Multiple conditions:

- name: Complex conditional task
  uri:
    url: "{{ endpoint }}"
    method: POST
  when:
    - service_uuid is defined
    - customer_id is defined
    - action != "deprovision"
    - enable_feature | default(true) | bool

Loops and Iteration

Simple loops:

- name: Create multiple balances
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "ApierV1.AddBalance",
        "params": [{
          "Account": "{{ service_uuid }}",
          "BalanceType": "{{ item.type }}",
          "Balance": {
            "Value": "{{ item.value }}"
          }
        }]
      }
  loop:
    - { type: "*voice", value: 3600 }
    - { type: "*data", value: 10737418240 }
    - { type: "*sms", value: 100 }

Looping over API responses:

- name: Get all customer sites
  uri:
    url: "{{ crm_config.crm.base_url }}/crm/site/customer_id/{{ customer_id }}"
    method: GET
    headers:
      Authorization: "Bearer {{ access_token }}"
  register: sites_response

- name: Configure equipment at each site
  debug:
    msg: "Configuring site at {{ item.address_line_1 }}"
  loop: "{{ sites_response.json }}"

Error Handling

Using ignore_errors:

- name: Optional SMS notification
  uri:
    url: "http://sms-gateway/send"
    method: POST
    body: {...}
  ignore_errors: true

Assertions for validation:

- name: Verify API response
  assert:
    that:
      - response.status == 200
      - response.json.result == "OK"
    fail_msg: "API call failed: {{ response.json }}"

Conditional error handling:

- name: Try to get existing service
  uri:
    url: "{{ crm_config.crm.base_url }}/crm/service/service_uuid/{{ service_uuid }}"
    method: GET
    headers:
      Authorization: "Bearer {{ access_token }}"
  register: service_lookup
  failed_when: false

- name: Create service if it doesn't exist
  uri:
    url: "{{ crm_config.crm.base_url }}/crm/service/"
    method: PUT
    body: {...}
  when: service_lookup.status == 404

Best Practices

Variable Naming

Use descriptive, consistent names:

# Good
service_uuid: "SVC_12345"
customer_name: "John Smith"
monthly_cost: 49.99

# Bad
svc: "SVC_12345"
name: "John Smith"
cost: 49.99

Prefix variables by source:

api_response_customer: {...}
api_response_product: {...}
cgr_account_info: {...}

Debugging

Print variables for troubleshooting:

- name: Print all variables
  debug:
    var: hostvars[inventory_hostname]

- name: Print specific variable
  debug:
    msg: "Service UUID: {{ service_uuid }}"

- name: Print API response
  debug:
    var: api_response_product.json

Validation

Always validate critical API responses:

- name: Create account
  uri:
    url: "{{ billing_endpoint }}"
    method: POST
    body: {...}
  register: response

- name: Verify account creation
  assert:
    that:
      - response.status == 200
      - response.json.result == "OK"
    fail_msg: "Failed to create account: {{ response.json }}"

Idempotency

Design tasks to be safely re-runnable:

# Check if resource exists first
- name: Check if account exists
  uri:
    url: "{{ ocs_endpoint }}/get_account"
    method: POST
    body: {"Account": "{{ service_uuid }}"}
  register: account_check
  failed_when: false

# Only create if doesn't exist
- name: Create account
  uri:
    url: "{{ ocs_endpoint }}/create_account"
    method: POST
    body: {...}
  when: account_check.status == 404

Security

Never hardcode credentials:

# Bad
mailjet_api_key: "abc123def456"

# Good - use environment variables
mailjet_api_key: "{{ lookup('env', 'MAILJET_API_KEY') }}"

# Good - use config file
mailjet_api_key: "{{ crm_config.email.api_key }}"

Always use HTTPS and authentication:

- name: Call external API
  uri:
    url: "https://api.example.com/endpoint"
    method: POST
    headers:
      Authorization: "Bearer {{ access_token }}"
    validate_certs: yes

Documentation

Document complex logic:

# Calculate pro-rata charge for partial month
# If customer signs up on the 15th and billing is on 1st,
# charge 50% of monthly cost for remaining days
- name: Calculate days until end of month
  command: "date -d 'last day of this month' +%d"
  register: days_in_month

- name: Get current day
  command: "date +%d"
  register: current_day

- name: Calculate pro-rata amount
  set_fact:
    days_remaining: "{{ (days_in_month.stdout | int) - (current_day.stdout | int) }}"
    pro_rata_cost: "{{ (monthly_cost | float) * (days_remaining | float) / (days_in_month.stdout | float) }}"

Testing Playbooks

Testing Approach

Dry Run First: Test with non-production systems
Verify Variables: Use debug tasks to confirm all required variables are present
Check Responses: Validate API responses before proceeding
Rollback Testing: Intentionally fail tasks to verify rescue blocks work
Deprovision Testing: Test with action: "deprovision" to verify cleanup

Example test playbook:

- name: Test Service Provisioning
  hosts: localhost
  gather_facts: no

  tasks:
    - name: Verify required variables
      assert:
        that:
          - product_id is defined
          - customer_id is defined
          - access_token is defined
        fail_msg: "Missing required variables"

    - name: Test API connectivity
      uri:
        url: "http://localhost:5000/crm/health"
        method: GET
      register: health_check

    - name: Verify health check
      assert:
        that:
          - health_check.status == 200

Common Pitfalls

Missing type conversions:

# Wrong - may be string
customer_id: "{{ customer_id }}"

# Correct - ensure integer
customer_id: {{ customer_id | int }}

Not handling undefined variables:

# Wrong - fails if not defined
service_uuid: "{{ service_uuid }}"

# Correct - provide default
service_uuid: "{{ service_uuid | default('') }}"

Forgetting validation:

# Wrong - doesn't check response
- name: Create account
  uri: ...
  register: response

# Correct - validate response
- name: Create account
  uri: ...
  register: response

- name: Verify creation
  assert:
    that:
      - response.json.result == "OK"

Provisioning Workflow

Generally, Omnitouch staff will work with the customer to:

Define the product requirements
Develop the necessary Ansible playbooks to automate the provisioning process
Test the playbooks in a staging environment
Deploy to production

This ensures that each service is deployed consistently and reliably, reducing the risk of errors and ensuring that all necessary steps are completed in the correct order.

Ansible Variables

The variables passed to Ansible playbooks include:

Product Variables
Derived from the OmniCRM product configurations and define how the service should be set up.

Inventory Variables
Selected from the inventory, these include items such as modems, SIM cards, IP address blocks, or phone numbers that are required for provisioning.

System Variables
Automatically added by OmniCRM:

product_id - The product being provisioned
customer_id - The customer receiving the service
service_id - The service being modified (for topups/changes)
access_token - JWT for API authentication

Deprovisioning

When a service is no longer needed, the Ansible Playbooks are also used to deprovision the service using the rescue block pattern. This:

Removes any configurations
Releases inventory back to the pool
Deletes billing accounts
Ensures the system is kept clean

Deprovisioning Methods

There are two primary ways deprovisioning is triggered:

1. Automatic Rollback (Provisioning Failure) When any task in the main provisioning block fails, the rescue section automatically executes to clean up partial changes.

2. Manual Deprovisioning Setting action: "deprovision" when running a playbook triggers the rescue block intentionally to remove a service.

Deprovisioning Pattern

All OmniCRM playbooks follow this structure for safe cleanup:

- name: Service Provisioning Playbook
  hosts: localhost
  gather_facts: no
  become: False

  tasks:
    - name: Main block
      block:
        # All provisioning tasks go here
        - name: Create OCS account
          uri: ...

        - name: Create service record
          uri: ...
          register: service_creation_response

        - name: Add transaction
          uri: ...

      rescue:
        # Deprovisioning/rollback tasks go here
        - name: Print all vars for debugging
          debug:
            var: hostvars[inventory_hostname]

        # Cleanup tasks execute in reverse order
        - name: Remove account in OCS
          uri: ...

        - name: Delete Service from CRM
          uri: ...

        - name: Fail if not intentional deprovision
          assert:
            that:
              - action == "deprovision"

Complete Deprovisioning Example

Here's a complete example from play_simple_service.yaml:

rescue:
  # 1. Debug information
  - name: Print all vars for Deprovision/Rollback
    debug:
      var: hostvars[inventory_hostname]

  - name: Get today's date
    set_fact:
      today: "{{ lookup('pipe', 'date +%Y-%m-%d') }}"

  # 2. Get service information if available
  - name: Try to get Service information from CRM API to Deprovision
    uri:
      url: "http://localhost:5000/crm/service/service_id/{{ service_creation_response.json.service_id }}"
      method: GET
      headers:
        Authorization: "Bearer {{ access_token }}"
      return_content: yes
    register: api_response_service
    ignore_errors: True
    when: service_creation_response is defined and service_creation_response.json is defined and service_creation_response.json.service_id is defined

  - name: Print api_response_service
    debug:
      var: api_response_service
    when: api_response_service is defined

  # 3. Set service_uuid for cleanup
  - name: Set service_uuid facts for Deprovision
    set_fact:
      service_uuid: "{{ api_response_service.json.service_uuid }}"
    when:
      - service_uuid is not defined
      - api_response_service is defined
      - api_response_service.json is defined

  # 4. Remove OCS account
  - name: Remove account in OCS
    uri:
      url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
      method: POST
      body_format: json
      headers:
        Content-Type: "application/json"
        Authorization: "Bearer {{ access_token }}"
      body:
        {
          "method": "ApierV2.RemoveAccount",
          "params": [{
            "Tenant": "{{ crm_config.ocs.ocsTenant }}",
            "Account": "{{ service_uuid }}",
            "ReloadScheduler": true
          }]
        }
      status_code: 200
    register: response
    ignore_errors: True
    when: service_uuid is defined

  - name: Print response from OCS account removal
    debug:
      var: response
    when: response is defined

  # 5. Delete service record from CRM
  - name: Delete Service from CRM if it was created
    uri:
      url: "http://localhost:5000/crm/service/service_id/{{ service_creation_response.json.service_id }}"
      method: DELETE
      headers:
        Authorization: "Bearer {{ access_token }}"
      status_code: 200
    register: delete_service_response
    ignore_errors: True
    when: service_creation_response is defined and service_creation_response.json is defined and service_creation_response.json.service_id is defined

  - name: Print delete service response
    debug:
      var: delete_service_response
    when: delete_service_response is defined

  # 6. Determine if this was intentional or a failure
  - name: Set status to "Success" if Manual deprovision / Fail if failed provision
    assert:
      that:
        - action == "deprovision"
      fail_msg: "Provisioning failed and was rolled back"
      success_msg: "Service deprovisioned successfully"

Key Deprovisioning Concepts

ignore_errors: True Most cleanup tasks use ignore_errors: True because resources might not exist (e.g., if provisioning failed before creating them).

Conditional Execution Use when clauses to only clean up resources that were created:

when: service_creation_response is defined

Reverse Order Cleanup happens in reverse order of creation:

Delete dependent resources first (transactions, inventory assignments)
Delete service record
Delete OCS/billing account
Release any held resources

Final Assertion The final assert distinguishes between:

Intentional deprovision (action == "deprovision") → Success
Failed provisioning (action not set) → Failure

Deprovisioning Different Resource Types

OCS/CGRateS Accounts:

- name: Remove account in OCS
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "ApierV2.RemoveAccount",
        "params": [{
          "Tenant": "{{ crm_config.ocs.ocsTenant }}",
          "Account": "{{ service_uuid }}",
          "ReloadScheduler": true
        }]
      }
    status_code: 200
  ignore_errors: True
  when: service_uuid is defined

CRM Service Records:

- name: Delete Service from CRM
  uri:
    url: "http://localhost:5000/crm/service/service_id/{{ service_id }}"
    method: DELETE
    headers:
      Authorization: "Bearer {{ access_token }}"
    status_code: 200
  ignore_errors: True
  when: service_id is defined

Inventory Items (Return to Pool):

- name: Release SIM card back to inventory pool
  uri:
    url: "http://localhost:5000/crm/inventory/inventory_id/{{ sim_inventory_id }}"
    method: PATCH
    headers:
      Authorization: "Bearer {{ access_token }}"
    body_format: json
    body:
      {
        "customer_id": null,
        "service_id": null,
        "item_state": "Available"
      }
    status_code: 200
  ignore_errors: True
  when: sim_inventory_id is defined

ActionPlans (Recurring Charges):

- name: Remove ActionPlan from account
  uri:
    url: "http://{{ crm_config.ocs.cgrates }}/jsonrpc"
    method: POST
    body_format: json
    body:
      {
        "method": "ApierV1.RemoveActionPlan",
        "params": [{
          "Tenant": "{{ crm_config.ocs.ocsTenant }}",
          "Id": "ActionPlan_Monthly_{{ service_uuid }}_{{ product_id }}"
        }]
      }
    status_code: 200
  ignore_errors: True

Payment Authorizations:

- name: Release payment authorization
  uri:
    url: "http://localhost:5000/crm/payments/release/{{ authorization_id }}"
    method: POST
    headers:
      Authorization: "Bearer {{ access_token }}"
    body_format: json
    body:
      {
        "metadata": {
          "release_reason": "provisioning_failed"
        }
      }
    return_content: yes
  ignore_errors: True
  when: authorization_id is defined

Manual Deprovisioning Workflow

To manually deprovision a service:

Identify the service to deprovision
Run the original provisioning playbook with action: "deprovision"
Playbook enters rescue block immediately
All cleanup tasks execute
Service is removed cleanly

Example API call:

POST /crm/provision/run_playbook
{
  "product_id": 123,
  "customer_id": 456,
  "service_id": 789,
  "action": "deprovision"
}

Partial Cleanup Scenarios

Scenario 1: OCS Account Created, Service Creation Failed

OCS account exists
Service record does NOT exist
Rescue block removes OCS account
No service to delete (safely skipped with when condition)

Scenario 2: Service Created, Transaction Failed

OCS account exists
Service record exists
Transaction does NOT exist
Rescue block removes both OCS account and service
No transaction to void (never created)

Scenario 3: Complete Provisioning, Payment Capture Failed

OCS account exists
Service record exists
Payment authorized but NOT captured
Rescue block:
- Releases payment authorization (customer not charged)
- Removes service record
- Removes OCS account

Best Practices for Deprovisioning

1. Use ignore_errors liberally Cleanup should be forgiving - don't fail if a resource doesn't exist.

2. Check resource existence with when clauses Only attempt cleanup if the resource was created:

when: service_creation_response is defined

3. Print debug information Always include debug tasks to help troubleshoot failed provisions:

- name: Print all vars for debugging
  debug:
    var: hostvars[inventory_hostname]

4. Clean up in reverse dependency order Delete children before parents:

Transactions before services
Services before OCS accounts
Inventory assignments before inventory releases

5. Handle payment authorizations Always release payment authorizations in rescue blocks to avoid charging customers for failed provisions.

6. Reload schedulers after cleanup When removing OCS resources, include ReloadScheduler: true to ensure CGRateS updates immediately.

Deprovisioning vs Deletion

Deprovisioning (via playbook rescue):

Removes service from all systems
Releases inventory
Cancels recurring charges
Leaves audit trail
Recommended approach

Direct Deletion (via API):

Only deletes CRM record
Does NOT clean up OCS account
Does NOT release inventory
Can leave orphaned resources
Not recommended for production

Rollback and Error Handling

Ansible's block/rescue feature is employed during both provisioning and deprovisioning to handle errors gracefully. If a task fails at any point during provisioning, the rescue section automatically rolls back changes to return to a consistent state. This ensures reliability and reduces the risk of partial or failed deployments.

Error Handling Example

- name: Main block
  block:
    - name: Step 1: Create account
      uri: ...
      register: response

    - name: Verify step 1
      assert:
        that:
          - response.status == 200

    - name: Step 2: Create service
      uri: ...

  rescue:
    # If any assert fails or any task errors:
    # 1. Execution jumps to rescue block
    # 2. Cleanup tasks execute
    # 3. Playbook fails with error message
    - name: Cleanup partial changes
      uri: ...

For complete details on the provisioning system, workflows, and authentication, see concepts_provisioning.

How Playbooks and Products Work Together​

Products Trigger Playbooks​

What Playbooks Can Do​

Example: Different Playbook Behaviors​

Plays vs Tasks​

Playbook Structure and Anatomy​

Basic Structure​

Header Explanation​

Configuration Loading​

Variable Access Patterns​

Common Playbook Patterns​

Service Provisioning Pattern​

Topup/Recharge Pattern​

CPE Provisioning Pattern​

Auto-Renewal Pattern​

Reusable Tasks​

Welcome Email Task​

Post Provisioning Tasks​

Common Operations​

Working with Inventory​

Date and Time Operations​

Generating Random Values​

Working with CGRateS/OCS​

Conditional Logic​

Loops and Iteration​

Error Handling​

Best Practices​

Variable Naming​

Debugging​

Validation​

Idempotency​

Security​

Documentation​

Testing Playbooks​

Testing Approach​

Common Pitfalls​

Provisioning Workflow​

Ansible Variables​

Deprovisioning​

Deprovisioning Methods​

Deprovisioning Pattern​

Complete Deprovisioning Example​

Key Deprovisioning Concepts​

Deprovisioning Different Resource Types​

Manual Deprovisioning Workflow​

Partial Cleanup Scenarios​

Best Practices for Deprovisioning​

Deprovisioning vs Deletion​

Rollback and Error Handling​

Error Handling Example​