Skip to main content

Monitoring & Metrics

OmniCRM API provides comprehensive Prometheus-based metrics for monitoring application performance, business operations, and external integrations. All metrics are exposed at the /crm/metrics endpoint in Prometheus exposition format.

Overview​

The metrics system tracks:

  • Provisioning Operations - Job execution, duration, success/failure rates
  • Database Performance - Query times, connection pool health
  • External Integrations - OCS/CGRateS, Stripe, Mailjet API calls
  • Background Jobs - Async task execution and performance
  • HTTP Requests - API endpoint usage and response times (auto-generated)

Metrics Endpoint​

URL: http://your-omnicrm-api:5000/crm/metrics

Format: Prometheus exposition format

Authentication: The metrics endpoint is publicly accessible for scraping by Prometheus. In production, it's recommended to restrict access using firewall rules or reverse proxy authentication.

Metric Categories​

1. Provisioning Metrics​

Provisioning metrics track the execution of Ansible playbooks that provision services, manage inventory, and configure external systems. See Provisioning System and Ansible Playbooks for more details.

omnicrm_provision_jobs_total​

Type: Counter

Labels:

  • status - Job completion status: success, failed, running

Description: Total number of provision jobs created. Incremented when provision jobs complete with their final status.

omnicrm_provision_job_duration_seconds​

Type: Histogram

Labels:

  • playbook - Name of the Ansible playbook executed
  • status - Job completion status: success, failed

Buckets: [1, 5, 10, 30, 60, 120, 180, 300, 600] seconds (1s to 10min)

Description: Time taken for provision jobs to complete. Records the duration of entire playbook execution from start to finish.

omnicrm_provision_jobs_active​

Type: Gauge

Description: Number of currently running provision jobs. Incremented when a job starts, decremented when it completes.

omnicrm_provision_tasks_total​

Type: Counter

Labels:

  • playbook - Name of the Ansible playbook
  • status - Task result: ok, failed, ignored

Description: Total number of Ansible tasks executed within playbooks. Incremented for each individual task that completes (success or failure).

omnicrm_provision_errors_total​

Type: Counter

Labels:

  • error_type - Type of error: fatal, task_failed, timeout
  • playbook - Name of the Ansible playbook

Description: Total number of provision errors by type. Incremented when provisioning tasks fail or fatal errors occur during execution.

2. Database Metrics​

Database metrics monitor query performance and connection pool health. OmniCRM uses SQLAlchemy with automatic instrumentation. See System Architecture for data model details.

omnicrm_db_query_duration_seconds​

Type: Histogram

Labels:

  • operation - SQL operation type: SELECT, INSERT, UPDATE, DELETE
  • table - Database table name

Buckets: [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0] seconds

Description: Database query execution time. Automatically tracked via SQLAlchemy event listeners on every query.

omnicrm_db_pool_size​

Type: Gauge

Description: Current database connection pool size. The total number of connections in the pool (both checked out and available).

omnicrm_db_pool_overflow​

Type: Gauge

Description: Current database connection pool overflow. The number of connections created beyond the normal pool size limit.

omnicrm_db_pool_connections_checked_out​

Type: Gauge

Description: Number of connections currently checked out from the pool and in use by application code.

omnicrm_db_errors_total​

Type: Counter

Labels:

  • error_type - Type of database error
  • operation - Operation that caused the error: connection_error, query_error, etc.

Description: Total number of database errors.

Status: Defined but not actively used in current code (reserved for future use)

3. OCS/CGRateS Metrics​

OCS (Online Charging System) metrics track interactions with the CGRateS billing engine. See Billing Overview and CGRateS Integration for more details.

omnicrm_ocs_api_calls_total​

Type: Counter

Labels:

  • method - OCS API method: GetBalance, SetBalance, SetAccount, RemoveAccount, etc.
  • status - Call result: success, failed

Description: Total number of OCS/CGRateS API calls. Incremented for every async API call to the OCS system.

omnicrm_ocs_api_duration_seconds​

Type: Histogram

Labels:

  • method - OCS API method: GetBalance, SetBalance, GetAccount, etc.

Buckets: [0.1, 0.25, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0] seconds

Description: OCS/CGRateS API call duration. Records the time taken for each API call including network latency.

omnicrm_ocs_api_errors_total​

Type: Counter

Labels:

  • method - OCS API method that failed
  • error_type - Error category: timeout, connection_error, json_error, http_error, etc.

Description: Total number of OCS/CGRateS API errors. Incremented when OCS API calls fail with specific error types.

omnicrm_ocs_balance_queries_total​

Type: Counter

Labels:

  • query_type - Type of balance query: single_account, multiple_accounts

Description: Total number of balance queries to OCS. Used for tracking usage of balance inquiry operations.

omnicrm_ocs_action_plan_operations_total​

Type: Counter

Labels:

  • operation - Action plan operation: create, remove, query

Description: Total number of action plan operations. Tracks creation, removal, and querying of CGRateS action plans for recurring charges.

4. Stripe Payment Metrics​

Stripe metrics track payment processing operations. See Payment System Guide and Payment Methods for integration details.

omnicrm_stripe_api_calls_total​

Type: Counter

Labels:

  • operation - Stripe operation: create_customer, charge, refund, update_payment_method, etc.
  • status - Operation result: success, failed

Description: Total number of Stripe API calls. Incremented for every payment processing operation.

omnicrm_stripe_api_duration_seconds​

Type: Histogram

Labels:

  • operation - Stripe operation type

Buckets: [0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0] seconds

Description: Stripe API call duration including network latency.

omnicrm_stripe_api_errors_total​

Type: Counter

Labels:

  • operation - Stripe operation that failed
  • error_type - Error category: card_declined, network_error, api_error, etc.

Description: Total number of Stripe API errors. Incremented when payment operations fail.

omnicrm_stripe_payment_amount_cents​

Type: Counter

Labels:

  • payment_type - Payment direction: charge, refund

Description: Total payment amount processed through Stripe in cents. Useful for tracking transaction volume and revenue.

omnicrm_stripe_payment_failures_total​

Type: Counter

Labels:

  • reason - Failure reason: card_declined, insufficient_funds, expired_card, etc.

Description: Total number of Stripe payment failures categorized by decline code.

5. Mailjet Email Metrics​

Mailjet metrics track email delivery operations. See Mailjet Integration for configuration details.

omnicrm_mailjet_api_calls_total​

Type: Counter

Labels:

  • email_type - Email template type: welcome, user_welcome, invoice, notification
  • status - Delivery result: success, failed

Description: Total number of Mailjet API calls. Tracked via the @track_mailjet_call decorator.

omnicrm_mailjet_api_duration_seconds​

Type: Histogram

Labels:

  • email_type - Email template type

Buckets: [0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0] seconds

Description: Mailjet API call duration. Time taken to submit email via Mailjet API (not delivery time).

omnicrm_mailjet_api_errors_total​

Type: Counter

Labels:

  • email_type - Email template type
  • error_type - Error category

Description: Total number of Mailjet API errors. Incremented when email sends fail.

omnicrm_mailjet_emails_sent_total​

Type: Counter

Labels:

  • email_type - Email template type
  • template_id - Mailjet template ID

Description: Total number of emails successfully sent via Mailjet. Distinct from API calls as one call can send to multiple recipients.

omnicrm_mailjet_email_recipients_total​

Type: Counter

Labels:

  • email_type - Email template type

Description: Total number of email recipients across all sent emails.

6. Background Job Metrics​

Background job metrics track async operations like playbook chains and scheduled tasks. See Provisioning System for background job details.

omnicrm_background_jobs_total​

Type: Counter

Labels:

  • job_type - Job category: playbook_single, playbook_chain, periodic_task

Description: Total number of background jobs started. Tracked via the BackgroundJobTimer context manager.

omnicrm_background_jobs_active​

Type: Gauge

Labels:

  • job_type - Job category

Description: Number of currently running background jobs. Incremented at job start, decremented at job completion.

omnicrm_background_job_duration_seconds​

Type: Histogram

Labels:

  • job_type - Job category
  • status - Job result: success, failed

Buckets: [1, 5, 10, 30, 60, 120, 180, 300, 600, 1800, 3600] seconds (1s to 1hr)

Description: Background job execution time. Includes the full duration of multi-step operations.

omnicrm_background_job_errors_total​

Type: Counter

Labels:

  • job_type - Job category
  • error_type - Error category

Description: Total number of background job errors. Incremented when background jobs fail with exceptions.

7. Flask HTTP Metrics (Auto-generated)​

These metrics are automatically generated by the prometheus-flask-exporter library and track all HTTP requests to the API.

flask_http_request_duration_seconds​

Type: Histogram

Labels:

  • method - HTTP method: GET, POST, PUT, DELETE, etc.
  • endpoint - Flask route name
  • status - HTTP status code

Description: HTTP request duration for all API endpoints. Automatically instrumented.

flask_http_request_total​

Type: Counter

Labels:

  • method - HTTP method
  • endpoint - Flask route name
  • status - HTTP status code

Description: Total HTTP requests by endpoint, method, and status code.

flask_http_request_exceptions_total​

Type: Counter

Labels:

  • method - HTTP method
  • endpoint - Flask route name

Description: Total unhandled exceptions in HTTP requests. Indicates bugs or unexpected errors.

8. API Error Metrics (Reserved)​

These metrics are defined but not currently instrumented. They are reserved for future use.

omnicrm_api_errors_total​

Type: Counter

Labels:

  • endpoint - API endpoint
  • error_type - Error category
  • status_code - HTTP status code

Status: Defined but not actively used

omnicrm_api_auth_failures_total​

Type: Counter

Labels:

  • auth_method - Authentication method: jwt, api_key, ip_whitelist
  • reason - Failure reason

Status: Defined but not actively used

9. Application Info Metric​

omnicrm_api​

Type: Info

Labels:

  • version - API version string
  • environment - Environment name: production, staging, development

Description: OmniCRM API information metric. Set once at application startup with version and environment information.

Periodic Updates​

update_db_pool_metrics(engine)

Automatically called every 30 seconds to update database connection pool gauges.

Prometheus Configuration​

Scrape Configuration​

Add OmniCRM to your prometheus.yml:

scrape_configs:
  - job_name: 'omnicrm-api'
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
      - targets: ['omnicrm-api:5000']
    metrics_path: '/crm/metrics'

Example Alerts​

High Provision Failure Rate

- alert: HighProvisionFailureRate
  expr: |
    (
      rate(omnicrm_provision_jobs_total{status="failed"}[5m]) /
      rate(omnicrm_provision_jobs_total[5m])
    ) > 0.1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High provision job failure rate"
    description: "{{ $value | humanizePercentage }} of provision jobs are failing"

Database Connection Pool Exhausted

- alert: DatabasePoolExhausted
  expr: omnicrm_db_pool_overflow > 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Database connection pool overflow detected"
    description: "Connection pool is using overflow connections, may indicate pool size is too small"

Slow Database Queries

- alert: SlowDatabaseQueries
  expr: |
    histogram_quantile(0.99,
      rate(omnicrm_db_query_duration_seconds_bucket[5m])
    ) > 1.0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Slow database queries detected"
    description: "99th percentile query time is {{ $value }}s"

OCS API Down

- alert: OCSAPIDown
  expr: |
    (
      rate(omnicrm_ocs_api_calls_total{status="failed"}[5m]) /
      rate(omnicrm_ocs_api_calls_total[5m])
    ) > 0.5
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "OCS API failure rate critical"
    description: "{{ $value | humanizePercentage }} of OCS API calls are failing"

Stripe Payment Issues

- alert: StripePaymentFailures
  expr: rate(omnicrm_stripe_payment_failures_total[5m]) > 5
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Elevated Stripe payment failures"
    description: "{{ $value }} payment failures per second"

Best Practices​

Monitoring Strategy​

  1. Set Up Core Alerts - Configure alerts for critical metrics:

    • Provision failure rate > 10%
    • Database connection pool exhaustion
    • OCS/CGRateS API failures
    • Stripe payment processing errors

  2. Track Business Metrics - Monitor operational KPIs:

    • Revenue processed via Stripe
    • Provisioning throughput
    • Customer email delivery rates

  3. Performance Monitoring - Watch for degradation:

    • API response time percentiles
    • Database query performance
    • External API latency

  4. Capacity Planning - Use metrics to forecast:

    • Database connection pool sizing
    • Background job worker scaling
    • API server capacity

Metric Retention​

Prometheus Retention Recommendations:

  • 15 days - High-resolution metrics (15s scrape interval)
  • 90 days - Downsampled metrics (5m aggregation)
  • 1 year - Long-term aggregated metrics (1h aggregation)

Use Thanos, Cortex, or VictoriaMetrics for long-term storage and global query.

Troubleshooting​

Metrics Not Appearing​

By default the /metrics endpoint is only exposed to internal (non public address space) sources. You can change this in your nginx config if required.

Check the metrics endpoint:

curl http://localhost:5000/crm/metrics

Verify Prometheus can scrape:

# Check Prometheus targets page
http://prometheus:9090/targets

Missing Specific Metrics​

Some metrics are only created after their first use:

  • Provisioning metrics appear after first provision job
  • Stripe metrics appear after first payment
  • OCS metrics appear after first billing operation

High Cardinality Issues​

If Prometheus is slow or consuming excessive memory, check for high-cardinality labels:

# Count unique time series per metric
count by (__name__) ({__name__=~".+"})

Metrics with >10,000 series may indicate a cardinality problem.

Metrics Summary​

Total Metrics: 34 (31 custom + 3 auto-generated Flask metrics)

By Type:

  • Counters: 20
  • Gauges: 5
  • Histograms: 8
  • Info: 1

By Category:

  • Provisioning: 5 metrics
  • Database: 5 metrics
  • OCS/CGRateS: 5 metrics
  • Stripe: 5 metrics
  • Mailjet: 5 metrics
  • Background Jobs: 4 metrics
  • HTTP/Flask: 3 metrics
  • Application Info: 1 metric
  • Reserved (future use): 2 metrics
Last updated on