Observability via Consolidated Tracing Facility (CTF) & OpenTelemetry
In complex rehosted environments where multi-step batch jobs (JCL) and online transaction processors (CICS/IMS) execute concurrently across multi-process Service Execution Processes (SEPs), isolating performance anomalies, race conditions, or intermittent database timeouts requires granular diagnostic visibility without attaching invasive debuggers or recompiling application code. [1]
Read the full briefing
1. Capability Overview
In complex rehosted environments where multi-step batch jobs (JCL) and online transaction processors (CICS/IMS) execute concurrently across multi-process Service Execution Processes (SEPs), isolating performance anomalies, race conditions, or intermittent database timeouts requires granular diagnostic visibility without attaching invasive debuggers or recompiling application code.
Rocket Enterprise Server 12.0 delivers a two-tiered observability framework:
- Consolidated Tracing Facility (CTF): An embedded diagnostic tracing infrastructure designed for low-overhead, component-level capture of runtime internals (COBOL Run-Time System, XA switch modules, JCL initiators, CICS/IMS engines, and communication listeners) (Consolidated Tracing Facility - Rocket Software Documentation).
- OpenTelemetry Integration (12.0): Native support for modern distributed tracing and metric collection, enabling teams to pipe runtime telemetry directly into enterprise Application Performance Monitoring (APM) platforms such as Jaeger, Prometheus, Dynatrace, Datadog, or Grafana via standard OpenTelemetry Protocol (OTLP) pipelines (Using OpenTelemetry - Rocket Software Documentation).
2. Architecture & How It Works
- Declarative Configuration & CTF Emitters: CTF behavior is governed by a declarative configuration file (typically
ctf.cfg) specified via theMFTRACE_CONFIGenvironment variable (To Enable CTF Tracing - Rocket Software Documentation).- Pluggable Emitters: Trace records are processed by modular emitters, primarily
binfile(high-performance binary output),textfile(human-readable text), orjavalogger(Emitter List - Rocket Software Documentation). Thebinfileemitter records events in an indexed, compact binary format that imposes minimal latency on active transaction threads and can be formatted offline. - Process-Isolated Logging: Because Enterprise Server operates across independent SEPs, Rocket documentation strongly recommends parameterizing trace file destinations with
$(PID)(e.g.,location = /var/log/ctf-$(PID).log) to prevent concurrent process file locks and write collisions (mfotl.exporter.#).
- Pluggable Emitters: Trace records are processed by modular emitters, primarily
- Surgical Component-Level Filtering: Rather than logging all events indiscriminately, CTF permits granular filtering by subsystem. For example:
mftrace.level.mf.rts = infocaptures runtime system lifecycle events.mftrace.comp.mf.rts#pgmload = trueaudits dynamic COBOL program load and resolution events (To Enable CTF Tracing - Rocket Software Documentation).mftrace.level.mf.esxa = infointercepts XA database switch modules, outputting exactSQLCODE,SQLSTATE, andxa_*function parameters during two-phase commit processing (RM Switch Module Diagnostics - Rocket Software Documentation).
- Modern Distributed Tracing with OpenTelemetry: Enterprise Server 12.0 adds the
mfotl.*configuration family to propagate trace context across network boundaries (Configuring OpenTelemetry - Rocket Software Documentation). It supportsotlp_httpexporters with configurable retry policies, exponential backoff, and gzip compression to ship spans and server performance metrics out to standard collectors without custom application hooks (mfotl.exporter.#).
3. Practical Configuration & Operational Tidbits
- Enabling CTF in Region Properties: In ESCWA (Enterprise Server Common Web Administration), go to
[Region] > General > Properties > Environment Variablesand define (To Enable CTF Tracing - Rocket Software Documentation): All child processes and SEPs instantiated by that region automatically inherit the tracing configuration upon startup.
MFTRACE_CONFIG=/etc/microfocus/ctf.cfg
MFTRACE_LOGS=/var/logs/es_traces- Troubleshooting XA Resource Failures: When an XA database connection or two-phase commit fails intermittently, create a minimal
ctf.cfgtargeting themf.esxacomponent. The resulting log files (cassi.textfile.<pid>.log) capture the underlying vendor client error codes without requiring database re-configuration (RM Switch Module Diagnostics - Rocket Software Documentation). - Production Tracing Guardrails: Avoid enabling global textfile emitters or verbose
debuglevels across all components in production environments. Instead, use thebinfileemitter paired with circular file size limits, or reserve fine-grained tracing for specific target components to prevent log directory exhaustion and preserve transaction throughput (Diagnostics in Enterprise Server - Rocket Software Documentation).
4. Reference Citations (Rocket Enterprise Server 12.0 Manuals)
- Consolidated Tracing Facility - Rocket Software Documentation — Enterprise Server 12.0 for UNIX
- Using OpenTelemetry - Rocket Software Documentation — Enterprise Server 12.0 for UNIX
- Configuring OpenTelemetry - Rocket Software Documentation — Enterprise Server 12.0 for UNIX
- To Enable CTF Tracing - Rocket Software Documentation — Enterprise Server 12.0 for UNIX
- RM Switch Module Diagnostics - Rocket Software Documentation — Enterprise Server 12.0 for UNIX
- Emitter List - Rocket Software Documentation — Enterprise Server 12.0 for UNIX
- mfotl.exporter.# — Enterprise Server 12.0 for UNIX
- Diagnostics in Enterprise Server - Rocket Software Documentation — Enterprise Server 12.0 for UNIX