Real-Time Data

What Is Real-Time Data?

Real-time data refers to information that becomes available almost instantaneously after the underlying event occurs. In practice, “real-time” exists on a spectrum:

True real-time: Sub-second latency (milliseconds)

• Stock trading systems
• Fraud detection
• Industrial process control

Near real-time: Seconds to minutes

• Live dashboards
• Operational monitoring
• Customer-facing applications

Frequent batch: Minutes to hours

• Business reporting
• Financial analytics
• Most business intelligence

Real-Time vs. Batch Processing

Aspect	Real-Time	Batch
Latency	Seconds	Hours to days
Complexity	Higher	Lower
Cost	Higher	Lower
Use case	Operational	Analytical
Data volume	Event-by-event	Bulk processing

When Do You Actually Need Real-Time?

Real-Time Is Essential

• Fraud detection (stop bad transactions)
• Operational alerts (system down)
• Customer-facing features (order status)
• Trading and pricing decisions
• Safety-critical systems

Near Real-Time Is Sufficient

• Executive dashboards (hourly refresh fine)
• Sales performance monitoring
• Inventory tracking
• Customer support metrics

Batch Is Appropriate

• Financial reporting
• Month-end close
• Historical analysis
• Compliance reporting
• Most management reports

The Real-Time Trade-Off

Real-time capability comes with costs:

Infrastructure complexity: Streaming architectures are more complex than batch

Higher costs: Processing event-by-event is more expensive

Data quality challenges: Less time to validate and clean

Skills requirements: Specialized engineering expertise needed

Maintenance burden: More components to monitor and maintain

Real-Time Architecture Components

Event Sources

Systems generating real-time events:

• Transaction systems
• IoT sensors
• User activity
• Application logs

Message Queues

Buffering and routing events:

• Apache Kafka
• Amazon Kinesis
• Google Pub/Sub
• Azure Event Hubs

Stream Processing

Analyzing events in flight:

• Apache Flink
• Apache Spark Streaming
• Amazon Kinesis Analytics

Real-Time Storage

Databases optimized for current state:

• Redis
• Apache Cassandra
• Time-series databases

Delivery

Getting data to consumers:

• WebSockets
• Server-sent events
• Push notifications
• Real-time dashboards

Real-Time Data Quality

Real-time data presents quality challenges:

Late-arriving data: Events may arrive out of order

Incomplete data: Not all information available immediately

Duplicates: Same event may be delivered multiple times

Corrections: Initial data may need adjustment

Best practice: Combine real-time operational view with batch-validated analytical view.

How Go Fig Handles Data Freshness

Go Fig provides appropriate freshness for finance use cases:

Configurable refresh: Choose hourly, daily, or on-demand updates

Near real-time dashboards: Key metrics update throughout the day

Scheduled workflows: Run pipelines on your required schedule

Event triggers: Start processing when new data arrives

Historical accuracy: Batch processing ensures data quality

For most financial analytics, near real-time (hourly or more frequent) provides the right balance of freshness and data quality.

Questions to Ask About Real-Time

Before investing in real-time capabilities:

1. What decision requires this speed? Can you act on data in seconds?

2. What’s the cost of delay? Is hourly data really a problem?

3. Is the data quality sufficient? Can you trust unvalidated data?

4. Do you have the skills? Can your team maintain streaming systems?

5. Is the ROI there? Do benefits justify the complexity?

Often, improving data accessibility with hourly updates delivers 90% of the value at 10% of the real-time cost.