Architecture is the advantage
Nominal Core's architecture equips engineers to handle their high-rate, high-volume data at real-world speed
Nominal Core is our flagship data platform that helps engineering teams ingest, manage, monitor, and analyze telemetry, logs, videos, and procedures in one place.
We win through our architecture. Every hardware engineering team faces overwhelming data complexity. Nominal has created a production-grade, multi-modal data warehouse for the extreme ingress and analytics requirements of high-rate, high-volume hardware test data. For years, the warehouse underpinning Nominal Core has been used in production by developers of advanced vehicles (land, air, sea, and space), nuclear reactors, and manufacturing systems. Below we detail some of the learnings and decisions from building, deploying, and supporting Nominal Core’s warehouse in mission-critical environments over the past several years.
1. Our architectural philosophy
Making critical decisions about complex physical systems requires a data platform that can keep pace with the speed and complexity of the real world. That platform must be built on a foundation of trust, performance, and scale. This document outlines the architecture of our platform, designed from the ground up to meet the demanding needs of enterprise engineering teams. We believe that transparency into our design choices will provide you with the confidence that our solution is not only powerful enough for today’s challenges, but is also the scalable, long-term partner for your most important data.
We believe that a robust telemetry analysis platform must excel in three core areas:
Reliability: Your data is invaluable and often irreplaceable. Think: years and millions of dollars invested before a first flight. Once data is ingested into our platform, it is protected against loss, corruption, and downtime, making it consistently available for your analysis.
Performance: Fast insights are critical. We prioritize low latency from ingestion to query, even across massive datasets, to ensure real-time visibility. Our architecture recognizes the importance of mission-critical streaming use cases, as well as aggregations of historical and simulation datasets.
Scalability: Your data volumes will inevitably grow. Our architecture is designed to scale seamlessly and cost-effectively, ensuring that performance never degrades as your usage increases, from your first project to an enterprise-wide deployment. We know that Scalability requires Efficiency.
2. Architecture overview
Our platform's strength lies in its multi-layered architecture, where each component is specialized for a specific task. Decoupled components ensure that functions like ingestion, processing, querying, and storage operate and scale independently. This prevents the resource contention common in monolithic systems and ensures that, for example, a heavy data merging operation will not impact the performance of your real-time queries. Independent scaling is crucial for organizations spinning up and down large test operations, where massive concurrent load is table-stakes, but is not expected between tests.
The following diagram provides a high-level view of our platform's architecture, illustrating the journey of data from initial ingestion to actionable insight
3. Resilient and flexible ingestion for complex hardware data
Whether your data is generated as a continuous real-time stream of telemetry, unstructured log data, or as discrete files from completed tests, our ingestion pipeline can handle it gracefully. Hardware data is fundamentally different from standard business intelligence, or even software observability data. Our ingestion system is engineered to handle the high frequency data, extreme cardinality, and out-of-order writes common to physical world systems. Finally, complex or even specialized data formats are common for our customers, and our platform natively handles standard formats while offering pluggable support for bespoke formats.
Two common paradigms we support are:
Streaming ingestion: For live data, all incoming telemetry and logs are first written to a resilient message bus. This acts as a protective buffer for streaming sources, guaranteeing data durability by isolating your systems from our backend. If any downstream component is slow or unresponsive, the bus holds the data safely, handling backpressure and preventing data loss. This design is ideal for telemetry workloads that emit frequent updates. The bus absorbs these high-frequency writes, and our processing layer intelligently batches them for optimal storage efficiency.
File-based ingestion & broad format support: For historical or batch data, we support direct file uploads. We eliminate brittle, time-consuming data transformation work by natively ingesting the formats your hardware produces, including CSV, TDMS, Chapter 10, ROS bag files, and more. Furthermore, you can handle any proprietary file format by providing your custom logic as a containerized extractor. This allows you to bring your own code directly to our platform, applying it automatically during ingestion. Our orchestration infrastructure manages the scaling and execution of your containers, ensuring your custom logic runs securely without you needing to manage the underlying infrastructure.
4. Contextual data modeling: unifying telemetry, logs, video, and more
Raw streams of telemetry and disconnected log files are difficult to interpret without context. Our platform's most powerful feature is its ability to unify all your raw engineering data – high-frequency telemetry, unstructured logs, and video—as well as crucial derived data and metadata, into a single, intuitive data model.
This is accomplished by organizing all incoming data into concepts that everyone can understand:
Assets: An "Asset" represents a specific physical entity you are testing or monitoring, such as a vehicle, a specific component like a valve or sensor, or a test bench.
Runs: A "Run" represents a specific sequence of events for an asset, such as a single test, a flight, or a simulation.
Events: An “Event” represents a specific moment or duration of significance. Events can be manually created by an engineer to mark a real-world observation (e.g., "audible click from actuator") or programmatically defined to automatically capture critical phases of a test, like "altitude between 0.1 and 40 feet" or a fault condition.
By automatically associating every data type with an Asset and a Run, our platform can readily provide context to users. This makes it possible not only to issue simple queries against your data ("show me all logs for vehicle 123") but to ask deeper questions (”show me the log messages and synchronized video feed from the exact moment this sensor anomaly occurred”).
Additionally, you can annotate raw channels with their corresponding units (e.g. volts, amps, meters/second). When you perform calculations, this unit information automatically propagates through the compute graph. Crucially, the system also validates operations, preventing or warning against invalid calculations – like adding volts to amps – which eliminates a common source of error and increases confidence in analytical outcomes.
5. High-performance analytics datastore
The analytics datastore at the center of the architecture is where raw data is transformed into queryable insights at incredible speed. Its performance stems from intelligent data handling, from initial write to on-disk storage and query execution.
Log-structured merge (LSM) tree for ingestion: High-frequency telemetry involves millions of rapid, frequent writes. Our engine handles this using an LSM-based approach where new data is written to a fast, in-memory structure before being efficiently merged into larger, optimized files on disk. This strategy avoids the "small file problem" common in other systems, where writing individual data batches as separate files creates massive fragmentation and metadata overhead that would cripple query performance.
Optimized data layout for compression and fast filtering: Data is stored by columns, not rows. For analytical queries that examine a few parameters over time, this means a query only reads the columns it needs, dramatically minimizing I/O. Furthermore, data is physically sorted on disk (by series, then timestamp), so time-range queries on a specific series read from a single contiguous block of disk, avoiding slow seeks. Unlike general-purpose formats, our engine employs a layered compression strategy ideal for time series. It first applies specialized codecs (like Delta for timestamps) to transform data into a more compressible representation, then uses a high-performance general algorithm, ZSTD. This layered approach typically yields a 10-50x reduction in storage size, significantly lowering costs and query latency. It works in concert with sparse primary indexes that allow the engine to quickly identify and skip over huge, irrelevant blocks of data, ensuring that queries process the minimal amount of data.
Horizontally scalable for high concurrency: We’ve all seen it: you test your data infrastructure repeatedly, but leading up to the critical event, the telemetry datastore cannot keep up with control room monitors at full data rate, or your dashboards become sluggish as every engineer tries to load the same chart at the same time. This can lead to painful operational delays. We understand that critical tests require scalable concurrency.
Vectorized query execution: Instead of processing data row-by-row, our engine uses vectorized execution, processing data in large batches (vectors). This technique leverages modern CPU architecture to its fullest potential, leading to massive improvements in computational throughput.
Arbitrary precision: We support high-resolution timestamps, ensuring you can precisely correlate events from high-frequency sensors without losing fidelity. Our platform allows for chosen precision of nanoseconds or picoseconds to enable storage optimization. We have learned from our customers – some of the most innovative teams in the world – that sub-picosecond precision would be wasteful. Our architecture supports further precision, when it is needed.
Embedded python analysis: Run custom Python scripts directly where your data lives. This eliminates the need for slow and expensive data copying between environments, dramatically speeding up your test validation and debugging.
This combination of architectural choices ensures that our platform can ingest tens of millions of events per second while simultaneously delivering query responses in milliseconds, even on petabyte-scale datasets.
6. Intelligent tiered storage for performance and cost control
To balance extreme performance with cost effectiveness at petabyte scale, our platform employs a managed, multi-tiered storage model.
In-memory cache: For real-time dashboards and alerting on live data, we use a high speed in-memory cache.
Hot storage (SSDs): Your "working set"– the most recent and frequently accessed data – resides on high performance Solid State Drives (SSDs) for instant query access.
Cold storage (object storage): As data ages and is accessed less frequently, it is seamlessly moved to highly durable, low-cost cloud object storage. When this data is queried, it is automatically cached on SSDs for faster subsequent access.
This model means your costs scale with the size of your active working set, not the total volume of data you store. This is further enhanced by advanced data compression, which significantly reduces the storage footprint across all tiers, directly lowering your total cost of ownership. Finally, our modular architecture allows for these tiers to be scaled independently, allowing different organizations to customize their caching spend.
7. Advanced and intuitive analysis for every engineer
Getting answers from your data should not require a degree in data science. Our platform empowers every engineer with intuitive, no-code analytical tools. This allows you to go beyond historical analysis and interact with your systems as they operate.
Crucially, this analysis can be performed on live, event-driven streams. Because we don't naively poll for new information, the time from event generation to actionable insight is measured in milliseconds, and the computational workload is significantly lower. For example, you can:
Perform live anomaly detection: Define a rule that alerts you if a sensor's value deviates from its 10-second rolling average by more than three standard deviations.
Monitor real-time system health: Calculate the live fuel efficiency of a vehicle during a test run by streaming fuel flow and velocity data simultaneously.
Automate actions: Configure alerts that call a webhook to trigger external workflows, such as initiating a safety shutdown procedure, creating a ticket in a project management system, or paging a responsible engineer.
8. Designed for enterprise scale and reliability
Our entire platform is designed to scale dynamically with your needs, managed entirely by us to eliminate operational complexity.
Automated scaling (Autoscaling): The platform continuously monitors key performance metrics for ingestion rates, query load, and processing queues. It automatically scales resources for each architectural layer – ingestion, processing, caching, and querying – up or down as needed. This ensures you always have the performance you require without over-provisioning, optimizing both performance and cost.
Fully managed reliability: Foundational tasks like data replication, sharding, and failover are automated and managed by our service. This abstracts away the immense operational complexity of running a distributed data system at scale, protecting against hot-spotting or data loss, and letting your team focus on engineering, not database administration.
9. Flexible deployment: Your cloud, your hardware
Our platform is engineered to run wherever your data is generated, from public clouds and on-premise hardware to fully air-gapped environments. This level of flexibility is achieved through foundational architectural decisions to embrace containerization and Kubernetes-native design. Every component of our stack is packaged into portable containers, and we leverage Kubernetes as a universal orchestration layer. This allows us to provide a consistent, reliable experience on any infrastructure and integrate with critical technologies purpose-built to run off-cloud.
We deliver our full stack as a set of standard Helm charts, and a Kubernetes Operator automates the entire lifecycle of the platform, from installation and configuration to upgrades and scaling. We support hybrid and multi-deployment models, and customers have installed our entire stack end-to-end by themselves. This is not a feature that can be duct-taped onto an architecture after the fact. We have designed for this since inception.
10. Future-ready: Your data, your AI
The future of engineering analysis involves leveraging Large Language Models (LLMs) as well as other AI techniques. Our platform is designed to be the central source of truth that feeds these next-generation tools, ensuring you can innovate freely without hidden costs.
Comprehensive open APIs: Your data is yours. We provide robust, open APIs that allow you to programmatically access all your data, enabling seamless integration with custom applications, third-party tools, and advanced analytical workflows. To further accelerate development, we also provide and support an open-source Python client, giving your teams a powerful and familiar way to interact with the platform.
Broad ecosystem compatibility: By using open standards for data accessibility, we ensure you can connect your data to any AI or ML tool you choose without facing punitive egress fees. Our platform becomes a flexible component in your broader data strategy, not a locked-in silo.
This approach gives you the power to build the exact solutions your team needs while retaining full control and ownership of your data.
11. Conclusion: Hardware engineers should build capability, not data pipelines
Most internal telemetry stacks begin as simple ideas – a Kafka cluster, Grafana dashboards – but quickly spiral into endless firefighting. Parsers break after vehicle software updates, dashboards stall during critical tests, and databases topple over during critical investigations.
Our architecture is the result of deliberate choices to solve the unique challenges of hardware data. We unify telemetry, logs, and video into a contextual data model, while our decoupled, auto-scaling infrastructure and high-performance analytics engine provide the flexibility, speed, and scale modern enterprises require. We’ve made the right calls upfront so your engineers can be free to build, not manage data pipelines.