Table of Contents
What are Big Objects?
In the field of data management, the term “Big Object” holds significant importance both universally and in specific platforms like Salesforce. In general, a Big Object refers to an advanced data structure specifically designed to store and manage extremely large datasets efficiently. These structures are essential in dealing with the complexities of big data, enabling scalable storage, improved query performance, and robust data analysis capabilities across various technology ecosystems.
However, within the Salesforce environment, a “Big Object” is defined more narrowly. Salesforce Big Objects are customized solutions tailored to archive and analyze historical data on a massive scale within the Salesforce platform. These are not typical database tables; they are structured to provide high-volume storage solutions that maintain performance without affecting the operational database. They focus on long-term data retention and accessibility for analytical purposes.
The Primary Types of Big Objects
There are two primary types of Big Objects: standard and custom. Standard Big Objects are pre-configured within various platforms to cater to general use cases. These objects are designed to integrate seamlessly into any system, providing robust solutions for common data challenges without the need for additional customization. They are particularly useful for businesses looking for out-of-the-box functionality that supports standard data storage, analysis, and retrieval processes.
On the other hand, Custom Big Objects offer a tailored approach, allowing organizations to design and structure data solutions that are precisely aligned with their specific business needs. This customization enables companies to handle unique data types or perform complex, specialized data operations that are not supported by standard configurations.
Both types of Big Objects have common use cases, including large-scale data archiving where long-term data retention is critical, comprehensive analytics on historical data to drive business decisions, and regulatory compliance where vast amounts of data need to be securely maintained and accessed. Standard Big Objects are often employed in industries with common data practices, such as healthcare for patient records. Custom Big Objects are favored in fields like digital marketing, where data needs are continually evolving and highly specific.
Salesforce Has Three Ways to Store Data On-Platform
Salesforce has three ways to store data on the platform: Standard Objects, Custom Objects, and Big Objects. Most CRM data is stored in Salesforce standard and custom objects. These types of data are frequently accessed for day-to-day customer transactions. Big Objects provide an alternative location for data storage, particularly for data with millions, even billions of records. To fully understand the benefits of Big Objects, we must first define Standard and Custom objects to highlight the critical differences between them, particularly their impact on data storage.
Salesforce Standard & Custom Objects
Standard Objects are the built-in structures provided by Salesforce. They are designed to handle everyday business operations such as managing user data, sales leads, and customer interactions. Standard objects include predefined entities like Accounts, Contacts, Leads, and Opportunities. These objects are integral to Salesforce and are tailored to cover common CRM functionalities across various industries.
Salesforce custom objects expand data model functionality by allowing you to create new database tables, aka objects. These are created each time a managed package from the Salesforce AppExchange is installed into an org or by a Salesforce administrator. Custom Objects allow organizations to store information unique to their processes. For instance, a university could create a custom object to manage student records, courses, or faculty interactions, which are not available as standard objects.
Salesforce’s standard and custom objects are not designed to handle massive data volumes. When data volumes get high, a Salesforce org will see performance issues, particularly in reporting. This is in addition to increased data storage costs! Any object containing over 1 million records can generally cause performance degradation, especially in combination with many custom fields. When data volumes on Salesforce get high, there are two options:
- Archive data off of the Salesforce platform
- Move data to a Big Object
Big Objects
Salesforce custom and standard objects are designed to capture transactional data, such as a customer’s interactions with a business’s sales and support team. This data is updated frequently, allowing a single record to connect to hundreds or thousands of related records. Conversely, Big Objects are created to aggregate and maintain massive data sets, and the data contained in Big Objects is rarely edited. Instead, Big Objects tend to grow as additional records are added to offset data storage costs and improve overall organization performance.
FieldHistoryArchive is the most common Big Object, and Salesforce automatically creates this object to contain record field history.
Below is a comparative table that outlines the key characteristics and differences between these three types of Salesforce objects:
Why You Should Use a Big Object
There are two common use cases that drive adoption of Salesforce Big Objects:
Store Big Data for Historical Analytics
Big Objects store billions of records, all in a way where the data is accessible natively inside Salesforce. This lets users create reports on this data without requiring additional off-platform storage or integrations. As a result, Big Objects can help a business analyze customer trends over a long period.
Key Features of Salesforce Big Object Storage:
- Scalability: Salesforce Big Objects can scale to store billions of records. This scalability is vital for organizations that accumulate large amounts of data over time and need a sustainable way to manage it.
- Performance: By using Big Objects, businesses can ensure that the performance of their everyday Salesforce applications remains unaffected, even as they handle large datasets. Big Objects are optimized for high-volume data storage and query operations on large datasets.
- Archiving: One of the primary uses of Big Objects is for data archiving. Organizations can archive old transactional data that must be retained for regulatory compliance or historical analysis. This offloads the data from the main CRM environment, helping to maintain system performance and reduce storage costs.
- Data Accessibility: Despite being designed for large-scale data, Big Objects are integrated within Salesforce, making the data accessible through Salesforce’s native tools. This allows for seamless querying and reporting, even on large historical datasets.
Salesforce Big Object storage provides a specialized, highly efficient means of storing and managing large volumes of data within the Salesforce platform. This capability is essential for companies that need to maintain large historical datasets for compliance, analytical, or operational needs, enabling them to leverage their data effectively without sacrificing performance.
Reduction of Data Costs
The per-record storage rate for a Big Object record is significantly lower than the per-record cost for a Salesforce organization exceeding its data storage limit. When this article was published, the general Salesforce pricing guide listed BigObjects as a $16,800 yearly investment or .0003 US cents per record. Alternatively, while not publicly published, Salesforce’s data storage rate is rumored to be $125 per month for 500 MB of extra data storage. Let’s do some simple math:
Big Object Cost:
.0003 cents per record
Salesforce Standard or Custom Object Cost:
.006 cents per record
- 500 MB = 500,000 KB
- Salesforce data records are counted as 2 KB
- 500,000 KB in an extra block of storage means that each storage block could contain a maximum of 250,000 records.
Annual cost per data storage block: $1,500 to contain 250,000 records
= .006 cents per record
With such a large cost discrepancy, it should be no surprise that Salesforce BigObjects are rapidly being adopted to support both data storage and data cost use cases!
BigObject Technical Consideration: Index
Unlike Salesforce’s custom objects, you determine what fields should be used as an index on Big Object creation. This is perhaps the most critical technical consideration as it determines the future use of the Big Object. In addition, the index determines how the data stored in the object is filtered and queried.
An index is set upon Big Object creation, and it can not change later.
Consider the implications of this statement. First, a Big Object can contain billions of records. An index that does not promote efficient data filtering or searching could negate the value of using this technology.
Example: A bad index
This index uses an Account lookup as the 1st custom index. Here’s why this could cause a big problem in the future:
There are three records in my BigObject today:
- Initech
- Massive Dynamic
- Umbrella Corporation
When a new record is created –– for this example, Stark Industries –– the record will be inserted into the index between Massive Dynamic and Umbrella Corporation. As the first index field is also a lookup, we could also have records that share the same first index field! This means that we have no efficient way to search the index to see what records are new or what records have changed!
Example: A good index
This index is designed for queries with our first indexed field based on the record creation data, creating an iterator. For example, we can then query records created since a specific date. The second part of our index is a unique case number, which functions as a unique identifier even if we end up with two records created on the same date and time. Finally, the Account lookup and phone number are captured much later in our index.
A good Big Object index must contain two core components:
- An iterator, like a timestamp
- A uniqueness indicator, like an ID
CapStorm Supports Full Backups of Salesforce Big Objects
When Big Objects are used, the data stored within them may need to be backed up to ensure the organization has a comprehensive Salesforce business continuity plan. At the time this article was published, Salesforce’s Backup Solution did not support the backup of Big Objects. FieldHistoryArchive, the most common Big Object, was not supported by Ownbackup (Knowledge Base: Salesforce Objects Included and Excluded in Backups), Grax, Spanning, and others.
CapStorm supports the replication of FieldHistoryArchive and other Big Objects. The CopyStorm application supports incremental replication from Salesforce Big Objects for those that have well-constructed indexes. This means that added or modified records are replicated during each backup, and the Big Object backups can run at a high frequency. For Big Objects that lack a uniqueness indicator and an iterator, CapStorm supports a full extract of these records.
Want to learn more? Check out a short demo video below:
For even more information on CapStorm’s Salesforce backup solutions, talk to one of our experts today!
