. Making Big Data management easier with Qlik Sense ODAG.

In the world of Big Data, where data sets are growing at an enormous rate, you need effective Big Data Management tools to deal with the challenges of processing huge amounts of information.

To address these challenges, Qlik Sense introduced the On-Demand App Generation (ODAG) functionality in the September 2017 version. Since then, the number of data generated every year has almost quadrupled. In 2023 we expect to reach 120 zettabytes of data! How can we analyze such huge data sets effectively with the tools available? In the article, I examine the capabilities of Qlik Sense, one of the most powerful Big Data technologies.

1. The Importance of Big Data Management
2. Big Data Management tools: Qlik Sense ODAG
3. Big Data Management best practices – the checklist for BI developers
4. What is ODAG in data analytics?
5. ODAG – the key components of a Big Data management tool
6. How to create On-Demand Applications (ODAG)
7. Use Case
8. Managing Big Data with ODAG
9. What is the Qlik Associative Big Data Index?
10. How does the Qlik Associative Big Data Index work?
11. Summary

The Importance of Big Data Management

The world today is dominated by Big Data, huge amounts of data generated in different sectors such as e-commerce, healthcare, science, social media, and more. The challenge most companies and data scientists face is to turn the raw and unstructured data into valuable information and analyze it as effectively as possible. But what data are we talking about? To help picture it, I have listed some interesting facts below.

Approximately 328.77 million terabytes of data are created every day
Approximately 120 zettabytes of data will be generated in 2023
In 2025, 181 zettabytes of data will be generated
Videos make up more than half of the data processed in the world

Big Data Management tools: Qlik Sense ODAG

In every industry, we deal with many types of data that we can call Big Data, which can be analyzed with tools such as Qlik Sense. There are plenty of examples of Big Data sets; for example, the aforementioned social media data (posts, photos, data on user preferences, behaviors or relationships), or data generated in e-commerce (purchase data, reviews, searches or customer behavior on the website).

Big Data Management best practices – the checklist for BI developers

Here are the steps you should consider following to maximize the potential of Qlik Sense when it comes to analyzing such data. The checklist below will be helpful for any BI developer, but I think it will also help business representatives plan their work (e.g. to better familiarize themselves with the process):

Make sure you have collected all the data you need from different data sources.
Take advantage of the connectors offered by tools for Big Data platforms such as Hadoop or Spark.
Design your Qlik Sense applications with ODAG in mind (an On-Demand App Generator, which I will discuss in detail later in the article). This includes identifying key dimensions that users wish to analyze and creating appropriate mechanisms for loading data on demand.
Think about what tables and fields are really needed. Avoid complex data models that can slow down performance (try to create a model called a “star” or “snowflake”). Make sure that the fact tables are as simple as possible, and transfer detailed information to the dimension tables.
Monitor performance and adjust the architecture for increasing data volume. This may include additional server resources, optimized data models, or application design best practices.
Ensure proper QS configuration in terms of safety. Define the necessary permission levels to ensure the protection of valuable and sensitive Big Data collections.
Take advantage of additional extensions available in the Qlik ecosystem. You can use those built by the community, or more professional extensions, such as the Vizlib library. They offer additional features or integrations to help you get the most out of the platform.
Technology and business needs are constantly changing. Update your Qlik Sense apps regularly, taking new features, data, and user requirements into account.
Don’t forget training. Users should know how to use the tool, how to interpret visualizations, and how to use ODAG features.

Also read: Business Intelligence outsourcing services

What is ODAG in data analytics?

What do we do when we are working with huge data sets that are too large to be entirely loaded into memory – for example, if we want to analyze only one day of sales at a hypermarket ? The ODAG functionality mentioned in the introduction comes in handy.

ODAG is one of the Big Data management solutions available on the market. It is a technique that allows you to dynamically create a Qlik application based on user choices. Thanks to ODAG, only those fragments of data that are needed at a given moment can be loaded and analyzed. So instead of loading the entire database, the user can choose given criteria or filters that will determine the piece of data that is of interest. Then, based on these choices, Qlik Sense generates a new application containing only the data that meets the selected criteria.

ODAG – the key components of a Big Data management tool

ODAG is based on three main components:

Parent App: the main Qlik application that contains all the available data or, most often, a representative subset of data. Here the user makes initial choices, specifying what data will be needed in the generated application.
On-Demand Filters: a collection of settings that determines what data is to be loaded into a new application, based on user choices.
On-Demand App: an application generated dynamically based on user choices in the parent application and filters. It contains only the data that has been specified by the user.

Elevate Your Data Strategy

Our customized Data solutions align with your business objectives. Consult with Marek Czachorowski, Head of Data and AI Solutions, for expert guidance.

Schedule a meeting

How to create On-Demand Applications (ODAG)

Creating an ODAG application is easy – it requires a developer to follow only a few steps.

Creation of a Parent App / Selection App. This is an application that usually contains filters to narrow down the data and aggregated values from the main data set.

2. Creating a Template App, which will be complemented with data after applying the selection from the parent application. This application should include the visualizations that users expect to have.

3. In the parent application, an On-Demand application link should be created. Its role is to generate an application for the user based on the Template App and data narrowed down by selections from the parent application.

4. The last step is to generate the application with the “Generate a new application” button.

Use Case

Thanks to the use of ODAG applications, users can focus on specific pieces of huge data sets instead of searching the entire database. Below I present 4 examples of the use of ODAG in Qlik Sense in different industries.

Retail Sector and Sales Analysis

A retail company has long-term sales data from all its stores around the world. Instead of analyzing the entire database, managers want to focus on data from a specific region and period of time. Using ODAG, the manager selects the region and date range he is interested in. Then, based on these criteria, the Qlik Sense application is generated, presenting only the relevant data.

Medical analysis and clinical trials

A hospital needs to analyze patient data to identify patterns in certain diseases. Physicians can use ODAG to generate reports on a specific diseases, based on selected criteria (e.g., patient age, gender, location). The application will be generated with only the data of patients meeting these criteria, which will facilitate the analysis and identification of patterns.

Market analysis and financial data

An investment company wants to analyze market data from the last 10 years, but instead of searching the entire database, an analyst wants to focus only on specific segments. Using ODAG, the analyst can select specific market segments and time range. As in the above examples, this will allow you to generate an application containing only those data that are relevant to an accurate analysis.

Transport Sectors and Logistics Analysis

A logistics company wants to optimize its delivery routes based on data from past years. They have hundreds of thousands of records for different routes, drivers, and road conditions. Thanks to ODAG, the planner can focus on selected routes, drivers or time ranges, and the generated Qlik Sense application will provide accurate data for analysis and optimization.

Managing Big Data with ODAG

On-Demand App Generation is the next step in the evolution of data analytics tools. As you can see, this solution allows for a more efficient and flexible use of resources in the world of Big Data. Its most important benefits include:

• Flexibility and scalability

Instead of trying to load huge amounts of data into one application, ODAG allows users to focus on the subset that is most relevant to them at given time. Thanks to this, Qlik Sense works quickly and efficiently even with huge data sets.

• Optimizing resources

Thanks to the on-demand approach, there is no need to constantly process and update all available data. Instead, data is processed only when the user needs it.

• Interactivity

Users can easily define what data they want to analyze, making Big Data analytics more interactive and directed .

• Better cost management

Big Data processing can be costly, especially when using cloud solutions based on the consumption of resources. With the on-demand approach, it is easier to control costs because you only pay for processing the data that is actually used.

• Improved performance

Thanks to the option to focus on specific data instead of the entire set, loading and analysis time is shorter.

DATA ANALYTICS SERVICES

Use data to your advantage

Discover our Data Management End-to-End offer! Find out more!

What is the Qlik Associative Big Data Index?

You may be wondering which technologies make it possible to work with huge data sets in Qlik Sense and how to further speed up analysis with the use of ODAG. One such feature is the Qlik Associative Big Data Index (QABDI). This is a technology introduced by Qlik to facilitate fast and interactive search and analysis of large data sets. QABDI is a solution aimed at Big Data environments, such as Hadoop or various data warehouse platforms. It allows users to analyze data without having to load it into memory, which is crucial when working with huge amounts of data.

How does the Qlik Associative Big Data Index work?

QABDI creates an associative index from huge data sets in source Big Data systems . This index is similar to the index in traditional databases, but it is adapted to the associative Qlik model, which allows you to search and analyze data faster.

When the user makes a selection or creates a query in the Qlik application, the system searches the QABDI index, instead of employing traditional data loading and analyzing methods. Thanks to this, responses to queries are delivered faster, even for very large data sets.

Remember that users can still use the Qlik interface in the traditional way (make selections, search data and create visualizations). However, thanks to QABDI, these operations are carried out directly on large data sets, without the need for prior processing or aggregation.

For more complex analyses where detailed data is needed, Qlik can automatically move from using QABDI to directly requesting detailed data in the source Big Data system . QABDI is also designed with scalability in mind. Thanks to this, as the amount of data grows, you can easily adjust the infrastructure and resources to ensure good performance at a continual level.

How to address Big Data Management challenges

ODAG or QABDI are powerful tools in the Qlik Sense arsenal that give users dynamic access to huge data sets. This makes it possible to conduct effective analysis and draw conclusions without the need to burden the system with unnecessary data, allowing analysts and managers to make informed decisions based on accurate and up-to-date information. In a world where data is no longer measured in terabytes but in zettabytes, this approach will help you save time and money.

Consult your project directly with a specialist

Book a meeting

1. The Importance of Big Data Management
2. Big Data Management tools: Qlik Sense ODAG
3. Big Data Management best practices – the checklist for BI developers
4. What is ODAG in data analytics?
5. ODAG - the key components of a Big Data management tool
6. How to create On-Demand Applications (ODAG)
7. Use Case
8. Managing Big Data with ODAG
9. What is the Qlik Associative Big Data Index?
10. How does the Qlik Associative Big Data Index work?
11. Summary