That was the starting point for Inetum’s webinar SAP Business Data Cloud & Databricks: Unlock AI & Data potential, which brought together practitioners from Inetum and special guests from SAP and Rolls-Royce. The session featured Jan Tretina (Ecosystem Development Manager, SAP), Sebastian Stefanowski (Databricks Practice Leader, Inetum), Raul Muñoz-Gutierrez (SAP Analytics Business Director, Inetum), and Andrew Lager (Program Manager and Digital Delivery Manager, Civil Digital and IT, Rolls-Royce), hosted by Oleh Hudym (SAP Growth Manager Manager, Inetum).

Less than 40% of leaders trust their own data – and that gap stalls AI

“Less than 40% of leaders report high confidence in their data,” Jan Tretina said at the opening of the session. “That gap directly impacts both decision speed and innovation.”

Three issues sit behind this figure. First, data quality: over half of organizations struggle to keep data accurate and consistent across systems. Second, misalignment between IT and business – finance teams need insights quickly, but IT landscapes can’t always deliver at that pace. Third, fragmentation: data spread across multiple systems, bringing it together in real time remains a major pain point.

The business consequence is concrete. Data-oriented organizations – those that have solved the trust problem – are, according to SAP’s analysis, four times more likely to succeed.

What SAP Business Data Cloud is – and how it closes the trust gap

Jan Tretina described SAP Business Data Cloud through a flywheel: AI is only as strong as the data behind it, data is only valuable when it is trusted and accessible, and both require a resilient platform underneath.

“SAP Business Data Cloud is our unified data foundation,” he said. “It brings clean, connected, and trusted business data together – unifying data across SAP and non-SAP systems and making that data immediately usable for AI, analytics, and planning.”

BDC consolidates SAP data services – SAP BW, SAP Datasphere, SAP Analytics Cloud, and extension partners including Databricks and Snowflake – under a single platform. The operational impact Tretina highlighted: instead of maintaining thousands of pipelines, custom integrations, and shadow data, BDC provides one consistent data foundation across all use cases. It embraces non-SAP data, open standards, and a broad partner ecosystem, letting organizations build multi-vendor landscapes without losing control of their data. This architecture makes BDC relevant across sectors – from retailers to manufacturing to financial services and banking.

Migration: BDC works with what organizations already have

A recurring concern when evaluating any new platform is the cost of transition – years of investment in SAP BW, Datasphere, or Analytics Cloud environments that can’t simply be discarded.

Raul Muñoz-Gutierrez addressed this directly. BDC includes all SAP data services within a single platform and is designed to absorb existing environments, not replace them. For SAP BW, BDC offers a “lift and shift” migration path – the existing environment moves in with data and connections preserved.

For SAP Datasphere or SAP Analytics Cloud, a “rewiring” process brings those solutions under the BDC umbrella without rebuilding. “All these tasks don’t require any work from the customers,” Muñoz-Gutierrez confirmed. “They can be carried out directly by SAP.”

Data products eliminate the 80% of a project that adds no value

“Do you know the effort and time required in a data project to perform data extraction, loading and transformation, as well as reconciling all the information?” Raul Muñoz-Gutierrez asked during the session. “All that work, that normally is the 80% of the project, is partially eliminated through a SAP data product.”

SAP data products contain the main data from key functional areas – financial, HR, supply chain – and they inherit all the data semantics from SAP. This makes them usable not just for reporting, but for AI and machine learning development from day one.

For CFOs specifically, the Financial Intelligence Package activates intelligent applications on SAP Analytics Cloud while simultaneously triggering data loading from SAP S/4HANA into BDC data products. “Regularly every 30 minutes, the data is going to travel from our SAP S/4 system to our BDC,” Muñoz-Gutierrez explained. Treasury management, financial planning, and forecasting become available and AI-ready from the moment of activation – with the option to build custom AI and machine learning on top.

SAP Databricks and standalone Databricks: the same engine, different purpose

If Databricks is already in the organization’s stack – or under evaluation – a natural question arises: how does SAP Databricks within BDC relate to the standard Databricks platform?

Sebastian Stefanowski, whose team works with both, explained the distinction. SAP Databricks is a modified release of the Databricks platform, tightly integrated with BDC – authentication, authorization, and billing are all managed through SAP, using SAP compute units. The most important aspect of the integration, in Stefanowski’s words, is a dedicated connector enabling zero-copy data sharing between SAP data products and the Databricks catalog: SAP financial data becomes directly queryable in Databricks without replication or transformation overhead.

“If SAP data is at the heart of your business, and financial data is really central to what you do, then SAP Databricks is definitely worth serious consideration,” Stefanowski said. “It would enable most of the useful features of Databricks with, generally speaking, no technological barrier.”

The trade-off is feature breadth. Organizations with IoT streaming requirements, multi-stage declarative pipeline orchestration (DLT), or a need to manage their own compute clusters will find standalone Databricks richer – with dedicated connectors for external cloud services and platforms like Salesforce. In SAP Databricks, those broader integrations run through SAP. Stefanowski’s summary was clear: for general data integration scenarios, standalone Databricks; for organizations where SAP financial data is the core, SAP Databricks removes the barriers.

“What seemed to be impossible a few years back in terms of AI capabilities is now possible with this amazing partnership between SAP and Databricks,” Oleh Hudym noted during the session.

Six years with Databricks at Rolls-Royce: 50% cost reduction and AI for every analyst

Andrew Lager has been on the Databricks journey at Rolls-Royce for over six years, including as an early adopter of many of its latest technologies – and he came to the webinar with numbers.

“We’ve reduced costs upwards of 50% compared to our previous solution,” he said, referring to the migration from a legacy warehouse to Databricks Lakehouse for engine health monitoring data. Three factors drove that reduction: cheaper compute via Spark, schema evolution that prevents jobs from breaking when table structures change during migrations, and Unity Catalog – which centralizes data access across sources so Lager can produce management information for internal stakeholders without involving additional teams.

On the AI side, his example was equally direct. “Those activities that would take me days can take me five minutes now,” he said, describing Databricks AI Genie – a feature that lets non-technical users query datasets in natural language instead of SQL.

“The biggest benefit, not just for me but for Rolls-Royce in general, has been opening up technical solutions to less technical people.”

What AI on SAP data actually looks like

Sebastian Stefanowski described Databricks as a complete environment for AI and machine learning, where SAP BDC data products are exposed as tables that plug directly into data pipelines. He grouped the platform’s AI and ML capabilities into several practical areas:

• AI Playground – run any off-the-shelf or externally sourced model on a serverless runtime; create custom prompts and test hypotheses before committing to a build
• Model serving endpoints – host and serve custom models at scale
• AI agent framework – build agentic solutions quickly, with a built-in vector search database for RAG implementations
• MLflow – manage and monitor the full training lifecycle; compare experiments and select the best-performing model
• AutoML – Databricks runs parallel experiments across multiple model architectures on your data automatically, then selects the model that best fits your test results
• AI functions in SQL – call AI models directly from a SQL SELECT statement and store predictions as part of the query output
• AI Gateway – govern model usage centrally: block unauthorized access, filter sensitive queries, and maintain full oversight of what models run and on what data

He then connected those capabilities to the most common use cases for SAP data: cash flow forecasting, prices forecasting, and stock level optimization – blending SAP financial and operational data with seasonality data, interest rate history, and interest rate predictions to anticipate future costs, prices, and inventory needs. Databricks AutoML fits naturally here, running statistical algorithms in parallel and surfacing the model that best fits historical test data.

Beyond forecasting, LLM integration opens a second category: automatically generating product descriptions from structured SAP product data; running sentiment analysis on customer feedback to understand the reasoning behind customer behavior; and identifying the most promising customers to approach based on that analysis.

“All these features can be really, really nicely integrated with SAP and blended with SAP data,” Stefanowski said. “I think that’s something which will bring SAP customers to another level.”

Flexible consumption: licensing that moves with the organization

Raul Muñoz-Gutierrez described the BDC commercial model as one of its differentiators. “SAP Business Data Cloud offers a unit subscription model that provides customers with flexible subscription pricing,” he said. “This allows them to subscribe to the services they currently need and easily modify them in the future.”

The example he gave: an organization starts with SAP BW integrated into BDC. Once that environment has been migrated to SAP Datasphere, the BW capacity is removed and reallocated – to SAP Databricks or SAP Snowflake, for instance. A separate data services licence layer complements the platform subscriptions, enabling activation of analytical applications with near real-time data, development of custom data products, and external data sharing via BDC Connect – the service that exposes SAP data to Databricks, Snowflake, and Microsoft Fabric through Delta Sharing.

What is coming in SAP Business Data Cloud in 2026

Jan Tretina outlined SAP’s roadmap across three areas.

First, deeper openness and connectivity: BDC Connect is expanding to Databricks, Snowflake, Google Cloud, Microsoft Fabric, and AWS. Bi-directional data sharing with SAP HANA Cloud is also in development – data flowing both ways, removing silos at the source rather than managing them downstream.

Second, data products: SAP is expanding coverage across more lines of business and releasing Data Product Studio, which will make modeling SAP and non-SAP data into governed, shareable data products “much easier than ever before” – turning data into “a true asset others can consume and trust,” in Tretina’s words.

Third, AI-native capabilities built directly into the data layer: a new AI Hub with new models, enhancements to the SAP HANA Cloud Knowledge Graph, and a fully agentic multi-modal database – including memory for agents. “This is the database AI was really looking for,” Tretina said.

“We are building the data foundation every enterprise will need to power next-generation AI. 2026 will be a breakthrough year for BDC.”

How CFOs should measure ROI on SAP Business Data Cloud

In the Q&A, Jan Tretina identified four areas where BDC generates measurable returns for finance leaders.

Efficiency and proactivity. Fewer manual reconciliations, reduced data preparation effort, faster close-to-forecast cycles.

Data trust. Higher data quality and fewer errors – directly addressing the confidence gap he described at the start of the session.

Agility. Faster time to insight and quicker scenario iterations.

Business impact. Improved forecast accuracy and tangible cost savings from AI-driven recommendations.

“CFOs should look at gains in proactivity, trust, and measurable financial outcomes,” Tretina said. “With SAP BDC, the ROI is visible both in operational efficiency and in the quality of decisions which can power the business.”

How to start – without committing to a licence first

For organizations that want to evaluate BDC before committing to a full rollout, Raul Muñoz-Gutierrez described a concrete entry point. Inetum has three years of experience with SAP Datasphere and related environments, and operates its own SAP Business Data Cloud environment – available to run proof-of-concept scenarios with customers in a live setting, testing whether a target architecture fits their real needs before any licence investment.

“You don’t have to invest in a licence right now,” Muñoz-Gutierrez said. “You have to know what you want, and we are the best company to accompany you to this final scenario – because we have SAP Data Specialists and also global data expertise from non-SAP solutions.”

On the Databricks side, Sebastian Stefanowski was unambiguous: “Databricks is for everyone.” The platform runs on a pay-per-use model – if workloads run once a day for an hour, the cost reflects that hour. If the platform is idle, the cost goes to zero. “The openness and scalability of this platform – which actually scales down to zero – that is what should encourage any company who wants to start cloud data processing,” he said.

The message across the webinar was consistent: before AI delivers value at scale, organizations need a data foundation that is clean, connected, and trusted. SAP Business Data Cloud is built to be that foundation. Databricks extends what can be done on top of it. And when the two are combined with the right approach, the gap between fragmented enterprise data and working AI closes faster than most organizations expect.

The full session is available on demand.

SAP, Databricks, Rolls-Royce, and Inetum practitioners covered migration strategy, data product architecture, and AI implementation – including the unedited live Q&A. Watch the webinar on demand →

To run a proof of concept in Inetum’s own SAP Business Data Cloud environment – without upfront licence commitment – talk to our team →

AI Applications Reshaping Business in 2025 

Generative AI sparked a wave of revolution that shows no signs of slowing down. Traditional AI already had an estimated global value potential of  $11–18 trillion, according to McKinsey. 2025 is undoubtedly the year of another LLM-based technology: agentic AI.  

By combining autonomous decision-making and system integration, AI agents transform generative AI into digital collaborators, gaining outstanding results.  In call centers alone, according to McKinsey, autonomous proactive agents can reduce resolution time by 60–90%, with 80% of cases being resolved automatically. While agents are commonly associated with chatbots, their applications extend far beyond that across many industries. I’ll aim to explain what agents are and explore typical use cases in selected sectors.  

What is an AI agent? 

Welcome to a world where AI isn’t just helping – you might say it’s working. AI agents are intelligent systems that go beyond scripted interactions. Powered often by large language models, they connect natural language comprehension, data access, and task logic to act proactively and independently. Companies are increasingly willing to deploy AI agents which are advanced AI systems to get competitive advantage.  

Also read:

• AI in project management
• Revolutionizing coding with AI coding agent

How can companies use AI agents? 

Imagine a system that not only understands your request – “prepare the sales forecast for Q3” – but fetches the data, runs the analysis, emails the report, and schedules a review meeting. These agents operate under specified goals, can learn over time, and may run visibly – through chat – or silently in the background. Their defining feature is goal-driven autonomy, transforming artificial assistants into digital coworkers. 

How AI agents work? 

Think of agents as the “turnkey automation engines.” They follow a precise perception–decision–action loop: they observe signals (like system alerts or emails), decide using reasoning engines or LLMs, and then act – updating records, triggering workflows, or escalating tasks. Today’s agents also incorporate memory components, tool orchestration, and task coordination, enabling them to manage complex multi-step processes without intervention. 

Implementing AI agents 

Modern implementations often use a hybrid architecture, featuring an orchestrator that directs specialized sub-agents. Picture a travel-booking agent working alongside calendar and ticketing agents – coordinating to book flights, reserve hotels, and notify participants. This multi-agent pattern is seen across enterprise automation platforms with growing frequency.  

Key types of AI agent system 

There are at least four powerful archetypes shaping agentic AI excellence: 

• Reactive agents are like digital reflexes: they respond instantly to input, without memory or planning. Think about transaction alert monitors or system uptime watchdogs.  
• Deliberative agents act like digital strategists: they reason and plan across multiple steps. Picture a forecasting assistant that builds internal models, evaluates scenarios, and adapts recommendations dynamically.  
• Conversational agents serve as user-facing colleagues: they hold fluid, multi-turn dialogue, pull in context, and even take action – like scheduling meetings or generating analytics. Many organizations are now layering LLMs on top of enterprise chat platforms.  
• Tool-using agents are digital operators: they connect to APIs or apps, generate charts, files, or record – bridging insight and outcome.  

In practice, these archetypes often collaborate – for example, a deliberative agent may trigger multiple tool-using agents to fulfill its plan, overseen by a reactive watchdog. 

Marek Czachorowski Do you want to learn more about the process of building digital hubs? Let’s talk! Book a meeting

Capabilities and risks of using AI 

Benefits of AI agents 

• AI agents are game-changers because they operate with autonomy, scale dynamically, learn from interaction, and integrate deeply with enterprise systems. AI agents offer a wide range of benefits. 
• Once launched, they relieve teams from repetitive work – running reporting, monitoring systems, or responding to emails – while maintaining 24/7 availability. 
• They reduce time spent, cut errors, and free professionals to focus on strategic tasks. In many early deployments, businesses have reported up to 50% gains in efficiency, especially in HR, finance, and customer service.  

Responsible AI – risks of AI assistants 

• With this power comes responsibility. Not without a reason, there are many discussions on responsible AI. AI agents – especially those relying on LLMs – raise explainability concerns. Decision logs must be traceable, with chain-of-thought audits that shed light on agent reasoning. 
• Accountability needs to be clear: which human is overseeing the agent’s actions? Guardrails, access controls, and regular testing are essential to prevent data leaks or manipulation. 
• Lastly, transparency is key – users must know they’re interacting with AI, in line with regulations like GDPR and the EU AI Act.  

Use cases across the enterprise 

Let’s bring these capabilities to life with practical examples: 

• Customer Service – a telecom provider deployed conversational agents that resolve routine support requests, freeing human agents to focus on high-touch cases and increasing customer satisfaction.  
• Finance & Risk – a financial institution uses a combination of reactive agents for fraud alerts and deliberative agents for compliance task automation – later validated through real-time internal audits.  
• Sales & Marketing – startup sales teams use lead-scoring agents that automatically rank prospects, generate outreach templates, and initiate cadences – removing manual steps from campaign execution.  
• HR – a global services firm adopted tool-using agents to manage onboarding paperwork, schedule training, and answer benefit inquiries – dramatically reducing admin time and increasing satisfaction.  
• R&D – life sciences company built agents that review clinical literature, extract findings, and surface key insights ahead of research meetings, cutting analyst prep time by 60%.  

Stories like these show how agents – working in context and across tools – are delivering efficiency, accuracy, and business agility. 

How to get started with AI agents? Start smart!  

Launching agents is as much about strategic discipline as it is about technology. Here’s our seven-step roadmap: 

1. Identify a focused use case – one that is measurable and meaningful. 

2. Run a short pilot (4–8 weeks) to surface technical and UX insights. 

3. Ensure clear, accessible data to power reliable agent reasoning. 

4. Build in human oversight, especially in critical areas like compliance or finance. 

5. Measure rigorously, tracking effectiveness and user satisfaction. 

6. Educate your team on agents’ roles, responses, and limits. 

7. Scale gradually, expanding only after pilots prove reliable and beneficial.  

Understanding AI agents – summary  

• AI agents are autonomous software entities designed to perform tasks and make decisions within various environments. These agents can be categorized into types such as reactive, deliberative, conversational, and tool-using agents.  

• AI systems often deploy multiple AI agents that can work together, leveraging their unique strengths. For instance, an AI assistant can act as a specialized agent to help users with specific tasks, while other, may focus on data processing or decision-making. 

• AI agents are designed to interact with their environment and can adapt based on feedback, showcasing the difference between traditional AI and more complex agentic AI systems. By utilizing AI agents, organizations can leverage AI capabilities to address real-world problems more effectively. 

Check our website for more insights on how we develop AI & Data projects!

What is an AI coding agent, and how does It work? 

An AI agent represents a paradigm shift in software development, moving beyond traditional coding assistance to autonomous programming capabilities. It’s an autonomous agent that understands the broader project context, plans multi-step tasks, and executes them independently. Unlike simple tools that operate on a single file, an agent with code access has insight into the entire codebase, allowing it to perform complex refactoring and implement new features in a way that’s consistent with the existing architecture. 

Its operation relies on advanced language models (LLMs) and a special agent framework, which allows it to interact with the development environment. 

AI coding agent ≠ chatty AI assistant 

Classical digital assistants are brilliant at sorting emails or juggling meetings. A coding agent, however, is built for serious engineering: 

• Whole-repo vision – not just the file you’re staring at. 

• Multi-step planning – think “replace our auth pipeline” rather than single-line completions. 

• Hands-on execution – it can run shell commands, execute tests and land patches – no human copy-paste required. 

GitHub’s Copilot Agent, for example, can launch a VM, clone your repo, add a feature, and open a pull request – all without leaving your IDE. 

How does agent mode in VS Code improve workflow? 

Since April 2025 every VS Code install ships with an Agent panel snug beside the editor. The agent monitors compiler output, terminal logs and file saves, so its advice is always in tune with what you’re doing. 

Why devs love it: 

• Inline roadmaps – before touching a line, the agent tells you the steps it plans to follow. 

• Self-healing loops – if a test bombs, it iterates until green. 

• Effortless tool-chaining – thanks to MCP, the agent can ping databases, cloud APIs or vector stores without you wiring things up manually. 

Why the Model Context Protocol (MCP) matters? 

LLM’s perform best with deep context. The Model Context Protocol is a vendor-neutral conduit that streams project graphs, tool specs and dynamic data into any AI agent. Anthropic’s open-source MCP servers expose file systems, cloud APIs or local Postgres databases without writing extra code. 

What MCP adds 

Popular MCP servers 

The open-source community maintains a growing list of servers that expose everything from file systems to cloud APIs. I encourage you to check the list of available MCP servers: modelcontextprotocol/servers: Model Context Protocol Servers 

What are the advantages of using MCP for code generation? 

MCP transforms code generation from isolated snippets to context-aware, full-stack solutions. Instead of generating isolated fragments, the agent gains the ability to create coherent, contextual code. MCP allows agents that use it to create high-quality code that is consistent with the rest of the project. 

The main benefits include: 

• Understand Complex Code Relationships: By accessing entire codebases through MCP, agents generate high-quality code that respects existing patterns, dependencies, and architectural decisions. The result is more efficient code. 

• Complex code generation: With access to tools (tool calling) and external APIs, the agent can generate code that integrates with other systems and works with actual data. 

• Improved code quality: The agent can use static analysis tools or run tests, ensuring that the code generated is not only functional but also adheres to best practices. 

Tackling coding challenges with AI agents 

Coding challenges can often be daunting, especially for beginners or when tackling unfamiliar problems. AI coding agents can provide step-by-step guidance, breaking down complex problems into manageable parts. AI coding agents can assist in several ways: 

• Generate solutions for complex coding tasks 

• Instant feedback & debugging  

• Adaptive learning 

• Repetitive tasks automation 

Using AI to generate solutions for complex coding tasks 

When faced with a complex problem, developers can describe it to the AI agent. The agent won’t just suggest a different code snippet; it can build code as a complete solution, which the developer can then iteratively refine. Using the code interpreter tool, a developer can analyse step-by-step how the agent arrived at the final solution. 

Debugging with code review agents: a new approach 

Debugging is traditionally a 20% diagnosis, 80% boredom. It can be one of the most frustrating and time-consuming aspects of coding. AI coding agents offer a fresh approach to this challenge. 

• Developers can receive immediate feedback on their code, helping them identify errors or inefficiencies quickly. 

• By understanding the context of the code, the AI agent can provide more relevant debugging suggestions, making it easier to resolve issues. 

• As developers interact with the AI, it can learn from their debugging patterns, improving its suggestions over time. 

AI agents can automatically pinpoint the root cause of compilation errors, suggest precise fixes and even generate unit tests to prevent regressions. 

Adaptive learning and skill building 

For developers looking to improve their skills, AI coding agents act as intelligent tutors. They provide detailed explanations for their suggestions, helping users understand best practices while building coding experience through hands-on practice with real-world scenarios. 

How AI can automate repetitive coding tasks 

Repetitive tasks can drain a developer’s productivity; boilerplate is the kryptonite of creativity. AI coding agents can relieve the pain in several ways. 

• Project scaffolding: Single-prompt generation of project structures, build scripts, and configuration files 

• Context-aware completions: Intelligent suggestions that understand your codebase and use code patterns consistently 

• Automated documentation: Generated inline comments, API summaries, and README files that keep projects well-documented 

• Integration assistance: Streamlined API and library integration with automatic configuration and error handling 

What is the Future of Software Development with AI Agents? 

The future of software development is likely to be heavily influenced by AI coding agents. It isn’t just about coding support from assistants like github copilot, but about fully autonomous agents. As these tools become more sophisticated, expect: 

• tighter IDE integration, richer MCP toolchains and cross-agent collaboration – e.g., a design agent handing Figma specs to a coding agent that ships production React components overnight. 

• agents to mediate merge conflicts, enforce security policies, and hand off tasks between design, data and DevOps agents. Newcomers will onboard in hours, not weeks, because every repo ships with an embedded tutor. 

• and so on… 

Agentic AI will manage the entire application lifecycle – from planning, through coding, testing, and deployment, to monitoring. 

Conclusion 

Pairing an AI coding agent with VS Code’s Agent Mode and the Model Context Protocol turns development from a tooling grind into true collaboration. With boilerplate outsourced, debugging streamlined and project context unified, you’re free to focus on solving real problems. Fire up your editor, toggle Agent Mode, plug in an MCP server – and see what happens when AI shoulders the grunt work so you don’t have to. 

Frequently Asked Questions About AI Coding Agents

1. 1. AI, outages, and more… What CEOs talked about in Q3 2024? 
2. 2. What is asset maintenance management? 
3. 3. What is reactive maintenance?
4. 4. What is preventive maintenance? 
5. 5. What is CMMS software?  
6. 6. Key challenges in asset maintenance
7. 7. Importance of asset maintenance plan across industries
8. 8. Asset maintenance strategy
9. 9. Benefits of asset maintenance management software 
10. 10. Asset management system success story

Over 80% of organizations experience unplanned downtime every three years, as per Forbes findings. At a time of economic uncertainty and the need to optimize costs, companies seek solutions to simplify asset management. Implementing programming is just the beginning of the challenges and decisions to be made.

AI, outages, and more… What CEOs talked about in Q3 2024? 

The end of 2024 has brought the next edition of the report on aspects of most interest to CEOs ‘What CEOs talked about’ by IoT Analytics. There are rarely any surprises, but the trends show what topics will most likely be invested in in 2025. The top three included: 

1) Practical AI applications – interest in AI capabilities is high in any industry looking to improve performance and accelerate tasks. Process automation, digital twins, and AI-based predictive maintenance solutions are in the spotlight. 

2) Renewable energy sources – in Q3 2024, interest in solar energy increased by almost 80% quarter-on-quarter and wind energy by almost 60%. Sustainability, green energy, and climate change are important themes globally, with some regional variations. 

3) Unpredictable IT outages – 2024 brought an outage that was on everyone’s lips: the CrowdStrike / Microsoft related to cyber security. Unpredictable outages such as this can cost thousands of dollars per hour, depending on the industry. This is why companies are prioritizing solutions to prevent unforeseen failures, such as Enterprise Asset Management Solutions. 

What is asset maintenance management? 

Asset Maintenance Management or Enterprise Asset Management (EAM) is a comprehensive approach to managing an organization’s physical assets across their entire lifecycle. It encompasses the processes, tools, and systems used to plan, acquire, maintain, operate, and dispose of assets in a way that maximizes their value, minimizes costs, and ensures operational efficiency.  

Asset Management is particularly relevant for organizations with significant investments in physical infrastructure or equipment, such as manufacturing, utilities, transportation, and healthcare.  

What is reactive maintenance? 

Reactive maintenance (also known as corrective maintenance) is the type of maintenance that involves repairing an asset once it is broken. This means responding to failures or breakdowns rather than taking preventive actions. 

What is preventive maintenance? 

Preventive maintenance is a wide subject, and the general aim is to prevent breakdowns before they happen and thus take care of the asset lifecycle. One of the types of PM is Condition-Based Monitoring. In the CBM approach, activities take place based on machine condition-related data, collected through sensors.  

What is CMMS software?  

CMMS stands for computerized maintenance management system. It refers to software that helps manage assets, plan maintenance work, and manage work orders. CMMs are inventory tracking systems that allow you to make informed decisions.  

What are the key challenges in asset maintenance management? 

1. Identifying inefficient maintenance – relying on reactive asset management techniques adds costs and gives a dim view of investment in this area. Companies that are unaware of this problem struggle with a vast number of work orders, fail to see the value in equipment data, and try to address staff shortages by temporarily subletting contractors. At the same time, the main problem remains unresolved and results in further investment.  

2. Creating an asset maintenance plan – creating a successful asset maintenance plan is no easy task. It’s a complex process that involves gathering asset inventory along with their value, calculating expected lifecycles, establishing SLAs and an optimum maintenance plan, and then sticking to the strategy over the long term.  

3. Choosing asset management software – such software is designed to follow the life cycle of solutions, whether software, hardware, or any machine. A properly chosen solution increases confidence in data security. It also improves confidence in the devices themselves. There are many asset management solutions on the market, many of which offer extensive customization and integration with external systems and take advantage of the cloud. One example allowing you to build an EAM is the Cumulocity IoT platform.  

4. Training maintenance team – investing in your technicians is crucial for implementing a successful maintenance strategy that incorporates Computerized Maintenance Management Systems (CMMS). Great training leads to well-prepared staff that can handle the maintenance process with ease. The trained team can effectively perform maintenance tasks and adhere to a preventive maintenance schedule. This ensures that maintenance needs are addressed proactively, thus preventing asset failures, and extending the asset life. 

Importance of asset maintenance plan across industries 

Asset management solutions are critical in industries that rely heavily on them. Such as, for example:  

Manufacturing  

IoT solutions collect real-time data from devices and parts for health monitoring purposes. This allows for the application of predictive maintenance solutions and the reduction of downtime and unnecessary repairs. With the right system, companies can schedule maintenance and engage maintenance teams to perform tasks more efficiently.  

Sales and procurement 

The area is focused on ensuring that the right assets are procured or produced at the right time to meet operational needs.  

Finance and controlling  

This area is related to financial planning, budgeting, and control mechanisms for managing the aspects of asset acquisition, its maintenance, and overall lifecycle costs.  

Compliance 

Regulatory adherence and compliance with legal standards related to asset management is an important part of managing organizational assets.  

How to develop an effective asset maintenance strategy? 

A successful and comprehensive Enterprise Asset Management strategy is built upon 3 main pillars:  

1. Asset business management  

2. Asset operations management  

3. Asset service management  

These areas are strictly connected and dependable upon each other to cover the whole landscape of the company’s asset-related activities from its “birth” (either by purchase or fabrication) until end-of-life and removal. The high degree of digitalization and automated integrations are crucial in taking advantage of this holistic approach.  

What are the benefits of asset maintenance management software 

• Extended asset lifespan: By providing the tools to maintain assets proactively, EAM systems help organizations extend the useful life of their assets.  
• Reduced downtime: Real-time monitoring, predictive maintenance, and better planning result in fewer breakdowns and less unplanned downtime.  
• Cost efficiency: By optimizing maintenance schedules, reducing emergency repairs, and managing spare parts more effectively, EAM can significantly reduce operational and maintenance costs.  
• Regulatory compliance: EAM systems ensure that organizations maintain compliance with industry standards and regulations, reducing legal and operational risks.  
• Better decision-making: With better data and analytics on asset performance, usage, and costs, EAM systems provide insights that allow for smarter decision-making regarding asset investment, replacement, or optimization.  

Want to dive deep into Enterprise Asset Management solutions? Prevent asset downtime with successful maintenance operations! Request a free demo presentation of a solution based on the Cumulocity IoT platform. 

Streamline Your IoT Operations

Andrzej Gumieniak, our Head of Practice IoT, is here to help you navigate the complexities of IoT solutions. Book a consultation to discuss your case.

Schedule a meeting

Asset management system success story 

Wind turbines monitoring  

A leader in green energy and wind turbine manufacturing faced the challenge of operating thousands of wind turbines worldwide with only a few operators.  

To achieve that, we helped them to build their command streamlined towards the devices and integrate everything into a central IoT solution based on Cumulocity IoT.  

Read the story! 

Asset management – summary 

Asset management is crucial for organizations aiming to maximize the lifecycle of their assets. A comprehensive asset maintenance management plan ensures that maintenance activities are scheduled and executed effectively 

This will allow you to make more informed decisions about needed maintenance work, whether that involves preventive and predictive maintenance or any other strategy. 

Mobile app development projects in the past 

When the mobile revolution started, most mobile applications were implemented in programming languages designed for a specific platform. Android developers were using Java while iOS developers dealt with Objective C which later was replaced by Swift.  

This approach made sense because the biggest challenge developers faced was performance. The apps were expected to run smoothly. Designers wanted to add some animations. And fast, at the time, mobile devices had less computing power than even the slowest ones we use nowadays.  

Why React Native development has become popular 

Later, with more powerful devices, and optimized technical solutions things started to change. Initially, cross-platform solutions were only suitable for simple, less demanding applications. However, today that is not an issue anymore. Mobile devices are powerful and cross-platform solutions went through an optimization process to be treated as “almost native”. 

Read also: Cloud-native applications: what do you need to know?

What is React Native app development, and why choose it for your development process? 

React Native is one of the cross-platform solutions that developers can use to create apps for multiple platforms.  

The benefits of using React Native: 

• React Native code is partially just a React code that can be reused to create a web application. 
• React Native deals with platform UI implementations and allows us to focus on developing features with React primitives. 
• In React Native, developers can use libraries, and SDKs or write their native code. 
• React Native is based on React, a popular framework for web development. It means it might be easier to find good React developers who only need to learn the native part of development. 
• React Native performs much better nowadays. The core team introduced a new communication method between JavaScript and native elements: “Bridge” asynchronous communication has been replaced with JSI (JavaScript Interface) communication.

React native framework vs. native app development: when to choose Swift or Kotlin for your app development project? 

Even though there are advantages to using React Native, some cons can be found as well. Due to their nature, cross-platform solutions cannot compete with native ones in some aspects. 

Potential React Native issues: 

• Near-native performance is still worse than native performance and some complex projects might need to be developed natively. Especially when they deal with many interactions. 
• Some parts of the app might still need to be developed using native programming languages. That means that either React developers need to learn how to do it, or some help from native developers might be needed from time to time. 
• React Native is a relatively new technology, therefore some modules might need to be built from scratch. It can slow down the software development if the development team is not aware of this limitation. 
• React Native was built by the Facebook team and has less support and backward compatibility than native technologies. It is used in many projects, so it won’t be abandoned in a day. Nonetheless, its popularity heavily depends on Facebook (Meta) and its use in their apps.

How React Native’s development time and cost compare to native mobile app development 

When we try to make a comparison between React Native and native app development, we should first focus on business requirements and company expectations. Some projects should be delivered as soon as possible to be later improved and extended. Other projects have already been developed as web applications and the goal is to port them to mobile applications. Those are probably the most common cases where React Native can come in handy. 

When React Native can speed up the software development process: 

• There is an existing web application built in React and many parts of the code can be reused. Only UI and native elements must be developed. 
• It is known from the very beginning that you need to serve your app on multiple platforms and app development process speed is the main factor. 
• Applications are not complex from the technological perspective and there probably won’t be a need to develop custom or native components. 
• The application is not expected to scale to a point where huge amounts of data could slow it and make it laggy. 
• The team in the company has a lot of React experience. It might be cheaper to allow them to learn native development than to hire dedicated teams for Android and iOS development.

When should you use React Native over progressive web apps (PWA)? 

PWA’s are simply applications running in browsers that use web technologies such as HTML, CSS, and JavaScript. They can mimic the behavior of native applications to some extent. There are many cases in which PWA’s are sufficient for our needs. They can be saved on the home screen, and they can work offline, using push notifications and some device features.  

React Native is a JavaScript-based framework that allows you to build mobile apps for iOS and Android. It has access to native rendering APIs and can take advantage of GPS, camera, and accelerometer. This allows developers to build applications that feel truly native and perform at a nearly native level. 

When should we choose React Native over PWA? 

PWA’s have limited access to certain hardware features of a device. This can impact the way the app functions. We should be aware of that and choose React Native when a specific device feature is a key factor to app success.

• Battery consumption is higher in PWA’s. It happens because they run in the browser which is less effective compared to native apps.
• Performance is better in React Native apps because the have access to native rendering APIs.
• React Native uses native UI components specific to each platform. That results in a seamless User Experience.

Read also: Angular – Let’s create our own Progressive Web Application

Best practices for React Native mobile app development

Most of the best practices for React mobile applications will be the same as in the case of React web app development best practices.

• Carefully plan code structure.
• Keep your components small and reusable.
• Extract reusable parts of logic to promote modular code.
• Try not to overcomplicate state management. Sometimes local state is all you need.
• Optimize performance where needed. Avoid unnecessary re-renders, and use optimized elements like FlatList.
• Avoid inline styling, use shared styles.
• Try to build dynamic components that can rely on server-side data. That might limit the number of app releases when some minor logic changes are introduced.
• Don’t forget to test your code.
• Use React Native’s built-in accessibility features
• Introduce crash analytics to ensure your app is healthy and the number of crashes is low.
  Log errors. That will help you with debugging issues and gathering analytics for future development.

Advantages of React Native cross-platform app development 

There are business and technical advantages of using React Native. Most important are the following. 

• React Native is gaining popularity. It holds around 13% of the market share among cross-platform mobile app development frameworks.
• There are many experienced React developers who can switch to React Native development relatively easily.
• React Native community is large.
• React Native targets many platforms at once.
• If there is a need for web application development with similar features, parts of the codebase can be shared between React and React Native.
• React Native is fast because it uses native APIs. It gives a near-mobile experience in terms of application smoothness.

Disadvantages of React Native cross-platform app development 

Just like every cross-platform solution built by a third party, React Native has its cons. The most important are: 

• Relying on third-party solutions can be risky even if the company that created is large and if it has a large community.
• Some custom modules are missing and others can be underdeveloped.
• Upgrades are not always easy to apply.
• Native developers might be needed from time to time to develop more sophisticated plugins written in native languages.
• It is easy to write an app that won’t perform as expected. You need to pay extra attention to developing an optimized application.
• Complex designs and interactions can run slowly in React Native.
• Even a well-optimized application won’t reach the speed of a native application.

12 reasons to choose React Native to develop mobile apps 

Let’s summarize it with a list of reasons that should encourage you to use React Native. 

1. It targets many platforms at once.
2. React Native can have a shared codebase with the React web application.
3. It has a large community.
4. React Native speed can be described as near-native.
5. The framework is gaining popularity in mobile development.
6. React Native uses native APIs and has access to specific device features.
7. React Native deals with platform UI implementations.
8. React Native developers are not limited to existing modules. They can write native modules.
9. The React community is big, and this might help to build a React Native team.
10. New features and improvements are often released.
11. Good developers’ experience during the development process (e.g. hot reloading).
12. React Native was developed by Facebook (Meta) and it is used in most of their mobile applications. Even though it’s a third-party solution, it might be considered a relatively safe one to use.

React Native case study 

Our client is a company in the heavy industry area. It is a European leader in laser metal processing equipment. End customers who buy the machines get a complete solution bundled with a control system for operators and service technicians. 
 
They needed a desktop application that would serve as a control panel to monitor cutting processes in the factory. The initial decision was to use React and Electron to build Windows desktop applications. During technical conversations the team decided to switch to React Native to leave a gate for native app development or performance improvements.   

This was due to the risk that the application displayed in the browser on slower tablets might not be efficient enough to correctly preview the live cutting process.  

Additionally, the client didn’t make a final decision whether the app should be made available in App Store and Google Play. In the end, the application was made available on the Windows display but the use of React Native creates opportunities for future mobile development. 

Summary 

This development framework allows developers to create cross-platform mobile apps that function seamlessly on both iOS and Android. By leveraging JavaScript and React, a React Native app development company can build an app using native features, ensuring a smoother User Experience. The react native development services enable the integration of native libraries, allowing for the use of native device capabilities throughout the development process. A native app development company can bring significant benefits resulting in faster development. If you are interested in React Native development services, contact us to see how your project can speed up.  

What is DevSecOps? 

In the realm of modern craft, there arose a practice known among the learned as DevSecOps, a union of three great disciplines: development, security, and operations. It was not unlike the forging of a mighty alliance, wherein each realm must contribute its strength, lest their endeavors fall prey to the shadows of vulnerability. 

In days past, many development teams would wait until the final hour, when the code was near release, before calling upon the wardens of security. Yet this path was fraught with peril, for to uncover weaknesses so late would often cost dear in time, gold, and effort. But the wisest of realms soon saw another way. 

Benefits of DevSecOps 

By weaving security into every phase of the software development lifecycle – whether in the laying of the first line of code or the final shaping of the product – teams could safeguard their work from the outset and prevent a number of security issues. Through deep collaboration and the magic of automation, they crafted a system where all shared the burden of protection. Thus, no longer was security the task of the few, but of all who toiled together, forging a shield that would hold fast even in the face of the darkest threats. 

In this way, DevSecOps was not merely a method, but a way of ensuring that no creation would leave the forges unguarded against the unseen dangers.  

DevSecOps vs DevOps 

In terms of traditional software development, the crafting of great projects was long governed by the old ways, where the work was divided into phases – like the seasons of the year – each flowing one after the other.  

There would come a time for planning, then design, and only after would the labor of development begin, followed by the testing and binding of all parts into a whole. Yet, this process, while orderly, was as slow as the march of the seasons themselves. In an age where customers’ desires are ever-shifting, such a pace no longer sufficed. 

Worse still, security specialists were oft called upon at the very end, when the work was near complete, to cast their protections over the product. Alas, this lateness was fraught with danger, for vulnerabilities uncovered at such a time could unravel much of the work that had come before. 

Thus, many turned to a new way: the DevOps model. Here, rather than waiting for the long seasons to pass, the DevOps teams delivered smaller, yet finely crafted parcels of work – each one polished and ready – rather than undertaking vast projects that stretched on for years.  

In this way, the teams of development and operations joined forces, testing and refining their work as they went. By using the tools of automation and forging standardized processes, they moved with great speed, yet kept the quality of their software product intact. 

DevSecOps for enhanced application security 

But in time, the guilds realized there was a further step to be taken. They sought not just swiftness, but security woven into every step of their journey. And so was born DevSecOps, where security was not an afterthought but a companion to every stage of the craft. From the very first whispers of planning, security was present, and in the fires of development, it was tested and shaped alongside the code. The burden of security threat protection no longer lay on the shoulders of a few, but on the entire fellowship of creators. This new way became known to some as ‘shift left security’, for it brought the guardians of protection to the forefront of the process, ensuring that no ill would befall their work from the very start of the journey to its end. 

Read also:

• Azure Cost Management 101
• Cloud-native applications: what do you need to know?

Why is DevSecOps important? 

Many foes are seeking to gain entry and plunder company’s most valuable treasures – its data and assets. One of the most common practices they can employ, is the exploitation of weaknesses, hidden deep within the folds of the organization’s own software. These vulnerabilities, if left unchecked, are like cracks in the foundation of a mighty fortress, and through them, adversaries can slip in unnoticed, wreaking havoc from within. 

Such breaches can be devastating. They consume both time and money, and in their wake, they leave scars upon a company’s name, causing trust to wither among clients and partners alike. 

DevSecOps culture 

But there is hope within DevSecOps. By following this path, the guilds of development, security, and operations stand together, ever watchful. The framework is like a vigilant sentry, reducing the risk of sending forth software riddled with flaws, misconfigurations, or vulnerabilities. Through constant vigilance and the weaving of security into every phase of creation, they close the gates through which bad actors might pass, fortifying their code so that it may stand strong against the onslaught of those who would seek to exploit it. Thus, the company stands firm, its reputation unshaken, its defenses prepared for the battles yet to come. 

Continuous Integration 

In the world of software development, there once was a time when the labor of many artisans was brought together only at the final hour, when all the pieces were near completion. But it was often in those moments, at the very end, that flaws were revealed, and the seams of their work would unravel, leaving them with a tangle of issues too great to swiftly resolve. 

But a new method arose, known as Continuous Integration, a practice where the builders of code did not wait for the end to unite their works. Instead, they would commit their efforts to a central repository many times throughout the day, like artisans returning to the hearth to meld their creations together piece by piece. Each time they did so, their work was automatically tested and integrated, ensuring that all parts fit together in harmony. 

By catching integration issues and bugs early, long before the final forge was set aflame, this approach saved the guilds from the chaos of last-minute discoveries.  

Continuous Delivery 

Building upon the foundation of Continuous Integration, a further practice emerged, known as Continuous Delivery. Where the former ensured that code was swiftly integrated and tested, Continuous Delivery took this one step further, automating the journey from the builder’s hands to a staging environment, where code would be tested. 

Once the code reached this staging ground, it did not rest. The system immediately set to work, not only with unit testing, but with a series of trials to ensure that all aspects of the creation were sound. The user interface was inspected to ensure it responded as intended, the seams of integration were examined for any weaknesses, and the APIs were tested to confirm they communicated well between systems. The code was also tested under the weight of simulated traffic, to see if it could bear the burden of the many users it would one day serve. 

The aim of Continuous Delivery is simple: to consistently deliver code that was not only complete, but of true value.  

Continuous Security 

DevSecOps practice, one of the most vital elements is the weaving of security into every step of the software’s journey, from the first step of design to the final unveiling. No longer could security be treated as an afterthought, called upon only when the work was nearly complete. Instead, it became a core part of the process, guiding the work like an unseen but ever-present hand. 

From the earliest stages, when the blueprints of the software were still taking shape, the guilds would engage in threat modeling, a process of foresight where they sought to uncover any potential dangers. They did not wait for the enemy to strike, but anticipated its moves, fortifying their code against unseen attacks before they could take root. 

As the work progressed, automated security testing was woven into every stage of the DevOps workflow. Automation played a key role, testing the code continuously, from the developers’ own environments to the farthest reaches of the deployment pipeline. No phase was left unguarded; no section of code went untested. 

Through this constant vigilance – testing early and testing often – the teams were able to find and mend weaknesses swiftly. And so, they delivered their software products with confidence, knowing that each line of code was protected. With DevSecOps, the road to production became one of fewer pitfalls and greater security, allowing organizations to deliver secured software swiftly and with minimal issues.   

Communication and collaboration 

In the practice of DevSecOps, the strength of the software does not lie solely in the application security tools or the processes but in the fellowship of those who undertake the journey together. It is a path that demands more than mere skill; it calls for deep collaboration and unity of purpose among individuals and teams. 

When the developers commit their work to the central repository in the practice of Continuous Integration, conflicts in code inevitably arise. But it is through collaboration – where minds come together, and voices are heard – that these challenges are resolved. Developers, security experts, and operations alike must work side by side, swiftly addressing these conflicts so the flow of progress is not hindered. 

But beyond the technical work, there is a greater need – communication. Teams must speak openly and often, sharing their visions and aligning around the same goals. Without this shared understanding, the work would drift in many directions, and the efforts of one might undo the labors of another. In DevSecOps approach, every hand contributes to the same creation, and every voice is heard in the great chorus.  

DevSecOps best practices: shift security left 

Planning and development 

Introducing security policies and addressing security risks early into the rhythm of development sprints is akin to fortifying the foundations of a great structure before the first stones are laid. By addressing vulnerabilities in the early stages, teams not only reduce the risk of future threats but also save valuable time, for it is far easier to mend potential flaws before the code has been built and added to the greater whole.  

At the start of every sprint, during planning and development, threat modeling becomes a crucial tool, a map guiding the teams to uncover and mitigate potential dangers long before they can occur. By identifying these threats early, security is no longer something that is added at the end, but something that is woven into the very fabric of the application from the outset. 

To ensure security, before the code is committed to the shared repository, automated checks are employed, acting as vigilant sentries. Integrated development environment (IDE) security plug-ins provide developers with immediate feedback, warning them if their code harbors a potential risk. These automated checks catch flaws early, empowering developers to address them before they can take root. 

As the code continues its journey, passing from one set of hands to the next, the software is further refined during the code review. Here, someone with the knowledge of security steps forward, offering their insight and making recommendations to bolster the work. This expertise ensures that, by the time the code is ready to move on, it is fortified against threats, functional, and valuable.   

Code commit 

A cornerstone of the DevSecOps process lies in the practice of continuous integration, a discipline that ensures code is not simply created in isolation but is constantly woven into the central repository, allowing teams to catch issues before they can fester. Developers, like diligent artisans, commit their code several times a day, ensuring that each piece of work seamlessly fits into the greater whole. This frequent integration allows potential conflicts or errors to be discovered early, long before they can threaten the stability of the project. 

However, to truly safeguard the craft, it is vital to introduce security into this phase. Automated security checks must stand guard alongside the integration process. These include scanning third-party libraries and dependencies – those external pieces of code that, while useful, may harbor unseen vulnerabilities. Unit testing ensures the smallest parts of the code function as they should, while static application security testing (SAST) reviews the code for weaknesses, searching for hidden threats that might otherwise go unnoticed. 

But safeguarding the code itself is not enough. The continuous integration (CI) and continuous delivery (CD) infrastructure, which carries this code from creation to deployment, must also be protected. Role-based access controls (RBAC) play a crucial role in this defense, limiting access to the system based on the specific roles of individuals. By ensuring that only those with the right permissions can interact with the CI/CD infrastructure, teams protect it from attackers who might seek to run malicious code or steal credentials. 

In this way, the continuous integration process becomes not only a means to unite code swiftly and efficiently but also a stronghold against external threats. Security is built into every layer, from the automated checks that scan the code to the protections guarding the very systems that bring the work to life. 

Building and testing 

In DevSecOps, where vigilance is paramount, the test environment serves as a proving ground for code before it ventures into production. Here, automated security scripts are seeking out potential threats that may have slipped past earlier defenses. By running these tests in a controlled environment, teams can uncover hidden vulnerabilities and ensure their work remains strong and secure.  
 
DAST 

Among the many tests that can be employed during this phase is Dynamic Application Security Testing (DAST), which simulates real-world attacks against the running application. Unlike static tests, DAST operates while the application is live, identifying vulnerabilities such as: cross-site scripting, SQL injection, and other dangerous flaws. 

Infrastructure scanning  

Infrastructure scanning follows, casting its gaze across the entire architecture, from servers to networks, searching for weaknesses in the foundational layers that might allow an attacker entry. For those employing containers as part of their deployment strategy, container scanning ensures that these lightweight units of software do not harbor vulnerabilities in their dependencies or configurations, fortifying them before they are deployed. 

Cloud configuration validation 

In the age of the cloud, where infrastructure is often abstracted and spread across vast digital environments, cloud configuration validation becomes crucial. By checking the configurations of cloud resources, teams can ensure that no misconfigurations – such as excessive permissions or insecure access points – expose their environments to unnecessary risk. 

Lastly, security acceptance testing ensures that all necessary security requirements are met. This step serves as the final safeguard, confirming that the code and infrastructure are not only functional but fortified against known threats and risks minimized. 

Production 

Once the application has been deployed to production and stands in the real world, some organizations take a proactive step to uncover any remaining weaknesses by engaging in penetration testing. This practice is more than just another test – it is a deliberate attempt to breach the application as an attacker might, with real-world tactics and determination. 

In penetration testing, skilled individuals, often referred to as ethical hackers, adopt the mindset of a potential adversary. They probe the application for weaknesses, using the same strategies and tools a malicious actor might employ. These tests can range from exploiting known vulnerabilities in third-party components to more sophisticated attacks aimed at bypassing the application’s defenses. 

The goal is simple: to expose any weaknesses that might have slipped through the earlier layers of security testing, those that could potentially be exploited. By simulating real-world attack scenarios, penetration testing reveals how the application holds up under direct assault, whether it’s vulnerable to unauthorized access, data breaches, or other forms of compromise. 

This phase is crucial for understanding not just theoretical vulnerabilities but how the system behaves in a live environment, where the s 1takes are highest. Penetration testing provides organizations with invaluable insights into the robustness of their defenses, allowing them to patch any remaining weaknesses before an actual attacker can exploit them. Thus, it becomes the final line of preparation, ensuring that the application is truly ready to stand firm against threats in the production environment. 

Operation 

Even with the most robust DevSecOps process, no system is entirely resistant to evolving threats. This is why continuous monitoring of applications becomes essential once they are deployed. By maintaining constant vigilance, organizations can quickly detect, respond to, and mitigate any new vulnerabilities, unforeseen threats and risks before they cause significant harm. 

Monitoring tools look for signs of irregularities, scanning for vulnerabilities, unauthorized access attempts, or other suspicious activities that might signal a breach or weakness. These tools provide real-time insights, alerting teams to potential issues the moment they arise.  

To further strengthen this defense, analytics data plays a key role. By analyzing patterns and trends in security events, teams can evaluate the effectiveness of their security posture. This data offers valuable insights into how well current defenses are performing, allowing organizations to track whether they are improving over time or if new vulnerabilities are emerging. It also highlights areas that may require optimization, guiding future efforts in reinforcing the system.  

Bear in mind, however, that in the world of security, the battle is never truly over. 

Read also:

• Cloud-Agnostic Applications: Pros and Cons
• Horizontal vs Vertical Scaling: A 101 Guide
• How to gain a cloud advantage? Here are 7 cloud computing trends for 2023 – 2025 

Implementing DevSecOps and automating security best practices 

I bid you to consider these tools as you embark upon the journey of DevSecOps automation within your organization. Some are like fruit hanging low upon the bough, easily gathered and swiftly put to use, while others may lie deeper within the forest, requiring more effort to attain. Yet, though the path to them may be more difficult, the rewards they yield are well worth the quest. 

Want to gain cost and competitive advantage in the cloud?

Consult with Marek Dobkowski, Head of Microsoft Practice, for expert guidance.

Let's talk!

Recommended DevSecOps tools 

Trivvy 

Trivy has risen to prominence as a trusted solution among open-source security scanners, valued for its reliability, swiftness, and simplicity. It offers a far-reaching array of security checks, making it a vital companion for those seeking to fortify their DevSecOps practices. For teams looking to secure their realms of code and infrastructure, Trivy stands as a steadfast tool, ever vigilant and ready to ensure the safety of their creations. 

Targets (what Trivy can scan): 

• Container Image 
• Filesystem 
• Git Repository (remote) 
• Virtual Machine Image 
• Kubernetes 
• AWS 

Scanners (what Trivy can find there): 

• OS packages and software dependencies in use (SBOM) 
• Known vulnerabilities (CVEs) 
• IaC issues and misconfigurations 
• Sensitive information and secrets 
• Software licenses 

Trufflehog 

TruffleHog tool is a masterful seeker of concealed passwords and keys. Like a skilled ranger, TruffleHog ventures where few dare to tread, unearthing the hidden secrets that, if left unchecked, could spell doom for the unwary. 

How TruffleHog wields its power: 

• Detect: TruffleHog scours the history of all platforms, much like a wise lorekeeper sifting through ancient scrolls, seeking out long-forgotten secrets. Yet it looks not only in the obvious places but also in the whispers of comments, the hidden folds of Docker images, and other obscure corners. 
• Analyze: TruffleHog reveals the true nature of the secrets it uncovers, discerning what resources and permissions are tied to API keys and other tokens. Remarkably, it achieves this without ever needing to peer into the provider’s vault. 
• Prevent: To stop the ill-fated inclusion of secrets from the very beginning, TruffleHog sets traps at key points, using pre-commit and pre-receive hooks. These safeguards ensure that no sensitive data is unintentionally leaked before it ever leaves the developer’s hand. 
• Remediate: TruffleHog continues to track the fate of discovered keys and secrets. It verifies that remediation is complete, sending reminders on preferred platforms and providing knowledge to users on how to properly manage and secure the keys that were once at risk. 

Why TruffleHog is a worthy ally: 

• Comprehensive multi-branch analysis: TruffleHog does not simply guard the main road but patrols every path. It scans all branches, not just the primary one, ensuring the same level of vigilance across the entire project. This is especially valuable in larger domains where many branches are being worked on in tandem. 
• Credential verification: TruffleHog employs programmatic verification, testing each credential using its own protocol or API. This removes the false trails, ensuring that only real threats are brought to light. 
• Open-source fellowship: As with any great alliance, TruffleHog thrives through the support of an open-source community. Many dedicated hands join together to audit and improve the tool, ensuring that no single voice carries undue weight. The community checks and balances each other’s work, so that trust is shared among all. 
   

Snyk 

This platform guards the entirety of an application’s journey – from the very first lines of code to its deployment in the cloud. Through its guidance, developers may discover and mend vulnerabilities before they are ever loosed upon the world. 

The powers of Snyk: 

• Snyk open source: Snyk scours open-source libraries and dependencies, seeking out vulnerabilities. And when such flaws are found, it does not merely warn the developer but offers a swift means to mend them, restoring the strength of the code. 
• Snyk code: As code is written, Snyk watches in real-time, finding and fixing vulnerabilities within the very heart of the application. It is like a companion at the developer’s side, ever watchful and ready to lend its aid.  
• Snyk container: In the context of containers and Kubernetes, where applications are housed, Snyk’s gaze does not falter. It delves into container images, finding and repairing potentially harmful vulnerabilities.  
• Snyk Infrastructure as Code: With great foresight, Snyk peers into the blueprints of infrastructure itself, examining the configurations of Terraform and Kubernetes code. Should it find any insecurity in the very foundation, it offers swift guidance on how to rectify these flaws, ensuring that the structure remains strong and secure. 

Pre-commit 

 Pre-commit is a powerful system for managing and maintaining pre-commit hooks across many programming languages. Pre-commit ensures that no errant detail is left unchecked before the code is sent for review. 

In the world of Git, hook scripts act as a safeguard, catching simple errors before they reach the eyes of a reviewer. Whenever a developer commits their work, these hooks spring into action, pointing out issues such as missing semicolons, trailing whitespace, or forgotten debug statements. By addressing these small matters early, Pre-commit allows the reviewer to focus on the grand architecture of the changes, rather than wasting time on trivial style errors. 

Wazuh 

Free and open to all, Wazuh is skilled in the arts of threat prevention, detection, and response. It is a protector capable of defending the realms of on-premises fortresses, virtualized strongholds, containerized ships, and vast cloud kingdoms alike. 

The strength of Wazuh lies in two parts: its endpoint security agents, which are deployed like watchful sentinels to the systems they protect, and its management server, a wise and ever-alert overseer. 

The agents gather knowledge and data from the systems they monitor, and the management server collects, analyzes, and interprets this information, ever vigilant for signs of danger. 

Wazuh, whem integrated with Elastic Stack, offers seamless navigation through security alerts, enhancing visibility and threat detection. By combining it with SIEM and XDR, you can gain protection for IT assets, responding to potential security dangers.  
 
Use-cases: 

• Configuration assessment 
• Malware detection 
• File integrity monitoring 
• Threat hunting 
• Log data analysis 
• Vulnerability detection 
• Incident response 
• Regulatory compliance 
• IT hygiene 
• Containers security 
• Posture management 
• Workload protection 
   

Successful DevSecOps and DevOps Integration 

The integration should be as natural as the turning of the seasons, an organic and seamless process that unfolds with time. It is not a single task to be completed and forgotten, but a continuous journey.   Though at times it may call for a shift in the very culture of the organization, such change is not forced, but arises naturally 

And as in all great works of creation, what we forge must be shaped by the needs of people. The processes we follow and the tools we wield must be chosen wisely, fitting the unique contours of our organization Only then can the integration thrive, as both DevOps and DevSecOps become not separate disciplines, but part of the same living tapestry, woven together in purpose and vision. 

Read also:

• DevOps monitoring systems
• Azure Durable Functions – the extension to Azure Functions

Despite the rapid advancement of technology and the constant release of new models and SDKs, it can be difficult for developers to know where to begin. While there are many polished end-to-end sample applications available for .NET developers to use as a reference, some may prefer to build their applications incrementally, starting with the basics and gradually adding more advanced features.

Building a console-based .NET chat app with solutions like Semantic Kernel

This post aims to guide developers in building a simple console-based .NET chat application from scratch, with minimal dependencies and fuss. The ultimate goal is to create an application that can answer questions based on both the data used to train the model and additional data provided dynamically. Each code sample provided in this post is a complete application, allowing developers to easily copy, paste, and run the code, experiment with it, and then incorporate it into their own applications for further refinement and customization.

How to get started with Semantic Kernel SDK. Learn how to use it

To begin, make sure you have .NET 8 installed, and create a simple console app.dotnet

new console -o chat-sample-app-01 --use-program-main 

cd chat-sample-app-01

This creates a new directory chat-sample-app-01 and populates it with two files: chat-sample-app-01.csproj and Program.cs. We then need to bring in one NuGet package: Microsoft.SemanticKernel.

dotnet add package Microsoft.SemanticKernel

Find out more at Microsoft Learn

Planning, orchestration, and multiple plugins

Instead of referencing specific AI-related packages such as Azure.AI.OpenAI, I have opted for an open source Semantic Kernel kit to streamline various interactions and easily switch between different implementations for faster experimentation. Semantic Kernel offers a collection of libraries that simplify working with Large Language Models (LLMs) by providing abstractions for various AI concepts, allowing for the easy substitution of different implementations. It also includes many concrete implementations of these abstractions, wrapping numerous other SDKs, and offers support for planning, orchestration, and multiple plugins. This post will explore various aspects of Semantic Kernel, but my primary focus is on its abstractions.

While I have tried to keep dependencies to a minimum for the purpose of this article, there is one more I cannot avoid: you need access to an LLM. The easiest way to get access is via either OpenAI or Azure OpenAI. For this post, I am using Azure OpenAI. You will need three pieces of information for the remainder of the post:

• Your API key and endpoint provided to you in the Azure portal
• A chat model, or to be more precise, the deployment name of your model. I use GPT-4-32k (0613), which as of this writing has a context window of 32K tokens. I’ll explain more about what it is later.
• An embedding model. I use text-embedding-3-large.

Let’s make it as easy as possible

With that out of the way, we can dive in. Believe it or not, we can create a simple chat app in just a few lines of code. Copy and paste this into your Program.cs:

using Microsoft.SemanticKernel;

namespace chatSampleApp01
{
    class Program
    {
        static string deploymentName = Environment.GetEnvironmentVariable("AI:OpenAI:DeploymentName")!;
        static string endpoint = Environment.GetEnvironmentVariable("AI:OpenAI:Endpoint")!;
        static string apiKey = Environment.GetEnvironmentVariable("AI:OpenAI:APIKey")!;

        static async Task Main(string[] args)
        {
            //Initialize Semantic Kernel
            var builder = Kernel.CreateBuilder();
            builder.AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey);
            
            var kernel = builder.Build();

            //Question and Answer loop
            string question;
            while (true)
            {
                Console.Write("Me: ");
                question = Console.ReadLine()!;
                Console.Write("Mine Copilot: ");
                Console.WriteLine(await kernel.InvokePromptAsync(question));
                Console.WriteLine();
            }
        }
    }
}

To prevent accidentally revealing my API key, which should be safeguarded like a password, I have stored it in an environment variable and accessed it using GetEnvironmentVariable. Then I created a new kernel using the Semantic Kernel APIs and added an OpenAI chat completion service to it. The Microsoft.SemanticKernel package we imported earlier includes references to client support for both OpenAI and Azure OpenAI, eliminating the need for additional components to communicate with these services. With this configuration, we can now run our chat app using dotnet run, enter questions, and receive responses from the service. 

The expression await kernel.InvokePromptAsync(question) is the core of the interaction with the LLM, where it captures the user’s input and sends it to the LLM, receiving a string response in return. Semantic Kernel is equipped to handle various function types, including prompt functions for text-based AI interactions and standard .NET methods capable of executing any C# code. These functions can be triggered directly by the user, as shown in this example, or as part of a “plan” where a set of functions is provided to the LLM to formulate a strategy to achieve a specified objective. Semantic Kernel can execute these functions as per the plan (I will show it later). Additionally, some models support a “function calling” feature, which is also simplified by Semantic Kernel.

Leveraging Semantic functions: reusable prompts, dynamic input handling, and plugins

In this instance, “function” refers to the user’s input, such as the question “What is Inetum Polska?” which is then processed by the LLM through the InvokePromptAsync method. To clarify the concept of “function,” we can extract it into a separate entity using the `CreateFunctionFromPrompt“ method, allowing us to reuse the same function for multiple inputs. This approach eliminates the need to create a new function for each input, but requires a way to incorporate the user’s input into the existing function. Semantic Kernel supports this through prompt templates, which include placeholders that are filled with the appropriate variables and functions. For example, if the sample is run again with a request for the current time, the LLM will not be able to provide an answer:

To anticipate such inquiries, we can equip the LLM with the necessary information within the prompt itself. I have registered a function with the kernel that provides the current date and time. Subsequently, I created a prompt function that utilizes a prompt template to invoke this time function during the prompt’s rendering. This template also incorporates the value of the $input variable. It is possible to pass any number of arguments with arbitrary names using a KernelArguments dictionary; in this case, I have chosen to name one “input”. Functions are organized into collections known as “plugins”.

using Microsoft.SemanticKernel;

namespace chatSampleApp02
{
    class Program
    {
        static string deploymentName = Environment.GetEnvironmentVariable("AI:OpenAI:DeploymentName")!;
        static string endpoint = Environment.GetEnvironmentVariable("AI:OpenAI:Endpoint")!;
        static string apiKey = Environment.GetEnvironmentVariable("AI:OpenAI:APIKey")!;

        static async Task Main(string[] args)
        {
            //Initialize Semantic Kernel
            var builder = Kernel.CreateBuilder();
            builder.AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey);

            // Create the prompt function as part of a plugin and add it to the kernel.
            builder.Plugins.AddFromFunctions(
                pluginName: "DateTimeHelpers",
                functions: [
                    KernelFunctionFactory.CreateFromMethod(
                        ()=> $"{DateTime.UtcNow:r}", "Now", "Gets the current date and time"
                    )
                ]);

            var kernel = builder.Build();

            var kernelFunction = KernelFunctionFactory.CreateFromPrompt(
                promptTemplate: @"
                    The current date and time is {{ datetimehelpers.now }}.
                    {{ $input }}"
                );

            //Question and Answer loop
            string question;
            while (true)
            {
                Console.Write("Me: ");
                question = Console.ReadLine()!;
                Console.Write("Mine Copilot: ");
                Console.WriteLine(await kernelFunction.InvokeAsync(kernel, new() { ["input"] = question }));
                Console.WriteLine();
            }
        }
    }
}

When the function is activated, it renders the prompt by calling the previously registered ‘Now’ function and integrating its output into the prompt. Now, posing the same question yields a more comprehensive answer. 

Does LLM have memory? How to use Semantic Kernel to overcome statelessness in chat agents 

We have made significant strides: with just a few lines of code, we have crafted a basic chat agent that can field repeated questions and provide responses. Moreover, we have managed to furnish it with extra prompt information to aid in answering questions it would otherwise be unable to tackle. Yet, in doing so, we have also fashioned a chat agent devoid of memory, lacking any awareness of prior conversations: 

To remedy the statelessness of LLMs and their lack of memory, we must maintain a record of our chat history and incorporate it into each prompt request. This can be done manually by integrating the chat history into the prompt, or we can rely on Semantic Kernel to handle it for us, which in turn can depend on the clients for Azure OpenAI, OpenAI, or any other chat service. The latter approach involves using the registered IChatCompletionService to create a new chat, which is essentially a compilation of all messages. This method not only processes requests and outputs responses but also archives them into the chat history. 

using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;

namespace chatSampleApp03
{
    class Program
    {
        static string deploymentName = Environment.GetEnvironmentVariable("AI:OpenAI:DeploymentName")!;
        static string endpoint = Environment.GetEnvironmentVariable("AI:OpenAI:Endpoint")!;
        static string apiKey = Environment.GetEnvironmentVariable("AI:OpenAI:APIKey")!;

        static async Task Main(string[] args)
        {
            //Initialize Semantic Kernel
            var builder = Kernel.CreateBuilder();
            builder.AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey);
            var kernel = builder.Build();

            //Create new chat
            var chatService = kernel.GetRequiredService<IChatCompletionService>();
            var chat = new ChatHistory(
                    systemMessage: "You are an AI assistant that helps people find information."
                );

            //Question and Answer loop
            string question;
            while (true)
            {
                Console.Write("Me: ");
                question = Console.ReadLine()!;
                chat.AddUserMessage(question);

                Console.Write("Mine Copilot: ");
                var answer = await chatService.GetChatMessageContentAsync(chat);
                chat.AddAssistantMessage(answer.Content!);
                Console.WriteLine(answer);
                
                Console.WriteLine();
            }
        }
    }
}

With that chat history rendered into an appropriate prompt, we then get back much more satisfying results: 

In a practical application, it is crucial to consider various additional factors, such as the data processing limitations of language models, referred to as the “context window.” The `GPT-4-32k (0613) model that I am using here can handle ~32000 tokens, where a token can be a full word, part of a word, or a single character. Additionally, each token incurs a cost for every interaction. Therefore, when transitioning from a trial phase to full production, it becomes essential to monitor the chat history’s data volume closely and manage it by removing unnecessary parts, etc.

We can enhance the user experience by adding a small segment of code that accelerates the interaction. These large language models (LLMs) generate responses by predicting the next token, so while we have been displaying the complete response once it is fully generated, we can actually present it in real time as it is being formulated. This functionality is available in Semantic Kernel through IAsyncEnumerable, which allows for convenient integration using await foreach loops to stream the response incrementally.

Does LLM know everything? Connectors and RAG with Semantic plugins and native functions

We have now reached a point where we can pose questions and receive answers, maintain a record of these exchanges to refine future responses, and even broadcast our findings. But is our work complete? Not quite.

As it stands, the only information available to LLM for providing answers is the data it was initially trained on, plus any additional information we explicitly include in the prompt (like

the current time, as previously mentioned). Consequently, if we inquire about topics outside the LLM’s training or areas where its knowledge is lacking, the responses we receive may be unhelpful, misleading, or entirely incorrect, which are often referred to as ‘hallucinations’.

The question is about the latest C# 12 changes, which were released after this version of the GPT-4-32k (0613) model was released (November 2023 vs October 2021). The model has no information about the newest capabilities, so it does not give a reasonable answer to the first question. We need to find a way to teach it about the things the user is asking about.

We know the way to teach LLM: include the necessary information in the prompt. For instance, the Microsoft Learn articles:

• Relationships between language features and library types 
• Version and update considerations for C# developers 
• The history of C# 
• What’s new in C# 11 
• Breaking changes in Roslyn after .NET 6 all the way to .NET 7 
• What’s new in C# 12 
• Breaking changes in Roslyn after .NET 7 all the way to .NET 8 
• What’s new in C# 13 
• Breaking changes in Roslyn after .NET 8 all the way to .NET 9 
• What’s new in .NET 8 
• What’s new in the .NET 8 runtime 
• What’s new in the SDK and tooling for .NET 8 
• What’s new in containers for .NET 8 

which were published after the training of this GPT-4-32k (0613) model have a detailed section regarding the new capabilities of C# 12. By incorporating this content into the prompt, we can supply the LLM with the required knowledge. In the following example, I have expanded the previous code to download the web page content and then insert it into a user message.

This approach ensures that the LLM is provided with the latest information to assist with the query.

using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;
using System.Text;

namespace chatSampleApp05
{
    internal class Program
    {
        static string deploymentName = Environment.GetEnvironmentVariable("AI:OpenAI:DeploymentName")!;
        static string endpoint = Environment.GetEnvironmentVariable("AI:OpenAI:Endpoint")!;
        static string apiKey = Environment.GetEnvironmentVariable("AI:OpenAI:APIKey")!;

        static async Task Main(string[] args)
        {
            //Initialize Semantic Kernel
            var builder = Kernel.CreateBuilder();
            builder.AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey);
            var kernel = builder.Build();

            //Create new chat
            var chatService = kernel.GetRequiredService<IChatCompletionService>();
            var chat = new ChatHistory(
                    systemMessage: "You are an AI assistant that helps people find information."
                );

            // Download a documents and add all of its contents to our chat
            var articleList = new List<string>
            {
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/relationships-between-language-and-library.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/version-update-considerations.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-version-history.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-11.md",
                "https://raw.githubusercontent.com/dotnet/roslyn/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20DotNet%207.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-12.md",
                "https://raw.githubusercontent.com/dotnet/roslyn/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20DotNet%208.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-13.md",
                "https://raw.githubusercontent.com/dotnet/roslyn/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20DotNet%209.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/overview.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/runtime.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/sdk.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/containers.md"

            };
            
            var articleStringBuilder = new StringBuilder();

            using (var httpClient = new HttpClient())
            {
                foreach (var article in articleList) {
                    articleStringBuilder.Append(await httpClient.GetStringAsync(article));
                }

                chat.AddUserMessage($"Here's some additional information: {articleStringBuilder.ToString()}");
            }

            string question;
            StringBuilder stringBuilder = new StringBuilder();

            //Question and Answer loop
            while (true)
            {
                Console.Write("Me: ");
                question = Console.ReadLine()!;
                chat.AddUserMessage(question);

                stringBuilder.Clear();
                Console.Write("Mine Copilot: ");

                await foreach (var message in chatService.GetStreamingChatMessageContentsAsync(chat))
                {
                    Console.Write(message);
                    stringBuilder.Append(message.Content);
                }
                Console.WriteLine();
                chat.AddAssistantMessage(stringBuilder.ToString());
                Console.WriteLine();
            }
        }
    }
}

and the result?

We have gone over the context window almost twice without adding any history to the conversation. We obviously need to include less information, but still need to ensure it is relevant information. RAG will help us… 

Interested in building AI-powered apps? Connect with Marek Dobkowski to explore Microsoft solutions that simplify your workflow and enhance productivity. Schedule a meeting

Using RAG

Retrieval Augmented Generation (RAG) essentially means looking up relevant information and incorporating it into the prompt. Instead of including all possible information in the prompt, we index the additional information we care about. When a question is asked, we use that question to find the most relevant indexed content and add just that specific content to the prompt. To facilitate this process, we need embeddings. 

An embedding can be thought of as a vector (array) of floating-point values that represents the content and its semantic meaning. We can use a model specifically designed for embeddings to generate such a vector for a given input, and then store both the vector and the original text in a database. Later, when a question is posed, we can process that question through the same model to produce a vector, which we then use to find the most relevant embeddings in our database. We are not necessarily looking for exact matches, but rather for sufficiently similar ones. The term ‘close’ here is quite literal, as the lookups are typically performed using distance measures like cosine similarity. For instance, consider this program: 

using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Embeddings;
using System.Numerics.Tensors;

namespace chatSampleApp06
{
    internal class Program
    {
        static string embeddingDeploymentName = Environment.GetEnvironmentVariable("AI:OpenAI:EmbeddingDeploymentName")!;
        static string endpoint = Environment.GetEnvironmentVariable("AI:OpenAI:Endpoint")!;
        static string apiKey = Environment.GetEnvironmentVariable("AI:OpenAI:APIKey")!;

        static async Task Main(string[] args)
        {
            //Experimental
            #pragma warning disable SKEXP0010,SKEXP0001

            //Initialize Semantic Kernel
            var builder = Kernel.CreateBuilder();

            builder.AddAzureOpenAITextEmbeddingGeneration(embeddingDeploymentName, endpoint, apiKey);

            var kernel = builder.Build();

            var input = "What is a reptile?";
            var examples = new string[]
            {
                "What is a reptile?",
                "¿Qué es un reptil?",
                "Was ist ein Reptil?",
                "A turtle is a reptile.",
                "Eidechse ist ein Beispiel für Reptilien.",
                "Crocodiles, lizards, snakes, and turtles are all examples.",
                "A frog is green.",
                "A grass is green.",
                "A cat is a mammal.",
                "A dog is a man's best friend.",
                "My best friend is Mike.",
                "I'm working at Inetum Polska since 2013."
            };

            // Generate embeddings for each piece of text
            var embeddingGenerator = kernel.GetRequiredService<ITextEmbeddingGenerationService>();
            var inputEmbedding = (await embeddingGenerator.GenerateEmbeddingsAsync([input])).First();

            var exampleEmbeddings = (await embeddingGenerator.GenerateEmbeddingsAsync(examples)).ToArray();
            var similarities = new List<Tuple<float, string>>();

            // Print the cosine similarity between the input and each example
            for (int i = 0; i < exampleEmbeddings.Length; i++)
            {
                similarities.Add(
                    new Tuple<float, string>(
                        TensorPrimitives.CosineSimilarity(exampleEmbeddings[i].Span, inputEmbedding.Span), 
                        examples[i]));
            }

            similarities.Sort((x,y) => y.Item1.CompareTo(x.Item1));

            Console.WriteLine("Similarity\tExample");
            foreach (var similarity in similarities) {
                Console.WriteLine($"{similarity.Item1:F6}\t{similarity.Item2}");
            }
            Console.ReadLine();
        }
    }
}

This process utilizes the AzureOpenAI embedding generation service to obtain an embedding vector (using the text-embedding-3-large model mentioned earlier in the post) for both an input and several other pieces of text. It then compares the resulting embedding for the input with the embeddings of those other texts, sorts the results based on similarity, and prints them out. 

Let’s incorporate this concept into the chat app. In this round, I have augmented the previous chat example with a few things: 

• In order for Semantic Kernel to handle the embedding generation through its abstractions, we need to include its Memory package. Please note the –prerelease flag, as this is an evolving area. While some Semantic Kernel components are stable, others are still in development and therefore marked as prerelease. 

dotnet add package Microsoft.SemanticKernel.Plugins.Memory --prerelease 

• Next, I need to create an ISemanticTextMemory for querying. I achieved this by using MemoryBuilder to combine an embeddings generator with a database. I specified the Azure OpenAI service as my embeddings generator using the WithAzureTextEmbeddingGenerationService method. For the store, I registered a VolatileMemoryStore instance using the WithMemoryStore method. Although we will change this later, it will suffice for now. VolatileMemoryStore is essentially an implementation of Semantic Kernel’s IMemoryStore abstraction that wraps an in-memory dictionary. 
• I downloaded the text and used Semantic Kernel’s TextChunker to break it into pieces. Then, I saved each piece to the memory store using `SaveInformationAsync“. This process generates an embedding for the text and stores the resulting vector along with the input text in the dictionary. 
• When it is time to ask a question, instead of just adding the question to the chat history and submitting it, we first use the question to perform a SearchAsync on the memory store. This generates an embedding vector for the question and searches the store for the closest vectors. I have it return the three closest matches, append the associated text together, add the results to the chat history, and submit it. After submitting the request, I remove this additional context from the chat history to avoid sending it again in subsequent requests, as it can consume much of the allowed context window. 

And full source code… 

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.Connectors.OpenAI;
using Microsoft.SemanticKernel.Memory;
using Microsoft.SemanticKernel.Text;
using System.Net;
using System.Text;
using System.Text.RegularExpressions;

namespace chatSampleApp07
{
    internal class Program
    {
        static string deploymentName = Environment.GetEnvironmentVariable("AI:OpenAI:DeploymentName")!;
        static string embeddingDeploymentName = Environment.GetEnvironmentVariable("AI:OpenAI:EmbeddingDeploymentName")!;
        static string endpoint = Environment.GetEnvironmentVariable("AI:OpenAI:Endpoint")!;
        static string apiKey = Environment.GetEnvironmentVariable("AI:OpenAI:APIKey")!;

        static async Task Main(string[] args)
        {
            //Experimental
            #pragma warning disable SKEXP0001, SKEXP0010, SKEXP0050

            //Initialize Semantic Kernel
            var builder = Kernel.CreateBuilder();
            builder.AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey);
            var kernel = builder.Build();

            //Initialize Memory Builder
            var memoryBuilder = new MemoryBuilder()
                .WithMemoryStore(new VolatileMemoryStore())
                .WithAzureOpenAITextEmbeddingGeneration(
                    deploymentName: embeddingDeploymentName,
                    endpoint: endpoint,
                    apiKey: apiKey
                );

            var memory = memoryBuilder.Build();

            // Download a documents and add all of its contents to our chat
            var articleList = new List<string>
            {
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/relationships-between-language-and-library.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/version-update-considerations.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-version-history.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-11.md",
                "https://raw.githubusercontent.com/dotnet/roslyn/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20DotNet%207.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-12.md",
                "https://raw.githubusercontent.com/dotnet/roslyn/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20DotNet%208.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-13.md",
                "https://raw.githubusercontent.com/dotnet/roslyn/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20DotNet%209.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/overview.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/runtime.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/sdk.md",
                "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/containers.md"
            };

            var collectionName = "microsoft-news";

            using (var httpClient = new HttpClient())
            {
                var allParagraphs = new List<string>();

                foreach (var article in articleList)
                {
                    var content = await httpClient.GetStringAsync(article);
                    var lines = TextChunker.SplitPlainTextLines(content, 64);
                    var paragraphs = TextChunker.SplitPlainTextParagraphs(lines, 512);

                    allParagraphs.AddRange(paragraphs);
                }

                for (var i = 0; i < allParagraphs.Count; i++)
                {
                    await memory.SaveInformationAsync(collectionName, allParagraphs[i], $"paragraph[{i}]");
                }
            }

            //Create new chat
            var chatService = kernel.GetRequiredService<IChatCompletionService>();
            var chat = new ChatHistory(
                    systemMessage: "You are an AI assistant that helps people find information."
                );

            string question;
            var responseBuilder = new StringBuilder();
            var contextBuilder = new StringBuilder();

            //Question and Answer loop
            while (true)
            {
                Console.Write("Me: ");
                question = Console.ReadLine()!;

                await foreach (var result in memory.SearchAsync(collectionName, question, limit: 3))
                {
                    contextBuilder.AppendLine(result.Metadata.Text);
                }

                var contextToRemove = -1;
                if (contextBuilder.Length > 0)
                {
                    contextBuilder.Insert(0, "Here's some additional information: ");
                    contextToRemove = chat.Count;
                    chat.AddUserMessage(contextBuilder.ToString());
                }

                chat.AddUserMessage(question);

                responseBuilder.Clear();
                Console.Write("Mine Copilot: ");

                await foreach (var message in chatService.GetStreamingChatMessageContentsAsync(chat,null, kernel))
                {
                    Console.Write(message);
                    responseBuilder.Append(message.Content);
                }

                Console.WriteLine();
                chat.AddAssistantMessage(responseBuilder.ToString());

                if (contextToRemove >= 0)
                {
                    chat.RemoveAt(contextToRemove);
                }

                Console.WriteLine();
            }
        }
    }
}

The text chunking code divided the documents into 104 “paragraphs” resulting in 104 embeddings being created and stored in the database. The exciting part is that with all these embeddings, when we pose our question, the database retrieves the most relevant material and adds the additional text to the prompt. Now, when we ask the same questions as before, we receive a much more helpful and accurate response:

Storing memories

Naturally, we do not want to reindex all documents every time the application starts. Imagine this was a public website facilitating chats with thousands of users and hundreds of documents reindexing all content each time – the application restart process would not only be time-consuming but also unnecessarily expensive. For instance, the Azure OpenAI embedding model I use costs €0.000121 per 1,000 tokens (Azure OpenAI Service pricing), meaning indexing just those documents costs a couple of cents (but remember: “scale makes a difference”).

Therefore, we should switch to using persistent storage. Semantic Kernel provides various IMemoryStore implementations, and we can easily switch to one that persists in the results. For example, let’s switch to one based on Sqlite. To do this, we need another NuGet package:

dotnet add package Microsoft.SemanticKernel.Connectors.Sqlite --prerelease

and with that, we can change just one line of code to switch from the VolatileMemoryStore:

.WithMemoryStore(new VolatileMemoryStore())

to the SqliteMemoryStore:

.WithMemoryStore(await SqliteMemoryStore.ConnectAsync("data\\rag-data.db"))

Sqlite is an embedded SQL database engine that operates within the same process and stores its data in standard disk files. In this case, it will connect to a rag-data.db file, creating it if it does not already exist. However, if we were to run this, we would still end up generating the embeddings again, as our previous example did not include a check to see if the data already existed. Therefore, our final step is to add a guard to prevent this redundant work.

var collectionName = "microsoft-news";
var collections = await memory.GetCollectionsAsync();

if (!collections.Contains(collectionName))
{                       
    ... // same code as before to download and process the documents
}
else
{
    Console.WriteLine($"Found '{collectionName}' in RAG database");
}

You get the idea. Here is the complete version using Sqlite:

using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.Connectors.OpenAI;
using Microsoft.SemanticKernel.Connectors.Sqlite;
using Microsoft.SemanticKernel.Memory;
using Microsoft.SemanticKernel.Text;
using System.Text;

namespace chatSampleApp08
{
    internal class Program
    {
        static string deploymentName = Environment.GetEnvironmentVariable("AI:OpenAI:DeploymentName")!;
        static string embeddingDeploymentName = Environment.GetEnvironmentVariable("AI:OpenAI:EmbeddingDeploymentName")!;
        static string endpoint = Environment.GetEnvironmentVariable("AI:OpenAI:Endpoint")!;
        static string apiKey = Environment.GetEnvironmentVariable("AI:OpenAI:APIKey")!;

        static async Task Main(string[] args)
        {
            //Experimental
            #pragma warning disable SKEXP0001, SKEXP0010, SKEXP0020, SKEXP0050

            //Initialize Semantic Kernel
            var builder = Kernel.CreateBuilder();
            builder.AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey);
            var kernel = builder.Build();

            //Initialize Memory Builder
            var memoryBuilder = new MemoryBuilder()
                .WithMemoryStore(await SqliteMemoryStore.ConnectAsync("data\\rag-data.db"))
                .WithAzureOpenAITextEmbeddingGeneration(
                    deploymentName: embeddingDeploymentName,
                    endpoint: endpoint,
                    apiKey: apiKey
                );

            var memory = memoryBuilder.Build();

            var collectionName = "microsoft-news";
            var collections = await memory.GetCollectionsAsync();
            
            if (!collections.Contains(collectionName))
            {                       
                // Download a documents and add all of its contents to our chat
                var articleList = new List<string>
                {
                    "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/relationships-between-language-and-library.md",
                    "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/version-update-considerations.md",
                    "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-version-history.md",
                    "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-11.md",
                    "https://raw.githubusercontent.com/dotnet/roslyn/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20DotNet%207.md",
                    "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-12.md",
                    "https://raw.githubusercontent.com/dotnet/roslyn/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20DotNet%208.md",
                    "https://raw.githubusercontent.com/dotnet/docs/main/docs/csharp/whats-new/csharp-13.md",
                    "https://raw.githubusercontent.com/dotnet/roslyn/main/docs/compilers/CSharp/Compiler%20Breaking%20Changes%20-%20DotNet%209.md",
                    "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/overview.md",
                    "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/runtime.md",
                    "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/sdk.md",
                    "https://raw.githubusercontent.com/dotnet/docs/main/docs/core/whats-new/dotnet-8/containers.md"
                };

                using (var httpClient = new HttpClient())
                {
                    var allParagraphs = new List<string>();

                    foreach (var article in articleList)
                    {
                        var content = await httpClient.GetStringAsync(article);
                        var lines = TextChunker.SplitPlainTextLines(content, 64);
                        var paragraphs = TextChunker.SplitPlainTextParagraphs(lines, 512);

                        allParagraphs.AddRange(paragraphs);
                    }

                    for (var i = 0; i < allParagraphs.Count; i++)
                    {
                        await memory.SaveInformationAsync(collectionName, allParagraphs[i], $"paragraph[{i}]");
                    }
                }
            }
            else
            {
                Console.WriteLine($"Found '{collectionName}' in RAG database");
            }

            //Create new chat
            var chatService = kernel.GetRequiredService<IChatCompletionService>();
            var chat = new ChatHistory(
                    systemMessage: "You are an AI assistant that helps people find information."
                );

            string question;
            var responseBuilder = new StringBuilder();
            var contextBuilder = new StringBuilder();

            //Question and Answer loop
            while (true)
            {
                Console.Write("Me: ");
                question = Console.ReadLine()!;

                await foreach (var result in memory.SearchAsync(collectionName, question, limit: 3))
                {
                    contextBuilder.AppendLine(result.Metadata.Text);
                }

                var contextToRemove = -1;
                if (contextBuilder.Length > 0)
                {
                    contextBuilder.Insert(0, "Here's some additional information: ");
                    contextToRemove = chat.Count;
                    chat.AddUserMessage(contextBuilder.ToString());
                }

                chat.AddUserMessage(question);

                responseBuilder.Clear();
                Console.Write("Mine Copilot: ");

                await foreach (var message in chatService.GetStreamingChatMessageContentsAsync(chat, null, kernel))
                {
                    Console.Write(message);
                    responseBuilder.Append(message.Content);
                }

                Console.WriteLine();
                chat.AddAssistantMessage(responseBuilder.ToString());

                if (contextToRemove >= 0)
                {
                    chat.RemoveAt(contextToRemove);
                }

                Console.WriteLine();
            }
        }
    }
}

Now, when we run it, the first invocation will still index everything, but after that, the data will already be indexed:

and subsequent invocations are able to simply use it.

While Sqlite is a fantastic tool, it is not specifically optimized for performing these types of searches. In fact, the code for this SqliteMemoryStore in SK merely enumerates the entire database and performs a `CosineSimilarity“ check on each entry.

// from: https://github.com/microsoft/semantic-kernel/blob/9264b3e0b42e184b7e9e8b2a073d8a721c4af92a/dotnet/src/Connectors/Connectors.Memory.Sqlite/SqliteMemoryStore.cs#L135

await foreach (var record in this.GetAllAsync(collectionName, cancellationToken).ConfigureAwait(false))
{
    if (record is not null)
    {
        double similarity = TensorPrimitives.CosineSimilarity(embedding.Span, record.Embedding.Span);
        ...
    }
}

For real scale and the ability to share data across multiple frontends, we need a dedicated ‘vector database’ designed for storing and searching embeddings. There are many such vector databases available now, including Azure AI Search, Chroma, Milvus, Pinecone,

Qdrant, Weaviate, and many more… We can easily set one of these up, change our `WithMemoryStore“ call to use the appropriate connector, and we are ready to go. Let’s proceed with that. For this example, I have chosen Azure AI Search.

I add the relevant Semantic Kernel “connector” to my project:

dotnet add package Microsoft.SemanticKernel.Connectors.AzureAISerach --prerelease 

and then add a couple of lines: 

static string azureAISearchEndpoint = Environment.GetEnvironmentVariable("AI:AzureAISearch:Endpoint")!;
static string azureAISearchApiKey = Environment.GetEnvironmentVariable("AI:AzureAISearch:APIKey")!;

change from:

.WithMemoryStore(await SqliteMemoryStore.ConnectAsync("data\\rag-data.db"))

to:

.WithMemoryStore(new AzureAISearchMemoryStore(azureAISearchEndpoint, azureAISearchApiKey)) 

And that’s it! The application works as before but much faster.  

Enhancing your Semantic integration: use the Semantic best practices and considerations

Wow! Clearly, I have left out many crucial details that any real application would need to address. For instance, how should the data being indexed be cleaned, normalized, and chunked? How should errors be managed? How can we limit the amount of data sent with each request, such as restricting chat history or the size of the found embeddings? How to make the application more secure (API Key vs Managed Identity)? Which service is the best for storing all the information?

And there are many other considerations, including making the UI much more attractive than my basic Console.WriteLine calls. Despite these missing details, I hope it is evident that you can start integrating this kind of functionality into your applications right away.

Also read: Introduction to Python Pandas Libraries

Data Warehouse vs Data Lake vs Data Mesh 

Initially (in the late 1980’s), the destination of data was a data warehouse. It was powered by ETL processes, which means that data was extracted from the source systems, transformed into the target form, and loaded into the target destination. What has been constantly changing over the past years is the target form of data to be analyzed.   

What is Data Lake 

Definition and purpose 

The Data Lake approach is a centralized repository that allows companies to store all structured and unstructured data at any scale. Unlike traditional databases that store data in tables and columns, Data Lake stores data in its raw form until it is needed for analysis. This means you can store everything from raw logs, photos, videos, to processed datasets in Data Lake. 

Key characteristics 

1. Scalability: Data Lake is highly scalable and allows you to store huge amounts of data from different sources. Using cloud solutions, it offers unlimited storage space for your data.  
2. Flexibility: As data is stored in raw format, Data Lake offers flexibility by allowing data scientists and data analysts to process data in different ways depending on their needs. 
3. Cost-effectiveness: Formats such as Parquet organizes data by columns rather than rows, which is very efficient for analytical queries that only access a subset of columns. This can lead to a significant reduction in storage requirements and costs.  
4. Data Variety: Data Lake can handle structured, semi-structured, and unstructured data, making it ideal for organizations that use a wide variety of data sources.  

Challenges in data governance in distributed data structure 

Data Lake architecture is not without the challenges. Lack of structure can lead to the so-called data swamp, in which data management and their effective use becomes difficult. Without proper management, data quality can worsen, making it difficult to find the data you need for analysis and treat it as trustworthy. 

What is Data Mesh?  

Definition and purpose 

Data Mesh is a relatively new concept that decentralizes data management by treating data as a product and assigning ownership to individual business domains. In the Data Mesh architecture, each domain – e.g. sales, marketing or finance – manages its own flows and data sets. This approach allows for more flexible and autonomous data management. 

Key characteristics 

1. Domain-oriented design: Data Mesh transfers data ownership to the domain teams that are closest to that data, making it easier to ensure its high quality.  
2. Scalability: Data Mesh provides scalability through decentralization. Each domain can be scaled independently, reducing bottlenecks and improving work efficiency on data. 
3. Data as a product: This principle emphasizes that data should be treated as a product, and dedicated teams care about its quality, availability and usability. 
4. Interoperability: Data Mesh promotes interoperability between different domains, ensuring that data can be easily shared and used across the organization. 

Challenges in Data Mesh strategy 

The implementation of the Data Mesh architecture requires significant cultural and organizational changes. This requires domain teams to have the necessary skills and resources to manage their own data, which can be challenging in organizations, in 

 which data management has traditionally been centralized. Moreover, a decentralized nature can sometimes lead to inconsistencies without proper management. 

Data Lake vs Data Mesh – key differences

Centralization vs decentralization 

The most significant difference between Data Lake and Data Mesh lies in the approach of these architectures to data management. Data Lake is a centralized repository. This means that all data is stored in one place and managed by a central team. Such centralization can facilitate the execution of standards and ensure consistency across the organization. 

Data Mesh, on the other hand, is about decentralization. Each domain in the organization is responsible for its own data, i.e. data management and ownership are distributed among different teams. This can lead to greater flexibility and specialization in the field, but requires robust management to avoid fragmentation. 

Data owners 

In the Data Lake architecture, the central IT or data team usually owns and manages the data. This can sometimes result in so-called bottlenecks, because the central team may not have the domain-specific knowledge needed to effectively manage data. 

On the other hand, Data Mesh assigns data ownership to domain teams. This not only empowers those who are closest to the data, but also ensures that the data is managed with the specific needs of the domain in mind. However, this requires domain teams to have the necessary skills and resources for data management. 

Data Management in Data Mesh and Data Lake 

Data management is a key factor in any data architecture. In Data Lake, management is often easier to enforce because all data is centralized. However, this can lead to rigid structures that may not meet the needs of all users. 

Data Mesh requires a management approach where each domain is responsible for managing its own data. This can lead to more flexible and domain-specific management policies, but it also calls for a strong, overarching management framework to ensure consistency and interoperability across the organization. 

Scalability 

Both Data Lake and Data Mesh offer scalability, but they do so in different ways. Data Lake is scaling by adding more space and computing power to a centralized repository. This can be cost-effective, but it can also impact performance negatively as the system develops.  

Scaling in Data Mesh is done by distributing data management across different domains. Each domain can be scaled independently, reducing the risk of bottlenecks and enabling more flexible and responsive data management. However, in this approach management can be more complicated because it requires coordination between various domains. 

Flexibility and agility 

Flexibility is another area in which these architectures differ significantly. Data Lake provides flexibility in terms of the types of data it can store and way of processing. However, its centralized nature can sometimes limit flexibility, as changes or new requirements may involve the need for a central team to process them. 

Data Mesh is inherently more flexible because each domain can manage its own data as needed. This can have a positive impact on innovation and enable the adaptation of data management to business needs. However, this flexibility comes at the expense of increased complexity and the need for robust coordination. 

Read also:

• Data-Driven Decision Making
• What is Data Quality?

Data Mesh integration vs. Data Lake Best Practices

Data Integration in Data Lake architecture 

In Data Lake, data integration typically involves centralizing data from different sources into a single repository. This process can be complex, especially for data from different systems that use different formats, schemas, and protocols. Best practices for data integration in Data Lake include: 

• ETL/ELT Processes:  Extracting, Transforming, Loading (ETL) or Extracting, Loading, Transforming (ELT) are critical in cleaning, transforming, and organizing the data being transferred to Data Lake. Properly designed ETL flows allow for usability and consistency of data. 
• Schema-on-Read: Unlike traditional databases, Data Lake often uses the “Schema-on-Read” approach, which means that data is stored in raw form and transformed into the required schema during a read or query. This provides flexibility, efficiency and usability, but involves the need for thorough design. 
• Data Catalog: Implementing a data catalog is essential to managing the massive volumes of data in Data Lake. Data Catalog  helps users to discover, understand, and trust the data being shared for analysis. 
• Data management: To maintain data quality, security, and compliance in a centralized environment, a robust governance framework is essential. This includes metadata management, access control, and audit mechanisms. 

Data Integration in Data Mesh architecture 

Data integration in the Data Mesh architecture is more decentralized and domain-based. Each business domain is responsible for integrating its own data to create standalone, interoperable data products.  
Best practices for data integration in Data Mesh include: 

• Domain-Driven Design: Data integration processes should be tailored to the specific needs of each domain. This includes designing data flows that are consistent with the workflows and data models within a given domain. 
• APIs and Interfaces: To ensure interoperability between domains, data products should be made available through well-defined APIs or data interfaces. This makes it easier to share data and integrate it across organization. 
• Federated Governance: While domains are responsible for their own data, this governance model ensures that integration standards, data quality, and compliance requirements are met across all domains. This often involves the need for a central management team to work closely with domain teams to define and enforce standards. 
• Event-Driven Architecture: Data Mesh often uses the Event-Driven architecture, where changes in one domain’s data can cause updates in another. This keeps your data in sync and up-to-date across the organization. 

What is the difference between Data Mesh vs Data Lake data products?

What is a data product 

A data product is a resource created from data that brings value to users, usually by providing practical insights, enabling decision making or automating processes. Unlike raw data, which is often unstructured and not immediately useful, data products are designed to be used directly by end users or applications. 

Data Products in Data Lake 

In a Data Lake architecture, data products are typically created and maintained by a centralized data team. These products often come from raw data stored in Data Lake, which is transformed and processed to meet the needs of different business units. The responsible team manages the entire lifecycle of these data products centrally, from their taking them for processing, storage, and eventual retrieval by users. 

While this centralized approach can ensure consistency and uniformity of data across the organization, it can also lead to delays in data delivery. A centralized team may lack the domain expertise required to create highly specialized data products, leading to generic, non-specialized solutions that may not fully meet the needs of specific business units. 

Data Products in Data Mesh 

Data Mesh treats data as a product from the very beginning. Each domain is responsible for creating and maintaining its own data products. These products are designed with specific users and use cases in mind, making them more useful for users and the business unit that created them. 

This decentralized approach allows for more flexibility and innovation as domain teams can quickly change their data products to meet changing business needs. However, it also requires robust governance and cross-domain collaboration to ensure the interoperability of data products and enable the overall needs of the organization to be met. 

The key difference lies in the approach to design and ownership. In Data Lake, data products are often reactive and standardized, created after data collection. In Data Mesh, data products are proactive and specialized, designed as part of the data lifecycle in the domain. 

When to choose Data Lake

Data Lake is a good choice for organizations that: 

• Need a centralized repository of all their data, especially in the case of big amounts of various data, the so-called “single source of truth”. 
• Use centralized teams of data scientists or analysts who need access to raw data for exploratory data analysis and the use of Machine Learning. 
• Prefer a more traditional, centralized approach to data management. 
• Have an experienced data management team that is able to handle the complexity of Data Lake without falling into the “data swamp” trap. 

Also read:

• Data Mesh vs Data Fabric
• 5 most popular data analytics tools 

When to choose Data Mesh

The Data Mesh architecture may be more appropriate for organizations that: 

• Have a complex organizational structure with multiple domains or business units that require autonomy in data management. 
• Strive to enable domain teams to take responsibility for their data, adapting their management to business needs. 
• Are looking for more flexibility and responsiveness in their data management practices. 
• Have the right resources and organizational culture to handle the transition to decentralized data management. 
• Need for high quality and specialized data products. 

Elevate Your Data Strategy Our customized Data solutions align with your business objectives. Consult with Marek Czachorowski, Head of Data and AI Solutions, for expert guidance. Schedule a meeting

Summary 

Data Lake is a traditional approach that allows for storing data in a centralized location. It is characterized by scalability but comes with certain challenges in data governance. Data Mesh is a decentralized approach in which data owners from various data domains are responsible for specific data, and its quality, making it available for data consumers from other domains.  

The choice between Data Lake and Data Mesh depends mostly on the structure of the organization, its needs, and its readiness for change. Data Lake offers a centralized, flexible, and cost-effective solution for storing large amounts of various data. However,  good data management is required. Data Mesh, although more complex and requiring significant changes in the culture of the organization, offers a decentralized approach that better adapts data management to business needs and allows for greater flexibility and scalability.  

Self-Service Business Intelligence (SSBI) is a modern approach that allows end-users to analyze data and generate reports without the need to involve IT specialists or data analysts. It is designed for business team members who do not have advanced technical skills but need access to data to make informed business decisions. Read our article to find out what makes it different from the traditional approach, how to implement SSBI and what are the possible software solutions for your business.

Traditional BI and self-service BI – what is the difference?

The traditional approach in reporting means that users order reports that the IT department creates. It’s engaging and quite time-consuming, especially if reports are needed right away.

Modern BI solutions allow you to gain insights from a variety of data sources, and access data for self-service analytics. Using self-service is a convenient way to ease the burden on business analyst departments. However – I advise against being over-enthusiastic. It’s worth bearing in mind that a self-service BI tool doesn’t do everything by itself. To prepare more advanced analyses and reports, you need to involve BI specialists or get training, for example, under the guidance of a solution provider. 

What is Self-Service BI?

Self-Service Business Intelligence is a set of tools and technologies that enable business users to independently collect, process, and analyze data. With SSBI, one can create their own reports, dashboards, and data visualizations, which speeds up the decision-making process. These tools are typically user-friendly and offer intuitive interfaces, minimizing the need for technical support. 

Also read:

• Data Quality Importance in Data Governance
• Data-Driven Decision Making
• Data Mesh vs Data Lake

Applications of Self-Service Business Intelligence

SSBI is used in many fields, such as:

• Marketing: Analyzing marketing campaigns, customer segmentation, monitoring ROI.
• Finance: Cost analysis, revenue forecasting, monitoring financial performance.
• Sales: Monitoring sales results, analyzing customer behaviors, predicting market trends.
• Operations: Optimizing processes, monitoring operational performance, analyzing supply chains.
• HR: Analyzing personal data, monitoring employment metrics, assessing employee performance.

Benefits of Self-Service BI

SSBI offers many benefits, including:

• Faster decision-making: Users have immediate access to the necessary data, allowing for quicker decisions.
• Cost reduction: Less involvement of IT teams in creating reports and analyzing data.
• Greater flexibility: Users can customize reports and analyses to meet their individual needs.
• Improved data quality: Users can continuously verify and correct data, which leads to higher-quality information.
• Increased innovation: Access to SSBI tools stimulates creativity and innovation in data analysis.

Risks of Self-Service Business Intelligence

Despite many benefits, SSBI also involves certain risks:

• Data security: Lack of proper security measures can lead to data leaks or unauthorized access.
• Data quality: Users may not have the necessary skills to correctly interpret data, leading to erroneous conclusions.
• Improper use of tools: Without proper training, users may use SSBI tools incorrectly.
• Information overload: Excessive data can lead to difficulties in analysis and interpretation.

Implementation of Self-Service Business Intelligence and BI best practices

The implementation process of SSBI requires a thoughtful approach, encompassing several key stages:

1. Data preparation: Identifying and preparing data that will be easy to analyze is crucial. Data must be accurate, consistent, and up-to-date. The best solution is a central data repository that users can access. Popular tools like Power BI can use this prepared data, facilitating analysis.
2. Selection of SSBI tools: Many SSBI tools are available on the market, such as Power BI, Tableau, or Qlik. It is important to choose a tool that best meets the organization’s needs and is easy for end-users to use.
3. User training: To effectively use SSBI tools, users need to be trained. Training should cover the basics of using tools, creating reports and dashboards, and data interpretation. This way, they will be able to prepare reports and analyses themselves without waiting for further support.
4. Monitoring and support: After implementing the SSBI tool, it is important to monitor its use and provide support to users. Regular data updates, training sessions, and technical support help maintain the high quality and efficiency of SSBI tools

Elevate Your Data Strategy

Our customized Data solutions align with your business objectives. Consult with Marek Czachorowski, Head of Data and AI Solutions, for expert guidance.

Schedule a meeting

Examples of Self-Service BI tools

Some traditional BI tools include complex BI software that requires specialized knowledge to operate, while self-service BI platforms are designed to be easy to use for non-technical users. These self-service BI tools provide a way for users to access and analyze business data without engaging BI specialists. Below we present the list of popular Self-Service tools for data analytics – get to know those BI tools capabilities in detail to choose the best self-service tool.

Tableau

Tableau is a self-service tool with a drag-and-drop function that enables users to easily create and manage visualizations and dashboards. The system is popular thanks to its simple interphase and intuitiveness.

Microsoft Power BI

Microsoft Power BI is integrated with the Azure cloud platform. This solution can be easily handled by anyone. Power BI comes with modern reports and dashboards made available on all devices. Data for reports is collected from various sources to create multidimensional data models, making it possible to conduct analyses in real time. 

Qlik 

Qlik analytics engine allows users to explore data and easily create sharable reports. Qlik Sense is a powerful platform that, thanks to AI and ML  capabilities, allows business users to use the full potential of the data.

Read more:

• Exploring the power of Qlik NPrinting
• Qlik + Talend – the future of data integration

The future of AI-powered BI and self-service BI

The future of SSBI looks promising, especially in the context of integration with Artificial Intelligence (AI) technologies. AI can significantly improve the functionality of SSBI tools by:

• Automation of analyses: AI can automatically analyze data and generate reports, further speeding up the decision-making process.
• Predictive analyses: AI can predict future trends and behaviors based on historical data, allowing for better planning and strategy.
• Personalization: AI can adapt SSBI tools to individual user needs, offering personalized ad hoc analysis and reports.
• Pattern recognition: AI can identify hidden patterns and correlations in data that are difficult for humans to detect.

Also read:

• What is Data Quality? Data Management Best Practices & More
• Data Mesh vs Data Fabric
• Data Warehouses management

BI strategy for enhanced analytics

Choosing the SSBI tool alone is not enough. To become a data-driven organization, you need a well-thought-out strategy. Which data will be analyzed? This is where the Master Data Management approach will help. Who will be responsible for the data project? Are you going to conduct it alone or in partnership with a Data Management services provider? 

Implementation of the self-service bi strategy should include the following steps: 

– Selecting a Chief Data Officer (CDO) to support the initiative.  
– Choosing a platform – if you are not sure which tool will meet your expectations, decide to work with a partner company. 
– Engaging users – representatives of different departments should support the implementation plan and be involved at every stage of the project. 
– Creating a Business Intelligence team. Typically, a BI team consists of:
small teams: analyst, developer, tester. Larger project teams: Release manager, administrator, team manager.  

 Summary

Self-Service Business Intelligence is a powerful tool that allows end-users to independently collect, process, and analyze data. While SSBI involves certain risks, the benefits of its application are significant. The future of SSBI, combined with AI technologies, seems even more promising, opening new possibilities in data analysis and business decision-making.

Elevate Your Data Strategy

Our customized Data solutions align with your business objectives. Consult with Marek Czachorowski, Head of Data and AI Solutions, for expert guidance.

Schedule a meeting

AI-enabled IoT. Key statistics on AI implementations

IoT – this technology is not just about smart cities, smart homes, and wearable gadgets. It is a transformative industrial advantage. In recent years, IoT has been quite hype, and we know the hype curve is high right now. IoT and AI increasingly often join forces to collect data and enable IoT devices to use their maximum potential. According to IoT Analytics Research, in 2022, only 17% of Artificial Intelligence solutions were implemented in IoT.

Future of AI-powered IoT applications

However, this state is expected to change by 2027 when nearly 50% of IoT solutions are predicted to have an AI component, with 13% being AI-based and 43% AI-augmented. That shows the real power of AI within Internet of Things solutions.   

Integration of AI and IoT for operational efficiency

Why do companies decide to integrate AI into the Internet of Things?  

So where does the huge popularity of combining AI and IoT take its roots?   

First, it is all about convenience and seeking ways to expedite operations. Humans naturally seek efficiency and convenience, often preferring to avoid performing repetitive and laborious tasks ourselves.  

Consequently, we delegate these tasks to computers, leveraging automation to handle them. By enabling automated systems to manage and control various processes, we can ensure that once we detect an issue or an opportunity, it can be addressed promptly and effectively by these systems.   

Now it is time to use AI  

Once we can control our connected devices, why not leave it to the computer to – using AI and Machine Learning techniques – analyze data generated by IoT devices and assess if something wrong is going on with those smart devices? This allows us to take appropriate actions. This way, IoT and AI-enabled data analysis ensure operational excellence.   

Real-world examples of AI and IoT implementation

Many companies that have decided to integrate AI into their IoT environment can already boast interesting solutions. Our client is a leader in green energy and wind turbine manufacturing.   They faced the challenge of operating thousands of wind turbines worldwide with only a few operators.  

Their goals and challenges   

• Centralized maintenance of worldwide wind fleet   
• Different technologies (some 25-years old)   
• Operator learning period and rotation    
• Number of devices and alarms to handle   

How did they prepare for their AI with IoT journey:

• Up to 300 built-in sensors   
• Different communication options – wired and wireless (mostly cellular)    
• Data delivery from periodical SMTP to real-time OPC-UA   

Artificial Intelligence in IoT solution  

For the client, the AI solution turned out to be the best one. It was based on sophisticated conditional rules working on a streaming analytics engine where the incoming data was compared with the enterprise data. This resulted in many alerts. So, to address that with a limited workforce, streaming analytics and integration with the enterprise systems (SAP and ServiceNow) were used. The simpler alerts of non-critical severity were managed automatically. By creating the proper work orders for the proper maintenance, human operators were relieved and could handle critical issues.   

Once the problem is resolved in the field, the status is automatically synchronized back from SAP systems to the IoT platform and the dashboard. This way, the operators know that the problems are resolved. With an autonomous approach, we reach very high productivity of the maintenance and operations teams.

Streamline Your IoT Operations Andrzej Gumieniak, our Head of Practice IoT, is here to help you navigate the complexities of IoT solutions. Book a consultation to discuss your case. Schedule a meeting

How does AI improve connectivity and decision-making in IoT systems?

For that, we can take advantage of Artificial Intelligence model assistance, pre-trained for specific use cases. This way we do not need to build up sophisticated conditional rules based on many parameters inside our business logic as AI supports us with certain routine tasks. That leads to an especially important and useful application: prescriptive maintenance. This maintenance strategy takes advantage of machine data to outline maintenance-related tasks.   

Also read: Predictive maintenance   

Machine Learning in IoT

Machine learning (ML) is playing a significant role in the Internet of Things (IoT) ecosystem. By leveraging AI and ML algorithms, companies can extract valuable insights from the vast amount of data generated by IoT sensors. Nowadays you can use IoT platforms (e.g. Cumulocity IoT) that have built-in ML components that make Machine Learning model training and implementation easier and quicker.   

AI in IoT streamlines data analytics

Using the incoming data, you can process it through analytics and predictive Machine Learning models. You no longer need to create alerts or work orders manually. Instead, you can utilize an AI advisor or a more sophisticated rules engine, taking advantage of contextual data to support issue resolution.  

IoT and AI for better alarm monitoring  

AI-enabled automation eliminates the need for operators to sift through thousands of alarms. Operators can then focus on the most challenging cases, while simple and obvious ones can be managed automatically by the central IoT platform.   

What are the benefits of AI and IoT?

• Faster issue resolution – data collected from the IoT sensors is analyzed in the instance by AI, which allows for quicker issue detection and resolution.  
• Improved security – IoT security is considered one of the biggest challenges and concerns. AI algorithms can monitor access to IoT networks and detect suspected activities.   
• Predictive & prescriptive maintenance capabilities for preventing downtime – by employing cognitive algorithms used in IoT, you can prescribe actions and instruct the maintenance staff on what to do or even issue the appropriate commands to the team automatically.   

Also read: The top challenges in IoT 

AI in IoT or Artificial Intelligence of Things?

Artificial intelligence in itself is a powerful tool. When it becomes the heart of an organism such as a network of connected IoT devices, you get an intelligent system capable of learning and making decisions. In this context, there is increasing talk of AIoT (Artificial Intelligence of Things) technology, in which IoT devices talk to each other and act independently of human intelligence.  

IoT and AI trends will become increasingly important

IoT applications increasingly leverage AI algorithms to analyze the vast amounts of data generated by industrial IoT devices. The combination of AI and IoT technologies allows for real-time data processing and the ability to make decisions based on the insights gathered in many industries now, including logistics, automotive, and manufacturing. Overall, the benefits of AI in IoT are vast and continue to expand as AI and IoT work together. We can expect this trend to gather pace, as the number of connected devices worldwide in 2025 is forecasted to hit 25 bn, according to IoT Analytics.