Monitoring and managing the performance of your data warehouse is critical to the overall health of your data estate. With the increase in data and query velocities, tracking query metrics pertaining to usage frequency, resource consumption, or regressions can impact your ability to efficiently draw meaningful insights from your data.
To increase your efficiency, we’re excited to reveal the preview of Query Store for Azure SQL Data Warehouse for both our Gen1 and Gen2 offers. Query Store is designed to help you with query performance troubleshooting by tracking queries, query plans, runtime statistics, and query history to help you monitor the activity and performance of your data warehouse. Query Store is a set of internal stores and Dynamic Management Views (DMVs) that allow you to:
Welcome to the Cloud Commercial Communities monthly webinar and podcast update. Each month the team focuses on core programs, updates, trends, and technologies that Microsoft partners and customers need to know to increase success using Azure and Dynamics. Make sure you catch a live webinar and participate in live Q&A. If you miss a session, you can review it on demand. Also consider subscribing to the industry podcasts to keep up to date with industry news.
Happening in December
Transform Your Business with AI at Microsoft
December 4, 2018 at 11:00 AM Pacific Time
Explore AI industry trends and how the Microsoft AI platform can empower your business processes with Azure AI Services including bots, cognitive services, and Azure machine learning.
Azure Marketplace and AppSource Pu... Read More
This blog continues our coverage of the solution guide published by Microsoft’s Industry Experiences team. The guide covers the following components:
- Ingesting data
- Hot path processing
- Cold path processing
- Analytics clients
We already covered the recommendation for processing data for an IoT application in the solution guide and suggested using Lambda architecture for data flow. To reiterate the data paths:
- A batch layer (cold path) stores all incoming data in its raw form and performs batch processing on the data. The result of this processing is stored as a batch view...
Although it’s not a typical use case for Azure Functions, a single Azure function is all it took to fully implement an end-to-end, real-time, mission-critical data pipeline for a fraud detection scenario. And it was done with a serverless architecture. Two blogs recently described this use case, “Considering Azure Functions for a serverless data streaming scenario,” and “A fast, serverless, big data pipeline powered by a single Azure Function.”
A large bank wanted to build a solution to detect fraudulent transactions... Read More
Apache Airflow is an open source platform used to author, schedule, and monitor workflows. Airflow overcomes some of the limitations of the cron utility by providing an extensible framework that includes operators, programmable interface to author jobs, scalable distributed architecture, and rich tracking and monitoring capabilities. Since its addition to Apache foundation in 2015, Airflow has seen great adoption by the community for designing and orchestrating ETL pipelines and ML workflows. In Airflow, a workflow is defined as a Directed Acyclic Graph (DAG), ensuring that the defined tasks are executed one after another managing the dependencies between tasks.
A simplified version of the Airflow architecture is shown below... Read More
Microsoft Connect(); 2018
On Tuesday, December 4th, Microsoft Connect(); 2018 provided a full day of developer-focused content—including updates on Azure and Visual Studio, keynotes, demos, and real-time coding with experts. Scott Guthrie’s keynote provided all the Azure and Visual Studio news on how you can stay productive and focus on what matters to you.
Guthrie’s keynote was followed by a demo-packed session hosted by Scott Hanselman and friends that provided a full tour on how you can build your apps on Azure using best-in-class developer tools.
The day continued with coding sessions hosted on Twitch and Mixer, where developers joined interactive Q&As to share what they’ve been building with Azure... Read More
Learn how to connect Power BI and Azure Data Services to share data and unlock new insights with a new tutorial. Business analysts who use Power BI dataflows can now share data with data engineers and data scientists, who can leverage the power of Azure Data Services, including Azure Databricks, Azure Machine Learning, Azure SQL Data Warehouse, and Azure Data Factory for advanced analytics and AI.
With the recently announced preview of Power BI dataflows, Power BI has enabled self-service data prep for business analysts. Power BI dataflows can ingest data from a large array of transactional and observational data sources, and cleanse, transform, enrich, schematize, and store the result... Read More
Today we are sharing an update to the Azure HDInsight integration with Azure Data Lake Storage Gen 2. This integration will enable HDInsight customers to drive analytics from the data stored in Azure Data Lake Storage Gen 2 using popular open source frameworks such as Apache Spark, Hive, MapReduce, Kafka, Storm, and HBase in a secure manner.
Azure Data Lake Storage Gen2
Azure Data Lake Storage Gen2 is the only data lake designed specifically for enterprises to run large scale analytics workloads in the cloud. It unifies the core capabilities from the first generation of Azure Data Lake with a Hadoop compatible file system endpoint now directly integrated into Azure Blob Storage... Read More
This blog post was co-authored by Nikisha Reyes-Grange, Senior Product Marketing Manager, Azure Marketing, and Balamurugan Balakreshnan, Cloud Solution Architect, CSU-Data and AI.
Johnson Controls has been a pioneer in building management solutions and services since founder Warren Johnson invented the first electric room thermostat in 1885. Johnson Controls has introduced many innovations to the building industry over the years and are now tackling a problem that costs the building industry billions each year.
Modern buildings include multiple systems that handle everything from building management to HVAC to security... Read More