Multicloud Journey- Service comparison of AWS, Azure, GCP

Service Name	AWS Service	Azure Service	GCP Service	Description
Marketplace	AWS Marketplace	Azure Marketplace	GCP Marketplace	Easy-to-deploy and automatically configured third-party applications, including single virtual machine or multiple virtual machine solutions
Compute (Virtual Servers)	EC2 instances	Virtual Machines	Compute Engine	Virtual servers allow users ti provision, manage, maintain OS & server software based on Pay-as-you-go/
Compute (Virtual Servers)	AWS Batch	Azure Batch	GCP Batch	Execute large scale parallel & high performance computing applications.
Compute (Virtual Servers)	AWS Auto-scaling	Azure VM Scale Sets	GCP Compute Engine Managed Instance Groups	Allows you to automatically scale the number of VM instances, based on defined metrices/thresholds scale out or scale in.
Compute (Virtual Servers)	VMWare on AWS	Azure VMWare by CloudSimple	VMware as a service	Redeploy & extend the VMware-based enterprise workloads to Azure by CloudSimple.
Compute (Virtual Servers)	Parallel Cluster	CycleCloud		Create, manage , optimize HPC & big compute clusters at scale.
Containers & Container Orchestrators	Elastic Container Service(ECS) AWS Fargate	Azure Container Instances(ACI)	Cloud Run	ACI is the flastest & Simplest way to run containers in Azure.
Containers & Container Orchestrators	Elastic Container Registry(ECR)	Azure Container Registry (ACR)	Container Registry Artifact Registry	Allows customers to store Docker formatted images. Used to create all types of container deployments on Azure.
Containers & Container Orchestrators	Elastic Kubernetes Service (EKS)	Azure Kubernetes Service (AKS)	Google Kubernetes Engine (GKE)	Deploy orchestrated containerized apps with CNCF Kubernetes at scale.
Containers & Container Orchestrators	AWS App Mesh	Azure Service Fabric Mesh	Anthos Service Mesh	Fully managed service that enables developers to deploy microservices applications without managing virtual machines, storage, or networking.
Containers & Container Orchestrators	EKS & Kubernetes Container Insights Metrices	Azure Monitor for containers	Kubernetes Engine Monitoring	Azure Monitor for containers is designed to monitor the performance of container workloads deployed to AKS, AKS Engine, ACI, Azure Stack.
Serverless (Functions)	AWS Lambda	Azure Functions	Cloud Functions	Provides FaaS (Function as a service) integrating systems & run backend processes in response to events without provisioning compute servers.
Database (Relational DB)	RDS	Azure SQL DB Azure Database for MySQL Azure Database for PostgreSQL	Cloud SQL (SQL Server, MySQL, PostgreSQL)	Managed relational database where scale, security, resiliency are handled by the platform
NoSQL/Document	DynamoDB SimpleDB Amazon DocumentDB	Azure Cosmos DB	Cloud Spanner	Managed relational db service with dynamic schema, security, scale, maintenance are handled by the cloud platform
NoSQL (PaaS)		Azure Cosmos DB	Cloud BigTable Cloud Firestore Firebase Realtime Database	Globally distributed multi-model db which natively supports multiple data-models, key-value, documents, graphs etc.
Caching	AWS ElastiCache	Azure Cache for Redis	Cloud Memorystore Redis Enterprise Cloud	An in-memory based, distributed caching service provides a high performance store typically store used to offload non-transactional work from a database.
Database migration	AWS DMS	Azure DMS	Open Source database Migration Tool/SQL Server Database Migration Tool	end to end migration of database migration schema & data from on-premise to cloud platform.
Networking Cloud Virtual Networking	AWS VPC	Azure Virtual Network (VNET)	GCP Virtual Private Network (VPC)	Provides an isolated, private environment in the cloud. Users have control over their virtual networking environment, including selection of their own IP address range, adding/updating address ranges, creation of subnets, and configuration of route tables and network gateways.
DNS Management	AWS Route 53	Azure DNS Azure Traffic Manager	Cloud DNS	Managing DNS records using the same credentials & billing and support contracts.
Dedicated Network (Hybrid Connectivity)	AWS Direct Connect	Azure ExpressRoute	Cloud Interconnect	Establishes a private network connection from a location to the cloud provider (not over the Internet).
Load Balancing	Network Load Balancer	Azure Load Balancer	Network Load Balancing	Azure Load Balancer load-balances traffic at layer 4 (TCP or UDP).
Load Balancing in Application layer	Application Load Balancer	Application Gateway Azure Front door Azure Traffic Manager	Global Load Balancing	Application Gateway is a layer 7 load balancer. IT takes backends with any IP that is reachable. It supports SSL termination, cookie-based session affinity, and round robin for load-balancing traffic.
Cross-premises connectivity	AWS VPN Gateway	Azure VPN Gateway Azure Virtual WAN	Cloud VPN Gateway	Connects Azure virtual networks to other Azure virtual networks, or customer on-premises networks (site-to-site). Allows end users to connect to Azure services through VPN tunneling (point-to-site).
Hybrid Connectivity	AWS Virtual Private Gateway	Azure VNET Gateway	Cloud Router	Enables dynamic routes exchange
CDN	AWS CloudFront	Azure CDN	Cloud CDN	A content delivery network (CDN) is a distributed network of servers that can efficiently deliver web content to users.
Firewall	AWS WAF	Azure WAF	Cloud Armor	Azure Web Application Firewall (WAF) provides centralized protection of your web applications from common exploits and vulnerabilities.
NAT Gateway	AWS NAT Gateway	Azure Virtual Network NAT	Cloud NAT	Virtual Network NAT (network address translation) provides outbound NAT translations for internet connectivity for virtual networks.
Private Connectivity to PaaS	AWS Private Link	Azure Private Link	VPC Service controls	Provides private connectivity between VPCs, AWS/Azure/GCP services, on-prem apps, securely on the network
Telemetry	VPC Flow Logs	NSG Flow Logs	VPC Flow Logs	Network security group (NSG) flow logs are a feature of Network Watcher that allows you to view information about ingress and egress IP traffic through an NSG.
Telemetry Network logs	VPC Flow Logs	NSG Flow Logs	Firewall Rules Logging	NSG logs are feature of Network Watcher that allows you to view info about traffic ingress & egress.
Telemetry (Monitoring)	AWS CloudWatch, X-Ray	Azure Monitor	Operations	Comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments.
Network Watcher	AWS CloudWatch	Azure Network Watcher	Network Intelligence Center	Azure Network Watcher provides tools to monitor, diagnose, view metrics, and enable or disable logs for resources in an Azure virtual network.
Security & IAM	AWS IAM	Azure AD	Cloud IAM	Allows users to securely control access to services and resources while offering data security and protection. Create and manage users and groups and use permissions to allow and deny access to resources.
IAM (Authentication & Authorization)	AWS IAM	Azure RBAC	Cloud IAM	Role-based access control (RBAC) helps you manage who has access to Azure resources, what they can do with those resources, and what areas they have access to
Multi-factor Authentication	AWS MFA	Azure AD MFA	GCP MFA	Safeguard access to data and applications while meeting user demand for a simple sign-in process.
Auth & Authoriation & Management	AWS Organizations	Azure Management Groups + RBAC	Resource Manager	Structure to organize and manage assets in Azure.
AD Domain Services	AWS Directory Service	Azure AD Domain Services	Managed Service for Microsoft Active Directory (AD)	Provides managed domain services such as domain join, group policy, LDAP, and Kerberos/NTLM authentication that are fully compatible with Windows Server Active Directory.
Identity managed service	AWS Cognito	Azure AD B2C	Firebase Authentication	A highly available, global, identity management service for consumer-facing applications that scales to hundreds of millions of identities.
Management Group	AWS Organizations	Azure Policy Azure Management Groups	Service Account
Encryption	Server side Encryption AWS S3 KMS	Azure Storage Service Encryption	Encryption by default at rest	Azure Storage Service Encryption helps you protect and safeguard your data and meet your organizational security and compliance commitments.
Hardware Security Module (HSM)	CloudHSM, KMS	Azure Key Vault	Cloud KMS	Provides security solution and works with other services by providing a way to manage, create, and control encryption keys stored in hardware security modules (HSM).
Security	AWS Inspector	Azure Security Center	Security Command Center	Automated Security assessment service provides security & compliance of applications.
Web Security with Certificates	AWS Certificate Manager	Azure App Service certificates	Web Security Scanner
Advanced Threat Management	AWS GuardDuty	Azure Advanced Threat Protection	Event Threat Protection	Detect and investigate advanced attacks on-premises and in the cloud.
Auditing	AWS Artifact	Service Trust Portal		Provides access to audit reports, compliance guides, and trust documents from across cloud services.
DDoS Protection	AWS Shield	Azure DDos Protection Service	DDoS Security with GCP Armor	Provides cloud services with protection from distributed denial of services (DDoS) attacks.
Storage (Object)	AWS S3	Azure Blob Storage	Cloud Storage	Object storage service, for use cases including cloud applications, content distribution, backup, archiving, disaster recovery, and big data analytics.
Storage (VHD)	AWS EBS	Azure Managed Disks	Persistant Disk Local SSD	SSD storage optimized for I/O intensive read/write operations. For use as high-performance Azure virtual machine storage.
Storage (File)	AWS EFS	Azure Files, Azure NetApp Files	GCP Filestore	File based storage and hosted NetApp Appliance Storage.
Data Archive	S3 Infrequent Access (IA)	Storage cool tier	Nearline
Deep Data Archive	S3 Glacier, Deep Archive	Storage archive access tier	Coldline	Archive storage has the lowest storage cost and higher data retrieval costs compared to hot and cool storage.
Data Backup	AWS Backup	Azure Backup	GCP Backup	Back up and recover files and folders from the cloud, and provide offsite protection against data loss.
Big Data & Analytics	Redshift	Azure Synapse Analytics (Formerly SQL DW)	GCP BigQuery	Cloud-based Enterprise Data Warehouse (EDW) that uses Massively Parallel Processing (MPP) to quickly run complex queries across petabytes of data.
Data warehouse & Lake	Lake Formation	Azure Data Share	Looker	Big data sharing service
Big Data Transformations	EMR	Azure Databricks	Cloud DataFlow	Managed Apache Spark-based analytics platform.
Big Data Transformations	EMR	HDInsight	GCP Dataproc	Managed Hadoop service.
Big Data Transformations	EMR	Azure Data Lake Storage Gen2	BigQuery	Massively scalable, secure data lake functionality built on Azure Blob Storage.
ETL/Data Orchestration	Data Pipeline, Glue	Azure Data Factory	Google Data Fusion	Processes and moves data between different compute and storage services, as well as on-premises data sources at specified intervals. Create, schedule, orchestrate, and manage data pipelines.
Enterprise Data discovery	AWS Glue	Azure Data Catalog	Cloud Data Catalog	A fully managed service that serves as a system of registration and system of discovery for enterprise data sources
NoSQL db	Dynamo DB	Azure Table Storage, Cosmos DB	Cloud Datastore	NoSQL key-value store for rapid development using massive semi-structured datasets.
Visualization & data Streaming	Kinesis Analytics AWS Athena	Azure Stream Analytics ADLA (Data Lake Analytics) ADLS Gen2	BigQuery	Storage and analysis platforms that create insights from large quantities of data, or data that originates from many sources.
Full text searching capability	Cloud Search	Cognitive Search Azure Search	Cloud Search	Delivers full-text search and related search analytics and capabilities.
BI tool for Visualization	Quicksight	PowerBI	Datastudio Looker	Business intelligence tools that build visualizations, perform ad hoc analysis, and develop business insights from data.
AI Hub	AWS SageMaker	Azure Machine Learning	AI Hub	A cloud service to train, deploy, automate, and manage machine learning models.
Bot Capability	Alexa Skills kit	Azure Bot Framework	Dialogflow	Build and connect intelligent bots that interact with your users using text/SMS, Skype, Teams, Slack, Office 365 mail, Twitter, and other popular services.
Conversational AI (Speech)	Lex	Speech Services	AI Building blocks- Conversation	API capable of converting speech to text, understanding intent, and converting text back to speech for natural responsiveness.
Conversational AI (NLP)	Lex	Azure LUIS	AI Building blocks -Language	A machine learning-based service to build natural language understanding into apps, bots, and IoT devices. Quickly create enterprise-ready, custom models that continuously improve.
Conversational AI(Speech to Text & vice versa)	Polly, Transcribe	Speech Services	AI Building blocks – Conversations	Enables both Speech to Text, and Text into Speech capabilities.
Enterprise AI (Computer Vision) (Face, Emotions detections)	Rekognition	Azure Cognitive Services	AI Building Blocks – Cloud AutoML AI Building Blocks – Sight	Customize and embed state-of-the-art computer vision for specific domains. Build frictionless customer experiences, optimize manufacturing processes, accelerate digital marketing campaigns-and more. No machine learning expertise is required.
Deep Learning	TensorFlow with SageMaker	ONNX ML.NET	TensorFlow	open source and cross-platform machine learning framework for both machine learning & AI
Data Science/Deep Learning VM	AWS Deep Learning AMIs	Azure DSVM	Deep Learning VM Image	Pre-Configured environments in the cloud for Data Science and AI Development.
Notebooks	AWS SageMaker Notebook instances	Azure Notebooks	AI Platform Notebooks	Develop and run code from anywhere with Jupyter notebooks on Azure.
Deep Learning Containers	AWS Deep Learning Containers	GPU Support on AKS	Deep Learning Containers	Graphical processing units (GPUs) are often used for compute-intensive workloads such as graphics and visualization workloads.
Automated Data Labeling	Automate Data Labeling with SageMaker	Azure ML – Data Labeling	Data Labeling Service	A central place to create, manage, and monitor labeling projects (public preview). Use it to coordinate data, labels, and team members to efficiently manage labeling tasks
ML Platform compute	AWS SageMaker ML Instance Types	Azure ML Compute Targets	AI Platform Training	Designated compute resource/environment where you run your training script or host your service deployment. This location may be your local machine or a cloud-based compute resource
ML Service Deployments	SageMaker Hosting Services-Model Deployment	Azure ML – Deployments	AI Platform Predictions	Deploy your machine learning model as a web service in the Azure cloud or to Azure IoT Edge devices
Monitor data drift	SageMaker Model Monitor	Azure ML – Data Drift	Continuous Evaluation	Monitor for data drift between the training dataset and inference data of a deployed model
TPU	AWS Inferencia	Azure ML – FPGA	Cloud TPU	FPGAs contain an array of programmable logic blocks, and a hierarchy of reconfigurable interconnects. The interconnects allow these blocks to be configured in various ways after manufacturing.
ML Ops	MLOps with SageMaker	Azure MLOps	GCP KubeFlow	MLOps, or DevOps for machine learning, enables data science and IT teams to collaborate and increase the pace of model development and deployment via monitoring, validation, and governance of machine learning models.
DevOps & App Monitoring	CloudWatch, X-Ray	Azure Monitor	Operations	Maximizes the availability and performance of your applications and services by delivering a comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments.
Code collaborations	Code Build CodeDeploy CodeCommit CodePipeline	Azure DevOps (Azure Board, Azure Pipeline, Azure Build & Release, Azure Repos)	Cloud Source Repositories	A cloud service for collaborating on code development.
Automation	OpsWorks	Azure Automations	Cloud Composer	Automation gives you complete control during deployment, operations, and decommissioning of workloads and resources.
Automated Infra Provisioning	CloudFormation	Azure Resource Manager VM extensions Azure Automation	Cloud Deployment Manager	Provides a way for users to automate the manual, long-running, error-prone, and frequently repeated IT tasks.
CLI, SDK interface	AWS CLI	Azure CLI, PowerShell	PowerShell on GCP GCloud SDK	Built on top of the native REST API across all cloud services, various programming language-specific wrappers provide easier ways to create solutions.
Building of Code	AWS CodeBuild	DevOps Build	Cloud Build	Fully managed build service that supports continuous integration and deployment
Managed Artifacts Repository	AWS CodeArtifact	Azure DevOps Artifacts	Artifact Registry	Add fully integrated package management to your continuous integration/continuous delivery (CI/CD) pipelines with a single click.
IoT Service	AWS IoT	Azure IoT Hub Azure Event Hub	Cloud IoT Core	A cloud gateway for managing bidirectional communication with billions of IoT devices, securely and at scale.
IoT data processing	AWS Kinesis Firehose, Kinesis Streams	Azure Event Hubs Azure Stream Analytics HDInsight Kafka	Cloud IoT core Cloud Pub/Sub GCP Dataflow	Process and route streaming data to subsequent processing engine or storage or database platform.
IoT on Edge	AWS GreenGrass	Azure IoT Edge	Edge TPU	Deploy cloud intelligence directly on IoT devices to run in on-premises scenarios.
IoT Things Graph/Digital Twins	IoT Things Graph	Azure Digital Twins	Device Registry	Create spatial intelligence graphs to model the relationships and interactions between people, places, and devices. Query data from a physical space rather than disparate sensors.
Messaging Storage	AWS SQS	Azure Queue Storage	Cloud Pub/Sub	Provides a managed message queueing service for communicating between decoupled application components.
Reliable Messaging	SQS	Service Bus Queue	Cloud Pub/Sub	Supports a set of cloud-based, message-oriented middleware technologies including reliable message queuing and durable publish/subscribe messaging
Messaging with notification	AWS SNS	Azure Event Grid	Cloud Pub/Sub	A fully managed event routing service that allows for uniform event consumption using a publish/subscribe model.
Cloud Management Advisory	Trusted Advisor	Advisor Azure Security Center	GCP Recommender	Provides analysis of cloud resource configuration and security so subscribers can ensure they’re making use of best practices and optimum configurations.
Billing API	AWS Usage & Billing Report AWS Budgets	Azure Billing API	Cloud Billing	Services to help generate, monitor, forecast, and share billing data for resource usage by time, organization, or product resources
Migrate on-prem workloads	Application Discovery Services	Azure Migrate	Assessment & Migration tool	Assesses on-premises workloads for migration to Azure, performs performance-based sizing, and provides cost estimations.
Telemetry Analysis of lift-shoft	EC2 Systems Manager	Azure Monitor	Operations (formerly StackDriver)	Comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments.
Trace	CloudTrail	Azure Monitor	Cloud Trace	Comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments.
Logging & Performance Monitoring	CloudWatch	Azure Application Insights	StackDriver Debugging/Logging	Application Insights, is an extensible Application Performance Management (APM) service for developers and DevOps professionals.
Cost Management	AWS Cost explorer	Azure Cost Management	GCP Cost Management	Optimize cloud costs while maximizing cloud potential.
Mobile Service	Mobile Hub Mobile SDK Mobile Analytics	Azure Xamarin Apps, App Center	GCP App Engine	Provides backend mobile services for rapid development of mobile solutions, identity management, data synchronization, and storage and notifications across devices.
Device Farm	AWS Device Farm	Azure App Center	Firebase Test Lab	Provides services to support testing mobile applications.
Bulk Data Transfer	Import/Export Disk,	Azure Import/Export	Transfer appliance	A data transport solution that uses secure disks and appliances to transfer large amounts of data. Also offers data protection during transit.
Petabyte to exabyte level data transfer to Cloud	Import/Export Snowball, SnowballEdge, Snowball Mobile	Azure DataBox	Transfer Appliance	Petabyte- to exabyte-scale data transport solution that uses secure data storage devices to transfer large amounts of data to and from Azure
Storage Gateway	AWS Storage Gateway	Azure StoreSimple	Google Cloud Storage	Integrates on-premises IT environments with cloud storage. Automates data management and storage, plus supports disaster recovery.
Data Sync	AWS Data Sync	Azure File Sync	Cloud Data Transfer	Data sync services
Serverless Workflow	AWS SWF	Azure Logic Apps	GCP Composer	Serverless technology for connecting apps, data and devices anywhere, whether on-premises or in the cloud for large ecosystems of SaaS and cloud-based connectors.
Hybrid	AWS Outposts	Azure Stack Azure ARC	GCP Anthos	For customers who want to simplify complex and distributed environments across on-premises, edge and multi-cloud
Media	AWS Elemental Media Convert Elastic Transcoder	Azure Media Services	GCP Anvato Zync Render Game Servers	Cloud-based media workflow platform to index, package, protect, and stream video at scale.
BlockChain	AWS BlockChain	Azure BlockChain Service	Digital Asset	Azure Blockchain Service is a fully managed ledger service that enables users the ability to grow and operate blockchain networks at scale in Azure
App Services	AWS ELB (Elastic BeanStalk)	Azure App Service	GCP App Engine	Managed hosting platform providing easy to use services for deploying and scaling web applications and services.
API Services	API Gateway	Azure API Management	Apigee API platform, API Analytics	A turnkey solution for publishing APIs to external and internal consumers.
Deploy Web apps	LightSail	Azure App Service	Cloud Run, App Engine	Build, deploy, and scale web apps on a fully managed platform.
Backend Serverless computation	AWS STEP Function	Azure Logic Apps	App Engine	Connect apps, data and devices on-premises or in the cloud.

Quick Installation of Single node Datazen Server in Azure Cloud Service & Sample Dashboards

The demo provides step by step guidance on quick setup of datazen server on single node server & connecting with publisher app to build custom visuals.

Pre-requisites for the demo : 

1. An active Azure subscription.
2. Windows 10 store(for installation of Datazen publisher)

Detailed steps are depicted as follows:

Steps	Screen Shot
1.     Go to https://manage.windowsazure.com/ 2.     Login with your Live ID. 3.     Click +New. 4.     Select Compute -> Virtual Machine -> From Gallery. (Fig. 2)
5.     Select the Windows Server 2012 R2 Datacenter image. 6.     Click the next arrow at the bottom right.
7.     Enter the required information for Virtual machine configuration. o   Virtual machine name o   Choose Basic or Standard Tier (recommended) o   Choose A4 as the machine size.  A4 has 8 cores, which is the minimum number of cores for a single machine setup. o   Enter the machine admin username/password. o   Click the next arrow.
8.     Enter the required information for Virtual machine configuration. o   Select option Create a new cloud service. Make sure the Cloud Service DNS Name is available. o   Choose the subscription it should be billed to, the region it should be deployed. Choose the one closest to your location o   Leave the Storage Account and Availability Set as is. o   Add an HTTP endpoint at a minimum. You may need to scroll to add the endpoint. o   Click the next arrow.
9.     Select Install VM Agent and leave other unchecked. 10.  Click the checkmark to start the deployment process. You’ll see it start the provisioning process in the list of the virtual machines you are responsible for in Azure.
11.   Wait for the status to change from Starting to Running in virtual machines.
12.   Select your VM then click Connect.
13.   Save the RDP file to your local machine.
14.   Open the saved Remote Desktop Connection file, then click Connect.
15.   Connect to the VM via Remote Desktop and enter the admin username/password.
16.   Click Yes to connect to Server.
17.   Click Configure this local server in the Server Manager dashboard that appears when you login.
18.  Then change the IE Enhanced Security Configuration to Off for Administrators. You can always change it back if you really want to when you’re done. 19.  Close the Server Manager.

 

 

Section 2: Install the Datazen Server

1.     Navigate to the following link and download the Datazen server software onto the VM. You may need to turn off IE Enhanced Security on the server to do so. 2.     The Datazen server files download as a zipped file. Extract all the files 3.     Open Datazen Enterprise Server.3.0.2562. 4.     Click run to start the install process.
5.     In the Datazen Enterprise Server Setup click Next.
6.     Click Next in the Setup wizard, accepting the terms in the License Agreement and moving through each screen.
7.     Click Next on the Features page.
8.     Click Next on the Core Service Credentials page.
9.     Once you get to the Admin Password page, type a Password for the Datazen admin user.  (Fig. 20) This doesn’t have to be the same password as you used for the server. 10.  Click Next.
11.  On the Authentication page, leave the Authentication Mode as Default. 12.  Click Next.
13.  On the Repository Encryption page, select Copy to Clipboard, then paste the key into a Notepad file. 14.  Save the Notepad file to a safe location.   15.  Click Next.
16.  On the Instance Id page, select Copy to Clipboard then paste the Id into a Notepad file. 17.  Save the Notepad file to a safe location.   18.  Click Next.
19.  On the Data Acquisition Service Credentials page, leave the credentials as is, then click Next.
20.  On the Web Applications IIS Settings page leave the default settings, then click Next.
21.  On the Control Panel Email Settings page, leave the default values since this is a test server. 22.  Click Next.
23.  On the Ready to Install page, click Install and wait until the installation is complete. This might take a few minutes.

Section 3: Configure the Datazen Server

1.   Open your browser in your local machine 2.   Navigate to http://mycloudservicename.cloudapp.net/cp. Make sure you replace the yourcloudservicename with the name of your cloud service. 3.   If you can successfully connect, you should see the Control Panel Log In screen. 4.   Enter the username admin and the password you entered in the Setup wizard, then select  Log In.
5.   You will need to create a new user to start creating dashboard hubs, since you need each hub to have an owner.  The owner can NOT be the admin user.  Click Create User to create your first user.
6.   Enter a value in the top three fields (the email address can be fake if you want) and select Create User.
7.   You will now see a new option to Create BI Hub.
8.   Enter Hub name whatever you’d like, but make sure you enter the username of the user you just created for owner username. 9.   Enter a maximum number of users that can connect to hub. 10.  Click Create.
11.   Finish the creation of the hub. It will be displayed in the list of available hubs.
12.  The new hub will also be shown in the navigation menu at the bottom left of the screen.
13.  Click the Server Users link on the left-hand side of the screen.
14.  Click Create User.
15.  Fill in the fields under Required Info. 16.  Click Create User.
17.  You will see the user and a Set password link option next to the username. 18.  Click on Set password link and then copy the link to your clipboard. Note: This step is only required as the email notification is not set up.
19.  Logout as the admin 20.  Open a new browser window and paste the URL to reset the password into the address bar. You can now finish setting up that user by entering the password for the account.
21.  In the Control Panel Activate User Account screen, then enter the new password, then re-type password 22.  Click Activate My Account. (Fig. 40) 23.  Logout as this user and log back in as the admin before proceeding.

 

Section 4: Apply a Custom Branding

1.  To add the Wide World Importers brand package to the server, save it locally. The package is provided with this demo. 2.  Click on the Branding link on the left-hand side and upload the brand package to the server.
3.  Make sure you choose the Server to upload it to. You will see the Server icon has the Wide World Importers branding associated.
4.  To make sure it was applied properly, open a new browser and navigate to the following URL (make sure you replace the mycloudservicename with whatever you named yours) http://mycloudservicename.cloudapp.net Your Server Login screen should look as shown on the right, now having the Wide World Importers brand package applied. (Fig. 43)

 

Section 5: Connect to the Datazen Server with Publisher

1.  Open the Datazen Publisher app If this is the first time using the app, you will have the option of connecting the Datazen demo server.  We recommend doing that, so you will have some nice demo dashboards to show immediately. 2.  To add new server Right-click in the dashboard, then click Connected. (Fig. 44)
3.  Click Add New Server Connection (Fig. 45)
4.  Provide the following information to connect to a Datazen server. (Fig. 46) Server Address: mycloudservicename.cloudapp.net User name: user name that created Password: provide user password 5.  Uncheck Use Secure Connection. 6.  Click Connect. (Fig. 46)
7.  When connected, you should be able to publish dashboards to your Datazen server. (Fig. 47) 8.  You will see a nice dashboard with KPIs for Wide World Importers and Fabrikam Insurance. (Fig. 47)

 

 

 

 

 

Azure Stream Analytics & Machine Learning Integration With RealTime Twitter Sentiment Analytics Dashboard on PowerBI

Recently, it has been introduced the integration of ASA & AML available as preview update & it’s possible to add AML web service URL & API key as ‘custom function‘ with ASA input. In this demo, realtime tweets are collected based on keywords like ‘#HappyHolidays2016‘, ‘#MerryChristmas‘, ‘#HappyNewYear2016‘ & those are directly stored on a .csv file saved on OneDrive. Here goes the solution architecture diagram of the POC.

 

 

Now, add the Service Bus event hub endpoint as input to the ASA job, while deploy the ‘Twitter Predictive Sentiment Analytics Model‘  & click on ‘Open in Studio‘ to start deploy the model. Don’t forget to run the solution before deploying.

 

Once the model is deployed, open the ‘Web Service‘ dashboard page to get the model URL & API key, click on default endpoint -> download the excel 2010 or earlier apps. Collect the URL & API key to apply it to ASA function credentials for AML deployment.

Next, create an ASA job & add the event hub credentials where the real world tweets are getting pushed & click on ‘Functions‘ tab of ASA job to add the AML credentials. Provide model name, URL & API key of the model & Once, it’s added, click on Save.

 

Now, add the following ASA SQL to aggregate the realtime tweets sentiment scores coming out from predictive twitter sentiment model.

 

Provide the output as Azure Blob storage, add a container name & serialization type as CSV & start the ASA job. Also, start importing data into PowerBI desktop from the ASA output Azure blob storage account.

 

 

PowerBI desktop contains in-built power Query to start preparing the ASA output data & processing data types. Choose the AML model sentiment score datatype as decimal type & TweetTexts as Text(String) type.

 

Start building the ‘Twitter Sentiment Analytics‘ dashboard powered by @AzureStreaming & Azure Machine Learning API with realworld tweet streaming, there’re some cool custom visuals are available on PowerBI.  I’ve used some visuals here like ‘wordcloud‘ chart which depicts some of the highly scored positive sentiment contained tweets with most specific keywords like ‘happynewyear2016‘, ‘MerryChristmas‘,’HappyHolidays‘ etc.

 

While, in the donut chart, the top 10 tweets with most positive sentiment counts are portrayed with the specific sentiment scores coming from AML predictive model experiment integrated with ASA jobs.

~Wish you HappyHolidays 2016!

A lap around Microsoft Azure IoT Hub with Azure Stream Analytics & IoT Analytics Suite

Last month on #AzureConf 2015, the Azure IoT Suite has been announced to be available for purchase along with the GA release of Azure IoT Hub. The IoT Hub helps to control, monitor & connect thousands of devices to communicate via cloud & talk to each other using suitable protocols. You can connect to your Azure IoT Hub using the IoT Hub SDKs available in different languages like C, C#, Java, Ruby etc. Also, there’re monitoring devices available like device explorer or iothub-explorer. In this demo, Weather Data Analytics is demonstrated using Azure IoT Hub with Stream Analytics powered by Azure IoT Suite & visualized using Azure SQL database with PowerBI.

You can provision your own device into Azure IoT analytics Suite using device explorer or iothub-explorer tool & start bi-directional communication through device-cloud & cloud-device.

First, create your Azure IoT Hub from Azure Preview Portal  by selecting New-> Internet of Things -> Azure IoT Hub. Provide hub name, select pricing & scale tier[F1 – free(1/subscription, connect 10 devices, 3000 messages /day), [S1 – standard (50,000 messages/day) & S2- standard(1.5 M messages/day)] for device to cloud communication. Select IoT Hub units, device to cloud partitions, resource group, subscription & finally location of deployment(currently it’s available only in three locations- ‘East Asia’, ‘East US’, ‘North Europe’.

 

 

Once the hub is created, next switch to device explorer to start creating a device, for details about to create a device & register, refer to this Github page. After registering the device, move back to  ‘Data‘ tab of device explorer tool & click on ‘Monitor‘ button to start receive device-cloud events sent to Azure IoT Hub from device.

 

The schema for the weather dataset looks like the following data & fresh data collected from various sensors & feed into Azure IoT Hub which can be viewed using Device Explorer tool.

 

In order to push data from weather data sensor device to Azure IoT hub, the following code snippet needs to be used. The full code-snipped is going to be available on my Github page.

 

using System;
using System.Text;
using System.Threading.Tasks;
using System.IO;
using System.Data;
using Newtonsoft.Json;
using Microsoft.VisualBasic;
using Microsoft.VisualBasic.FileIO;

namespace Microsoft.Azure.Devices.Client.Samples
{
class Program
{
private const string DeviceConnectionString = “Your device connection-string”;
private static int MESSAGE_COUNT = 5;
static string data = string.Empty;

static void Main(string[] args)
{
try
{
DeviceClient deviceClient = DeviceClient.CreateFromConnectionString(DeviceConnectionString);

if (deviceClient == null)
{
Console.WriteLine(“Failed to create DeviceClient!”);
}
else
{
SendEvent(deviceClient).Wait();
ReceiveCommands(deviceClient).Wait();
}

Console.WriteLine(“Exited!\n”);
}
catch (Exception ex)
{
Console.WriteLine(“Error in sample: {0}”, ex.Message);
}
}

static async Task SendEvent(DeviceClient deviceClient)
{
string[] filePath = Directory.GetFiles(@”\Weblog\”,”*.csv”);
string csv_file_path = string.Empty;
int size = filePath.Length;
for(int i=0; i< size; i++)
{
Console.WriteLine(filePath[i]);
csv_file_path = filePath[i];
}

DataTable csvData = GetDataTableFromCSVFile(csv_file_path);
Console.WriteLine(“Rows count:” + csvData.Rows.Count);
DataTable table = csvData;
foreach(DataRow row in table.Rows)
{
foreach(var item in row.ItemArray)
data = item.ToString();
Console.Write(data);

try
{
foreach(DataRow rows in table.Rows)
{
var info = new WeatherData
{
weatherDate = rows.ItemArray[0].ToString(),
weatherTime = rows.ItemArray[1].ToString(),
apperantTemperature = rows.ItemArray[2].ToString(),
cloudCover = rows.ItemArray[3].ToString(),
dewPoint = rows.ItemArray[4].ToString(),
humidity = rows.ItemArray[5].ToString(),
icon = rows.ItemArray[6].ToString(),
pressure = rows.ItemArray[7].ToString(),
temperature = rows.ItemArray[8].ToString(),
timeInterval = rows.ItemArray[9].ToString(),
visibility = rows.ItemArray[10].ToString(),
windBearing = rows.ItemArray[11].ToString(),
windSpeed = rows.ItemArray[12].ToString(),
latitude = rows.ItemArray[13].ToString(),
longitude = rows.ItemArray[14].ToString()
};

var serializedString = JsonConvert.SerializeObject(info);
var message = data;
Console.WriteLine(“{0}> Sending events: {1}”, DateTime.Now.ToString(), serializedString.ToString());
await deviceClient.SendEventAsync(new Message(Encoding.UTF8.GetBytes(serializedString.ToString())));
}
}

catch(Exception ex)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine(“{0} > Exception: {1}”, DateTime.Now.ToString(), ex.Message);
Console.ResetColor();
}
// Task.Delay(200);

}

Console.WriteLine(“Press Ctrl-C to stop the sender process”);
Console.WriteLine(“Press Enter to start now”);
Console.ReadLine();

//string dataBuffer;

//Console.WriteLine(“Device sending {0} messages to IoTHub…\n”, MESSAGE_COUNT);

//for (int count = 0; count < MESSAGE_COUNT; count++)
//{
// dataBuffer = Guid.NewGuid().ToString();
// Message eventMessage = new Message(Encoding.UTF8.GetBytes(dataBuffer));
// Console.WriteLine(“\t{0}> Sending message: {1}, Data: [{2}]”, DateTime.Now.ToLocalTime(), count, dataBuffer);

// await deviceClient.SendEventAsync(eventMessage);
//}
}

private static DataTable GetDataTableFromCSVFile(string csv_file_path)
{
DataTable csvData = new DataTable();
string data = string.Empty;
try
{
using (TextFieldParser csvReader = new TextFieldParser(csv_file_path))
{
csvReader.SetDelimiters(new string[] { “,” });
csvReader.HasFieldsEnclosedInQuotes = true;

//read column names
string[] colFields = csvReader.ReadFields();
foreach (string column in colFields)
{
DataColumn datecolumn = new DataColumn(column);
datecolumn.AllowDBNull = true;
csvData.Columns.Add(datecolumn);
}
while (!csvReader.EndOfData)
{
string[] fieldData = csvReader.ReadFields();

for (int i = 0; i < fieldData.Length; i++)
{
if (fieldData[i] == “”)
{
fieldData[i] = null;
}
}
csvData.Rows.Add(fieldData);

}
}
}
catch (Exception ex)
{
Console.WriteLine(“Exception” + ex.Message);
}
return csvData;
}

static async Task ReceiveCommands(DeviceClient deviceClient)
{
Console.WriteLine(“\nDevice waiting for commands from IoTHub…\n”);
Message receivedMessage;
string messageData;

while (true)
{
receivedMessage = await deviceClient.ReceiveAsync(TimeSpan.FromSeconds(1));

if (receivedMessage != null)
{
messageData = Encoding.ASCII.GetString(receivedMessage.GetBytes());
Console.WriteLine(“\t{0}> Received message: {1}”, DateTime.Now.ToLocalTime(), messageData);

await deviceClient.CompleteAsync(receivedMessage);
}
}
}
}
}

You could check output to events sending from device to cloud on console.

Next, start pushing the device data into Azure IoT Hub & monitor the events receiving process through device explorer. Now, start provisioning an Azure Stream Analytics Job on Azure portal. Provide ‘Azure IoT Hub‘ as an input to the job like as the followings.

 

Now provide Azure Stream Analytics Query to connect incoming unstructured datasets from device to cloud to pass into Azure SQL database. So, first, provision a SQL database on Azure & connect to as output to Stream Analytics job.

create table input(
weatherDate nvarchar(max),
weatherTime datetime,
apperantTemperature nvarchar(max),
cloudCover nvarchar(max),
dewPoint nvarchar(max),
humidity nvarchar(max),
icon nvarchar(max),
pressure nvarchar(max),
temperature nvarchar(max),
timeInterval nvarchar(max),
visibility nvarchar(max),
windBearing nvarchar(max),
windSpeed nvarchar(max),
latitude nvarchar(max),
longitude nvarchar(max)
)
select input.weatherDate, input.weatherTime,input.apperantTemperature,input.cloudCover,
input.dewPoint, input.humidity,input.icon,input.pressure,count(input.temperature) as avgtemperature, input.timeInterval, input.visibility, input.windBearing,
input.windSpeed,input.latitude,input.longitude

into weathersql
from input
group by input.weatherDate, input.weatherTime, input.apperantTemperature,input.cloudCover,
input.dewPoint, input.humidity,input.icon, input.pressure,input.timeInterval,input.visibility, input.windBearing,
input.windSpeed,input.latitude,input.longitude, TumblingWindow(second,2)

Specify the output of ‘WeatherIoT’ ASA job as ‘Azure SQL Database‘, alternatively, you can select any of the rest of the connectors like ‘Event Hub’, ‘DocumentDB’ etc.

 

Make sure that , to create the necessary database & table first on SQL before adding as output to ASA job. For this demo, I have created the ‘weatheriot‘ table on Azure SQL database. The t-sql query looks like this.

 

Next, start the ASA job & receive the final Azure IoT hub(device to cloud) data processed to IoT hub ->ASA -> Azure SQL database pipeline. Once you receive data on your Azure SQL table. Start building the PowerBI ‘Weather IoT Data Analytics’ dashboard for visualization & to leverage the power of Azure IoT momentum.

Connect to PowerBI connected through same account of Azure subscription where you provisioned the ASA job & start importing data from Azure SQL database. Create stunning reports using funnel, donut, global map charts with live data refresh.

For this demo, I’ve populated charts on average weather temperature, pressure, humidity, dew point forecasting analysis over specific areas based on latitude & longitude values, plotted & pinned into PowerBI ‘Weather Data Azure IoT Analytics’ dashboard.

 

What’s new in Azure Data Catalog

The Azure Data Catalog (aka previously PowerBI Data Catalog) has released in public preview on last monday(July 13th) @WPC15, which typically reveals a new world of storing & connecting #Data across on-prem & azure SQL database. Lets hop into a quick jumpstart on it.

Connect through Azure Data Catalog through this url  https://www.azuredatacatalog.com/ by making sure you are logging with your official id & a valid Azure subscription. Currently , it’s free for first 50 users & upto 5000 registered data assets & in standard edition, upto 100 users & available upto 1M registered data assets.

 

Lets start with the signing of the official id into the portal.

Once it’s provisioned, you will be redirected to this page to launch a windows app of Azure Data Catalog.

 

It would start downloading the app from clickonce deployed server.

 

After it downloaded & would prompt to select server , at this point it has capacity to select data from SQL Server Analysis service, Reporting Service, on-prem/Azure SQL database & Oracle db.

For this demo, we used on-prem SQL server database to connect to Azure Data Catalog.

We selected here ‘AdventureWorksLT’ database & pushed total 8 tables like ‘Customer’, ‘Product’, ‘ProductCategory’, ‘ProductDescription’,’ProductModel’, ‘SalesOrderDetail’ etc. Also, you can tags to identify the datasets on data catalog portal.

Next, click on ‘REGISTER’ to register the dataset & optionally, you can include a preview of the data definition as well.

 

Once the object registration is done, it would allow to view on portal. Click on ‘View Portal’ to check the data catalogs.

Once you click , you would be redirected to data catalog homepage where you can search for your data by object metaname.

 

in the data catalog object portal, all of the registered metadata & objects would be visible with property tags.

You can also open the registered object datasets in excel to start importing into PowerBI.

Click on ‘Excel’ or ‘Excel(Top 1000)’ to start importing the data into Excel. The resultant data definition would in .odc format.

 

Once you open it in Excel, it would be prompted to enable custom extension. Click on ‘Enable’.

From Excel, the dataset is imported to latest Microsoft PowerBI Designer Preview app to build up a custom dashboard.

Login into https://app.powerbi.com & click to ‘File’ to get data from .pbix file.

Import the .pbix file on ‘AdventureWorks’ customer details & product analytics to powerbi reports & built up a dashboard.

The PowerBI preview portal dashboard has some updates on tile details filter like extension of custom links.

 

The PowerBI app for Android is available now, which is useful for quick glance of real-time analytics dashboards specially connected with Stream analytics & updating  real time.

 

 

 

What’s new in Azure SDK 2.5 & Visual Studio 2013 Update 4

Recently, after playing enough with Azure Stream Analytics , it’s time to move on with azure .net development & a new version of Azure sdk is published. Let’s have a quick overview on latest azure sdk.

First of all, lets download the sdk from webpi console, as directed ‘Microsoft Azure SDK 2.5 for .NET(VS 2013)

In this edition, there are few new components added like as:

i) EnvironmentTools.VS.msi

ii) HiveODBC32.msi

iii)HiveODBC64.msi

iv) Microsoft.Azure.HDInsightTools-x64.msi

v) Microsoft.Azure.HDInsightTools-x86.msi

so on…

Now, after installing sdk 2.5 , lets start with Visual Studio 2013.

Expand on ‘QuickStart’ under ‘Cloud’ & start exploring options to create AppService , Compute & DataService directly from VS 2013 /2012 itself.

 

The default ‘DataBlobStorage1’ sample would be created in VS to create blob container, create a block blob/page blob, upload a new blob , delete a blob (all basic CRUD operations on blob using REST)

Next, the major improvements is done on Azure HDinsight shell integration into Visual Studio onto which you can now run your custom Hive table queries on HDFS of HDInsight clusters. Lets create a sample Hive query file on VS 2013.

Lets move into HDInsight tab on left side of VS installed menu & select HDInsight’ & select ‘HiveApplication’ to start with new Hive-ql. For this demo, I am selecting Hive Sample from VS.

 

On selecting Hive sample, I would be able to open the sample Hive queries on ‘weblogAnalysis.hql‘  & ‘sensordataAnalysis.hql’ from Azure HDinsight cluster.

Here goes a sample weblogAnalysis.hql:

DROP TABLE IF EXISTS weblogs;
— create table weblogs on space-delimited website log data.
— In this sample we will use the default container. You could also use ‘wasb://[container]@[storage account].blob.core.windows.net/Path/To/Data/’ to access the data in other containers.
CREATE EXTERNAL TABLE IF NOT EXISTS weblogs(s_date date, s_time string, s_sitename string, cs_method string, cs_uristem string,
cs_uriquery string, s_port int, cs_username string, c_ip string, cs_useragent string,
cs_cookie string, cs_referer string, cs_host string, sc_status int, sc_substatus int,
sc_win32status int, sc_bytes int, cs_bytes int, s_timetaken int )
ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘ ‘
STORED AS TEXTFILE LOCATION ‘/HdiSamples/WebsiteLogSampleData/SampleLog/’
TBLPROPERTIES (‘skip.header.line.count’=’2’);

 

Before proceeding with the realtime hive queries, we need to make sure that the Azure HDI cluster is already provisioned & it might be either a simple Hadoop HDI cluster, HBase HDI cluster or Storm HDI cluster to build hive tables on top of it.

There’s a new option came out for Azure HDI cluster to add custom powershell scripts while provisioning a HDI cluster using azure portal. Also, new additions of HDI cluster is exploration of R(official cran packages) & Apache Spark on hdinsight hdfs cluster which will be covered with demo next.

A Quick Walk-through on Azure Storage(SQL, NoSQL, NewSQL)

It’s always imaginable that developers are always flexible to proceed with relational databases while migrating an existing on-premise app to Azure platform while leveraging best possible architectural guidelines on migration to cloud. But, still forth in real-time cases , typical scenarios like suboptimal performance, high expenses, or worse case scenario because, NOSQL db can handle some tasks more efficiently than relational databases can. In few enterprise cases, it’s encountered a critical data storage problem, as because NOSQL solution implementation have been better off before deploying its app to production.

Moreover, there’s no single best data management choice for all data storage tasks, different data management solutions are optimized for different tasks. Let’s have a quick walk-through on various data storage option models supported on Microsoft Azure.

 

 

Let’s start first by four types of NOSQL db supported now Azure.

• Key/value pair databases: store a single serialized object for each key value. They’re good for storing large volumes of data in situations where you want to get one item for a given key value and you don’t have to query based on other properties of the item.

• Azure Blob Storage : It’s also a key/value based data storage which is same like as file system in functionality where you could search a file based on it’s folder/file name as key not file content as key. Blob offers read-write storage options (aka Block Blob) for storing large media files as well as for standard streaming purpose facilitates the usage of VHDs as Page Blob(aka Azure Drive).

• Azure Table Storage : A standard key-value pair based NOSQL storage option prevailed from Azure storage inception phase. Each value is called an entity (similar to a row, identified by a partition key and row key) and contains multiple properties (similar to columns, but not all entities in a table have to share the same columns). Querying on columns other than the key is extremely inefficient and should be avoided.

• Document Databases : Popular key/value databases in which the values are documents. “Document” here isn’t used in the sense of a Word or an Excel document but means a collection of named fields and values, any of which could be a child document. For example, in an order history table, an order document might have order number, order date, and customer fields, and the customer field might have name and address fields. The database encodes field data in a format such as XML, YAML, JSON, or BSON, or it can use plain text. One feature that sets document databases apart from other key/value databases is the capability they provide to query on nonkey fields and define secondary indexes, which makes querying more efficient. This capability makes a document database more suitable for applications that need to retrieve data on the basis of criteria more complex than the value of the document key.

Example : Mongo DB.

• Column-family databases : key/value pair based data storage enables to structure data based on collections of columns called ‘Column families‘. For example, a population database consists of one group of column called ‘Persons’ (containing firstname, middlename, lastname) , one group for person’s address & another for profile info. The database can then store each column family in a separate partition while keeping all of the data for one person related to the same key. You can then read all profile information without having to read through all of the name and address information as well.

Example : Cassendra , Apache HBase (in preview supported with HDInsight as NOSQL Blob Storage)

• Graph databases : Stores data in form of objects & relationships.The purpose of a graph database is to enable an application to efficiently perform queries that traverse the network of objects and the relationships between them. For example, the objects might be employees in a human resources database, and you might want to facilitate queries such as “find all engineers who directly or indirectly work for Product Manager.”

Example : Neo4j Graph Database.

Compared with relational databases, the NoSQL options offer far greater scalability and are more cost effective for storage and analysis of unstructured data. The tradeoff is that they don’t provide the rich querying and robust data integrity capabilities of relational databases. NoSQL options would work well for IIS log data, which involves high volume with no need for join queries. NoSQL options would not work so well for banking transactions, which require absolute data integrity and involve many relationships to other account-related data.

• A brief about NewSQL : Combines the scalability features of NOSQL along with distributed querying & transactional integrity of OldSQL.
• The first type of NewSQL systems are completely new database platforms. These are designed to operate in a distributed cluster of shared-nothing nodes, in which each node owns a subset of the data. Though many of the new databases have taken different design approaches, there are two primary categories evolving. The first type of system sends the execution of transactions and queries to the nodes that contain the needed data. SQL queries are split into query fragments and sent to the nodes that own the data. These databases are able to scale linearly as additional nodes are added.

• General-purpose databases

  These maintain the full functionality of traditional databases, handling all types of queries. These databases are often written from scratch with a distributed architecture in mind, and include components such as distributed concurrency control, flow control, and distributed query processing. This includes Google Spanner, Clustrix, FoundationDB, NuoDB,TransLattice, ActorDB,andTrafodion.

  In-memory databases

  The applications targeted by these NewSQL systems are characterized as having a large number of transactions that (1) are short-lived (i.e., no user stalls), (2) touch a small subset of data using index lookups (i.e., no full table scans or large distributed joins), and (3) are repetitive (i.e. executing the same queries with different inputs).

  These NewSQL systems achieve high performance and scalability by eschewing much of the legacy architecture of the original IBM System R design, such as heavyweight recovery or concurrency control algorithms.

  Example systems in this category are:VoltDB, Pivotal‘s SQLFire and GemFire XD, SAP HANA.

  Example : NuoDB is supported in Azure as NewSQL.

  • Key Points to Consider while choosing the Data Storage Options :

Data semantic

---

What is the core data storage and data access semantic (are you storing relational or unstructured data)?
Unstructured data such as media files fits best in Blob storage; a collection of related data such as products, inventories, suppliers, customer orders, etc., fits best in a relational database.
Query support

---

How easy is it to query the data?
What types of questions can be efficiently asked?
Key/value data stores are very good at getting a single row when given a key value, but they are not so good for complex queries. For a user-profile data store in which you are always getting the data for one particular user, a key/value data store could work well. For a product catalog from which you want to get different groupings based on various product attributes, a relational database might work better.
NoSQL databases can store large volumes of data efficiently, but you have to structure the database around how the app queries the data, and this makes ad hoc queries harder to do. With a relational database, you can build almost any kind of query.
Functional projection

---

Can questions, aggregations, and so on be executed on the server?
If you run SELECT COUNT(*) from a table in SQL, the DBMS will very efficiently do all the work on the server and return the number you’re looking for. If you want the same calculation from a NoSQL data store that doesn’t support aggregation, this operation is an inefficient “unbounded query” and will probably time out. Even if the query succeeds, you have to retrieve all the data from the server and bring it to the client and count the rows on the client.
What languages or types of expressions can be used?
With a relational database, you can use SQL. With some NoSQL databases, such as Azure Table storage.

Ease of scalability

---

How often and how much will the data need to scale?
Does the platform natively implement scale-out?
How easy is it to add or remove capacity (size and throughput)?
Relational databases and tables aren’t automatically partitioned to make them scalable, so they are difficult to scale beyond certain limitations. NoSQL data stores such as Azure Table storage inherently partition everything, and there is almost no limit to adding partitions. You can readily scale Table storage up to 200 terabytes, but the maximum database size for Azure SQL Database is 500 gigabytes. You can scale relational data by partitioning it into multiple databases, but setting up an application to support that model involves a lot of programming work.
Instrumentation and Manageability

---

How easy is the platform to instrument, monitor, and manage?
You need to remain informed about the health and performance of your data store, so you need to know up front what metrics a platform gives you for free and what you have to develop yourself.
Operations

---

How easy is the platform to deploy and run on Azure? PaaS? IaaS? Linux?
Azure Table storage and Azure SQL Database are easy to set up on Azure. Platforms that aren’t built-in Azure PaaS solutions require more effort.
API Support

---

Is an API available that makes it easy to work with the platform?
The Azure Table Service has an SDK with a .NET API that supports the .NET 4.5 asynchronous programming model. If you’re writing a .NET app, the work to write and test the code will be much easier for the Azure Table Service than for a key/value column data store platform that has no API or a less comprehensive one.
Transactional integrity and data consistency

---

Is it critical that the platform support transactions to guarantee data consistency?
For keeping track of bulk emails sent, performance and low data-storage cost might be more important than automatic support for transactions or referential integrity in the data platform, making the Azure Table Service a good choice. For tracking bank account balances or purchase orders, a relational database platform that provides strong transactional guarantees would be a better choice.
Business continuity

---

How easy are backup, restore, and disaster recovery?
Sooner or later production data will become corrupted and you’ll need an undo function. Relational databases often have more fine-grained restore capabilities, such as the ability to restore to a point in time. Understanding what restore features are available in each platform you’re considering is an important factor to consider.
Cost

---

If more than one platform can support your data workload, how do they compare in cost?
For example, if you use ASP.NET Identity, you can store user profile data in Azure Table Service or Azure SQL Database. If you don’t need the rich querying facilities of SQL Database, you might choose Azure Table storage in part because it costs much less for a given amount of storage.

A Lap around the New Microsoft Azure Management Portal & Preview Programs

Well, a few months back during Build 2014, the new Microsoft Azure Management Portal (Preview) announced for public access. There’s a quite lot interesting features are included to it to make it more appealing & enchanting. One of the unique feature is ‘Parts‘ quite similar like metro ‘Tiles’ to make developer’s life more easier conglomerate all services under one single UI pane.

 

• Select , Virtual Machine pane while list of all standard VM images along with available data centers are available.

 

• Similarly , Web , Mobile & Developer Services , you would be able to see the respective services while in Data, Cache & Storage+backup service consists of latest Azure Redis Cache(Preview), HDInsight, MySQL, Mongo Labs, SQL Server 2014 Standard, Oracle DB 12.1.0.1 standard & 11G R2, WebLogic server, Azure Backup & Hyper-V Recovery Manager.

• Now, check for App+ Data Services, a lot of new app services including ‘Notification Hub’ , ‘Pusher’ , ‘SendGrid‘ is available in new Azure Portal.

 

• The ‘Parts’ of new Azure Portal is self -customizable , so when you right-click on any of tiles , you would get the ‘Customize‘ option from when you can select the appropriate size of blocks of ‘Service Health’, ‘Gallery’, ‘What’s New’ etc.

 

• There is a self -notification-able ‘Notification’ hub is added in new portal which summarizes any of ‘Service Health’ report /errors related to service for last 24 hours.

•  While, from ‘Browse’ option, you should be able to see the list of ‘Resource Group Services’, ‘Virtual Machine’, ‘Websites’, ‘Team Server projects.

• Click on ‘What’s New’ tile while a brilliant  grid view of latest updates from Azure blog is available.

 

• Now, take a focus on latest ‘Preview Features‘ available which are ‘Windows Azure Automation’, ‘New Service Tiers for SQL Database’, ‘Azure RemoteApp’, ‘Visual Studio Online Account Access’, ‘Windows Azure Files’, ‘Billing Alert Service’.

First, lets check out the ‘Azure Automation API‘ .

• Windows Azure Automation allows you to automate the creation, monitoring, deployment, and maintenance of resources in your Windows Azure environment using a highly-available workflow execution engine. Orchestrate time-consuming, error-prone, and frequently repeated tasks against Windows Azure and third party systems to decrease time to value for your cloud operations.
• SQL DB Premium Tiers(P1 & P2) are already in preview mode since 2013 , now there are three tiers available in this mode Basic, Standard & Premium since Web & Business edition of SQL Azure db will be deprecated on April 2015 onwards.

Details of each tiers :

• Basic (Preview): Designed for applications with a light transactional workload. Performance objectives for Basic provide a predictable hourly transaction rate.
• Standard (Preview): Standard is the go-to option for getting started with cloud-designed business applications. It offers mid-level performance and business continuity features. Performance objectives for Standard deliver predictable per minute transaction rates.
• Premium (Preview): Designed for mission-critical databases, Premium offers the highest performance levels for SQL Database and access to advanced business continuity features. Performance objectives for Premium deliver predictable per second transaction rates.

 

• Azure RemoteApp helps employees stay productive anywhere, and on a variety of devices – Windows, Mac OS X, iOS, or Android. Your company’s applications run on Windows Server in the Azure cloud, where they’re easier to scale and update. Employees install Microsoft Remote Desktop clients on their Internet-connected laptop, tablet, or phone—and can then access applications as if they were running locally. You need to sign up here

 

• You’ll need to have Azure Active Directory (Azure AD) set up in order to work with Visual Studio Online Access. You’ll also need an organizational account, which is an email address associated with either your directory or an Office 365 subscription. If you don’t use Azure AD, go here to find out how to set it up. If you use Azure AD but can’t access your organization’s directory, work with your system administrator to get set up.

 

• Windows Azure Files allows VMs in a Windows Azure Data Center to mount a shared file system using the SMB protocol. These VMs will then be able to access the file system using standard Windows file APIs (CreateFile, ReadFile, WriteFile, etc). Many VMs (or PaaS roles) can attach to these file systems concurrently, allowing you to share persistent data easily between various roles and instances. In addition to accessing your files through the Windows file APIs, you can access your data using the file REST API, which is similar to the familiar blob interface.

 

• If you’re the Account Administrator for a Windows Azure subscription, you can set up email alerts when a subscription reaches a spending threshold you choose. Alerts are currently not available for subscriptions associated with a commitment plan using Billing Services.To view and set up alerts, go to the Account Center click Subscriptions, select the subscription you want to work with, and then click Alerts. You can set up a total of five billing alerts per subscription, with a different threshold and up to two email recipients for each alert.

 

 

A Lap Around of Cloud Design Patterns – Microsoft Patterns & Practices

Patterns & Practices are key indicators of any application development life-cycle whether it’s Windows Store / Desktop app, Web , Cloud , mobility or LOB app. Microsoft P & P leverages few new patterns as well for Cloud based apps. Lets have a summation as followed:

1. Cache-aside Pattern.

2. Circuit Breaker Pattern.

3. Compensating Transaction Pattern.

4. Compensating Consumers Pattern.

5. Compute Resource Consolidation Pattern.

6. Command and Query Responsibility Segregation(CQRS) Pattern.

7. Event Sourcing Pattern.

8. External Configuration Pattern.

9. Federated Identity Pattern.

10. Gatekeeper Pattern.

11. Health Endpoint Monitoring Pattern.

12. Index Table Pattern.

13. Leader Election Pattern.

14. Materialized View Pattern.

15. Pipes and Filters Pattern.

16. Priority Queue Pattern.

17. Queue based Load leveling Pattern.

18. Retry Pattern.

19. Run time Reconfiguration Pattern.

20. Scheduler Agent Supervisor Pattern.

21. Sharding Pattern.

22. Static Content Hosting Pattern.

23. Throttling Pattern.

24. Valet Key Pattern.

25. Asynchronous messaging Pattern.

26. Autoscaling Guidance.

27. Caching Guidance

28.Compute Partitioning Guidance.

29. Data Consistency Primer.

30. Data Partitioning Guidance.

31. Data Replication & Synchronization Guidance.

32. Instrumentation & Telemetry Guidance.

33. Multiple Data Center Deployment Guidance.

34. Service Metering Guidance.

 

Take a look of this link to deep dive into each of pattern’s architecture , key scenario considerations.

To have a complete overview on the projects executed by P&P team, have a look at this site.

 

 

A Quick Review on my Latest Book – Microsoft Tabular Modelling CookBook

Last December 2013, my next book ‘Microsoft Tabular Modelling CookBook‘ where I worked as Technical Reviewer is published. Lets have a quick overview of the book.

 

Overview

 

• Develop tabular models for personal use
• Learn about the modeling techniques which are required to overcome commonly encountered problems and master advanced modeling scenarios
• Understand the techniques required to promote personal models to corporate enterprise environments and manage the models in a corporate environment
• Learn tips and tricks for querying tabular models and learn how to access the data within them for dynamic reporting including the development of interactive workbooks

Encourage to all of my blog readers to explore more about the book here