Performance

Azure Cosmos DB–Multi Master

October 8, 2018 .NET, .NET Core, .NET Framework, ASP.NET, Azure, Azure CLI, Azure Cosmos DB, CosmosDB, Data Consistancy, Data Integrity, Microsoft, Multi-master, Performance, Reliability, Resilliancy, Scalability, Scale Up No comments

During the Ignite 2018, Microsoft has announced the general availability of Multi-Master feature being introduced to Azure Cosmos DB to provide more control into data redundancy and elastic scalability for your data from different regions with multiple writes and read instances.

What is Multi-Master essentially?

Multi-master is a capability that provided as part of Cosmos DB, that would provide you multiple write regions and provides an option to handle conflict resolution automatically through different options provided by the platform. Most of the major scenarios you would encounter the conflict can be resolved with these simple configurations.

A sample diagram depicting a use case of load balanced web app writing to respective regional master:-

image

With multi-master, Azure Cosmos DB delivers a single digit millisecond write latency at the 99th percentile anywhere in the world, and now offers 99.999 percent write availability (in addition to 99.999 percent read availability) backed by the industry-leading SLAs.

image

Wow! That’s an amazing performance Cosmos DB guarantees to provide so that your mission-critical systems will have zero downtime, if they start using Cosmos DB.

 

How to Enabled Multi-Master support in your Cosmos DB solutions?

Currently multi-master can only be enabled for new Cosmos DB instances using “Enable Multi-Master” option in Azure Portal or through PowerShell or ARM templates or through SDK.

These options are detailed below with necessary examples:

1.) Azure Portal – Enable Multi-region writes and Enable geo-redundancy

image

2.) Azure CLI 
Set the “enable-multiple-write-locations” parameter to “true”

az cosmosdb create \
   –-name "thingx-cosmosdb-dev" \
   --resource-group "consmosify-dev" \
   --default-consistency-level "Session" \
   --enable-automatic-failover "true" \
   --locations "EastUS=0" "WestUS=1" \
   --enable-multiple-write-locations true \

3.) AzureRM PowerShell
In AzureRM PowerShell cmdlet – Set enableMultipleWriteLocations parameter to “true”

$locations = @(@{"locationName"="East US"; "failoverPriority"=0},
             @{"locationName"="West US"; "failoverPriority"=1})

$iprangefilter = ""

$consistencyPolicy = @{"defaultConsistencyLevel"="Session";
                       "maxIntervalInSeconds"= "10";
                       "maxStalenessPrefix"="200"}

$CosmosDBProperties = @{"databaseAccountOfferType"="Standard";
                        "locations"=$locations;
                        "consistencyPolicy"=$consistencyPolicy;
                        "ipRangeFilter"=$iprangefilter;
                        "enableMultipleWriteLocations"="true"}

New-AzureRmResource -ResourceType "Microsoft.DocumentDb/databaseAccounts" `
  -ApiVersion "2015-04-08" `
  -ResourceGroupName "consmosify-dev" `
  -Location "East US" `
  -Name "thingx-cosmosdb-dev" `
  -Properties $CosmosDBProperties

4.) Through CosmosDB SDK
Setting connection policy in DocumentDBClient and set UseMultipleWriteLocations to true.

ConnectionPolicy policy = new ConnectionPolicy
{
   ConnectionMode = ConnectionMode.Direct,
   ConnectionProtocol = Protocol.Tcp,
   UseMultipleWriteLocations = true,
};
policy.PreferredLocations.Add("East US");
policy.PreferredLocations.Add("West US");
policy.PreferredLocations.Add("West Europe");
policy.PreferredLocations.Add("North Europe");
policy.PreferredLocations.Add("Southeast Asia");
policy.PreferredLocations.Add("Japan East");
policy.PreferredLocations.Add("Japan West");

Azure Cosmos DB multi-master configuration is the game changes that really makes it a true global scale database with automatic conflict resolution capabilities for data synchronization and consistancy.

In my later sessions I will write examples to cover how conflict resolutions can be configured and used in realtime scenarios.

Useful Refs:

Azure Cosmos DB – 429 Too Many Requests

October 6, 2018 .NET, Azure, CosmosDB, Document DB, Microsoft, Performance, Reliability, Resilliancy, Scalability, Visual Studio 2017, VisualStudio, VS2017 No comments

Recently while I was doing Performance Testing in one of the APIs interacting with Cosmos DB, I encountered a problem as Azure Cosmos DB API’s started returning Http Code 429.  Http Status Code 429 indicates that too many request been received or request rate is very large. This error would happen when we have concurrent users trying to write or read from same cosmos db collection.

Following diagram covers the architecture of the performance test I am performing:

image

Based on analysis it found out to be the Throttling happening from Azure Cosmos DB, as we make requests that may use more than provisioned Request Units(RU) per second. We were using default Cosmos DB configuration for a fixed collection of 1000 RU’s per second which is sufficient enough for a 500 reads and 100 writes for a 1 kb file. You can refer more about Request Units from Azure Docs.

image

 

 

 

Solution(s):

1. Now first logical step we can do is to get rid off this error by increasing the Throughput for the collection.  I am going to increase to 10000 RU/s maximum allocatable for a Storage Capacity: Fixed.   This should ideally improve the Throughput for 250 or more virtual users hitting.

image

2. Second logical step is to improve the code: Improve the connection parameters in the Document DB SDK –> DocumentDbClient. For this I referred to the Microsoft Docs: Performance tips for Azure Cosmos DB and .NET

Providing optimum values to the following Properties in RetryOption class   to be passed as parameter to Connection Policy.

image

 

In my case I provided a value of 30 to give ultimate results:

new RetryOptions() { MaxRetryAttemptsOnThrottledRequests = 30, MaxRetryWaitTimeInSeconds = 30  }

That should resolve most of the 429 issues when dealing with Cosmos DB SDK

Azure Cosmos DB – Consistency Levels

June 2, 2018 Azure, CosmosDB, Data Consistancy, Data Integrity, Higher Availability, Microsoft, Reliability, Resilliancy, Scalability No comments

CosmosDB is a planet scale multi model, multi-region NoSQL database service provided as part of Azure Platform. Azure Cosmos DB is designed to provide global distribution for every data model you choose while creating Cosmos DB.  It is promised to provide low latency and various well-defined consistency models to ensure data redundancy and high availability.

In this short diagram I will be covering the different consistency models available with Cosmos DB and their benefits:

image

There said depending on your data criticality and needs of faster accessibility, you can choose between any of the above consistency models. I strongly trusts in session consistency, as it ensures a balance b/w both.  But again it is totally depending on your business case and how critical is your system depends on the accuracy of this data.

Hope you enjoyed this short article!.

Further reads: https://docs.microsoft.com/en-us/azure/cosmos-db/consistency-levels

Azure Functions App–Run OnDemand Serverless code – a path way to Serverless Computing

June 18, 2017 App Service, Azure, Azure Functions, CosmosDB, Microsoft, Resilliancy, Scalability, Windows Azure Development, Windowz Azure No comments

Azure Functions is a new cloud solution from Azure that would let you execute small pieces code or “functions” in the cloud.  This means you do not have to worry about the infrastructure or environment to execute your little piece of code to solve any of your business problems.

functions-logo

Functions can make development even more productive, and you can use your development language of choice.

Benefits:

  • Pay only for the time your code runs and trust Azure to scale as needed.
  • Azure Functions lets you develop serveries applications on Microsoft Azure.
  • Supports wide variety of development language choices , such as C#, F#, Node.js, Python or PHP.
  • Bring your own dependencies – you can bring any of your Nuget/NPM dependencies for your functional logic.

What can we do with Azure Functions?

Azure Functions is a very good  solution for processing data, integrating systems, working with the internet-of-things (IoT), and building simple APIs and micro services.

Functions provides templates to help you  get started with some useful scenarios, including the following:

  • BlobTrigger – Process Azure Storage blobs when they are added to containers. You might use this function for image resizing.
  • EventHubTrigger – Respond to events delivered to an Azure Event Hub. Particularly useful in application instrumentation, user experience or workflow processing, and Internet of Things (IoT) scenarios.
  • Generic Webhook – Process webhook HTTP requests from any service that supports webhooks.
  • GitHub Webhook – Respond to events that occur in your GitHub repositories.
  • HTTPTrigger – Trigger the execution of your code by using an HTTP request.
  • QueueTrigger – Respond to messages as they arrive in an Azure Storage queue.
  • ServiceBusQueueTrigger – Connect your code to other Azure services or on-premises services by listening to message queues.
  • ServiceBusTopicTrigger – Connect your code to other Azure services or on-premises services by subscribing to topics.
  • TimerTrigger – Execute cleanup or other batch tasks on a predefined schedule.

Integration Support with other Azure Services:

Following are the services integration supported by Azure Functions app.

  • Azure Cosmos DB
  • Azure Event Hubs
  • Azure Mobile Apps (tables)
  • Azure Notification Hubs
  • Azure Service Bus (queues and topics)
  • Azure Storage (blob, queues, and tables)
  • GitHub (webhooks)
  • On-premises (using Service Bus)
  • Twilio (SMS messages)

Costing:

Azure functions will be charged based on two pricing plans below:

  1. App Service Plan – if you already have an Azure App Service running with Logic, Web, Mobile or Web Job, you can use the same environment for your Azure functions execution without needing to pay for extra resources.  You will be charged based on regular app service rates.
  2. Consumption plan  – with this plan you only need to pay for how long and how many times your functions runs and computational needs/resource usage during that execution time. Consumption plan pricing includes a monthly free grant of 1 million requests and 400,000 GB-s of resource consumption per month.

You can find further pricing related info here

Support and SLA:

  • Free billing and subscription management support
  • Flexible support plans starting at $29/month. Find a plan
  • 99.95% guaranteed up time. Read the SLA

Useful Links:

.NET Framework 4.7–Released for All versions of Windows

May 3, 2017 .NET, .NET 4.7, .NET Framework, .NET Framework 4.7, ASP.NET, ASP.NET MVC, C#.NET, Caching, Cryptography, Extensions, Microsoft, Performance, Security, Visual Studio 2013, Visual Studio 2015, Visual Studio 2017, VisualStudio, VS2012, VS2013, VS2015, WCF, Web API, Web API v2.0, Windows, Windows 10, Windows 7, Windows 8, Windows 8.1, WinForms, WPF No comments

Microsoft has released next version of .NET Framework (do not get confused with .NET Core) .  Though the .NET Framework 4.7 was released as part of Windows 10 Creators Update a month ago. You can now install the .NET Framework 4.7 on other versions of Windows

Download the: .NET Framework 4.7  – Web installer  |  Offline Installer

.NET Framework 4.7 Developer Pack  In order to add support for .NET Framework 4.7 in Visual Studio 2012 or later we need to install Developer Pack.

Windows Versions and Support:

The .NET Framework 4.7 is supported on the following Windows versions:

  • Windows 10 Creators Update (included in-box)
  • Windows 10 Anniversary Update
  • Windows 8.1
  • Windows 7 SP1

The .NET Framework 4.7 is supported on the following Windows Server versions:

  • Windows Server 2016
  • Windows Server 2012 R2
  • Windows Server 2012
  • Windows Server 2008 R2 SP1

New Features in .NET Framework 4.7:

On a high-level below are the set of new features introduced in following areas:

Core

Networking  Default operating system support for TLS protocols*

ASP.NET

  • Object Cache Extensibility  (plug in new implementations of an object cache for an ASP.NET application by using the new ICacheStoreProvider interface. )
  • Memory monitoring (Developers can now write their own memory monitors to replace the default by using the ApplicationMonitors.MemoryMonitor property.)
  • Memory Limit Reactions. (Developers can now replace or supplement the default behavior by subscribing IObserver implementations to the application’s memory monitor.

Windows Communication Foundation (WCF) 

  • Ability to configure the default message security settings to TLS 1.1 or TLS 1.2
  • Improved reliability of WCF applications and WCF serialization

Windows FormsHigh DPI support

Windows Presentation Foundation (WPF)

  • Support for a touch/stylus stack based on Windows WM_POINTER messages
  • New implementation for WPF printing APIs

Also improvements in :

  • High DPI support for Windows Forms applications on Windows 10
  • Touch support for WPF applications on Windows 10
  • Enhanced cryptography support
  • Support for C# 7 and VB 15, including ValueTuple
  • Support for .NET Standard 1.6
  • Performance and reliability improvements

 

Additional References:

IoT Security–Essentials–Part 01

February 1, 2017 Cloud to Device, Communication Protocols, Connected, Connectivity, Contrained Networks/Devices, Device to Cloud, Geolocation, Identity of Things (IDoT), Internet Appliance, Internet of Things, IoT, IoT Privacy, IoT Security, machine-to-machine (M2M), Machines, Tech-Trends No comments , , , , ,

Security(Cyber Security) is an essential requirement for any IoT platform or devices or end users and the communication infrastructure.  In order to achieve or design best possible security solutions,  to avoid some external entity or hacker gaining access to your IoT device or infrastructure, every architect or system designer should do Threat Modeling exercise.  As the system is designed and architected, we can minimize the exposure to external threats to our IoT architecture.

With this article I am trying to provide you relevant bits and pieces essential for your understanding:

What is Cyber Security?

As per WhatIs.com – “Cybersecurity is the body of technologies, processes and practices designed to protect networks, computers, programs and data from attack, damage or unauthorized access. In a computing context, security includes both cybersecurity and physical security.”

To make it more clear and simpler – Cyber Security also known as Computer security, or IT security, is the protection of computer systems from the theft or damage to their hardware, software or information, as well as from disruption or misdirection of the services they provide. Cyber security includes controlling physical access to the hardware, as well as protecting against harm that may come via network access, data and code injection.

What is Threat Modeling?

The objective of threat modeling is to understand how an attacker might be able to compromise a system and then make sure appropriate mitigations are in place. Threat modeling forces the design team to consider mitigations as the system is designed rather than after a system is deployed. This fact is critically important, because retrofitting security defenses to a myriad of devices in the field is infeasible, error prone and will leave customers at risk.

[Content courtesy:  Microsoft]

In order to optimize security best practices, it is recommended that a proposed IoT architecture be divided into several component/zones as part of the threat modeling exercise.

Relevant Important  Zones  for an IoT architecture  :

  • Device,
  • Field Gateway,
  • Cloud gateways, and
  • Services.

Each zone is separated by a Trust Boundary, which is noted as the dotted red line in the diagram below. It represents a transition of data/information from one source to another. During this transition, the data/information could be subject to Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service and Elevation of Privilege (STRIDE).

[Content courtesy:  Microsoft]

This diagram like below provides a full 360 view you any proposed solution:

iot-security-architecture-fig1

Summary of important Sections/Zones:

  1. The Device Zonerepresents a thing or device where device to device or local user physical access is possible.
  2. The Field Gateway Zone –  Field gateway is a device/appliance (Embedded/Hardware) or some general-purpose software that runs on a Physical Server, and acts as communication enabler and potentially, as a device control system and device data processing hub.
  3. The Cloud Gateway ZoneCloud gateway is a system that enables remote communication from and to devices or field gateways from several different sites across public network space, typically towards a cloud-based control and data analysis system, a federation of such systems.
  4. The Services Zone –  A “service” is  any software component or module that is interfacing with devices through a field- or cloud gateway for data collection and analysis, as well as for command and control. Services are mediators.

Once we identified threat boundaries we should be able to provide fail safe security measures each associated zones, to meet the business needs and global information exchange and data compliance  standards.  It is also important to design the product from the start with security in mind because understanding how an attacker might be able to compromise a system helps make sure appropriate mitigations are in place from the beginning.

In next session, we will go through Microsoft’s IoT Reference architecture and associated security measures been put together across each zones. 

Additional Resources: