Intelligent Cloud

Azure Cosmos DB – TTL (Time to Live) – Reference Usecase

October 9, 2018 .NET, .NET Core, .NET Framework, Analytics, Architecture, Azure, Azure, Azure Cosmos DB, Azure Functions, Azure IoT Suite, Cloud Computing, Cold Path Analytics, CosmosDB, Emerging Technologies, Hot Path Analytics, Intelligent Cloud, Intelligent Edge, IoT Edge, IoT Hub, Microsoft, Realtime Analytics, Visual Studio 2017, VisualStudio, VS2017, Windows No comments

TTL capability within Azure Cosmos DB is a live saver, as it would take necessary steps to purge redudent data based on the configurations you may. 

Let us think in terms of an Industrial IoT scenario, devices can produce vast amounts of telemetry information, logs and user session information that is only useful until we operate on them and take action on them, to be specific up to finate period of time. Once that data becomes surplus, we need an application logic that purges these old records.

With the “Time to Live” or TTL, Microsoft Cosmos DB provides an ability to have your documents automatically purged from database storage after a certian period if time(which you configured)

  • This TTL by default can be set on a document collection level and later can be overridden on a per document basis.
  • Once the TTL is set, Cosmos DB service will automatically remove the documents when its lifetime is over.
  • Inorder to track TTL, Cosmos DB uses an offset field to check when it was last modified.  This field is identifiable as “_ts”, which exists in every document you create.  Basically it is a UNIX epoch timestamp. This field is updated everytime when the document is modified. [Ref: Picture1]

image

[Picture1]

Enabling TTL on Cosmos DB Collection:

You can enable TTL on a Cosmos DB collection simply by using Azure Portal –> Cosmos DB collection setting for existing or during creation of  a new collection)

TTL value needs to be set in seconds – if you need 90 days => 60 sec * 60 min * 24 hour * 90 days = 7776000 seconds

image

[Picture2]

Below is a one of the reference architecture in which Cosmos DB – TTL would be essentially useful and viable to any Iot business case:

image

[Picture3]

Hope that was helpful to get some understanding. For more references visit:  Cosmos DB Documentation

Azure in China

May 17, 2017 .NET, AppFabric, Azure, Azure In China, Azure IoT Suite, Cloud Computing, Cloud Services, IaaS, Intelligent Cloud, Intelligent Edge, IoT Hub, Media Services, Microsoft, Mobile Services, PaaS, SQL Azure, SQL Server, VisualStudio, Windowz Azure 1 comment

Microsoft Azure presence in China is always a question when there is a need for any customer to deploy azure applications specifically for Chinese Regional customers.

Recently I had an interaction with a Microsoft Certified Trainer, who carelessly said Azure only uses Chinese partner based environment is only for serving CDN (Content Delivery Network) needs, partial knowledge is difficult to convince. Being a Certified Trainer, he should have known these. As I recollect my earlier experience with azure, I was aware that Microsoft has partnered with 21Vianet , to setup a Chinese instance of Azure with most possible IaaS, PaaS and SaaS offerings are available to Chinese customers. At the same time data would reside within Chinese governments data privacy and storage regulations.

Similarly other competitor cloud providers such as IBM also partnered with 21Vianet to setup their BlueMix instances in China.

Microsoft Azure in China

General availability of Azure in China through 21vianet partnership has been announced in March 26th 2014. These Azure Services in Chinese region will be operated and sold by 21vianet.

Quoting to Doug Hauger GM- National Cloud Programs, Microsoft

Windows Azure, operated by 21Vianet, is based on the same technology as Microsoft’s global Windows Azure service and offers a reliable, flexible and value-based service to bring Microsoft’s latest cloud technologies and benefits to Chinese customers and propel forward technological development in China as a whole.

Why Partnership?

As per Chinese Government(PRC) Cyber Security Law and Governance policies there is a clear restriction for a foreign entrant to obtain license and provide services to Chinese customers.  All the data relating to Chinese customers should reside within Chinese Data Centres, and foreign companies would have to pass on the operational ownership to a Chinese company in order to operate within PRC(Peoples Republic of China).

In Additional to that on November’2016 new amendments attached to new PRC Cyber Security Law (effective 1st June 2017).

As per DLA PIPER

The new law will come into force on 1 June 2017 and has significant implications for the data privacy and cybersecurity practices of both Chinese companies and international organisations doing business in China.

The new PRC Cybersecurity Law intends to combat online fraud and protect China against Internet security risks. In short, it imposes new security and data protection obligations on “network operators”; puts restrictions on transfers of data outside China by “key information infrastructure operators”; and introduces new restrictions on critical network and cybersecurity products.  [Read more from Source]

Where is Azure in China?

You can access Azure in China instance through a separate portal https://www.azure.cn. Azure Portal is available at https://portal.azure.cn/. Pricing information is available on http://www.azure.cn/pricing/pia/  (click on English to get an English version).

Below are two data centre regions Azure is available in China.

image

Services Offered in Azure China Instances:

Not all all services of Azure including Mobile Services, Analytics and IoT Suite are available for Chinese regions. I will try to cover a specific list of services that are available for China region.Azure China is a few versions behind regular Azure and not all services are supported. Some of the services supported are:

  • Compute – Virtual Machines (For IaaS Services)
  • Compute – Cloud Service (For PaaS Services)
  • Compute – Web Site
  • Data Services – Storage (For Blobs, tables and Queues)
  • Data Services – SQL Database
  • Data Service – HDInsight
  • SQL Database – SQL Import/ExportNetworks – Virtual Network (To communicate between cloud services via Local networks)
  • Service Bus
  • Active Directory
  • Access Control Service – which has its own UI and namespace
  • Caching – only dedicated cache is supported
  • Mobile Service
  • Media Service
  • CDN
  • Traffic Manager
  • Azure IoT Suite
  • Notification Center
  • Event Center
  • Service Fabric
  • Batch Service
  • Flow Analytics, Power BI
  • Document DB, SQL Database, SQL Data Warehouse, MySQL
  • and many more

image

A list of supported and not supported services can be found at (This is in Chinese, and it is a working document, use google translate to take you through all steps necessary for you to understand services offered in Azure China) http://www.windowsazure.cn/documentation/articles/developerdifferences/#readyonwacn

This also includes all the endpoints that are required for developers to integrate/develop applications targeting China region.

Hope this information was helpful to you. I would assume it would be helpful for some who are specifically looking forward for providing technology solutions for Chinese customers through Azure Cloud.

Sources:

Introducing Azure IoT Edge

May 13, 2017 .NET, Analytics, Artificial Intelligence(AI), Augmented Reality, Azure, Azure IoT Suite, Cloud Computing, Data Analytics, Edge Analytics, Embedded, Emerging Technologies, Event Hubs, Industrial Automation, Intelligent Cloud, Intelligent Edge, IoT, IoT Edge, IoT Hub, Linux, Mac OSX, Machine Learning(ML), Microsoft, Robotics, Self Driven Cars, Stream Analytics, Windows, Windowz Azure No comments

During Build! 2017 Microsoft has announced the availability of Azure IoT Edge, which would bring in some of the cloud capabilities to edge devices/networks within your Enterprise. This would enable industrial devices to utilize the capabilities of IoT in Azure within their constrained resources . 

With this Microsoft now makes it easier for developers to move some of their computing needs to these devices.  Edge devices are mostly having small foot print based to high end machines within your company network.

The essential capabilities to be supported by Azure IoT edge  include:

  • Perform Edge Analytics (a cut down version of Azure Stream Analytics)- Instead of doing analytics in cloud developer/implementer can move the basic cloud data processing and analytical capabilities to Edge Device. Run your machine learning algorithms in Edge device and take predictive analytics steps.
  • Perform Artificial Intelligence processing at edge device itself. Availability of Microsoft Cognitive Service on edge device would bring in whole lot of automation capabilities. Imagine Alexa/Siri working without internet connection, it should be able to provide you reminders etc.
  • Perform RealTime Decision making locally based on predefined rules.
  • Reduce bandwidth costs
  • Connect to other Edge devices and legacy devices within the constrained/corporate network.
  • Deploy IoT solutions to Edge Device from Cloud and provide updates as needed.
  • Operate offline without the need of real-time internet connectivity or intermittent connectivity. Doesn’t have to rely on Cloud to provide commands for processing, can do offline data capture and processing of information from other devices connected and take decisions without the need to rely on a connected cloud service.

Azure IoT Edge enables seamless deployment of cloud services such as:

Along with sharing the image represents Azure’s Enterprise Digital Vision, we will discuss about the same in later sessions:

Digital-Enterprise-Vision_png

Getting Started & More information:

IoT Central–Microsoft’s SaaS solution for IoT

April 25, 2017 AMQP, Analytics, Azure, Azure IoT Suite, Cloud Computing, Cloud Services, Cloud to Device, Communication Protocols, Connected, Connectivity, Device to Cloud, Emerging Technologies, HTTP 1.1, Identity of Things (IDoT), Intelligent Cloud, Intelligent Edge, Internet of Things, IoT, IoT Central, IoT Devices, IoT Edge, IoT Hub, IoT Privacy, IoT Security, Machines, MQTT, PaaS, SaaS, Stream Analytics No comments

Microsoft has today released their IoT SaaS offering for customers and partners called as “Microsoft IoT Central”.  IoT Central enables powerful IoT scenarios without requiring cloud solution expertise and also simplifies the development process and makes customers to make quick time to market solutions, making digital transformation more accessible to everyone without overhead of implement solutions end to end.

As per Microsoft :

“IoT Central provides an easier way to create connected products that propel digital business. Take the complexity out of the Internet of Things (IoT) with a true, end-to-end IoT software as a service (SaaS) solution in the cloud that helps you build, use, and maintain smart products.”

Benefits of IoT Central:

  • Proven platform and technology with enterprise grade security.
  • Reduced complexities of setting up and maintaining IoT infrastructure and solutions.
  • Building smart connected products with lesser cost  and lesser overhead would ensure higher customer satisfaction.
  • Quickly adapt to changing environments.

For those would need control on implementing end to end can still choose the PaaS solution Azure IoT Suite.

Below is a picture from @JanakiramMSV’s article from forbes.com, to help you have a high level look at all the IoT offerings from Microsoft.

az-iot

Sources:

IoT Hub vs Event Hub–A quick comparison

December 11, 2016 Azure, Cloud Computing, Cloud to Device, Communication Protocols, Connectivity, Contrained Networks/Devices, Data Hubs, Device Shadow, Device to Cloud, Device Twin, Emerging Technologies, Event Hubs, HTTP2, Identity of Things (IDoT), Intelligent Cloud, Internet of Things, Interoperability, IoT, IoT Hub, IoT Privacy, IoT Security, Messaging, Microsoft, Performance, Protocols, Reliability, Scalability, Tech-Trends No comments

With this article I am trying to provide you a birds eye view comparison of IoT Hub and Azure Event Hub, so that some of you may stop feeling that there is nothing new in IoT Hub.

For the interest of this article, I put together a table with side-by-side comparison of some important features/desired features from an IoT Hub like platform.

Feature IoT Hub Event Hub
Communication Supports both device-to-cloud and cloud-to-device bidirectional communication Supports only device-to-cloud communication
State Management Can maintain device state using Device Twins and query them whenever needed. Not Supported
Protocol Support AMQP 1.1, AMQP over Web Sockets, MQTT 3.2, MQTT over Web Sockets, HTTP 1.1, Web Sockets. AMQP 1.1, AMQP over Web Sockets, HTTP 11 , Web Sockets only
Protocol Extensions Provides IoT protocol gateway a customizable implementation for industrial protocol channelling. Not Supported
Security Provides identity to each device and easily revocable through IoT Hub Device Management portal. Shared access policies with limited revocation capabilities are provided.
Monitoring/ Operations Provides a rich set of features through Device Management capability. Includes individually enable/disable or provision new device. Change security keys as needed. View/identify individual device problems easily. Does not provide individual performance metrics. Can provide only a high level aggregated metrics only.
Scalability Scalable to thousands/millions of simultaneous devices Limited number of simultaneous connections up to 5000 connections per Azure Service Bus Quotas. Event Hub provides a capability to partition your message to channel it in to associated Service Bus quotas.
SDK Support/ Developer Support Provides very good Integration SDK and developer support. Both Azure IoT  Device SDK and IoT Gateway SDK are the most essential kits provided for almost all devices/OS platforms. It also support all the latest programming languages such as C#, Node.js, Java and Python.
Also provides  direct MQTT, AMQP and REST based HTTP APIs.
Very detail oriented documentation provided.
.NET, Java and C apart from protocols such as AMQP, HTTP API interfaces.
Files/Images Upload Capability Supports IoT devices/solutions to upload files/images/snapshots to cloud and define a workflow for processing them. Not Available
Message Routing Very decent message routing capability is available out of the box. Up to 10 end points can be defined and Advanced Rules can be defined on how routing should occur. Requires additional programming and hosting to support as per the need.

From this comparison table, you can analyse that IoTHub is the right candidate for your IoT solution needs, as Event Hub lacking certain capabilities that are essential for an IoT Ingestion point. If you are only requiring to send messages to cloud and doesn’t require any fancy stuff as IoTHub provides, you can choose Event Hub.

Remember with more power comes more responsibility, that’s what IotHub intend to provide to you.

Hope this overview was helpful. Please feel free to comment or initiate a discussion any time. Please share your feedbacks on this article as well.