Preview 2-708.pptx

1.0x

Preview 2-708.pptx

Created 2 years ago

Duration 0:00:00
lesson view count 99
Select the file type you wish to download
Slide Content
  1. Clemens Szyperski

    Slide 1 - Clemens Szyperski

    • Principal Group Engineering Manager
    • Dipanjan Banik
    • Program Manager II
    • Gaining Real-Time IoT InsightsUsing Azure Stream Analytics, Azure ML, and Power BI
    • 2-708
    • //build/ content is being presented by Microsoft Office Mix The video for this session will be available shortly
  2. What is Streaming Data?

    Slide 2 - What is Streaming Data?

    • Data in Motion
    • Data at Rest
  3. Demo

    Slide 3 - Demo

    • How real time analytics changes the business dynamics in the healthcare industry
  4. What are customers wanting to do?

    Slide 4 - What are customers wanting to do?

    • Smart grid
    • CRM alerting sales to customer case
    • Data and identity protection services
    • Smart retail
    • Real-time fraud detection
    • Click-stream analysis
    • Real-time financial portfolio alerts
    • Connected carsSmart cities
  5. PoC inFujitsu Akisai Plant Factorypowered by Microsoft AzureRichard McCormack, Fujitsu Americarmccormack@us.fujitsu.com, @RichardMcCor

    Slide 5 - PoC inFujitsu Akisai Plant Factorypowered by Microsoft AzureRichard McCormack, Fujitsu Americarmccormack@us.fujitsu.com, @RichardMcCor

  6. What is Akisai?

    Slide 6 - What is Akisai?

    • One-stop ICT solution by Fujitsu’s Food & Agriculture Cloud.
    • Variety of innovative solutions and services for agribusiness.
    • Greenhouse Horticulture
    • High Value Crops
    • Sales
    • Production
    • Management
    • Biz Analysis
    • Sales Delivery
    • Accounting
    • Store & Analyze Sensing Data
    • One-stop ICT Solutions and their Support Services
    • Collect Environment Data
    • Optimize Each Operations
    • Open field Cultivation
    • Animal Husbandry
    • 4
    • 1. Production Management
    • 3. Greenhouse Horticulture
    • 2. Remote Sensing Network
    • 4. Animal Husbandry
  7. Slide 7

    • Video
  8. Slide 8

    • Innovation in Akisai Plant Factory
    • Cultivation
    • Technology
    • Akisai Plant Factory in Fukushima
    • 1st  Innovation
    • Semi-conductor
    • Manufacturing
    • ICT
    • Mass production of Clean Lettuce with expertise
    • ×
    • ×
    • 2nd  Innovation powered by Microsoft Azure
    • Excel
    • product quality
    • Excel
    • Productivity
    • NOW
    • Data
    • consolidation
    • Copyright 2015 FUJITSU LIMITED
  9. Future Direction of Akisai Plant Factory

    Slide 9 - Future Direction of Akisai Plant Factory

    • Copyright 2015 FUJITSU LIMITED
    • ・Improve production & reduce costs
    • ・Expand channels and business
    • Improve Business
    • ・Collaborate with local communities/businesses
    • ・To be an incubation center
    • ・Promote an advanced agriculture to the world
    • ・Expand line-ups of low-Potassium vegetables
    • ・Pursue tastier, healthier products
    •  ⇒ Control quality/ingredients with optimized envs
    • ・A reference model with FJ solutions
    • ・World-class showcase
    • ・Visitors impressed with Fujitsu!
    • PoC of ICT Solutions
    • Contribute to Tohoku Recovery
    • Provide Foods with pleasure
  10. PoC in Akisai Plant Factory

    Slide 10 - PoC in Akisai Plant Factory

    • OperationsOptimization
    • Overall optimization of production & management
    • Management Dashboard
    • Akisai, facility management system, etc.
    • Visualization
    • Energy
    • Shipments
    • Rejects
    • Potassium
    • Yield
    • Man-hours
    • Harvests
    • Weight
    • Primary Data
    • Secondary Data
    • Decision Making
    • Visualization /Realization
    • Temperature
    • Humidity
    • Lighting
    • CO2
    • Airflow
    • Nutrient Solution
    • Lots
    • Production
    • control
    • Operational
    • Level
    • Management Level
    • Executive Level
    • Overall optimization of production and management at the Akisai Plant Factory in Fukushima
    • Copyright 2015 FUJITSU LIMITED
  11. Slide 11

    • Demo
  12. System landscape

    Slide 12 - System landscape

    • Copyright 2015 FUJITSU LIMITED
    • Management Dashboard
    • M2M/IoT
    • Platform
    • Stream Analytics
    • Cloud gateways
    • Event hubs
    • Hot Path
    • Cold Path
    • Management Daashborad
    • Hybrid
    • Event
    • Data
    • M2M
    • Azure Cloud
    • Office 365
    • Teamsite
    • System
    • On-Premises
    • (AIZU Factory)
    • Machine Learning
    • Teamsite
    • Copyright 2015 FUJITSU LIMITED
  13. Lessons learned

    Slide 13 - Lessons learned

    • IoT technology ecosystem and co-innovation partnership matter
    • IoT and Big Data are key enablers of business innovation
    • but complexity of architecture and E2E stack is a big challenge
    • Equally important success factors
    • Careful selection of IoT ecosystem
    • Effective co-innovation partnership
    • End users want to focus just on IoT Apps that enable business growth
    • Need agile IoT App development, deployment and business process integration
    • Enabled by Cloud-based IoT Platform-as-a-Service functionality
    • Co-innovation of Fujitsu, Microsoft & Customers support rapid Proof-of-Business
    • To accelerate the business innovation learning curve
    • Copyright 2015 FUJITSU LIMITED
  14. Project “Inception”Microsoft Technology Center and NEC

    Slide 15 - Project “Inception”Microsoft Technology Center and NEC

    • Todd Van Nurden
    • Chief Architect, Technical Solutions, MTC
  15. Problem:How do we deliver ambient intelligence?

    Slide 16 - Problem:How do we deliver ambient intelligence?

    • Engage quests, users, or visitors more naturally
    • Make the environment sensitive to a users needs
    • Augment existing infrastructure to support transparent user engagement
    • Understand a users intent
    • Make it SimpleDo it in weeks not years and don’t require a PhD
  16. Slide 17

    • Passive Attract
    • Active Attract
    • Passive
    • Engaged
    • Engaged
    • Customer Needs Assistance
    • How it works
  17. Inception Framework

    Slide 18 - Inception Framework

    • Azure
    • Hadoop
    • Kiosk
    • Biometrics Services
    • Telemetry Services
    • Kinect
    • Sensor
    • Azure Event Hub
    • ASA
    • Interactions
    • ASA
    • Biometrics
    • ASA
    • Telemetry
    • Interactions
    • Biometrics
    • Telemetry
    • Hive
    • Script
    • Interactions
    • Hive Table
    • Biometrics
    • Hive Table
    • Telemetry
    • Hive Table
    • Excel
    • Inception – Logical/Physical
    • Interaction
    • Services
  18. Inception

    Slide 19 - Inception

    • A transparent computing experiencedesigned to allow people to be people and have the environment change and react based on what they are doing
    • Initial collaboration between Target and the Microsoft Technology Center – Minneapolis
    • NEC developed an extension to the Inception work that allows the system to acquire anonymous demographics as well as supporting the world-best face recognition API
    • The original prototype was completed in 3 weeks;the pilot version was delivered in 3 months
  19. Demo

    Slide 20 - Demo

    • Inception Framework
  20. Introducing the Inception Framework

    Slide 21 - Introducing the Inception Framework

    • Kinect Telemetry Capture
    • NEC Telemetry Capture
    • Azure Scripts to fully automate the deployment of Inception components: Storage, ASA, HD Insight
    • Kiosk Reference App and Code
    • Biometrics and Face Recognition Reference App and Code
    • Basic Object Detection and Interaction Components
  21. Go Do’s

    Slide 22 - Go Do’s

    • Inception + Windows 10 will let you quickly create experiences that boarder on magic
    • Sign up to take Inception for a spin(Get the code – Stop by the Kinect Booth!)
    • Make amazing things!
  22. Canonical scenarios

    Slide 23 - Canonical scenarios

    • Archiving
    • Dashboarding
    • Triggering Workflows
  23. Canonical Stream Analytics Pattern

    Slide 24 - Canonical Stream Analytics Pattern

    • Presentation and action
    • Storage and
    • Batch Analysis
    • StreamAnalysis
    • Ingestion
    • Collection
    • Event production
    • Event hubs
    • Cloud gateways(web APIs)
    • Field gateways
    • Applications
    • Legacy IOT (custom protocols)
    • Devices
    • IP-capable devices(Windows/Linux)
    • Low-power devices (RTOS)
    • Search and query
    • Data analytics(Power BI)
    • Web/thick client dashboards
    • Event Hubs
    • SQL DB
    • Storage Tables
    • Power BI
    • Storage Blobs
    • Stream Analytics
    • Devices to take action
    • Machine
    • Learning
    • more to come…
  24. Introducing Azure Stream Analytics

    Slide 25 - Introducing Azure Stream Analytics

    • Mission critical reliability and scale
    • Enables rapid development
    • Fully managed real-time analytics
  25. Real-time analytics

    Slide 26 - Real-time analytics

    • Fully managed real-time analytics
    • Real-time Analytics
    • Intake millions of events per second (up to 1 GB/s)
    • Low processing latency, auto adaptive (sub-second to seconds)
    • Correlate between different streams, or with reference data
    • Find patterns or lack of patterns in data in real-time
    • Fully Managed Cloud Service
    • No hardware acquisition and maintenance
    • No platform/infrastructure deployment and maintenance
    • Easily expand your business globally leveraging Azure regions
  26. Mission critical

    Slide 27 - Mission critical

    • Mission Critical Reliability
    • Guaranteed event delivery
    • Guaranteed business continuity: Automatic and fast recovery
    • Effective Audits
    • Privacy and security properties of solutions are evident
    • Azure integration for monitoring and ops alerting
    • Easy To Scale
    • Scale from small to large on demand
    • Mission critical reliability and scale
  27. Rapid development

    Slide 28 - Rapid development

    • Rapid Development with SQL like language
    • High-level: focus on stream analytics solution
    • Concise: less code to maintain
    • Fast test: Rapid development and debugging
    • First-class support for event streams and reference data
    • Built in temporal semantics
    • Built-in temporal windowing and joining
    • Simple policy configuration to manage out-of-order eventsand late arrivals
    • Enables rapid development
  28. Customers using Azure Stream Analytics

    Slide 29 - Customers using Azure Stream Analytics

    • Infrastructure – Procure and setup
    • Develop solution (code) for ingress, processing and egress
    • Develop solutions to integrate with other components like ML, BI etc
    • Develop solutions to manage resiliency, such as infrastructure failures
    • Develop solutions and infrastructure for increasing scale with business growth
    • Monitoring and troubleshooting of solution
    • Focus on building solutions
    • … not on solution infrastructure
    • … and get there faster
  29. DML

    Slide 30 - DML

    • SELECT
    • FROM
    • WHERE
    • GROUP BY
    • HAVING
    • CASE WHEN THEN ELSE
    • INNER/LEFT OUTER JOIN
    • UNION
    • CROSS/OUTER APPLY
    • CAST
    • INTO
    • ORDER BY ASC, DSC
    • SAQL – Language & Library
    • Scaling Extensions
    • WITH
    • PARTITION BY
    • OVER
    • Date and Time Functions
    • DateName
    • DatePart
    • Day
    • Month
    • Year
    • DateTimeFromParts
    • DateDiff
    • DateAdd
    • Windowing Extensions
    • TumblingWindow
    • HoppingWindow
    • SlidingWindow
    • Aggregate Functions
    • Sum
    • Count
    • Avg
    • Min
    • Max
    • StDev
    • StDevP
    • Var
    • VarP
    • String Functions
    • Len
    • Concat
    • CharIndex
    • Substring
    • PatIndex
    • Temporal Functions
    • Lag, IsFirst
    • CollectTop
  30. Scenario – Twitter Analytics

    Slide 31 - Scenario – Twitter Analytics

    • ID
    • CreatedAt
    • UserName
    • TimeZone
    • Text
    • Language
    • Topic
    • 1
    • 2015-04-30T20:45:30
    • Joshua X
    • Eastern Time (US & Canada)
    • Oh, joy! More forced @Xbox Live updates
    • en
    • XBox
    • 2
    • 2015-04-30T20:45:31
    • Cristabel Y
    • London
    • RT @verge: Streaming Xbox One games ..
    • en
    • XBox
    • “A news media website wants to increase site traffic by covering trending topics on social media.”
    • To determine which topics are immediately relevant to customers, they need real-time analytics about the tweet volume and sentiment for each topic.
    • TwitterStream
  31. Filters

    Slide 32 - Filters

    • SELECT UserName, TimeZone
    • FROM InputStream
    • WHERE Topic = 'XBox'
    • Show me the user name and time zone of tweets on the topic XBox
    • "Haroon”, “Eastern Time (US & Canada)”
    • "XO", “London”
    • “Zach Dotseth“, “London”, “Football”,(…)
    • "Haroon”, “Eastern Time (US & Canada)”
    • “XBox”,(…)
    • "XO",”London”,
    • “XBox“, (…)
    • time
  32. Windowing Concepts

    Slide 33 - Windowing Concepts

    • Windows can be tumbling, hopping, or sliding
    • Windows are fixed length
    • Must be used in a GROUP BY clause
    • Output event will have the timestamp of the end of the window
    • 1
    • 5
    • 4
    • 2
    • 6
    • 8
    • 6
    • 4
    • t1
    • t2
    • t5
    • t6
    • t3
    • t4
    • Time
    • Window 1
    • Window 2
    • Window 3
    • Aggregate
    • Function (Sum)
    • 18
    • 14
    • Output Events
  33. Hopping Windows

    Slide 34 - Hopping Windows

    • SELECT Topic, Count(*) AS TotalTweets
    • FROM TwitterStream TIMESTAMP BY CreatedAt
    • GROUP BY Topic, HoppingWindow(second, 10, 5)
    • “Every 5 seconds give me the count of tweets over the last 10 seconds”
    • 1
    • 5
    • 4
    • 2
    • 6
    • 8
    • 6
    • 0
    • 5
    • 20
    • 10
    • 15
    • Time
    • (secs)
    • 25
    • A 10-second Hopping Window with a 5-second “Hop”
    • 30
    • 4
    • 2
    • 6
    • 8
    • 6
    • 5
    • 3
    • 6
    • 1
    • 1
    • 5
    • 4
    • 2
    • 6
    • 8
    • 6
    • 5
    • 3
    • 6
    • 1
    • 5
    • 3
  34. Joining multiple streams

    Slide 35 - Joining multiple streams

    • {“XO”, 4, “Win10”}
    • {“Jo”, 0, “Surface”}
    • {“Foo”,4, “Bing”}
    • Twitter Stream:
    • {“Dip”, 2, “XBox”}
    • {“XO”, 0, “Win10”}
    • {“Dip”, 0, “Xbox”}
    • {“Jo”, 4, “Surface”}
    • {“Foo”, 0, “Bing”}
    • Twitter Stream:
    • (same stream,
    • further down the timeline)
    • SELECT TS1.UserName, TS1.Topic
    • FROM TwitterStream TS1 TIMESTAMP BY CreatedAt
    • JOIN TwitterStream TS2 TIMESTAMP BY CreatedAt
    • ON TS1.UserName = TS2.UserName AND TS1.Topic = TS2.Topic AND DateDiff(second, TS1, TS2) BETWEEN 1 AND 60
    • WHERE TS1.SentimentScore != TS2.SentimentScore
    • time
    • “List all users and the topics on which they switched their sentiment within a minute“
  35. Slide 36

    • Detecting absence of events
    • “Show me if a topic is not tweeted for 10 seconds since it was last tweeted”
    • SELECT TS1.CreatedAt, TS1.Topic, TS1.UserName
    • FROM TwitterStream TS1 TIMESTAMP BY CreatedAt
    • LEFT OUTER JOIN TwitterStream TS2 TIMESTAMP BY CreatedAt
    • ON TS1.Topic = TS2.Topic AND DateDiff(second, TS1, TS2) BETWEEN 1 AND 10
    • WHERE TS2.Topic IS NULL
    • {“XO”, 4, “Win10”}
    • {“WAA”, 2, “Microsoft”}
    • {“AB”, 0, “Bing}
    • {“Dip”, 4, “Xbox”}
    • {“Foo”, 0, “Win10”}
    • {“Tim”, 2, “Microsoft”}
    • {“AB”, 0, “Bing”}
    • time
    • Twitter Stream:
    • Twitter Stream:
    • (same stream,
    • further down the timeline)
  36. Slide 37

    • Reference Data
    • Seamless correlation of event streams with reference data
    • Static or slowly-changing data stored in blobs
    • CSV and JSON files in Azure Blobs;scanned for new snapshots on a settable cadence
    • JOIN (INNER or LEFT OUTER) between streams and reference data sources
    • Reference data appears like another input:
    • SELECT myRefData.Name, myStream.Value
    • FROM myStream
    • JOIN myRefData
    • ON myStream.myKey = myRefData.myKey
  37. Stream Analytics

    Slide 38 - Stream Analytics

    • Scaling using Partitions
    • Partitioning allows for parallel execution over scaled-out resources
    • SELECT Count(*) AS Count, Topic
    • FROM TwitterStream PARTITION BY PartitionId
    • GROUP BY TumblingWindow(minute, 3), Topic, PartitionId
    • Query
    • Result 1
    • Query
    • Result 2
    • Query
    • Result 3
    • PartitionId = 1
    • PartitionId = 3
    • PartitionId = 2
    • PartitionId = 1
    • PartitionId = 2
    • PartitionId = 3
    • Event Hub
  38. Multiple steps, multiple outputs

    Slide 39 - Multiple steps, multiple outputs

    • WITH Step1 AS (
    • SELECT Count(*) AS CountTweets, Topic
    • FROM TwitterStream PARTITION BY PartitionId
    • GROUP BY TumblingWindow(second, 3), Topic, PartitionId
    • ),
    • Step2 AS (
    • SELECT Avg(CountTweets)
    • FROM Step1
    • GROUP BY TumblingWindow(minute, 3)
    • )
    • SELECT * INTO Output1 FROM Step1
    • SELECT * INTO Output2 FROM Step2
    • SELECT * INTO Output3 FROM Step2
    • A query can have multiple steps to enable pipeline execution
    • A step is a sub-query defined using WITH (“common table expression”)
    • Can be used to develop complex queries more elegantly by creating a intermediary named result
    • Creates unit of execution for scaling out when PARTITION BY is used
    • Each step’s output can be sent to multiple output targets using INTO
  39. Slide 40

    • Machine Learning
    • Azure ML and Stream Analytics are now integrated
    • The integration is in limited preview as of today!
    • (See team blog for sign-up information.)
    • Azure ML can publish web endpoints for operationalized models
    • Azure Stream Analytics can bind custom function names to such web endpoints
    • Example: apply bound function event-by-event
    • sentiment mapped to endpoint/API key
    • SELECT text, sentiment(text) AS score
    • FROM myStream
  40. Twitter Sentiment Analysis

    Slide 41 - Twitter Sentiment Analysis

    • Demo
  41. Stream Analytics is priced on two variables:

    Slide 42 - Stream Analytics is priced on two variables:

    • Volume of data processed
    • Streaming units required to process the data stream
    • Meter
    • Price (USD)
    • Volume of Data Processed
    • Volume of data processed by the streaming job (in GB)
    • $.001 per GB
    • Streaming Unit
    • Blended measure of cores, memory, and bandwidth
    • $0.031 per hour
    • * Streaming unit is a unit of compute capacity with a maximum throughput of 1MB/s
    • Pricing
  42. Daily Azure Stream Analytics cost for 1 MB/sec of average processing

    Slide 43 - Daily Azure Stream Analytics cost for 1 MB/sec of average processing

    • Volume of Data Processed Cost -
    • $0.001 /GB * 84.375 GB = $0.08 per day, streaming max 1 MB/s non-stop
    • Streaming Unit Cost -
    • $.031 /hr * 24 hrs = $0.74 per day, for 1 MB/sec max. throughput
    • Total cost -
    • $0.74 + $0.08 = $0.82 per day -or- $24.60 per month
    • Example Pricing
  43. Resource Library

    Slide 44 - Resource Library

    • Business Overview http://azure.microsoft.com/en-us/services/stream-analytics/
    • Documentation http://azure.microsoft.com/en-us/documentation/services/stream-analytics/
    • Samples https://github.com/streamanalytics/samples
    • ASA Blog http://blogs.msdn.com/b/streamanalytics/rss.aspx
    • Follow us on Twitter https://twitter.com/AzureStreaming (follow @AzureStreaming)
    • ASA Forum https://social.msdn.microsoft.com/Forums/en-US/home?forum=AzureStreamAnalytics
    • Vote for ideas http://feedback.azure.com/forums/270577-azure-stream-analytics
    • Email ASA Team azstream@microsoft.com
  44. Improve your skills by enrolling in our free cloud development courses at the Microsoft Virtual Academy.

    Slide 45 - Improve your skills by enrolling in our free cloud development courses at the Microsoft Virtual Academy.

    • Try Microsoft Azure for free and deploy your first cloud solution in under 5 minutes!
    • Easily build web and mobile apps for any platform with AzureAppService for free.
    • Resources