All around the cloud: Here’s everything AWS has announced so far at re:Invent

At the same time, it has significantly deepened its support for fast development, training, and deployment of high-performance artificial intelligence and other sophisticated enterprise applications in the cloud. Chief AWS announcements on re:Invent’s first day fell into several categories:

• Boosting cloud storage efficiencies;
• Simplifying cloud data movement;
• Enforcing enterprise controls on cloud workloads;
• Accelerating the pipeline for high-performance cloud apps;
• Speeding development of apps for cloud-to-edge deployment; and
• Enhancing the manageability of apps in serverless, hybrid and edge clouds

Here are Wikibon’s comprehensive dissections of these announcements:

Boosting cloud storage efficiencies

Many AWS customers store a lot of data in S3, Elastic File System and Elastic Block Storage. The vendor announced several new features that help users to improve the economics of managing all this data in AWS’ infrastructure as a service public cloud:

• Accelerated batch operations for object management in S3: AWS announced the preview of a new feature that enables customers to manage billions of objects stored in Amazon S3 with a single API request or a few clicks in the S3 Management Console. Within minutes and for any number of S3 objects, users can make changes to object properties and metadata, and perform other storage management tasks, such as copying objects between buckets, replacing tag sets, modifying access controls, and restoring archived objects from Amazon S3 Glacier. Users can also run custom AWS Lambda functions across potentially trillions of S3 objects to execute more complex tasks, such as processing data or transcoding image files. S3 Batch Operations manages retries, tracks progress, sends notifications, generates completion reports, and delivers events to AWS CloudTrail for all changes made and tasks executed.
• Intelligent tiering for nondisruptive storage cost optimization in S3: AWS announced Intelligent-Tiering, a new S3 storage class that helps customers optimize storage costs automatically when their data access patterns change. This new storage class automatically moves data between two tiers–frequent access and infrequent access–without performance impact or operational overhead. It is available today in all commercial AWS Regions and AWS GovCloud (US). For a per-object monthly monitoring and automation fee, S3 Intelligent-Tiering monitors access patterns and moves objects that have not been accessed for 30 consecutive days to the infrequent access tier, but imposes no retrieval fees. If an object in the infrequent access tier is accessed later, it is automatically moved back to the frequent access tier. It applies no additional tiering fees when objects are moved between access tiers within the S3 Intelligent-Tiering storage class.
• Boosted efficiencies for storage of infrequently access files in Elastic File System: AWS announced a forthcoming new EFS Infrequent Access Storage Class that is cost-optimized for files that are accessed less frequently. The new storage class will be generally available in all regions where EFS is available in 2019. Customers will be able to use EFS IA to store larger amounts of data in their file systems. It will reduce storage costs for files not accessed every day, with savings up to 85 percent compared to the EFS Standard storage class. Users will be able to start using EFS IA when available by creating a new file system and enabling Lifecycle Management. The new storage class will automatically move files that have not been accessed for 30 days from the Standard storage class to the Infrequent Access storage class.
• Performance enhancement for data persisted on solid-state drives in Elastic Block Storage: AWS announced peak-performance improvements in Provisioned IOPS SSD Volumes for EBS. The peak performance of these volumes has doubled from 32,000 IOPS to 64,000 IOPS and from 500 MB/s to 1,000 MB/s of throughput per volume when attached to Nitro system EC2 instances. These faster volumes are available now in commercial regions worldwide and AWS GovCloud (US) regions. The performance improvements support applications that high performance storage, such as large transactional databases, big data analytics, and log processing.

Simplifying cloud data movement

AWS customers are continually moving data into, out of, and within the vendor’s portfolio of cloud services, as well as to and from on-premises platforms and other cloud providers’ environments. To simplify, accelerate, and automate movement of this data, while boosting the efficiency of customer data management operations, AWS made the following announcements:

• Speeding cloud data synchronization: AWS announced availability of DataSync, a new online service that simplifies, automates, and accelerates data transfers. AWS DataSync automate movement of data between on-premises storage and Amazon S3 or EFS. It accelerates and secures data transfers over the Internet or AWS Direct Connect. It uses an on-premises software agent to connect to users’ existing storage, and a fully managed in-cloud service to automate, scale, and validate transfers. This spares users from having to write scripts, modify applications or manage infrastructure. It enables customers to perform one-time data migrations, transfer on-premises data for timely in-cloud analysis, and automate replication to AWS for data protection and recovery. Users pay only for the data copied to or from S3 or EFS.
• Facilitating secure file transfers in the cloud: AWS announced Transfer for SFTP, a fully managed service that enables integration of Secure Shell File Transfer Protocol-based file transfers into the AWS cloud. AWS Transfer for SFTP enables users to easily move SFTP file transfer workloads to AWS without needing to modify applications or manage any SFTP servers. Users can continue to operate with their existing SFTP clients or applications. Data uploaded or downloaded using SFTP is available in the customer’s Amazon S3 bucket, and can be used for archiving or processing in AWS. With AWS SFTP, users pay only for the use of the SFTP server endpoint, and data uploaded and downloaded.

Enforcing enterprise controls on cloud data workloads

AWS customers have bet their businesses on the public cloud and require robust security, compliance, availability, performance, and other service-level guarantees. To address these customer requirements, AWS made the following announcements:

• More stringent user control over key management: AWS announced the Key Management Service or KMS custom key store This allows users to create a dedicated single-tenant key store in KMS using AWS CloudHSM. It helps users to gain more control over their KMS keys. KMS is a fully managed service that generates encryption keys and helps users manage their use across more than 45 AWS services. The KMS custom key store integrates KMS with AWS CloudHSM to help meet compliance obligations that would otherwise require the use of on-premises hardware security modules. AWS cautioned that this new service comes at increased cost and has a potential impact on performance and availability.
• Enhanced cloud application availability and performance: AWS announcedGlobal Accelerator, a network layer service that customers can deploy in front of their internet applications to improve availability and performance. It enables customers to seamlessly route traffic to multiple regions and improve availability and performance for their end users. It directs internet traffic from users to enterprise cloud applications running in AWS regions. Users are directed to workloads based on their geographic location, application health, and configurable weights. It allocates static Anycast IP addresses that are globally unique for applications and do not change, thereby removing the need to update clients as cloud applications scale.
• More flexible networking of virtual private clouds and on-premises networks: AWS announced Transit Gateway, a new service that allows customers to simplify interconnect and management of thousands of virtual private clouds and on-premises networks. Users can leverage AWS Transit Gateway to build hub-and-spoke network topologies, connecting existing VPCs, data centers, remote offices and remote gateways to a managed Transit Gateway. They have full control over network routing and security, even if VPCs, Active Directories, shared services and other resources span multiple AWS accounts. User can also employ Transit Gateways to consolidate existing edge connectivity and route it through a single ingress/egress point.

Accelerating the pipeline for high-performance cloud apps

AWS customers are running AI in the cloud as well as other high-performance computing, networking and storage applications that require very low latency. To address these customer requirements, AWS made the following announcements:

• New marketplace availability of containerized cloud apps: AWS announcedthat customers can choose from more than 160 curated and trusted container products in AWS Marketplace. It announced the availability of 160 new trusted container products, from itself and partners, that can be quickly and easily deployed on its Amazon Elastic Container Service and Amazon ECS for Kubernetes services. The containerized products now available in AWS Marketplace consist of developer tools as well as high-performance computing and security offerings. Available in AWS Marketplace simplifies user access, set up, deployment, and billing for these offerings in AWS’ public cloud. Amazon is also adding a new feature called the AWS Private Marketplace, which enables companies to create approved lists of containerized cloud-ready software that aligns with their individual policies for streamlined acquisition and deployment by employees as needed.
• New instances supporting high-performance cloud applications: AWS announced new EC2 P3dn and C5n instances, which are the latest additions to the C5 family that are optimized for high-performance computing, networking and storage applications that require very low latency. The new instances, which are built on AWS Nitro System for very low latency, support 100 Gbps networking performance. Available next week, P3dn instances will provide powerful GPU instances in the cloud for accelerated training of machine learning models. The new P3dn instances deliver a 4X increase in network throughput compared to existing P3 instances, providing up to 100 Gbps of networking throughput, fast NVMe instance storage, custom Intel CPUs with 96 vCPUs and support for AVX512 instructions, and NVIDIA Tesla V100 GPUs each with 32 GB of memory. The new C5n instances, available now, increase the maximum throughput performance available in AWS’s compute-intensive instance family. C5 instances offer up to 25 Gbps of network bandwidth addressing the requirements of a wide range of workloads, offer 100 Gbps of network bandwidth and provide four times as much throughput as existing C5 instances. Powered by custom-designed AWS Graviton processors based on the Arm architecture, the new EC2 A1 instances support high-performance, low-cost scale-out workloads — such as containerized microservices — that are shared across a group of smaller instances. The A1 instances, available now, are EBS-optimized by default.
• New managed service, framework and library for building AI-powered robotics: AWS announced RoboMaker, which is a new managed service that enables developers to build intelligent robotics applications quickly using cloud services. It extends the open-source Robot Operating System with connectivity to such AWS services as machine learning, monitoring, and analytics. This enables robots to stream data, navigate, communicate, comprehend, and learn. It provides an AWS Cloud9-based robotics integrated development environment for a development and large-scale parallel simulation. Developers can start application development with a single click in the AWS Management Console, with AWS RoboMaker automatically provisioning the underlying infrastructure. It also supports enables over-the-air robotics fleet application deployment, update, and management in integration with AWS Greengrass. AWS RoboMaker cloud extensions for ROS include Amazon Kinesis Video Streams ingestion, Amazon Rekognition image and video analysis, Amazon Lex speech recognition, Amazon Polly speech generation, and Amazon CloudWatch logging and monitoring.

Speeding development of apps for cloud-to-edge deployment

AWS customers are running containerized microservices in “internet of things” and other cloud-to-edge application scenarios. To address these customer requirements, AWS made the following announcements:

• Collect, structure, label and search data scalably from industrial equipment: The new AWS IoT SiteWise, available in preview, is a managed service that collects data from the plant floor, structures and labels the data, and generates real time key performance indicators and metrics to help customers make better, data-driven decisions.
• Detect and respond to events many IoT sensors and applications: The new AWS IoT Events, available in preview, is a managed service that gives customers the ability to easily AWS IoT Events, available in preview, is a managed service that makes it easy for industrial, consumer and commercial customers to detect and respond to events from many different IoT sensors and applications.
• Build new IoT applications through visual, no-code tooling: The new AWS IoT Things Graph, available in preview, enables developers to build IoT applications by representing devices and cloud services as reusable models that can be combined through a visual drag-and-drop interface, instead of writing low-level code. It provides a visual way to represent complex real-world systems, and deploys IoT applications to the edge on devices running AWS Greengrass so that applications can respond more quickly, even if not connected to the Internet.
• Connect devices to third-party applications, on-premises software and AWS services: The new AWS IoT Greengrass Connectors, available today, helps developers create powerful applications that enable AWS IoT Greengrass devices to connect to third-party applications, such as ServiceNow, Splunk and Twilio, while securely handling access control and credentials.

Enhancing the manageability of apps in serverless, hybrid and edge clouds

AWS customers have integrated its services into serverless, hybrid and edge computing environments. To address these customer requirements, AWS made the following announcements:

• DevOps for Web and serverless back-ends: AWS announced the availability of Amplify Console, which provides a continuous deployment and hosting service for single-page Web applications with serverless backends. AWS Amplify Console accelerates the web and serverless application release cycle. It automatically deploys changes to front-end and back-end nodes on every code commit. Integrating with GitHub repos, it automatically determines the build settings for both the front-end framework and any serverless back-end resources that were configured with the Amplify CLI. It simplifies production and development workflows by creating new front- and back-end environments every time a developer connects a new feature branch.
• Deployment of public cloud services into on-premises environments: AWS announced the general availability of its Snowball Edge Compute Optimized AWS Snowball Edge is a pre-packaged, pre-optimized hardware solution incorporates 52 vCPUs, 208GB of memory, 7.68TB of NVMe SSD and 42TB of S3-compatible storage for running compute-intensive applications in physically harsh or offline locations. It also comes with the option to add an Nvidia Corp. TESLA V100 graphics processing unit for scenarios such as full motion video processing when connectivity is a challenge. It is optimized for running compute-intensive apps at in edge and hybrid cloud environments, being configured with more GPU compute power. It enables AWS users to do some local data processing and filtering locally, often at disconnected sites, before sending the devices and the data back to AWS. It enables users to do employ AWS Lambda functions for local processing and incorporates support for EC2 Compute Instances. The new Snowball Edge Compute Optimized with GPU option includes an on-board GPU for can real-time full-motion video analysis and processing, machine learning inferencing, and other high-performance computing workloads.
• Virtualization of secure multitenant containers across serverless clouds: AWS announced Firecracker, a lightweight, open-source virtual machine monitor that uses the Linux Kernel-based Virtual Machine. Firecracker enables creation and management of secure multitenant containers and lambda functions in serverless clouds. It is engineered to optimize the transient and stateless workloads characteristic of serverless environments. AWS Lambda uses Firecracker as the foundation for provisioning and running sandboxes upon which AWS executes customer code. Firecracker provides a secure device model that reduces memory footprint and attack surface area while providing a high density of microVMs on each server. Its minimal device model enables faster startup times (less than 125 ms on an i3.metal with the default microVM size) while reducing the attack surface. It enables packing of thousands of microVMs onto a single machine. Users can access its in-process rate limiter to control, with fine granularity, how network and storage resources are shared, even across thousands of microVMs. Firecracker runs on Intel processors today, with support for AMD and Arm coming in 2019. Firecracker will also enable popular container runtimes such as containerd to manage containers as microVMs, thereby enabling Kubernetes to use the technology. Firecracker is licensed under Apache v2.0 and downloadable from the Firecracker GitHub repo.

To catch what AWS executives, partners and customers are saying now, get drill-downs on their forthcoming announcements and see compelling glimpses into their roadmap, be sure to tune into theCUBE live this week.