Getting started, and benchmarking Spark RAPIDS on Kubernetes and fast S3 — Quite a lot of big data and most machine learning workloads are compute-bond. While machine learning, especially deep learning, has been leveraging GPU to accelerate the computation for years, GPU acceleration for big data is relatively new. It is good to see active developments of Spark RAPIDS in this area. …

A review from a field data and machine learning architect — It is a beautiful sunny Friday morning today. I am sitting at a table near the window in a nice cafe, listening to the peaceful music, looking at the trees through the window. I am starting my last workday of 2022. There are only some system-generated reports in my email…

Consolidating and accelerating big data with fast S3, Kubernetes and Spark RAPIDS — A few years ago, I worked on a project to build a large-scale Hadoop & Spark cluster from scratch. We ended up using 1000+ servers to create a 50+ PBs HDFS cluster and run tens of hundreds of Hive and Spark jobs there. I enjoyed the whole process — from…

Combining the strength of data lake and warehouse in a way that is open, simple, and runs anywhere — Data lakes represent the first step towards gaining insights from ever-growing data. In many cases, it is the first place collected data lands on to the data system. Data lake creates two challenges: Data quality and governance. Everything is just a file/object in a data lake. Performance. Limited query optimisation…

Big data benchmarks using TPC-DS and YCSB with HDFS, FlashBlade S3, and Amazon S3 — Every big data user I talk to has a data lake and data warehouse use case. It typically starts with Hadoop, using HDFS as the data lake and Apache Spark for distributed processing. A data warehouse is always there because everyone likes SQL. The big trend in this area is…

Understand, protect, and leverage metadata in big data systems — Back in 2009, when I built my first big data system using Hadoop, it was less than 10 nodes, processing single-digit terabytes of data. Nowadays, that amount of data can be stored in a laptop. Data is getting bigger. We get excited about the hundreds of terabytes, or even petabytes…

Best ways to run Spark jobs on Kubernetes for development, data exploration and in production — During the years of working on Apache Spark applications, I have always been swtiching my environment between development and production. I would use an IDE such as Visual Studio Code to write the Scala or PySpark code, test it locally against a small piece of the data, submit the Spark…

Address challenges of managing and sharing GPU and data in AI infrastructure with a couple of clicks. — How easy is it to access your AI infrastructure? Pure Storage has partnered with Lablup to make on-premise AI accessible and fast. Lablup’s Backend.ai is an open-source computing resource orchestrator designed for AI/ML. It makes AI easier to access for your data scientists. Adding Pure Storage’s FlashBlade®, an industry-leading flash-based…

Building a scalable and reliable logging solution for large Kubernetes cluster with scalable tools and fast object storage. — Building a basic logging solution for Kubernetes could be as easy as running a couple of commands. However, to support large-scale Kubernetes clusters, the logging solution itself needs to be scalable and reliable. In my previous blog, I described an overview of my Kubernetes monitoring and logging solution. At that…