Following the groundbreaking impact of DeepSeek R1<\/a>, DeepSeek AI continues to push the boundaries of innovation with its latest offering: Smallpond. This lightweight data processing framework combines the power of DuckDB<\/a> for SQL analytics<\/a> and 3FS for high-performance distributed storage, designed to efficiently handle petabyte-scale datasets. Smallpond promises to simplify data processing for AI and big data applications, eliminating the need for long-running services and complex infrastructure, marking another significant leap forward from the DeepSeek team. In this article, we will explore the features, components, and applications of DeepSeek AI\u2019s Smallpond framework, and also learn how to use it.<\/p>\n This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\n Smallpond is an open-source, lightweight data processing framework developed by DeepSeek AI, designed to extend the capabilities of DuckDB\u2014a high-performance, in-process analytical database\u2014into distributed environments.<\/p>\n By integrating DuckDB with the Fire-Flyer File System (3FS), Smallpond offers a scalable solution for handling petabyte-scale datasets without the overhead of traditional big data frameworks like Apache Spark.<\/p>\n Released on February 28, 2025, as part of DeepSeek\u2019s Open Source Week, Smallpond targets data engineers and scientists who need efficient, simple, and high-performance tools for distributed analytics.<\/p>\n Learn More: DeepSeek Releases 3FS & Smallpond Framework<\/a><\/em><\/p>\n Now let\u2019s understand the core components of DeepSeek\u2019s Smallpond framework.<\/p>\n DuckDB is an embedded, in-process SQL OLAP<\/a> database optimized for analytical workloads. It excels at executing complex queries on large datasets with minimal latency, making it ideal for single-node analytics. Smallpond extends DuckDB\u2019s capabilities to distributed systems, retaining its performance benefits.<\/p>\n 3FS is a distributed file system designed by DeepSeek for AI and high-performance computing (HPC) workloads. It leverages modern SSDs and RDMA networking to deliver low-latency, high-throughput storage (e.g., 6.6 TiB\/s read throughput in a 180-node cluster). Unlike traditional file systems, 3FS prioritizes random reads over caching, aligning with the needs of AI training and analytics.<\/p>\n Smallpond uses DuckDB as its compute engine and 3FS as its storage backbone. Data is stored in Parquet format on 3FS, partitioned manually by users, and processed in parallel across nodes using DuckDB instances coordinated by Ray. This integration combines DuckDB\u2019s query efficiency with 3FS\u2019s scalable storage, enabling seamless distributed analytics.<\/p>\n Now, let\u2019s learn how to install and use Smallpond.<\/p>\n Smallpond is Python-based and installable via pip available only for Linux distros. Ensure Python 3.8\u20133.11 is installed, along with a compatible 3FS cluster (or local filesystem for testing).<\/p>\n For 3FS, clone and build from the GitHub repository:<\/p>\n Initialize a ray instance for ray clusters if using 3FS, follow the codes below:<\/p>\n Running the above code will produce output similar to the image below:<\/p>\n Now we can initialize Ray with 3FS by using the address we got as shown above. To initialize Ray in smallpond, Configure a compute cluster (e.g., AWS EC2<\/a>, on-premises) with 3FS deployed on SSD-equipped nodes or For local testing (Linux<\/a>\/Ubuntu), use a filesystem path.<\/p>\n Smallpond primarily supports Parquet files, optimized for columnar storage and DuckDB compatibility. Other formats (e.g., CSV) may be supported via DuckDB\u2019s native capabilities.<\/p>\n Load and save data using Smallpond\u2019s high-level API.<\/p>\n Manual partitioning is key to Smallpond\u2019s scalability. Choose a strategy based on your data and workload:<\/p>\n The high-level API simplifies data loading, transformation, and saving:<\/p>\n For advanced use, Smallpond integrates DuckDB\u2019s SQL engine and Ray\u2019s task distribution directly:<\/p>\n Smallpond\u2019s performance shines in benchmarks like GraySort, sorting 110.5 TiB across 8,192 partitions in 30 minutes and 14 seconds (3.66 TiB\/min throughput) on a 50-node compute cluster with 25 3FS storage nodes.<\/p>\n Smallpond redefines distributed data processing by combining DuckDB\u2019s analytical prowess with 3FS\u2019s high-performance storage. Its simplicity, scalability, and open-source nature make it a compelling choice for modern data workflows. Whether you\u2019re preprocessing AI datasets or analyzing terabytes of logs, Smallpond offers a lightweight yet powerful solution. Dive in, experiment with the code, and join the community to shape its future!<\/p>\n The media shown in this article is not owned by Analytics Vidhya and is used at the Author\u2019s discretion<\/strong>.<\/p>\n A. DeepSeek Smallpond is an open-source, lightweight data processing framework that extends DuckDB\u2019s capabilities to distributed environments using 3FS for scalable storage and Ray for parallel processing.<\/p>\n<\/p><\/div>\n A. Smallpond is a lightweight alternative to Spark, offering high-performance distributed analytics without complex dependencies. However, it requires manual partitioning and infrastructure setup, unlike Spark\u2019s built-in resource management.<\/p>\n<\/p><\/div>\n A. Smallpond requires Python (3.8\u20133.12), a Linux-based OS, and a compatible 3FS cluster or local storage. For distributed workloads, a Ray cluster with SSD-equipped nodes is recommended.<\/p>\n<\/p><\/div>\n A. Smallpond primarily supports Parquet files for optimized columnar storage but can handle other formats through DuckDB\u2019s native capabilities.<\/p>\n<\/p><\/div>\n A. Best practices include manual data partitioning based on workload, leveraging 3FS for high-speed storage, and minimizing data shuffling across nodes to reduce network overhead.<\/p>\n<\/p><\/div>\n A. Smallpond excels at batch processing but may not be ideal for real-time analytics. For low-latency streaming data, alternative frameworks like Apache Flink or Kafka Streams might be better suited.<\/p>\n<\/p><\/div>\n<\/p><\/div>\nLearning Objectives<\/h4>\n
\n
What is DeepSeek Smallpond?<\/h2>\n
Key Features of Smallpond<\/h3>\n
\n
Core Components of DeepSeek Smallpond<\/h2>\n
DuckDB<\/h4>\n
3FS (Fire-Flyer File System)<\/h4>\n
Integration of DuckDB and 3FS in Smallpond<\/h4>\n
![\"Integration](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/Untitled-2025-03-08-1231_1-thumbnail_webp-600x300-1.webp\")
<\/figure>\nGetting Started with Smallpond<\/h2>\n
Step 1: Installation<\/h3>\n
# Install Smallpond with dependecies\npip install smallpond\n\n# Optional: Install development dependencies (e.g., for testing)\npip install \"smallpond[dev]\"\n\n# Install Ray Clusters\npip install 'ray[default]'<\/code><\/pre>\ngit clone https:\/\/github.com\/deepseek-ai\/3fs\ncd 3fs\ngit submodule update --init --recursive\n.\/patches\/apply.sh\n# Install dependencies (Ubuntu 20.04\/22.04 example)\nsudo apt install cmake libuv1-dev liblz4-dev libboost-all-dev\n# Build 3FS (refer to 3FS docs for detailed instructions)<\/code><\/pre>\nStep 2: Setting Up the Environment<\/h3>\n
#intialize ray accordingly\nray start --head --num-cpus=![\"Ray](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/Screenshot_from_2025-03-17_19-03-22.webp\")
<\/figure>\nimport smallpond\n\n# Initialize Smallpond session (local filesystem for testing)\nsp = smallpond.init(data_root=\"Path\/to\/local\/Storage\",ray_address=\"192.168.214.165:6379\")# Enter your own ray address \n\n# For 3FS cluster (update with your 3FS endpoint and ray address)\nsp = smallpond.init(data_root=\"3fs:\/\/cluster_endpoint\",ray_address=\"192.168.214.165:6379\")# Enter your own ray address\n<\/code><\/pre>\nStep 3: Data Ingestion and Preparation<\/h3>\n
Supported Data Formats<\/h4>\n
Reading and Writing Data<\/h4>\n
# Read Parquet file\ndf = sp.read_parquet(\"data\/input.prices.parquet\")\n\n# Process data (example: filter rows)\ndf = df.map(\"price > 100\") # SQL-like syntax\n\n# Write results back to Parquet\ndf.write_parquet(\"data\/output\/filtered.prices.parquet\")<\/code><\/pre>\nData Partitioning Strategies<\/h4>\n
\n
# Partition by file count\ndf = df.repartition(3)\n\n# Partition by rows\ndf = df.repartition(3, by_row=True)\n\n# Partition by column hash (e.g., ticker)\ndf = df.repartition(3, hash_by=\"ticker\")<\/code><\/pre>\nStep 4: API Referencing<\/h3>\n
High-Level API Overview<\/h4>\n
\n
Low-Level API Overview<\/h4>\n
\n
Detailed Function Descriptions<\/h4>\n
\n
df = sp.read_parquet(\"3fs:\/\/data\/input\/*.parquet\")<\/code><\/pre>\n\n
# SQL-like\ndf = df.map(\"SELECT ticker, price * 1.1 AS adjusted_price FROM {0}\")\n# Python function\ndf = df.map(lambda row: {\"adjusted_price\": row[\"price\"] * 1.1})<\/code><\/pre>\n\n
df = sp.partial_sql(\"SELECT ticker, MIN(price), MAX(price) FROM {0} GROUP BY ticker\", df)<\/code><\/pre>\nPerformance Benchmarks<\/h2>\n
![\"Performance](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/Screenshot_2025-03-10_200124.webp\")
<\/figure>\nBest Practices for Optimizing Performance<\/h2>\n
\n
Scalability Considerations<\/h2>\n
\n
Applications of Smallpond<\/h2>\n
\n
Advantages and Disadvantages of Smallpond<\/h2>\n
\n\n
\n \nFeature<\/th>\n Advantages<\/th>\n Disadvantages<\/th>\n<\/tr>\n<\/thead>\n \n Scalability<\/strong><\/td>\n Handles petabyte-scale data efficiently<\/td>\n Cluster management overhead<\/td>\n<\/tr>\n \n Performance<\/strong><\/td>\n Excellent benchmark performance<\/td>\n May not optimize single-node performance<\/td>\n<\/tr>\n \n Cost<\/strong><\/td>\n Open-source and cost-effective<\/td>\n Dependence on external frameworks<\/td>\n<\/tr>\n \n Usability<\/strong><\/td>\n User-friendly API for ML developers<\/td>\n Security concerns related to DeepSeek\u2019s AI models<\/td>\n<\/tr>\n \n Architecture<\/strong><\/td>\n Distributed computing with DuckDB and Ray Core<\/td>\n None<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n Conclusion<\/h2>\n
Key Takeaways<\/h4>\n
\n
Frequently Asked Questions<\/h2>\n
![\"Kabyik](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_5Adsf54.webp\")
<\/p>\n