notebooks/Python - Polars dataframe complete user guide.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Ultimate Guide to Data Wrangling with Python | Rust Polars Data Frame\n",
    "---\n",
    "[Download PDF version of this notebook](https://github.com/AmitXShukla/RPA/blob/main/SampleData/The%20Ultimate%20Guide%20to%20Data%20Wrangling%20with%20Python%20-%20Rust%20Polars%20Data%20Frame.pdf)\n",
    "\n",
    "[Video Tutorials](https://www.youtube.com/playlist?list=PLp0TENYyY8lHJaY4t5bAihnFS5TBUQYV1)\n",
    "\n",
    "    Author: Amit Shukla\n",
    "[https://github.com/AmitXShukla](https://github.com/AmitXShukla)\n",
    "\n",
    "[https://x.com/ashuklax](https://x.com/AShuklaX)\n",
    "\n",
    "[https://youtube.com/@Amit.Shukla](https://youtube.com/@Amit.Shukla)\n",
    "\n",
    "\n",
    "The aim of this comprehensive user guide is to equip you with all the necessary knowledge and skills required to utilize Python Polars Data Frame effectively for financial and supply chain data science analytics. \n",
    "\n",
    "It provides an in-depth overview of the most commonly used functions and capabilities of the package.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "I'm Amit Shukla, and I specialize in training neural networks for finance supply chain analysis, enabling them to identify data patterns and make accurate predictions.\n",
    "During the challenges posed by the COVID-19 pandemic, I successfully trained GL and Supply Chain neural networks to anticipate supply chain shortages. The valuable insights gained from this effort have significantly influenced the content of this tutorial series.\n",
    "\t\n",
    "#### Objective:\n",
    "By delving into this powerful tool, we will master the fundamental techniques of Data Wrangling. This knowledge is crucial in preparing finance and supply chain data for advanced analytics, visualization, and predictive modeling using neural networks and machine learning.\n",
    "\t\n",
    "#### Subject\n",
    "It's important to note that this particular series will concentrate solely on `Data Wrangling`. \n",
    "Throughout this series, our main focus will be on learning `Rust Polars Data Frame Python library`.\n",
    "\t\n",
    "#### Following\n",
    "However, in future installments, we will explore Data Analytics and delve into the realm of machine learning for predictive analytics.\n",
    "\tThank you for joining me, and I'm excited to embark on this educational journey together.\n",
    "\t\n",
    "Let's get started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of content\n",
    "---\n",
    "\n",
    "- Why another DataFrame\n",
    "- Installation\n",
    "- Finance and Supply chain Data Analytics\n",
    "- Creating Polars DataFrame\n",
    "- Data Types & Casting\n",
    "- IO\n",
    "- About Contexts\n",
    "    - selection\n",
    "    - filtering\n",
    "    - group by\n",
    "- About Expressions\n",
    "- Functions, User defined functions and Windows function\n",
    "- Transformation\n",
    "- Polar SQLs\n",
    "- Lazy API | Eager execution"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why another DataFrame\n",
    "---\n",
    "Despite the numerous state-of-the-art dataframe packages available in the market, the Polar dataframe, which is built on RUST, boasts the fastest execution speed, enabling it to handle complex data science operations on tabular datasets.\n",
    "\n",
    "- Execution on larger-than-memory (RAM) data analytics\n",
    "- Lazy API vs Eager execution\n",
    "- Automatic Optimization\n",
    "- Embarrassingly Parallel\n",
    "- Easy to learn consistent, predictable API that has strict schema\n",
    "- SQLs like expressions\n",
    "\n",
    "#### Efficient Execution of Analytics on Large-than-Memory (RAM) Data\n",
    "\n",
    "RAM is not a big deal these days as most computers and VMs offer inexpensive GBs of RAM. In fact, the availability of affordable RAM is the primary reason why Pandas-like DataFrames remain the go-to choice, and it is unlikely that Pandas or R Tables will become obsolete anytime soon.\n",
    "\n",
    "However, Polars DataFrames are increasingly gaining popularity among developers due to their ability to harness the horsepower of Apache Spark, the backend support of DuckDB and Apache Arrow, and the ease-of-use of Pandas-like data frame functionalities. \n",
    "\n",
    "Additionally, Polars comes with built-in multi-core, multi-threaded parallel processing, making it a highly preferred choice.\n",
    "\n",
    "#### Lazy API vs Eager execution\n",
    "\n",
    "Just because an API is referred to as \"lazy\" does not necessarily imply that there will be a delay in processing or execution, and conversely, \"eager\" execution doesn't necessarily mean that the programming language will process data transformations or begin execution immediately and more quickly.\n",
    "\n",
    "In simpler terms, using a Lazy API implies that the API will first take the time to optimize the query before execution, which often results in improved performance.\n",
    "\n",
    "To illustrate this concept, consider running SQL on an RDBMS database. If the statistics, indexes, and data partitions have been appropriately optimized and the SQL is written in an optimized manner that utilizes the available statistics, indexes, and data partitions, the results will be delivered more quickly.\n",
    "\n",
    "#### Automatic Optimization\n",
    "\n",
    "We will learn few automation techniques to efficiently optimize queries.\n",
    "\n",
    "#### Embarrassingly Parallel\n",
    "\n",
    "#### Easy to learn consistent, predictable API that has strict schema\n",
    "\n",
    "#### SQLs like expressions\n",
    "\n",
    "`Let's get started`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U numpy"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finance and Supply chain Data Analytics\n",
    "---\n",
    "\n",
    "#### Finance data model\n",
    "\n",
    "A finance data model is a comprehensive and structured framework used to represent and organize financial information within an organization.\n",
    "\n",
    "It serves as the blueprint for how financial data is collected, stored, processed, and analyzed, ensuring accuracy, consistency, and efficiency in managing financial operations. \n",
    "\n",
    "The model defines the relationships between various financial entities such as assets, liabilities, revenues, expenses, and equity, enabling financial professionals to gain insights into the company's financial health, performance, and risk exposure. \n",
    "\n",
    "It typically encompasses multiple dimensions, including time, currency, and geographical locations, chart of accounts, departments / cost centers, fiscal years and reporting accounting periods providing a holistic view of the organization's financial landscape. \n",
    "\n",
    "A well-designed finance data model is critical for generating accurate financial reports, facilitating financial planning and forecasting, and supporting strategic decision-making at all levels of the business.\n",
    "\n",
    "As stated above, since our objective is learn Data Science operations on Finance and Supply chain dataset, we will focus on creating\n",
    "few real life examples which are similar to Finance and Supply chain.\n",
    "\n",
    "For more information, please learn more about [Finance and Supply chain ERP data](https://amitxshukla.github.io/GeneralLedger.jl/tutorials/erd/).\n",
    "\n",
    "Objective of following section is to understand ERP GL like data. \n",
    "\n",
    "A sample of data structure and ERD relationship diagram can be seen in this diagram below.\n",
    "\n",
    "`Finance ER Diagram`\n",
    "\n",
    "![ERD Diagram](https://github.com/AmitXShukla/AmitXShukla.github.io/raw/master/blogs/PlutoCon/gl_erd.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Supply chain data model\n",
    "\n",
    "A supply chain data model is a structured representation of the various elements and interactions within a supply chain network.\n",
    "\n",
    "It encompasses critical components such as customers, orders, receipts, products, invoices, vouchers, and ship-to locations. \n",
    "\n",
    "Customers form the foundation of the supply chain, as they drive demand for products. Orders and receipts represent the flow of goods and services, capturing the movement of inventory throughout the supply chain. \n",
    "\n",
    "The product entity accounts for the diverse range of items being handled, from raw materials to finished goods.\n",
    "\n",
    "Invoices and vouchers track financial transactions, ensuring transparent and accurate billing processes. \n",
    "\n",
    "Ship-to locations specify the destinations of goods during the distribution process.\n",
    "\n",
    "By establishing relationships and attributes between these elements, the supply chain data model aids in optimizing inventory management, forecasting demand, enhancing order fulfillment, and ultimately, improving overall operational efficiency within the supply chain ecosystem.\n",
    "\n",
    "`Supply Chain ER Diagram`\n",
    "\n",
    "![ERD Diagram](../SampleData/ER_Flow.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Polars DataFrame\n",
    "---\n",
    "\n",
    "#### Polars Data Structure\n",
    "\n",
    "The core base data structures provided by Polars are Series and DataFrames. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finance DataSet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Location Type: Series 1:  shape: (5,)\n",
      "Series: '' [str]\n",
      "[\n",
      "\t\"CA\"\n",
      "\t\"OR\"\n",
      "\t\"WA\"\n",
      "\t\"TX\"\n",
      "\t\"NY\"\n",
      "]\n",
      "Location Type: Series 2:  shape: (5,)\n",
      "Series: 'location' [str]\n",
      "[\n",
      "\t\"CA\"\n",
      "\t\"OR\"\n",
      "\t\"WA\"\n",
      "\t\"TX\"\n",
      "\t\"NY\"\n",
      "]\n",
      "Location Type: DataFrame 1:  shape: (5, 1)\n",
      "┌──────────┐\n",
      "│ column_0 │\n",
      "│ ---      │\n",
      "│ str      │\n",
      "╞══════════╡\n",
      "│ CA       │\n",
      "│ OR       │\n",
      "│ WA       │\n",
      "│ TX       │\n",
      "│ NY       │\n",
      "└──────────┘\n",
      "Location Type: DataFrame 2:  shape: (5, 1)\n",
      "┌──────────┐\n",
      "│ location │\n",
      "│ ---      │\n",
      "│ str      │\n",
      "╞══════════╡\n",
      "│ CA       │\n",
      "│ OR       │\n",
      "│ WA       │\n",
      "│ TX       │\n",
      "│ NY       │\n",
      "└──────────┘\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(polars.series.series.Series,\n",
       " polars.series.series.Series,\n",
       " polars.series.series.Series)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#########################\n",
    "## Series and DataFrames\n",
    "#########################\n",
    "import polars as pl\n",
    "\n",
    "# with a tuple\n",
    "location_1 = pl.Series([\"CA\", \"OR\", \"WA\", \"TX\", \"NY\"]) \n",
    "# location_1 series when will converted to DataFrame will not have a column name\n",
    "# later is used to create a DataFrame will assign a column name like column_xx\n",
    "    \n",
    "location_2 = pl.Series(\"location\", [\"CA\", \"OR\", \"WA\", \"TX\", \"NY\"])\n",
    "\n",
    "print(f\"Location Type: Series 1: \", location_1)\n",
    "print(f\"Location Type: Series 2: \", location_2)\n",
    "\n",
    "location_1_df = pl.DataFrame(location_1)\n",
    "location_2_df = pl.DataFrame(location_2)\n",
    "print(f\"Location Type: DataFrame 1: \", location_1_df)\n",
    "print(f\"Location Type: DataFrame 2: \", location_2_df)\n",
    "# type(location_1_df[\"location\"]) \n",
    "# will error out, because location_1 series didn't had column name\n",
    "type(location_2_df[\"location\"]), type(location_1), type(location_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>AS_OF_DATE</th><th>DESCRIPTION</th><th>REGION</th><th>TYPE</th><th>CATEGORY</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>0</td><td>21</td><td>2022-01-01 00:00:00</td><td>&quot;Dallas&quot;</td><td>&quot;Region C&quot;</td><td>&quot;Physical&quot;</td><td>&quot;Recv&quot;</td></tr><tr><td>1</td><td>12</td><td>2022-01-01 00:00:00</td><td>&quot;New York&quot;</td><td>&quot;Region B&quot;</td><td>&quot;Physical&quot;</td><td>&quot;Recv&quot;</td></tr><tr><td>2</td><td>13</td><td>2022-01-01 00:00:00</td><td>&quot;Philadelphia&quot;</td><td>&quot;Region C&quot;</td><td>&quot;Physical&quot;</td><td>&quot;Mfg&quot;</td></tr><tr><td>3</td><td>16</td><td>2022-01-01 00:00:00</td><td>&quot;Atlanta&quot;</td><td>&quot;Region B&quot;</td><td>&quot;Physical&quot;</td><td>&quot;Mfg&quot;</td></tr><tr><td>4</td><td>20</td><td>2022-01-01 00:00:00</td><td>&quot;Kansas City&quot;</td><td>&quot;Region B&quot;</td><td>&quot;Physical&quot;</td><td>&quot;Ship&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 7)\n",
       "┌───────┬─────┬─────────────────────┬──────────────┬──────────┬──────────┬──────────┐\n",
       "│ Row # ┆ ID  ┆ AS_OF_DATE          ┆ DESCRIPTION  ┆ REGION   ┆ TYPE     ┆ CATEGORY │\n",
       "│ ---   ┆ --- ┆ ---                 ┆ ---          ┆ ---      ┆ ---      ┆ ---      │\n",
       "│ u32   ┆ i64 ┆ datetime[μs]        ┆ str          ┆ str      ┆ str      ┆ str      │\n",
       "╞═══════╪═════╪═════════════════════╪══════════════╪══════════╪══════════╪══════════╡\n",
       "│ 0     ┆ 21  ┆ 2022-01-01 00:00:00 ┆ Dallas       ┆ Region C ┆ Physical ┆ Recv     │\n",
       "│ 1     ┆ 12  ┆ 2022-01-01 00:00:00 ┆ New York     ┆ Region B ┆ Physical ┆ Recv     │\n",
       "│ 2     ┆ 13  ┆ 2022-01-01 00:00:00 ┆ Philadelphia ┆ Region C ┆ Physical ┆ Mfg      │\n",
       "│ 3     ┆ 16  ┆ 2022-01-01 00:00:00 ┆ Atlanta      ┆ Region B ┆ Physical ┆ Mfg      │\n",
       "│ 4     ┆ 20  ┆ 2022-01-01 00:00:00 ┆ Kansas City  ┆ Region B ┆ Physical ┆ Ship     │\n",
       "└───────┴─────┴─────────────────────┴──────────────┴──────────┴──────────┴──────────┘"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating DataFrame from a dict or a collection of dicts.\n",
    "# let's create a more sophisticated DataFrame\n",
    "# in real world, Organization maintain dozens of record structure to store \n",
    "# different type of locations, like ShipTo Location, Receiving, \n",
    "# Mailing, Corp. office, head office,\n",
    "# field office etc. etc.\n",
    "\n",
    "########################\n",
    "## LOCATION DataFrame ##\n",
    "########################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "location = pl.DataFrame({\n",
    "    \"ID\":  list(range(11, 23)),\n",
    "    \"AS_OF_DATE\" : datetime(2022, 1, 1),\n",
    "    \"DESCRIPTION\" : [\"Boston\",\"New York\",\"Philadelphia\",\"Cleveland\",\"Richmond\",\n",
    "                     \"Atlanta\",\"Chicago\",\"St. Louis\",\"Minneapolis\",\"Kansas City\",\n",
    "                     \"Dallas\",\"San Francisco\"],\n",
    "    \"REGION\": [\"Region A\",\"Region B\",\"Region C\",\"Region D\"] * 3,\n",
    "    \"TYPE\" : \"Physical\",\n",
    "    \"CATEGORY\" : [\"Ship\",\"Recv\",\"Mfg\"] * 4\n",
    "})\n",
    "location.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 9)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>AS_OF_DATE</th><th>DESCRIPTION</th><th>REGION</th><th>TYPE</th><th>STATUS</th><th>CLASSIFICATION</th><th>CATEGORY</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>0</td><td>27000</td><td>2022-01-01 00:00:00</td><td>&quot;Liabilities&quot;</td><td>&quot;Region C&quot;</td><td>&quot;L&quot;</td><td>&quot;Active&quot;</td><td>&quot;LIABILITIES&quot;</td><td>&quot;Return&quot;</td></tr><tr><td>1</td><td>38000</td><td>2022-01-01 00:00:00</td><td>&quot;Operating Expe…</td><td>&quot;Region D&quot;</td><td>&quot;E&quot;</td><td>&quot;Active&quot;</td><td>&quot;OPERATING_EXPE…</td><td>&quot;Temp&quot;</td></tr><tr><td>2</td><td>20000</td><td>2022-01-01 00:00:00</td><td>&quot;Liabilities&quot;</td><td>&quot;Region A&quot;</td><td>&quot;L&quot;</td><td>&quot;Active&quot;</td><td>&quot;LIABILITIES&quot;</td><td>&quot;Depreciation&quot;</td></tr><tr><td>3</td><td>40000</td><td>2022-01-01 00:00:00</td><td>&quot;Assets&quot;</td><td>&quot;Region A&quot;</td><td>&quot;A&quot;</td><td>&quot;Active&quot;</td><td>&quot;ASSETS&quot;</td><td>&quot;Sales&quot;</td></tr><tr><td>4</td><td>18000</td><td>2022-01-01 00:00:00</td><td>&quot;Non Operating …</td><td>&quot;Region D&quot;</td><td>&quot;E&quot;</td><td>&quot;Active&quot;</td><td>&quot;NON-OPERATING_…</td><td>&quot;Investment&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 9)\n",
       "┌───────┬───────┬───────────────┬───────────────┬───┬──────┬────────┬───────────────┬──────────────┐\n",
       "│ Row # ┆ ID    ┆ AS_OF_DATE    ┆ DESCRIPTION   ┆ … ┆ TYPE ┆ STATUS ┆ CLASSIFICATIO ┆ CATEGORY     │\n",
       "│ ---   ┆ ---   ┆ ---           ┆ ---           ┆   ┆ ---  ┆ ---    ┆ N             ┆ ---          │\n",
       "│ u32   ┆ i64   ┆ datetime[μs]  ┆ str           ┆   ┆ str  ┆ str    ┆ ---           ┆ str          │\n",
       "│       ┆       ┆               ┆               ┆   ┆      ┆        ┆ str           ┆              │\n",
       "╞═══════╪═══════╪═══════════════╪═══════════════╪═══╪══════╪════════╪═══════════════╪══════════════╡\n",
       "│ 0     ┆ 27000 ┆ 2022-01-01    ┆ Liabilities   ┆ … ┆ L    ┆ Active ┆ LIABILITIES   ┆ Return       │\n",
       "│       ┆       ┆ 00:00:00      ┆               ┆   ┆      ┆        ┆               ┆              │\n",
       "│ 1     ┆ 38000 ┆ 2022-01-01    ┆ Operating     ┆ … ┆ E    ┆ Active ┆ OPERATING_EXP ┆ Temp         │\n",
       "│       ┆       ┆ 00:00:00      ┆ Expenses      ┆   ┆      ┆        ┆ ENSES         ┆              │\n",
       "│ 2     ┆ 20000 ┆ 2022-01-01    ┆ Liabilities   ┆ … ┆ L    ┆ Active ┆ LIABILITIES   ┆ Depreciation │\n",
       "│       ┆       ┆ 00:00:00      ┆               ┆   ┆      ┆        ┆               ┆              │\n",
       "│ 3     ┆ 40000 ┆ 2022-01-01    ┆ Assets        ┆ … ┆ A    ┆ Active ┆ ASSETS        ┆ Sales        │\n",
       "│       ┆       ┆ 00:00:00      ┆               ┆   ┆      ┆        ┆               ┆              │\n",
       "│ 4     ┆ 18000 ┆ 2022-01-01    ┆ Non Operating ┆ … ┆ E    ┆ Active ┆ NON-OPERATING ┆ Investment   │\n",
       "│       ┆       ┆ 00:00:00      ┆ Expenses      ┆   ┆      ┆        ┆ _EXPENSES     ┆              │\n",
       "└───────┴───────┴───────────────┴───────────────┴───┴──────┴────────┴───────────────┴──────────────┘"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "########################\n",
    "## ACCOUNTS DataFrame ##\n",
    "########################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "accounts = pl.DataFrame({\n",
    "    \"ID\":  list(range(10000, 45000, 1000)),\n",
    "    \"AS_OF_DATE\" : datetime(2022, 1, 1),\n",
    "    \"DESCRIPTION\" : [\"Operating Expenses\",\"Non Operating Expenses\",\"Assets\",\n",
    "                     \"Liabilities\",\"Net worth accounts\", \"Statistical Accounts\",\n",
    "                     \"Revenue\"] * 5,\n",
    "    \"REGION\": [\"Region A\",\"Region B\",\"Region C\",\"Region D\", \"Region E\"] * 7,\n",
    "    \"TYPE\" : [\"E\",\"E\",\"A\",\"L\",\"N\",\"S\",\"R\"] * 5,\n",
    "    \"STATUS\" : \"Active\",\n",
    "    \"CLASSIFICATION\" : [\"OPERATING_EXPENSES\",\"NON-OPERATING_EXPENSES\", \n",
    "                        \"ASSETS\",\"LIABILITIES\",\"NET_WORTH\",\"STATISTICS\",\n",
    "                        \"REVENUE\"] * 5,\n",
    "    \"CATEGORY\" : [\n",
    "       \t\t\"Travel\",\"Payroll\",\"non-Payroll\",\"Allowance\",\"Cash\",\n",
    "       \t\t\"Facility\",\"Supply\",\"Services\",\"Investment\",\"Misc.\",\n",
    "       \t\t\"Depreciation\",\"Gain\",\"Service\",\"Retired\",\"Fault.\",\n",
    "       \t\t\"Receipt\",\"Accrual\",\"Return\",\"Credit\",\"ROI\",\n",
    "       \t\t\"Cash\",\"Funds\",\"Invest\",\"Transfer\",\"Roll-over\",\n",
    "       \t\t\"FTE\",\"Members\",\"Non_Members\",\"Temp\",\"Contractors\",\n",
    "       \t\t\"Sales\",\"Merchant\",\"Service\",\"Consulting\",\"Subscriptions\"\n",
    "       \t],\n",
    "})\n",
    "accounts.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 9)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>AS_OF_DATE</th><th>DESCRIPTION</th><th>REGION</th><th>STATUS</th><th>CLASSIFICATION</th><th>TYPE</th><th>CATEGORY</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>0</td><td>2200</td><td>2022-01-01 00:00:00</td><td>&quot;Information Te…</td><td>&quot;Region A&quot;</td><td>&quot;Active&quot;</td><td>&quot;IT&quot;</td><td>&quot;I&quot;</td><td>&quot;IT_Staff&quot;</td></tr><tr><td>1</td><td>1100</td><td>2022-01-01 00:00:00</td><td>&quot;Human Resource…</td><td>&quot;Region B&quot;</td><td>&quot;Active&quot;</td><td>&quot;HR&quot;</td><td>&quot;H&quot;</td><td>&quot;human_resource…</td></tr><tr><td>2</td><td>1500</td><td>2022-01-01 00:00:00</td><td>&quot;Sales &amp; Market…</td><td>&quot;Region C&quot;</td><td>&quot;Active&quot;</td><td>&quot;SALES&quot;</td><td>&quot;S&quot;</td><td>&quot;sales&quot;</td></tr><tr><td>3</td><td>2000</td><td>2022-01-01 00:00:00</td><td>&quot;Sales &amp; Market…</td><td>&quot;Region B&quot;</td><td>&quot;Active&quot;</td><td>&quot;SALES&quot;</td><td>&quot;S&quot;</td><td>&quot;sales&quot;</td></tr><tr><td>4</td><td>1800</td><td>2022-01-01 00:00:00</td><td>&quot;Business leade…</td><td>&quot;Region C&quot;</td><td>&quot;Active&quot;</td><td>&quot;BUSINESS&quot;</td><td>&quot;B&quot;</td><td>&quot;business&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 9)\n",
       "┌───────┬──────┬───────────────┬───────────────┬───┬────────┬───────────────┬──────┬───────────────┐\n",
       "│ Row # ┆ ID   ┆ AS_OF_DATE    ┆ DESCRIPTION   ┆ … ┆ STATUS ┆ CLASSIFICATIO ┆ TYPE ┆ CATEGORY      │\n",
       "│ ---   ┆ ---  ┆ ---           ┆ ---           ┆   ┆ ---    ┆ N             ┆ ---  ┆ ---           │\n",
       "│ u32   ┆ i64  ┆ datetime[μs]  ┆ str           ┆   ┆ str    ┆ ---           ┆ str  ┆ str           │\n",
       "│       ┆      ┆               ┆               ┆   ┆        ┆ str           ┆      ┆               │\n",
       "╞═══════╪══════╪═══════════════╪═══════════════╪═══╪════════╪═══════════════╪══════╪═══════════════╡\n",
       "│ 0     ┆ 2200 ┆ 2022-01-01    ┆ Information   ┆ … ┆ Active ┆ IT            ┆ I    ┆ IT_Staff      │\n",
       "│       ┆      ┆ 00:00:00      ┆ Technology    ┆   ┆        ┆               ┆      ┆               │\n",
       "│ 1     ┆ 1100 ┆ 2022-01-01    ┆ Human         ┆ … ┆ Active ┆ HR            ┆ H    ┆ human_resourc │\n",
       "│       ┆      ┆ 00:00:00      ┆ Resource      ┆   ┆        ┆               ┆      ┆ e             │\n",
       "│ 2     ┆ 1500 ┆ 2022-01-01    ┆ Sales &       ┆ … ┆ Active ┆ SALES         ┆ S    ┆ sales         │\n",
       "│       ┆      ┆ 00:00:00      ┆ Marketing     ┆   ┆        ┆               ┆      ┆               │\n",
       "│ 3     ┆ 2000 ┆ 2022-01-01    ┆ Sales &       ┆ … ┆ Active ┆ SALES         ┆ S    ┆ sales         │\n",
       "│       ┆      ┆ 00:00:00      ┆ Marketing     ┆   ┆        ┆               ┆      ┆               │\n",
       "│ 4     ┆ 1800 ┆ 2022-01-01    ┆ Business      ┆ … ┆ Active ┆ BUSINESS      ┆ B    ┆ business      │\n",
       "│       ┆      ┆ 00:00:00      ┆ leaders       ┆   ┆        ┆               ┆      ┆               │\n",
       "└───────┴──────┴───────────────┴───────────────┴───┴────────┴───────────────┴──────┴───────────────┘"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "##########################\n",
    "## DEPARTMENT DataFrame ##\n",
    "##########################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "dept = pl.DataFrame({\n",
    "    \"ID\":  list(range(1000, 2500, 100)),\n",
    "    \"AS_OF_DATE\" : datetime(2022, 1, 1),\n",
    "    \"DESCRIPTION\" : [\"Sales & Marketing\",\"Human Resource\",\n",
    "                     \"Information Technology\",\"Business leaders\",\"other temp\"] * 3,\n",
    "    \"REGION\": [\"Region A\",\"Region B\",\"Region C\"] * 5,\n",
    "    \"STATUS\" : \"Active\",\n",
    "    \"CLASSIFICATION\" : [\"SALES\",\"HR\", \"IT\",\"BUSINESS\",\"OTHERS\"] * 3,\n",
    "    \"TYPE\" : [\"S\",\"H\",\"I\",\"B\",\"O\"] * 3,\n",
    "    \"CATEGORY\" : [\"sales\",\"human_resource\",\"IT_Staff\",\"business\",\"others\"] * 3,\n",
    "})\n",
    "dept.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 9)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>LEDGER</th><th>ORG</th><th>FISCAL_YEAR</th><th>PERIOD</th><th>ACCOUNT</th><th>DEPT</th><th>LOCATION</th><th>POSTED_TOTAL</th></tr><tr><td>u32</td><td>str</td><td>str</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>0</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>1</td><td>26000</td><td>1700</td><td>14</td><td>635256</td></tr><tr><td>1</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2020</td><td>8</td><td>31000</td><td>2300</td><td>21</td><td>502139</td></tr><tr><td>2</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2020</td><td>9</td><td>31000</td><td>1300</td><td>22</td><td>729446</td></tr><tr><td>3</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2020</td><td>6</td><td>15000</td><td>2400</td><td>22</td><td>769327</td></tr><tr><td>4</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>8</td><td>31000</td><td>1700</td><td>17</td><td>49468</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 9)\n",
       "┌───────┬─────────┬──────────┬─────────────┬───┬─────────┬──────┬──────────┬──────────────┐\n",
       "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL │\n",
       "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
       "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64     ┆ i64  ┆ i64      ┆ i64          │\n",
       "╞═══════╪═════════╪══════════╪═════════════╪═══╪═════════╪══════╪══════════╪══════════════╡\n",
       "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 26000   ┆ 1700 ┆ 14       ┆ 635256       │\n",
       "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 31000   ┆ 2300 ┆ 21       ┆ 502139       │\n",
       "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 31000   ┆ 1300 ┆ 22       ┆ 729446       │\n",
       "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 15000   ┆ 2400 ┆ 22       ┆ 769327       │\n",
       "│ 4     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 31000   ┆ 1700 ┆ 17       ┆ 49468        │\n",
       "└───────┴─────────┴──────────┴─────────────┴───┴─────────┴──────┴──────────┴──────────────┘"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "######################\n",
    "## LEDGER DataFrame ##\n",
    "######################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "sampleSize = 100_000\n",
    "org = \"ABC Inc.\"\n",
    "ledger_type = \"ACTUALS\" # BUDGET, STATS are other Ledger types\n",
    "fiscal_year_from = 2020\n",
    "fiscal_year_to = 2023\n",
    "random.seed(123)\n",
    "\n",
    "ledger = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, \n",
    "                                          fiscal_year_to+1, 1)),k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ACCOUNT\" : random.choices(accounts[\"ID\"], k=sampleSize),\n",
    "\t\"DEPT\" : random.choices(dept[\"ID\"], k=sampleSize),\n",
    "\t\"LOCATION\" : random.choices(location[\"ID\"], k=sampleSize),\n",
    "\t\"POSTED_TOTAL\": random.sample(range(1000000), sampleSize)\n",
    "})\n",
    "ledger.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 9)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>LEDGER</th><th>ORG</th><th>FISCAL_YEAR</th><th>PERIOD</th><th>ACCOUNT</th><th>DEPT</th><th>LOCATION</th><th>POSTED_TOTAL</th></tr><tr><td>u32</td><td>str</td><td>str</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>0</td><td>&quot;BUDGET&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>11</td><td>30000</td><td>1100</td><td>11</td><td>74976</td></tr><tr><td>1</td><td>&quot;BUDGET&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>9</td><td>42000</td><td>2300</td><td>22</td><td>115846</td></tr><tr><td>2</td><td>&quot;BUDGET&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2022</td><td>10</td><td>17000</td><td>1300</td><td>16</td><td>156533</td></tr><tr><td>3</td><td>&quot;BUDGET&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>8</td><td>20000</td><td>1100</td><td>14</td><td>212062</td></tr><tr><td>4</td><td>&quot;BUDGET&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>5</td><td>38000</td><td>1600</td><td>21</td><td>204052</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 9)\n",
       "┌───────┬────────┬──────────┬─────────────┬───┬─────────┬──────┬──────────┬──────────────┐\n",
       "│ Row # ┆ LEDGER ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL │\n",
       "│ ---   ┆ ---    ┆ ---      ┆ ---         ┆   ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
       "│ u32   ┆ str    ┆ str      ┆ i64         ┆   ┆ i64     ┆ i64  ┆ i64      ┆ i64          │\n",
       "╞═══════╪════════╪══════════╪═════════════╪═══╪═════════╪══════╪══════════╪══════════════╡\n",
       "│ 0     ┆ BUDGET ┆ ABC Inc. ┆ 2021        ┆ … ┆ 30000   ┆ 1100 ┆ 11       ┆ 74976        │\n",
       "│ 1     ┆ BUDGET ┆ ABC Inc. ┆ 2021        ┆ … ┆ 42000   ┆ 2300 ┆ 22       ┆ 115846       │\n",
       "│ 2     ┆ BUDGET ┆ ABC Inc. ┆ 2022        ┆ … ┆ 17000   ┆ 1300 ┆ 16       ┆ 156533       │\n",
       "│ 3     ┆ BUDGET ┆ ABC Inc. ┆ 2021        ┆ … ┆ 20000   ┆ 1100 ┆ 14       ┆ 212062       │\n",
       "│ 4     ┆ BUDGET ┆ ABC Inc. ┆ 2021        ┆ … ┆ 38000   ┆ 1600 ┆ 21       ┆ 204052       │\n",
       "└───────┴────────┴──────────┴─────────────┴───┴─────────┴──────┴──────────┴──────────────┘"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ledger_type = \"BUDGET\" # ACTUALS, STATS are other Ledger types\n",
    "\n",
    "ledger_budg = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, fiscal_year_to+1, 1))\n",
    "                               ,k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ACCOUNT\" : random.choices(accounts[\"ID\"], k=sampleSize),\n",
    "\t\"DEPT\" : random.choices(dept[\"ID\"], k=sampleSize),\n",
    "\t\"LOCATION\" : random.choices(location[\"ID\"], k=sampleSize),\n",
    "\t\"POSTED_TOTAL\": random.sample(range(1000000), sampleSize)\n",
    "})\n",
    "ledger_budg.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 9)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>LEDGER</th><th>ORG</th><th>FISCAL_YEAR</th><th>PERIOD</th><th>ACCOUNT</th><th>DEPT</th><th>LOCATION</th><th>POSTED_TOTAL</th></tr><tr><td>u32</td><td>str</td><td>str</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>0</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2023</td><td>2</td><td>43000</td><td>2000</td><td>15</td><td>216541</td></tr><tr><td>1</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2022</td><td>5</td><td>14000</td><td>2000</td><td>15</td><td>910167</td></tr><tr><td>2</td><td>&quot;BUDGET&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2020</td><td>11</td><td>23000</td><td>2000</td><td>13</td><td>866285</td></tr><tr><td>3</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>3</td><td>31000</td><td>2400</td><td>19</td><td>609860</td></tr><tr><td>4</td><td>&quot;BUDGET&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2022</td><td>8</td><td>36000</td><td>1300</td><td>19</td><td>24617</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 9)\n",
       "┌───────┬─────────┬──────────┬─────────────┬───┬─────────┬──────┬──────────┬──────────────┐\n",
       "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL │\n",
       "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
       "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64     ┆ i64  ┆ i64      ┆ i64          │\n",
       "╞═══════╪═════════╪══════════╪═════════════╪═══╪═════════╪══════╪══════════╪══════════════╡\n",
       "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ … ┆ 43000   ┆ 2000 ┆ 15       ┆ 216541       │\n",
       "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 14000   ┆ 2000 ┆ 15       ┆ 910167       │\n",
       "│ 2     ┆ BUDGET  ┆ ABC Inc. ┆ 2020        ┆ … ┆ 23000   ┆ 2000 ┆ 13       ┆ 866285       │\n",
       "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 31000   ┆ 2400 ┆ 19       ┆ 609860       │\n",
       "│ 4     ┆ BUDGET  ┆ ABC Inc. ┆ 2022        ┆ … ┆ 36000   ┆ 1300 ┆ 19       ┆ 24617        │\n",
       "└───────┴─────────┴──────────┴─────────────┴───┴─────────┴──────┴──────────┴──────────────┘"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#########################################\n",
    "# combined ledger for Actuals and Budget\n",
    "#########################################\n",
    "dfLedger = pl.concat([ledger, ledger_budg], how=\"vertical\")\n",
    "dfLedger.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Supply Chain DataSet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 8)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>AS_OF_DATE</th><th>DESCRIPTION</th><th>REGION</th><th>STATUS</th><th>CLASSIFICATION</th><th>TYPE</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>0</td><td>1000</td><td>2022-01-01 00:00:00</td><td>&quot;Rx&quot;</td><td>&quot;Region A&quot;</td><td>&quot;Active&quot;</td><td>&quot;Rx&quot;</td><td>&quot;R&quot;</td></tr><tr><td>1</td><td>1800</td><td>2022-01-01 00:00:00</td><td>&quot;Construction&quot;</td><td>&quot;Region C&quot;</td><td>&quot;Active&quot;</td><td>&quot;Constructions&quot;</td><td>&quot;C&quot;</td></tr><tr><td>2</td><td>2200</td><td>2022-01-01 00:00:00</td><td>&quot;Consulting&quot;</td><td>&quot;Region A&quot;</td><td>&quot;Active&quot;</td><td>&quot;Services&quot;</td><td>&quot;S&quot;</td></tr><tr><td>3</td><td>2100</td><td>2022-01-01 00:00:00</td><td>&quot;Material&quot;</td><td>&quot;Region C&quot;</td><td>&quot;Active&quot;</td><td>&quot;Material&quot;</td><td>&quot;M&quot;</td></tr><tr><td>4</td><td>1400</td><td>2022-01-01 00:00:00</td><td>&quot;un-assigned&quot;</td><td>&quot;Region B&quot;</td><td>&quot;Active&quot;</td><td>&quot;OTHERS&quot;</td><td>&quot;O&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 8)\n",
       "┌───────┬──────┬─────────────────────┬──────────────┬──────────┬────────┬────────────────┬──────┐\n",
       "│ Row # ┆ ID   ┆ AS_OF_DATE          ┆ DESCRIPTION  ┆ REGION   ┆ STATUS ┆ CLASSIFICATION ┆ TYPE │\n",
       "│ ---   ┆ ---  ┆ ---                 ┆ ---          ┆ ---      ┆ ---    ┆ ---            ┆ ---  │\n",
       "│ u32   ┆ i64  ┆ datetime[μs]        ┆ str          ┆ str      ┆ str    ┆ str            ┆ str  │\n",
       "╞═══════╪══════╪═════════════════════╪══════════════╪══════════╪════════╪════════════════╪══════╡\n",
       "│ 0     ┆ 1000 ┆ 2022-01-01 00:00:00 ┆ Rx           ┆ Region A ┆ Active ┆ Rx             ┆ R    │\n",
       "│ 1     ┆ 1800 ┆ 2022-01-01 00:00:00 ┆ Construction ┆ Region C ┆ Active ┆ Constructions  ┆ C    │\n",
       "│ 2     ┆ 2200 ┆ 2022-01-01 00:00:00 ┆ Consulting   ┆ Region A ┆ Active ┆ Services       ┆ S    │\n",
       "│ 3     ┆ 2100 ┆ 2022-01-01 00:00:00 ┆ Material     ┆ Region C ┆ Active ┆ Material       ┆ M    │\n",
       "│ 4     ┆ 1400 ┆ 2022-01-01 00:00:00 ┆ un-assigned  ┆ Region B ┆ Active ┆ OTHERS         ┆ O    │\n",
       "└───────┴──────┴─────────────────────┴──────────────┴──────────┴────────┴────────────────┴──────┘"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "################################\n",
    "## PRODUCT_CATEGORY DataFrame ##\n",
    "################################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "category = pl.DataFrame({\n",
    "    \"ID\":  list(range(1000, 2500, 100)),\n",
    "    \"AS_OF_DATE\" : datetime(2022, 1, 1),\n",
    "    \"DESCRIPTION\" : [\"Rx\",\"Material\",\"Consulting\",\"Construction\",\n",
    "                     \"un-assigned\"] * 3,\n",
    "    \"REGION\": [\"Region A\",\"Region B\",\"Region C\"] * 5,\n",
    "    \"STATUS\" : \"Active\",\n",
    "    \"CLASSIFICATION\" : [\"Rx\",\"Material\", \"Services\",\"Constructions\",\n",
    "                        \"OTHERS\"] * 3,\n",
    "    \"TYPE\" : [\"R\",\"M\",\"S\",\"C\",\"O\"] * 3,\n",
    "})\n",
    "category.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 6)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>AS_OF_DATE</th><th>DESCRIPTION</th><th>STATUS</th><th>CATEGORY</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>str</td><td>str</td><td>i64</td></tr></thead><tbody><tr><td>0</td><td>220</td><td>2022-01-01 00:00:00</td><td>&quot;Item 3&quot;</td><td>&quot;Active&quot;</td><td>1900</td></tr><tr><td>1</td><td>110</td><td>2022-01-01 00:00:00</td><td>&quot;Item 2&quot;</td><td>&quot;Active&quot;</td><td>1900</td></tr><tr><td>2</td><td>190</td><td>2022-01-01 00:00:00</td><td>&quot;Item 5&quot;</td><td>&quot;Active&quot;</td><td>1800</td></tr><tr><td>3</td><td>130</td><td>2022-01-01 00:00:00</td><td>&quot;Item 4&quot;</td><td>&quot;Active&quot;</td><td>1900</td></tr><tr><td>4</td><td>150</td><td>2022-01-01 00:00:00</td><td>&quot;Item 1&quot;</td><td>&quot;Active&quot;</td><td>1900</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 6)\n",
       "┌───────┬─────┬─────────────────────┬─────────────┬────────┬──────────┐\n",
       "│ Row # ┆ ID  ┆ AS_OF_DATE          ┆ DESCRIPTION ┆ STATUS ┆ CATEGORY │\n",
       "│ ---   ┆ --- ┆ ---                 ┆ ---         ┆ ---    ┆ ---      │\n",
       "│ u32   ┆ i64 ┆ datetime[μs]        ┆ str         ┆ str    ┆ i64      │\n",
       "╞═══════╪═════╪═════════════════════╪═════════════╪════════╪══════════╡\n",
       "│ 0     ┆ 220 ┆ 2022-01-01 00:00:00 ┆ Item 3      ┆ Active ┆ 1900     │\n",
       "│ 1     ┆ 110 ┆ 2022-01-01 00:00:00 ┆ Item 2      ┆ Active ┆ 1900     │\n",
       "│ 2     ┆ 190 ┆ 2022-01-01 00:00:00 ┆ Item 5      ┆ Active ┆ 1800     │\n",
       "│ 3     ┆ 130 ┆ 2022-01-01 00:00:00 ┆ Item 4      ┆ Active ┆ 1900     │\n",
       "│ 4     ┆ 150 ┆ 2022-01-01 00:00:00 ┆ Item 1      ┆ Active ┆ 1900     │\n",
       "└───────┴─────┴─────────────────────┴─────────────┴────────┴──────────┘"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#######################\n",
    "## PRODUCT DataFrame ##\n",
    "#######################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "product = pl.DataFrame({\n",
    "    \"ID\":  list(range(100, 250, 10)),\n",
    "    \"AS_OF_DATE\" : datetime(2022, 1, 1),\n",
    "    \"DESCRIPTION\" : [\"Item 1\",\"Item 2\",\"Item 3\",\"Item 4\",\"Item 5\"] * 3,\n",
    "    \"STATUS\" : \"Active\",\n",
    "    \"CATEGORY\" : random.choices(category[\"ID\"], k=15),\n",
    "})\n",
    "product.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 10)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>AS_OF_DATE</th><th>DESCRIPTION</th><th>ADDRESS</th><th>PHONE</th><th>EMAIL</th><th>STATUS</th><th>TYPE</th><th>CATEGORY</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>i64</td></tr></thead><tbody><tr><td>0</td><td>100</td><td>2022-01-01 00:00:00</td><td>&quot;Customer 1&quot;</td><td>&quot;Address 1&quot;</td><td>&quot;0000000001&quot;</td><td>&quot;1@email&quot;</td><td>&quot;Active&quot;</td><td>&quot;Corp&quot;</td><td>1600</td></tr><tr><td>1</td><td>170</td><td>2022-01-01 00:00:00</td><td>&quot;Customer 3&quot;</td><td>&quot;Address 3&quot;</td><td>&quot;0000000003&quot;</td><td>&quot;3@email&quot;</td><td>&quot;Active&quot;</td><td>&quot;Gov&quot;</td><td>1400</td></tr><tr><td>2</td><td>190</td><td>2022-01-01 00:00:00</td><td>&quot;Customer 5&quot;</td><td>&quot;Address 5&quot;</td><td>&quot;0000000005&quot;</td><td>&quot;5@email&quot;</td><td>&quot;Active&quot;</td><td>&quot;Corp&quot;</td><td>1100</td></tr><tr><td>3</td><td>130</td><td>2022-01-01 00:00:00</td><td>&quot;Customer 4&quot;</td><td>&quot;Address 4&quot;</td><td>&quot;0000000004&quot;</td><td>&quot;4@email&quot;</td><td>&quot;Active&quot;</td><td>&quot;Corp&quot;</td><td>1200</td></tr><tr><td>4</td><td>140</td><td>2022-01-01 00:00:00</td><td>&quot;Customer 5&quot;</td><td>&quot;Address 5&quot;</td><td>&quot;0000000005&quot;</td><td>&quot;5@email&quot;</td><td>&quot;Active&quot;</td><td>&quot;Gov&quot;</td><td>1200</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 10)\n",
       "┌───────┬─────┬─────────────────────┬─────────────┬───┬─────────┬────────┬──────┬──────────┐\n",
       "│ Row # ┆ ID  ┆ AS_OF_DATE          ┆ DESCRIPTION ┆ … ┆ EMAIL   ┆ STATUS ┆ TYPE ┆ CATEGORY │\n",
       "│ ---   ┆ --- ┆ ---                 ┆ ---         ┆   ┆ ---     ┆ ---    ┆ ---  ┆ ---      │\n",
       "│ u32   ┆ i64 ┆ datetime[μs]        ┆ str         ┆   ┆ str     ┆ str    ┆ str  ┆ i64      │\n",
       "╞═══════╪═════╪═════════════════════╪═════════════╪═══╪═════════╪════════╪══════╪══════════╡\n",
       "│ 0     ┆ 100 ┆ 2022-01-01 00:00:00 ┆ Customer 1  ┆ … ┆ 1@email ┆ Active ┆ Corp ┆ 1600     │\n",
       "│ 1     ┆ 170 ┆ 2022-01-01 00:00:00 ┆ Customer 3  ┆ … ┆ 3@email ┆ Active ┆ Gov  ┆ 1400     │\n",
       "│ 2     ┆ 190 ┆ 2022-01-01 00:00:00 ┆ Customer 5  ┆ … ┆ 5@email ┆ Active ┆ Corp ┆ 1100     │\n",
       "│ 3     ┆ 130 ┆ 2022-01-01 00:00:00 ┆ Customer 4  ┆ … ┆ 4@email ┆ Active ┆ Corp ┆ 1200     │\n",
       "│ 4     ┆ 140 ┆ 2022-01-01 00:00:00 ┆ Customer 5  ┆ … ┆ 5@email ┆ Active ┆ Gov  ┆ 1200     │\n",
       "└───────┴─────┴─────────────────────┴─────────────┴───┴─────────┴────────┴──────┴──────────┘"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "########################\n",
    "## CUSTOMER DataFrame ##\n",
    "########################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "customer = pl.DataFrame({\n",
    "    \"ID\":  list(range(100, 250, 10)),\n",
    "    \"AS_OF_DATE\" : datetime(2022, 1, 1),\n",
    "    \"DESCRIPTION\" : [\"Customer 1\",\"Customer 2\",\"Customer 3\",\n",
    "                     \"Customer 4\",\"Customer 5\"] * 3,\n",
    "    \"ADDRESS\" : [\"Address 1\",\"Address 2\",\"Address 3\",\n",
    "                 \"Address 4\",\"Address 5\"] * 3,\n",
    "    \"PHONE\" : [\"0000000001\",\"0000000002\",\"0000000003\",\n",
    "               \"0000000004\",\"0000000005\"] * 3,\n",
    "    \"EMAIL\" : [\"1@email\",\"2@email\",\"3@email\",\"4@email\",\"5@email\"] * 3,\n",
    "    \"STATUS\" : \"Active\",\n",
    "    \"TYPE\" : [\"Corp\",\"Gov\",\"Individual\"] * 5,\n",
    "    \"CATEGORY\" : random.choices(category[\"ID\"], k=15),\n",
    "})\n",
    "customer.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>AS_OF_DATE</th><th>CUSTOMER</th><th>ITEM</th><th>QTY</th><th>POSTED_TOTAL</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>0</td><td>1000</td><td>2024-01-01 00:00:00</td><td>150</td><td>180</td><td>966405</td><td>835603</td></tr><tr><td>1</td><td>1070</td><td>2024-01-01 00:00:00</td><td>140</td><td>120</td><td>253823</td><td>928636</td></tr><tr><td>2</td><td>1060</td><td>2024-01-01 00:00:00</td><td>150</td><td>160</td><td>581636</td><td>37077</td></tr><tr><td>3</td><td>1030</td><td>2024-01-01 00:00:00</td><td>100</td><td>240</td><td>883691</td><td>505463</td></tr><tr><td>4</td><td>1040</td><td>2024-01-01 00:00:00</td><td>110</td><td>230</td><td>17190</td><td>27331</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 7)\n",
       "┌───────┬──────┬─────────────────────┬──────────┬──────┬────────┬──────────────┐\n",
       "│ Row # ┆ ID   ┆ AS_OF_DATE          ┆ CUSTOMER ┆ ITEM ┆ QTY    ┆ POSTED_TOTAL │\n",
       "│ ---   ┆ ---  ┆ ---                 ┆ ---      ┆ ---  ┆ ---    ┆ ---          │\n",
       "│ u32   ┆ i64  ┆ datetime[μs]        ┆ i64      ┆ i64  ┆ i64    ┆ i64          │\n",
       "╞═══════╪══════╪═════════════════════╪══════════╪══════╪════════╪══════════════╡\n",
       "│ 0     ┆ 1000 ┆ 2024-01-01 00:00:00 ┆ 150      ┆ 180  ┆ 966405 ┆ 835603       │\n",
       "│ 1     ┆ 1070 ┆ 2024-01-01 00:00:00 ┆ 140      ┆ 120  ┆ 253823 ┆ 928636       │\n",
       "│ 2     ┆ 1060 ┆ 2024-01-01 00:00:00 ┆ 150      ┆ 160  ┆ 581636 ┆ 37077        │\n",
       "│ 3     ┆ 1030 ┆ 2024-01-01 00:00:00 ┆ 100      ┆ 240  ┆ 883691 ┆ 505463       │\n",
       "│ 4     ┆ 1040 ┆ 2024-01-01 00:00:00 ┆ 110      ┆ 230  ┆ 17190  ┆ 27331        │\n",
       "└───────┴──────┴─────────────────────┴──────────┴──────┴────────┴──────────────┘"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#####################\n",
    "## ORDER DataFrame ##\n",
    "#####################\n",
    "import random\n",
    "from datetime import datetime\n",
    "sampleSize = 10\n",
    "\n",
    "order = pl.DataFrame({\n",
    "    \"ID\":  list(range(1000, 1100, sampleSize)),\n",
    "    \"AS_OF_DATE\" : datetime(2024, 1, 1),\n",
    "    \"CUSTOMER\": random.choices(customer[\"ID\"], k=sampleSize),\n",
    "    \"ITEM\": random.choices(product[\"ID\"], k=sampleSize),\n",
    "    \"QTY\": random.sample(range(1000000), sampleSize),\n",
    "\t\"POSTED_TOTAL\": random.sample(range(1000000), sampleSize)\n",
    "})\n",
    "order.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>AS_OF_DATE</th><th>ORDER</th><th>STATUS</th></tr><tr><td>u32</td><td>datetime[μs]</td><td>i64</td><td>str</td></tr></thead><tbody><tr><td>0</td><td>2024-01-01 00:00:00</td><td>1040</td><td>&quot;open&quot;</td></tr><tr><td>1</td><td>2024-01-01 00:00:00</td><td>1010</td><td>&quot;paid&quot;</td></tr><tr><td>2</td><td>2024-01-01 00:00:00</td><td>1000</td><td>&quot;cancelled&quot;</td></tr><tr><td>3</td><td>2024-01-01 00:00:00</td><td>1010</td><td>&quot;cancelled&quot;</td></tr><tr><td>4</td><td>2024-01-01 00:00:00</td><td>1030</td><td>&quot;paid&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 4)\n",
       "┌───────┬─────────────────────┬───────┬───────────┐\n",
       "│ Row # ┆ AS_OF_DATE          ┆ ORDER ┆ STATUS    │\n",
       "│ ---   ┆ ---                 ┆ ---   ┆ ---       │\n",
       "│ u32   ┆ datetime[μs]        ┆ i64   ┆ str       │\n",
       "╞═══════╪═════════════════════╪═══════╪═══════════╡\n",
       "│ 0     ┆ 2024-01-01 00:00:00 ┆ 1040  ┆ open      │\n",
       "│ 1     ┆ 2024-01-01 00:00:00 ┆ 1010  ┆ paid      │\n",
       "│ 2     ┆ 2024-01-01 00:00:00 ┆ 1000  ┆ cancelled │\n",
       "│ 3     ┆ 2024-01-01 00:00:00 ┆ 1010  ┆ cancelled │\n",
       "│ 4     ┆ 2024-01-01 00:00:00 ┆ 1030  ┆ paid      │\n",
       "└───────┴─────────────────────┴───────┴───────────┘"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#######################\n",
    "## INVOICE DataFrame ##\n",
    "#######################\n",
    "import random\n",
    "from datetime import datetime\n",
    "sampleSize = 10\n",
    "\n",
    "invoice = pl.DataFrame({\n",
    "    \"AS_OF_DATE\" : datetime(2024, 1, 1),\n",
    "    \"ORDER\": random.choices(order[\"ID\"], k=sampleSize),\n",
    "    \"STATUS\" : [\"open\",\"paid\",\"cancelled\",\"shipped\",\"hold\"] * 2,\n",
    "})\n",
    "invoice.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Types\n",
    "---\n",
    "\n",
    "Polars is entirely based on Arrow data types and backed by Arrow memory arrays. This makes data processing cache-efficient and well-supported for Inter Process Communication.\n",
    "\n",
    "Please read official [Polars Data Type documentation](https://pola-rs.github.io/polars-book/user-guide/concepts/data-types/) for more details."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IO\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#################\n",
    "### csv files ###\n",
    "#################\n",
    "dfLedger.write_csv(\"../downloads/ledger.csv\")\n",
    "dfLedger_c = pl.read_csv(\"../downloads/ledger.csv\")\n",
    "# Polars allows you to scan a CSV input. \n",
    "# Scanning delays the actual parsing of the file \n",
    "# and instead returns a lazy computation holder called a LazyFrame.\n",
    "dfLedger_c = pl.scan_csv(\"../downloads/ledger.csv\")\n",
    "\n",
    "#####################\n",
    "### parquet files ###\n",
    "#####################\n",
    "dfLedger.write_parquet(\"../downloads/ledger.parquet\")\n",
    "dfLedger_c = pl.read_parquet(\"../downloads/ledger.parquet\")\n",
    "# Polars allows you to scan a parquet input. \n",
    "# Scanning delays the actual parsing of the file and instead \n",
    "# returns a lazy computation holder called a LazyFrame.\n",
    "dfLedger_c = pl.scan_parquet(\"../downloads/ledger.parquet\")\n",
    "\n",
    "######################################################\n",
    "### json files ### ndjson: new line delimited json ###\n",
    "######################################################\n",
    "dfLedger.write_json(\"../downloads/ledger.json\")\n",
    "dfLedger_c = pl.scan_json(\"../downloads/ledger.json\")\n",
    "# Polars allows you to scan a json input. \n",
    "# Scanning delays the actual parsing of the file and instead \n",
    "# returns a lazy computation holder called a LazyFrame.\n",
    "dfLedger_c = pl.scan_json(\"../downloads/ledger.json\")\n",
    "\n",
    "####################\n",
    "## multiple files ##\n",
    "####################\n",
    "for i in range(5):\n",
    "    dfLedger.write_csv(f\"../downloads/my_many_files_{i}.csv\")\n",
    "pl.scan_csv(\"../downloads/my_many_files_*.csv\").show_graph() \n",
    "# see how query optimization/parallelism works\n",
    "df = pl.read_csv(\"../downloads/my_many_files_*.csv\")\n",
    "print(df.shape)\n",
    "\n",
    "###############\n",
    "## databases ##\n",
    "###############\n",
    "import polars as pl\n",
    "\n",
    "connection_uri = \"postgres://username:password@server:port/database\"\n",
    "query = \"SELECT * FROM foo\"\n",
    "\n",
    "pl.read_database(query=query, connection_uri=connection_uri)\n",
    "\n",
    "# Polars doesn't manage connections and data transfer from databases by itself. \n",
    "# Instead external libraries (known as engines) handle this.\n",
    "#  At present Polars can use two engines to read from databases:\n",
    "# ConnectorX and ADBC\n",
    "# $  pip install connectorx\n",
    "# $  pip install adbc-driver-sqlite\n",
    "\n",
    "# As with reading from a database above Polars uses an engine \n",
    "# to write to a database. \n",
    "# The currently supported engines are:\n",
    "# SQLAlchemy and\n",
    "# Arrow Database Connectivity (ADBC)\n",
    "# $  pip install SQLAlchemy pandas\n",
    "\n",
    "# AWS & Big Query - API - WIP as of 07/11/23"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Polars DataFrame Context\n",
    "---\n",
    "\n",
    "The two core components of the Polars DataFrame DSL (domain specific language) are Contexts and Expression.\n",
    "\n",
    "A context, as implied by the name, refers to the context in which an expression needs to be evaluated. There are three main contexts:\n",
    "\n",
    "    Selection: df.select([..]), df.with_columns([..])\n",
    "    Filtering: df.filter()\n",
    "    Groupby / Aggregation: df.groupby(..).agg([..])\n",
    "\n",
    "Additional Contexts\n",
    "    \n",
    "    List, Arrays and SQL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 8)\n",
      "┌────────┬──────────┬─────────────┬────────┬─────────┬──────┬──────────┬──────────────┐\n",
      "│ LEDGER ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL │\n",
      "│ ---    ┆ ---      ┆ ---         ┆ ---    ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
      "│ str    ┆ str      ┆ i64         ┆ i64    ┆ i64     ┆ i64  ┆ i64      ┆ i64          │\n",
      "╞════════╪══════════╪═════════════╪════════╪═════════╪══════╪══════════╪══════════════╡\n",
      "│ BUDGET ┆ ABC Inc. ┆ 2023        ┆ 12     ┆ 20000   ┆ 2100 ┆ 12       ┆ 739002       │\n",
      "│ BUDGET ┆ ABC Inc. ┆ 2023        ┆ 9      ┆ 19000   ┆ 1400 ┆ 15       ┆ 317636       │\n",
      "│ BUDGET ┆ ABC Inc. ┆ 2020        ┆ 11     ┆ 31000   ┆ 2200 ┆ 12       ┆ 226428       │\n",
      "│ BUDGET ┆ ABC Inc. ┆ 2022        ┆ 12     ┆ 29000   ┆ 1500 ┆ 14       ┆ 282944       │\n",
      "│ BUDGET ┆ ABC Inc. ┆ 2023        ┆ 10     ┆ 41000   ┆ 1200 ┆ 12       ┆ 712123       │\n",
      "└────────┴──────────┴─────────────┴────────┴─────────┴──────┴──────────┴──────────────┘\n",
      "shape: (200_000, 9)\n",
      "┌────────┬─────────────┬────────┬─────────┬───┬─────────┬──────┬──────────┬──────────────┐\n",
      "│ Row #  ┆ FISCAL_YEAR ┆ PERIOD ┆ LEDGER  ┆ … ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ in Thousands │\n",
      "│ ---    ┆ ---         ┆ ---    ┆ ---     ┆   ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
      "│ u32    ┆ i64         ┆ i64    ┆ str     ┆   ┆ i64     ┆ i64  ┆ i64      ┆ f64          │\n",
      "╞════════╪═════════════╪════════╪═════════╪═══╪═════════╪══════╪══════════╪══════════════╡\n",
      "│ 0      ┆ 2020        ┆ 1      ┆ ACTUALS ┆ … ┆ 12000   ┆ 2400 ┆ 22       ┆ 753.956      │\n",
      "│ 1      ┆ 2020        ┆ 1      ┆ ACTUALS ┆ … ┆ 10000   ┆ 1900 ┆ 14       ┆ 826.906      │\n",
      "│ 2      ┆ 2020        ┆ 1      ┆ ACTUALS ┆ … ┆ 21000   ┆ 1700 ┆ 17       ┆ 454.574      │\n",
      "│ 3      ┆ 2020        ┆ 1      ┆ ACTUALS ┆ … ┆ 34000   ┆ 1300 ┆ 12       ┆ 334.989      │\n",
      "│ …      ┆ …           ┆ …      ┆ …       ┆ … ┆ …       ┆ …    ┆ …        ┆ …            │\n",
      "│ 199996 ┆ 2023        ┆ 12     ┆ BUDGET  ┆ … ┆ 18000   ┆ 2200 ┆ 18       ┆ 65.437       │\n",
      "│ 199997 ┆ 2023        ┆ 12     ┆ BUDGET  ┆ … ┆ 17000   ┆ 1300 ┆ 13       ┆ 960.254      │\n",
      "│ 199998 ┆ 2023        ┆ 12     ┆ BUDGET  ┆ … ┆ 35000   ┆ 2000 ┆ 20       ┆ 41.157       │\n",
      "│ 199999 ┆ 2023        ┆ 12     ┆ BUDGET  ┆ … ┆ 41000   ┆ 1900 ┆ 21       ┆ 265.241      │\n",
      "└────────┴─────────────┴────────┴─────────┴───┴─────────┴──────┴──────────┴──────────────┘\n"
     ]
    }
   ],
   "source": [
    "print(dfLedger.sample(5))\n",
    "##################\n",
    "## select context\n",
    "##################\n",
    "out = dfLedger.select(\n",
    "    pl.col(\"FISCAL_YEAR\").sort(),\n",
    "    pl.col(\"PERIOD\").sort(),\n",
    "    pl.col([\"LEDGER\", \"ORG\", \"ACCOUNT\", \"DEPT\", \"LOCATION\"]),\n",
    "    (pl.col(\"POSTED_TOTAL\") / 1000).alias(\"in Thousands\"),\n",
    ").with_row_count(\"Row #\")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (200_000, 10)\n",
      "┌────────┬─────────┬──────────┬─────────────┬───┬──────┬──────────┬──────────────┬──────────────┐\n",
      "│ Row #  ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL ┆ in Thousands │\n",
      "│ ---    ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---  ┆ ---      ┆ ---          ┆ ---          │\n",
      "│ u32    ┆ str     ┆ str      ┆ i64         ┆   ┆ i64  ┆ i64      ┆ i64          ┆ f64          │\n",
      "╞════════╪═════════╪══════════╪═════════════╪═══╪══════╪══════════╪══════════════╪══════════════╡\n",
      "│ 0      ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 2400 ┆ 22       ┆ 753956       ┆ 753.956      │\n",
      "│ 1      ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 1900 ┆ 14       ┆ 826906       ┆ 826.906      │\n",
      "│ 2      ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 1700 ┆ 17       ┆ 454574       ┆ 454.574      │\n",
      "│ 3      ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 1300 ┆ 12       ┆ 334989       ┆ 334.989      │\n",
      "│ …      ┆ …       ┆ …        ┆ …           ┆ … ┆ …    ┆ …        ┆ …            ┆ …            │\n",
      "│ 199996 ┆ BUDGET  ┆ ABC Inc. ┆ 2022        ┆ … ┆ 2200 ┆ 18       ┆ 65437        ┆ 65.437       │\n",
      "│ 199997 ┆ BUDGET  ┆ ABC Inc. ┆ 2020        ┆ … ┆ 1300 ┆ 13       ┆ 960254       ┆ 960.254      │\n",
      "│ 199998 ┆ BUDGET  ┆ ABC Inc. ┆ 2022        ┆ … ┆ 2000 ┆ 20       ┆ 41157        ┆ 41.157       │\n",
      "│ 199999 ┆ BUDGET  ┆ ABC Inc. ┆ 2020        ┆ … ┆ 1900 ┆ 21       ┆ 265241       ┆ 265.241      │\n",
      "└────────┴─────────┴──────────┴─────────────┴───┴──────┴──────────┴──────────────┴──────────────┘\n"
     ]
    }
   ],
   "source": [
    "##########################\n",
    "## with_columns context ##\n",
    "##########################\n",
    "out = dfLedger.with_columns(\n",
    "    (pl.col(\"POSTED_TOTAL\") / 1000).alias(\"in Thousands\"),\n",
    ").with_row_count(\"Row #\")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (25_136, 9)\n",
      "┌───────┬─────────┬──────────┬─────────────┬───┬─────────┬──────┬──────────┬──────────────┐\n",
      "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL │\n",
      "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
      "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64     ┆ i64  ┆ i64      ┆ i64          │\n",
      "╞═══════╪═════════╪══════════╪═════════════╪═══╪═════════╪══════╪══════════╪══════════════╡\n",
      "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ … ┆ 14000   ┆ 2100 ┆ 11       ┆ 290813       │\n",
      "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ … ┆ 16000   ┆ 1600 ┆ 21       ┆ 28323        │\n",
      "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ … ┆ 42000   ┆ 1600 ┆ 16       ┆ 106856       │\n",
      "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ … ┆ 37000   ┆ 1600 ┆ 17       ┆ 145223       │\n",
      "│ …     ┆ …       ┆ …        ┆ …           ┆ … ┆ …       ┆ …    ┆ …        ┆ …            │\n",
      "│ 25132 ┆ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ … ┆ 20000   ┆ 1400 ┆ 17       ┆ 831737       │\n",
      "│ 25133 ┆ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ … ┆ 32000   ┆ 1700 ┆ 16       ┆ 131723       │\n",
      "│ 25134 ┆ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ … ┆ 33000   ┆ 2200 ┆ 17       ┆ 187186       │\n",
      "│ 25135 ┆ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ … ┆ 14000   ┆ 1800 ┆ 18       ┆ 642420       │\n",
      "└───────┴─────────┴──────────┴─────────────┴───┴─────────┴──────┴──────────┴──────────────┘\n"
     ]
    }
   ],
   "source": [
    "##########################\n",
    "## filter context ##\n",
    "##########################\n",
    "out = dfLedger.filter(\n",
    "    ((pl.col(\"LEDGER\") == \"ACTUALS\") & (pl.col(\"FISCAL_YEAR\") == 2023))\n",
    ").with_row_count(\"Row #\")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (1, 3)\n",
      "┌───────┬─────────┬───────┐\n",
      "│ Row # ┆ LEDGER  ┆ count │\n",
      "│ ---   ┆ ---     ┆ ---   │\n",
      "│ u32   ┆ str     ┆ u32   │\n",
      "╞═══════╪═════════╪═══════╡\n",
      "│ 0     ┆ ACTUALS ┆ 25136 │\n",
      "└───────┴─────────┴───────┘\n",
      "shape: (8, 4)\n",
      "┌───────┬─────────┬─────────────┬───────┐\n",
      "│ Row # ┆ LEDGER  ┆ FISCAL_YEAR ┆ count │\n",
      "│ ---   ┆ ---     ┆ ---         ┆ ---   │\n",
      "│ u32   ┆ str     ┆ i64         ┆ u32   │\n",
      "╞═══════╪═════════╪═════════════╪═══════╡\n",
      "│ 0     ┆ ACTUALS ┆ 2023        ┆ 25136 │\n",
      "│ 1     ┆ ACTUALS ┆ 2020        ┆ 24957 │\n",
      "│ 2     ┆ BUDGET  ┆ 2022        ┆ 24885 │\n",
      "│ 3     ┆ ACTUALS ┆ 2021        ┆ 25036 │\n",
      "│ 4     ┆ BUDGET  ┆ 2023        ┆ 24815 │\n",
      "│ 5     ┆ BUDGET  ┆ 2020        ┆ 25159 │\n",
      "│ 6     ┆ ACTUALS ┆ 2022        ┆ 24871 │\n",
      "│ 7     ┆ BUDGET  ┆ 2021        ┆ 25141 │\n",
      "└───────┴─────────┴─────────────┴───────┘\n",
      "shape: (8, 6)\n",
      "┌───────┬─────────┬─────────────┬───────┬──────────────┬─────────────────────────┐\n",
      "│ Row # ┆ LEDGER  ┆ FISCAL_YEAR ┆ count ┆ POSTED_TOTAL ┆ Posted Total in Million │\n",
      "│ ---   ┆ ---     ┆ ---         ┆ ---   ┆ ---          ┆ ---                     │\n",
      "│ u32   ┆ str     ┆ i64         ┆ u32   ┆ i64          ┆ f64                     │\n",
      "╞═══════╪═════════╪═════════════╪═══════╪══════════════╪═════════════════════════╡\n",
      "│ 0     ┆ BUDGET  ┆ 2021        ┆ 25141 ┆ 12639396311  ┆ 12639.396311            │\n",
      "│ 1     ┆ ACTUALS ┆ 2020        ┆ 24957 ┆ 12430170980  ┆ 12430.17098             │\n",
      "│ 2     ┆ ACTUALS ┆ 2023        ┆ 25136 ┆ 12526897831  ┆ 12526.897831            │\n",
      "│ 3     ┆ BUDGET  ┆ 2020        ┆ 25159 ┆ 12611880007  ┆ 12611.880007            │\n",
      "│ 4     ┆ BUDGET  ┆ 2023        ┆ 24815 ┆ 12446847043  ┆ 12446.847043            │\n",
      "│ 5     ┆ ACTUALS ┆ 2021        ┆ 25036 ┆ 12506846602  ┆ 12506.846602            │\n",
      "│ 6     ┆ BUDGET  ┆ 2022        ┆ 24885 ┆ 12386337109  ┆ 12386.337109            │\n",
      "│ 7     ┆ ACTUALS ┆ 2022        ┆ 24871 ┆ 12420814798  ┆ 12420.814798            │\n",
      "└───────┴─────────┴─────────────┴───────┴──────────────┴─────────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "######################\n",
    "## group by context ##\n",
    "######################\n",
    "out = dfLedger.filter(\n",
    "    ((pl.col(\"LEDGER\") == \"ACTUALS\") & (pl.col(\"FISCAL_YEAR\") == 2023))\n",
    "        ).group_by(\"LEDGER\").agg(\n",
    "                                pl.count()\n",
    "                                ).with_row_count(\"Row #\")\n",
    "print(out)\n",
    "\n",
    "out = dfLedger.group_by(\"LEDGER\", \"FISCAL_YEAR\").agg(\n",
    "                                pl.count()\n",
    "                                ).with_row_count(\"Row #\")\n",
    "print(out)\n",
    "\n",
    "# sort group by data by FISCAL_YEAR\n",
    "out = dfLedger.group_by(\"LEDGER\", \"FISCAL_YEAR\").agg(\n",
    "                                pl.count(),\n",
    "                                pl.sum(\"POSTED_TOTAL\"),\n",
    "                                (pl.sum(\"POSTED_TOTAL\") / 1_000_000)\n",
    "                                .alias(\"Posted Total in Million\"),\n",
    "                                ).with_row_count(\"Row #\")\n",
    "print(out)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Polars DataFrame Expressions\n",
    "---\n",
    "\n",
    "#### using Expression to select columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 7)\n",
      "┌───────┬──────┬─────────────────────┬──────────┬──────┬────────┬──────────────┐\n",
      "│ Row # ┆ ID   ┆ AS_OF_DATE          ┆ CUSTOMER ┆ ITEM ┆ QTY    ┆ POSTED_TOTAL │\n",
      "│ ---   ┆ ---  ┆ ---                 ┆ ---      ┆ ---  ┆ ---    ┆ ---          │\n",
      "│ u32   ┆ i64  ┆ datetime[μs]        ┆ i64      ┆ i64  ┆ i64    ┆ i64          │\n",
      "╞═══════╪══════╪═════════════════════╪══════════╪══════╪════════╪══════════════╡\n",
      "│ 0     ┆ 1050 ┆ 2024-01-01 00:00:00 ┆ 130      ┆ 230  ┆ 440813 ┆ 510808       │\n",
      "│ 1     ┆ 1010 ┆ 2024-01-01 00:00:00 ┆ 180      ┆ 150  ┆ 295337 ┆ 266870       │\n",
      "│ 2     ┆ 1020 ┆ 2024-01-01 00:00:00 ┆ 120      ┆ 170  ┆ 962406 ┆ 997515       │\n",
      "│ 3     ┆ 1030 ┆ 2024-01-01 00:00:00 ┆ 100      ┆ 240  ┆ 883691 ┆ 505463       │\n",
      "│ 4     ┆ 1070 ┆ 2024-01-01 00:00:00 ┆ 140      ┆ 120  ┆ 253823 ┆ 928636       │\n",
      "└───────┴──────┴─────────────────────┴──────────┴──────┴────────┴──────────────┘\n",
      "shape: (10, 7)\n",
      "┌───────┬──────┬─────────────────────┬──────────┬──────┬────────┬─────────────────────┐\n",
      "│ Row # ┆ ID   ┆ AS_OF_DATE          ┆ CUSTOMER ┆ ITEM ┆ QTY    ┆ Amount in thousands │\n",
      "│ ---   ┆ ---  ┆ ---                 ┆ ---      ┆ ---  ┆ ---    ┆ ---                 │\n",
      "│ u32   ┆ i64  ┆ datetime[μs]        ┆ i64      ┆ i64  ┆ i64    ┆ f64                 │\n",
      "╞═══════╪══════╪═════════════════════╪══════════╪══════╪════════╪═════════════════════╡\n",
      "│ 0     ┆ 1000 ┆ 2024-01-01 00:00:00 ┆ 150      ┆ 180  ┆ 966405 ┆ 835.603             │\n",
      "│ 1     ┆ 1010 ┆ 2024-01-01 00:00:00 ┆ 180      ┆ 150  ┆ 295337 ┆ 266.87              │\n",
      "│ 2     ┆ 1020 ┆ 2024-01-01 00:00:00 ┆ 120      ┆ 170  ┆ 962406 ┆ 997.515             │\n",
      "│ 3     ┆ 1030 ┆ 2024-01-01 00:00:00 ┆ 100      ┆ 240  ┆ 883691 ┆ 505.463             │\n",
      "│ …     ┆ …    ┆ …                   ┆ …        ┆ …    ┆ …      ┆ …                   │\n",
      "│ 6     ┆ 1060 ┆ 2024-01-01 00:00:00 ┆ 150      ┆ 160  ┆ 581636 ┆ 37.077              │\n",
      "│ 7     ┆ 1070 ┆ 2024-01-01 00:00:00 ┆ 140      ┆ 120  ┆ 253823 ┆ 928.636             │\n",
      "│ 8     ┆ 1080 ┆ 2024-01-01 00:00:00 ┆ 160      ┆ 200  ┆ 877957 ┆ 793.188             │\n",
      "│ 9     ┆ 1090 ┆ 2024-01-01 00:00:00 ┆ 220      ┆ 170  ┆ 660229 ┆ 512.122             │\n",
      "└───────┴──────┴─────────────────────┴──────────┴──────┴────────┴─────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "######################\n",
    "# selection context ##\n",
    "######################\n",
    "print(order.sample(5).with_row_count(\"Row #\"))\n",
    "\n",
    " # using numerical operators\n",
    "dfOrder = order.select(\n",
    "    pl.col(\"ID\"),\n",
    "    pl.col(\"AS_OF_DATE\"),\n",
    "    pl.col(\"CUSTOMER\"),\n",
    "    pl.col(\"ITEM\"),\n",
    "    pl.col(\"QTY\"),\n",
    "    (pl.col(\"POSTED_TOTAL\") / 1000).alias(\"Amount in thousands\"),\n",
    ").with_row_count(\"Row #\")\n",
    "print(dfOrder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>AS_OF_DATE</th><th>CUSTOMER</th><th>ITEM</th><th>QTY</th><th>Amount in thousands</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>i64</td><td>i64</td><td>i64</td><td>f64</td></tr></thead><tbody><tr><td>9</td><td>1090</td><td>2024-01-01 00:00:00</td><td>220</td><td>170</td><td>660229</td><td>512.122</td></tr><tr><td>1</td><td>1010</td><td>2024-01-01 00:00:00</td><td>180</td><td>150</td><td>295337</td><td>266.87</td></tr><tr><td>2</td><td>1020</td><td>2024-01-01 00:00:00</td><td>120</td><td>170</td><td>962406</td><td>997.515</td></tr><tr><td>5</td><td>1050</td><td>2024-01-01 00:00:00</td><td>130</td><td>230</td><td>440813</td><td>510.808</td></tr><tr><td>4</td><td>1040</td><td>2024-01-01 00:00:00</td><td>110</td><td>230</td><td>17190</td><td>27.331</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 7)\n",
       "┌───────┬──────┬─────────────────────┬──────────┬──────┬────────┬─────────────────────┐\n",
       "│ Row # ┆ ID   ┆ AS_OF_DATE          ┆ CUSTOMER ┆ ITEM ┆ QTY    ┆ Amount in thousands │\n",
       "│ ---   ┆ ---  ┆ ---                 ┆ ---      ┆ ---  ┆ ---    ┆ ---                 │\n",
       "│ u32   ┆ i64  ┆ datetime[μs]        ┆ i64      ┆ i64  ┆ i64    ┆ f64                 │\n",
       "╞═══════╪══════╪═════════════════════╪══════════╪══════╪════════╪═════════════════════╡\n",
       "│ 9     ┆ 1090 ┆ 2024-01-01 00:00:00 ┆ 220      ┆ 170  ┆ 660229 ┆ 512.122             │\n",
       "│ 1     ┆ 1010 ┆ 2024-01-01 00:00:00 ┆ 180      ┆ 150  ┆ 295337 ┆ 266.87              │\n",
       "│ 2     ┆ 1020 ┆ 2024-01-01 00:00:00 ┆ 120      ┆ 170  ┆ 962406 ┆ 997.515             │\n",
       "│ 5     ┆ 1050 ┆ 2024-01-01 00:00:00 ┆ 130      ┆ 230  ┆ 440813 ┆ 510.808             │\n",
       "│ 4     ┆ 1040 ┆ 2024-01-01 00:00:00 ┆ 110      ┆ 230  ┆ 17190  ┆ 27.331              │\n",
       "└───────┴──────┴─────────────────────┴──────────┴──────┴────────┴─────────────────────┘"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select context the selection applies expressions over columns.\n",
    "# The expressions in this context must produce Series that are\n",
    "# all the same length or have a length of 1.\n",
    "\n",
    "# select all cols\n",
    "dfOrder.select(pl.col(\"*\")).sample(5)\n",
    "# dfOrder.select(pl.all()).sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 5)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>AS_OF_DATE</th><th>ITEM</th><th>Amount in thousands</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>i64</td><td>f64</td></tr></thead><tbody><tr><td>0</td><td>1000</td><td>2024-01-01 00:00:00</td><td>180</td><td>835.603</td></tr><tr><td>8</td><td>1080</td><td>2024-01-01 00:00:00</td><td>200</td><td>793.188</td></tr><tr><td>6</td><td>1060</td><td>2024-01-01 00:00:00</td><td>160</td><td>37.077</td></tr><tr><td>3</td><td>1030</td><td>2024-01-01 00:00:00</td><td>240</td><td>505.463</td></tr><tr><td>5</td><td>1050</td><td>2024-01-01 00:00:00</td><td>230</td><td>510.808</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 5)\n",
       "┌───────┬──────┬─────────────────────┬──────┬─────────────────────┐\n",
       "│ Row # ┆ ID   ┆ AS_OF_DATE          ┆ ITEM ┆ Amount in thousands │\n",
       "│ ---   ┆ ---  ┆ ---                 ┆ ---  ┆ ---                 │\n",
       "│ u32   ┆ i64  ┆ datetime[μs]        ┆ i64  ┆ f64                 │\n",
       "╞═══════╪══════╪═════════════════════╪══════╪═════════════════════╡\n",
       "│ 0     ┆ 1000 ┆ 2024-01-01 00:00:00 ┆ 180  ┆ 835.603             │\n",
       "│ 8     ┆ 1080 ┆ 2024-01-01 00:00:00 ┆ 200  ┆ 793.188             │\n",
       "│ 6     ┆ 1060 ┆ 2024-01-01 00:00:00 ┆ 160  ┆ 37.077              │\n",
       "│ 3     ┆ 1030 ┆ 2024-01-01 00:00:00 ┆ 240  ┆ 505.463             │\n",
       "│ 5     ┆ 1050 ┆ 2024-01-01 00:00:00 ┆ 230  ┆ 510.808             │\n",
       "└───────┴──────┴─────────────────────┴──────┴─────────────────────┘"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "# select all cols excluding some\n",
    "dfOrder.select(pl.col(\"*\").exclude(\"CUSTOMER\", \"QTY\")).sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>QTY</th></tr><tr><td>u32</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>0</td><td>1000</td><td>966405</td></tr><tr><td>9</td><td>1090</td><td>660229</td></tr><tr><td>7</td><td>1070</td><td>253823</td></tr><tr><td>3</td><td>1030</td><td>883691</td></tr><tr><td>4</td><td>1040</td><td>17190</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 3)\n",
       "┌───────┬──────┬────────┐\n",
       "│ Row # ┆ ID   ┆ QTY    │\n",
       "│ ---   ┆ ---  ┆ ---    │\n",
       "│ u32   ┆ i64  ┆ i64    │\n",
       "╞═══════╪══════╪════════╡\n",
       "│ 0     ┆ 1000 ┆ 966405 │\n",
       "│ 9     ┆ 1090 ┆ 660229 │\n",
       "│ 7     ┆ 1070 ┆ 253823 │\n",
       "│ 3     ┆ 1030 ┆ 883691 │\n",
       "│ 4     ┆ 1040 ┆ 17190  │\n",
       "└───────┴──────┴────────┘"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select certain cols\n",
    "dfOrder.select(pl.col(\"Row #\", \"ID\", \"QTY\")).sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>AS_OF_DATE</th><th>Row #</th><th>ID</th><th>QTY</th></tr><tr><td>str</td><td>u32</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>&quot;2024-Jan-01&quot;</td><td>0</td><td>1000</td><td>966405</td></tr><tr><td>&quot;2024-Jan-01&quot;</td><td>6</td><td>1060</td><td>581636</td></tr><tr><td>&quot;2024-Jan-01&quot;</td><td>2</td><td>1020</td><td>962406</td></tr><tr><td>&quot;2024-Jan-01&quot;</td><td>3</td><td>1030</td><td>883691</td></tr><tr><td>&quot;2024-Jan-01&quot;</td><td>9</td><td>1090</td><td>660229</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 4)\n",
       "┌─────────────┬───────┬──────┬────────┐\n",
       "│ AS_OF_DATE  ┆ Row # ┆ ID   ┆ QTY    │\n",
       "│ ---         ┆ ---   ┆ ---  ┆ ---    │\n",
       "│ str         ┆ u32   ┆ i64  ┆ i64    │\n",
       "╞═════════════╪═══════╪══════╪════════╡\n",
       "│ 2024-Jan-01 ┆ 0     ┆ 1000 ┆ 966405 │\n",
       "│ 2024-Jan-01 ┆ 6     ┆ 1060 ┆ 581636 │\n",
       "│ 2024-Jan-01 ┆ 2     ┆ 1020 ┆ 962406 │\n",
       "│ 2024-Jan-01 ┆ 3     ┆ 1030 ┆ 883691 │\n",
       "│ 2024-Jan-01 ┆ 9     ┆ 1090 ┆ 660229 │\n",
       "└─────────────┴───────┴──────┴────────┘"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# working with date columns\n",
    "dfOrder.select(\n",
    "    pl.col(\"AS_OF_DATE\").dt.to_string(\"%Y-%h-%d\"),\n",
    "    pl.col(\"Row #\", \"ID\", \"QTY\")\n",
    "    ).sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>ID</th><th>QTY</th><th>Amount in thousands</th></tr><tr><td>i64</td><td>i64</td><td>f64</td></tr></thead><tbody><tr><td>1070</td><td>253823</td><td>928.636</td></tr><tr><td>1010</td><td>295337</td><td>266.87</td></tr><tr><td>1060</td><td>581636</td><td>37.077</td></tr><tr><td>1080</td><td>877957</td><td>793.188</td></tr><tr><td>1040</td><td>17190</td><td>27.331</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 3)\n",
       "┌──────┬────────┬─────────────────────┐\n",
       "│ ID   ┆ QTY    ┆ Amount in thousands │\n",
       "│ ---  ┆ ---    ┆ ---                 │\n",
       "│ i64  ┆ i64    ┆ f64                 │\n",
       "╞══════╪════════╪═════════════════════╡\n",
       "│ 1070 ┆ 253823 ┆ 928.636             │\n",
       "│ 1010 ┆ 295337 ┆ 266.87              │\n",
       "│ 1060 ┆ 581636 ┆ 37.077              │\n",
       "│ 1080 ┆ 877957 ┆ 793.188             │\n",
       "│ 1040 ┆ 17190  ┆ 27.331              │\n",
       "└──────┴────────┴─────────────────────┘"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select cols by regex\n",
    "dfOrder.select(pl.col(\"^.*(ID|QT|Amount).*$\")).sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (1, 7)\n",
      "┌───────┬──────┬─────────────────────┬──────────┬──────┬────────┬─────────────────────┐\n",
      "│ Row # ┆ ID   ┆ AS_OF_DATE          ┆ CUSTOMER ┆ ITEM ┆ QTY    ┆ Amount in thousands │\n",
      "│ ---   ┆ ---  ┆ ---                 ┆ ---      ┆ ---  ┆ ---    ┆ ---                 │\n",
      "│ u32   ┆ i64  ┆ datetime[μs]        ┆ i64      ┆ i64  ┆ i64    ┆ f64                 │\n",
      "╞═══════╪══════╪═════════════════════╪══════════╪══════╪════════╪═════════════════════╡\n",
      "│ 3     ┆ 1030 ┆ 2024-01-01 00:00:00 ┆ 100      ┆ 240  ┆ 883691 ┆ 505.463             │\n",
      "└───────┴──────┴─────────────────────┴──────────┴──────┴────────┴─────────────────────┘\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 5)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>CUSTOMER</th><th>ITEM</th><th>QTY</th></tr><tr><td>u32</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>7</td><td>1070</td><td>140</td><td>120</td><td>253823</td></tr><tr><td>1</td><td>1010</td><td>180</td><td>150</td><td>295337</td></tr><tr><td>8</td><td>1080</td><td>160</td><td>200</td><td>877957</td></tr><tr><td>3</td><td>1030</td><td>100</td><td>240</td><td>883691</td></tr><tr><td>4</td><td>1040</td><td>110</td><td>230</td><td>17190</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 5)\n",
       "┌───────┬──────┬──────────┬──────┬────────┐\n",
       "│ Row # ┆ ID   ┆ CUSTOMER ┆ ITEM ┆ QTY    │\n",
       "│ ---   ┆ ---  ┆ ---      ┆ ---  ┆ ---    │\n",
       "│ u32   ┆ i64  ┆ i64      ┆ i64  ┆ i64    │\n",
       "╞═══════╪══════╪══════════╪══════╪════════╡\n",
       "│ 7     ┆ 1070 ┆ 140      ┆ 120  ┆ 253823 │\n",
       "│ 1     ┆ 1010 ┆ 180      ┆ 150  ┆ 295337 │\n",
       "│ 8     ┆ 1080 ┆ 160      ┆ 200  ┆ 877957 │\n",
       "│ 3     ┆ 1030 ┆ 100      ┆ 240  ┆ 883691 │\n",
       "│ 4     ┆ 1040 ┆ 110      ┆ 230  ┆ 17190  │\n",
       "└───────┴──────┴──────────┴──────┴────────┘"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select cols by data types\n",
    "print(dfOrder.sample(1)) # original - take a note of dtypes\n",
    "dfOrder.select(pl.col(pl.UInt32, pl.Int64)).sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (1, 7)\n",
      "┌───────┬──────┬─────────────────────┬──────────┬──────┬────────┬─────────────────────┐\n",
      "│ Row # ┆ ID   ┆ AS_OF_DATE          ┆ CUSTOMER ┆ ITEM ┆ QTY    ┆ Amount in thousands │\n",
      "│ ---   ┆ ---  ┆ ---                 ┆ ---      ┆ ---  ┆ ---    ┆ ---                 │\n",
      "│ u32   ┆ i64  ┆ datetime[μs]        ┆ i64      ┆ i64  ┆ i64    ┆ f64                 │\n",
      "╞═══════╪══════╪═════════════════════╪══════════╪══════╪════════╪═════════════════════╡\n",
      "│ 9     ┆ 1090 ┆ 2024-01-01 00:00:00 ┆ 220      ┆ 170  ┆ 660229 ┆ 512.122             │\n",
      "└───────┴──────┴─────────────────────┴──────────┴──────┴────────┴─────────────────────┘\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 5)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>CUSTOMER</th><th>ITEM</th><th>QTY</th></tr><tr><td>u32</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>5</td><td>1050</td><td>130</td><td>230</td><td>440813</td></tr><tr><td>1</td><td>1010</td><td>180</td><td>150</td><td>295337</td></tr><tr><td>2</td><td>1020</td><td>120</td><td>170</td><td>962406</td></tr><tr><td>7</td><td>1070</td><td>140</td><td>120</td><td>253823</td></tr><tr><td>6</td><td>1060</td><td>150</td><td>160</td><td>581636</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 5)\n",
       "┌───────┬──────┬──────────┬──────┬────────┐\n",
       "│ Row # ┆ ID   ┆ CUSTOMER ┆ ITEM ┆ QTY    │\n",
       "│ ---   ┆ ---  ┆ ---      ┆ ---  ┆ ---    │\n",
       "│ u32   ┆ i64  ┆ i64      ┆ i64  ┆ i64    │\n",
       "╞═══════╪══════╪══════════╪══════╪════════╡\n",
       "│ 5     ┆ 1050 ┆ 130      ┆ 230  ┆ 440813 │\n",
       "│ 1     ┆ 1010 ┆ 180      ┆ 150  ┆ 295337 │\n",
       "│ 2     ┆ 1020 ┆ 120      ┆ 170  ┆ 962406 │\n",
       "│ 7     ┆ 1070 ┆ 140      ┆ 120  ┆ 253823 │\n",
       "│ 6     ┆ 1060 ┆ 150      ┆ 160  ┆ 581636 │\n",
       "└───────┴──────┴──────────┴──────┴────────┘"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select cols by data types\n",
    "print(dfOrder.sample(1)) # original - take a note of dtypes\n",
    "dfOrder.select(pl.col(pl.UInt32, pl.Int64)).sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (9, 2)\n",
      "┌──────────┬───────┐\n",
      "│ CUSTOMER ┆ count │\n",
      "│ ---      ┆ ---   │\n",
      "│ i64      ┆ u32   │\n",
      "╞══════════╪═══════╡\n",
      "│ 130      ┆ 1     │\n",
      "│ 120      ┆ 1     │\n",
      "│ 220      ┆ 1     │\n",
      "│ 110      ┆ 1     │\n",
      "│ 150      ┆ 2     │\n",
      "│ 180      ┆ 1     │\n",
      "│ 140      ┆ 1     │\n",
      "│ 160      ┆ 1     │\n",
      "│ 100      ┆ 1     │\n",
      "└──────────┴───────┘\n",
      "9\n"
     ]
    }
   ],
   "source": [
    "# select cols to pull unique # of column values\n",
    "# for example, pull # of distint | unique customers\n",
    "\n",
    "dfOrderSample = dfOrder.select(pl.col(\"CUSTOMER\"))\n",
    "\n",
    "# print row count by CUSTOMER\n",
    "print(dfOrderSample.group_by(\"CUSTOMER\").agg(pl.count()))\n",
    "\n",
    "# print unique # of rows by CUSTOMER\n",
    "print(dfOrderSample.select(pl.col(\"CUSTOMER\")).n_unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (10, 2)\n",
      "┌──────────┬─────────────┐\n",
      "│ CUSTOMER ┆ conditional │\n",
      "│ ---      ┆ ---         │\n",
      "│ i64      ┆ str         │\n",
      "╞══════════╪═════════════╡\n",
      "│ 150      ┆ 0           │\n",
      "│ 180      ┆ 0           │\n",
      "│ 120      ┆ 0           │\n",
      "│ 100      ┆ Preferred   │\n",
      "│ …        ┆ …           │\n",
      "│ 150      ┆ 0           │\n",
      "│ 140      ┆ 0           │\n",
      "│ 160      ┆ 0           │\n",
      "│ 220      ┆ 0           │\n",
      "└──────────┴─────────────┘\n"
     ]
    }
   ],
   "source": [
    "# using conditional expression\n",
    "\n",
    "df_conditional = dfOrder.select(\n",
    "    pl.col(\"CUSTOMER\"),\n",
    "    pl.when(pl.col(\"CUSTOMER\") == 100)\n",
    "    .then(pl.lit(\"Preferred\"))\n",
    "    .otherwise(pl.lit(False))\n",
    "    .alias(\"conditional\"),\n",
    "    \n",
    ")\n",
    "print(df_conditional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### select columns using selectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "dtype_columns([Datetime(Milliseconds, None), Datetime(Microseconds, Some(\"*\")), Datetime(Milliseconds, Some(\"*\")), Duration(Microseconds), Datetime(Microseconds, None), Duration(Microseconds), Duration(Nanoseconds), Time, Date, Duration(Milliseconds), Datetime(Nanoseconds, None), Datetime(Nanoseconds, Some(\"*\")), Datetime(Microseconds, None)]).to_string().sample_n()\n"
     ]
    }
   ],
   "source": [
    "#########################################\n",
    "## select columns using selectors\n",
    "## using cs selector for column selection\n",
    "#########################################\n",
    "\n",
    "import polars.selectors as cs\n",
    "\n",
    "dfOrder.select(pl.all())\n",
    "dfOrder.select(cs.integer(), cs.string()).sample(5) \n",
    "# all int and string cols\n",
    "\n",
    "dfOrder.select(cs.numeric() - cs.first()) \n",
    "# all cols except first col\n",
    "\n",
    "dfOrder.select(cs.by_name(\"CUSTOMER\") | cs.numeric()) \n",
    "# col=CUSTOMER or all numeric cols\n",
    "\n",
    "dfOrder.select(~cs.numeric()) \n",
    "# everything else which is not numeric\n",
    "\n",
    "dfOrder.select(cs.contains(\"ID\"), cs.matches(\".*_.*\")) \n",
    "# select cols by pattern\n",
    "\n",
    "dfOrder.select(cs.temporal().as_expr().dt.to_string(\"%Y-%h-%d\")) \n",
    "# cols by converting to expressions\n",
    "\n",
    "#####################\n",
    "# debugging selectors\n",
    "#####################\n",
    "\n",
    "# is_selector\n",
    "from polars.selectors import is_selector\n",
    "\n",
    "out = cs.temporal()\n",
    "print(is_selector(out))\n",
    "\n",
    "# selector_column_names\n",
    "from polars.selectors import selector_column_names\n",
    "\n",
    "out = cs.temporal().as_expr().dt.to_string(\"%Y-%h-%d\")\n",
    "print(out.sample(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Data Type Casting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 7)\n",
      "┌───────┬──────┬─────────────────────┬──────────┬──────┬────────┬─────────────────────┐\n",
      "│ Row # ┆ ID   ┆ AS_OF_DATE          ┆ CUSTOMER ┆ ITEM ┆ QTY    ┆ Amount in thousands │\n",
      "│ ---   ┆ ---  ┆ ---                 ┆ ---      ┆ ---  ┆ ---    ┆ ---                 │\n",
      "│ u32   ┆ i64  ┆ datetime[μs]        ┆ i64      ┆ i64  ┆ i64    ┆ f64                 │\n",
      "╞═══════╪══════╪═════════════════════╪══════════╪══════╪════════╪═════════════════════╡\n",
      "│ 8     ┆ 1080 ┆ 2024-01-01 00:00:00 ┆ 160      ┆ 200  ┆ 877957 ┆ 793.188             │\n",
      "│ 1     ┆ 1010 ┆ 2024-01-01 00:00:00 ┆ 180      ┆ 150  ┆ 295337 ┆ 266.87              │\n",
      "│ 2     ┆ 1020 ┆ 2024-01-01 00:00:00 ┆ 120      ┆ 170  ┆ 962406 ┆ 997.515             │\n",
      "│ 7     ┆ 1070 ┆ 2024-01-01 00:00:00 ┆ 140      ┆ 120  ┆ 253823 ┆ 928.636             │\n",
      "│ 4     ┆ 1040 ┆ 2024-01-01 00:00:00 ┆ 110      ┆ 230  ┆ 17190  ┆ 27.331              │\n",
      "└───────┴──────┴─────────────────────┴──────────┴──────┴────────┴─────────────────────┘\n",
      "shape: (10, 2)\n",
      "┌────────────────────┬────────────────────┐\n",
      "│ integers_as_floats ┆ floats_as_integers │\n",
      "│ ---                ┆ ---                │\n",
      "│ f32                ┆ f32                │\n",
      "╞════════════════════╪════════════════════╡\n",
      "│ 1000.0             ┆ 966405.0           │\n",
      "│ 1010.0             ┆ 295337.0           │\n",
      "│ 1020.0             ┆ 962406.0           │\n",
      "│ 1030.0             ┆ 883691.0           │\n",
      "│ …                  ┆ …                  │\n",
      "│ 1060.0             ┆ 581636.0           │\n",
      "│ 1070.0             ┆ 253823.0           │\n",
      "│ 1080.0             ┆ 877957.0           │\n",
      "│ 1090.0             ┆ 660229.0           │\n",
      "└────────────────────┴────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "#####################\n",
    "## DataType Casting #\n",
    "#####################\n",
    "# Polar provide casting method to convert the underlying DataType \n",
    "# of a column to a new one\n",
    "# strict=True # means, DataType conversion will throw an error\n",
    "# strict=False # means, DataType conversion will convert value to null\n",
    "\n",
    "# cast\n",
    "print(dfOrder.sample(5))\n",
    "out = dfOrder.select(\n",
    "    pl.col(\"ID\").cast(pl.Float32).alias(\"integers_as_floats\"),\n",
    "    pl.col(\"QTY\").cast(pl.Float32).alias(\"floats_as_integers\"),\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (10, 3)\n",
      "┌─────────────────────────┬───────────────────────┬────────────┐\n",
      "│ integers_smallfootprint ┆ floats_smallfootprint ┆ Amount     │\n",
      "│ ---                     ┆ ---                   ┆ ---        │\n",
      "│ i16                     ┆ i16                   ┆ f32        │\n",
      "╞═════════════════════════╪═══════════════════════╪════════════╡\n",
      "│ 1000                    ┆ 150                   ┆ 835.603027 │\n",
      "│ 1010                    ┆ 180                   ┆ 266.869995 │\n",
      "│ 1020                    ┆ 120                   ┆ 997.515015 │\n",
      "│ 1030                    ┆ 100                   ┆ 505.463013 │\n",
      "│ …                       ┆ …                     ┆ …          │\n",
      "│ 1060                    ┆ 150                   ┆ 37.077     │\n",
      "│ 1070                    ┆ 140                   ┆ 928.635986 │\n",
      "│ 1080                    ┆ 160                   ┆ 793.187988 │\n",
      "│ 1090                    ┆ 220                   ┆ 512.122009 │\n",
      "└─────────────────────────┴───────────────────────┴────────────┘\n"
     ]
    }
   ],
   "source": [
    "# downncast\n",
    "# casting from Int64 to Int16 and from Float64 to Float32 can be \n",
    "# used to lower memory usage\n",
    "out = dfOrder.select(\n",
    "    pl.col(\"ID\").cast(pl.Int16).alias(\"integers_smallfootprint\"),\n",
    "    pl.col(\"CUSTOMER\").cast(pl.Int16).alias(\"floats_smallfootprint\"),\n",
    "    pl.col(\"Amount in thousands\").cast(pl.Float32).alias(\"Amount\"),\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "strict conversion from `f64` to `i8` failed for column: Amount in thousands, value(s) [266.87, 505.463, … 997.515]; if you were trying to cast Utf8 to temporal dtypes, consider using `strptime`\n",
      "shape: (10, 1)\n",
      "┌─────────────────────┐\n",
      "│ Amount in thousands │\n",
      "│ ---                 │\n",
      "│ i8                  │\n",
      "╞═════════════════════╡\n",
      "│ null                │\n",
      "│ null                │\n",
      "│ null                │\n",
      "│ null                │\n",
      "│ …                   │\n",
      "│ 37                  │\n",
      "│ null                │\n",
      "│ null                │\n",
      "│ null                │\n",
      "└─────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "# overflow\n",
    "try:\n",
    "    out = dfOrder.select(pl.col(\"Amount in thousands\").cast(pl.Int8))\n",
    "    print(out)\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "\n",
    "# to supress above error\n",
    "# run with strict=False\n",
    "out = dfOrder.select(pl.col(\"Amount in thousands\").cast(pl.Int8, strict=False))\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (10, 7)\n",
      "┌───────┬─────────────────────┬──────────┬──────┬────────┬─────────────────────┬──────┐\n",
      "│ Row # ┆ AS_OF_DATE          ┆ CUSTOMER ┆ ITEM ┆ QTY    ┆ Amount in thousands ┆ ID   │\n",
      "│ ---   ┆ ---                 ┆ ---      ┆ ---  ┆ ---    ┆ ---                 ┆ ---  │\n",
      "│ u32   ┆ datetime[μs]        ┆ i64      ┆ i64  ┆ i64    ┆ f64                 ┆ str  │\n",
      "╞═══════╪═════════════════════╪══════════╪══════╪════════╪═════════════════════╪══════╡\n",
      "│ 0     ┆ 2024-01-01 00:00:00 ┆ 150      ┆ 180  ┆ 966405 ┆ 835.603             ┆ 1000 │\n",
      "│ 1     ┆ 2024-01-01 00:00:00 ┆ 180      ┆ 150  ┆ 295337 ┆ 266.87              ┆ 1010 │\n",
      "│ 2     ┆ 2024-01-01 00:00:00 ┆ 120      ┆ 170  ┆ 962406 ┆ 997.515             ┆ 1020 │\n",
      "│ 3     ┆ 2024-01-01 00:00:00 ┆ 100      ┆ 240  ┆ 883691 ┆ 505.463             ┆ 1030 │\n",
      "│ …     ┆ …                   ┆ …        ┆ …    ┆ …      ┆ …                   ┆ …    │\n",
      "│ 6     ┆ 2024-01-01 00:00:00 ┆ 150      ┆ 160  ┆ 581636 ┆ 37.077              ┆ 1060 │\n",
      "│ 7     ┆ 2024-01-01 00:00:00 ┆ 140      ┆ 120  ┆ 253823 ┆ 928.636             ┆ 1070 │\n",
      "│ 8     ┆ 2024-01-01 00:00:00 ┆ 160      ┆ 200  ┆ 877957 ┆ 793.188             ┆ 1080 │\n",
      "│ 9     ┆ 2024-01-01 00:00:00 ┆ 220      ┆ 170  ┆ 660229 ┆ 512.122             ┆ 1090 │\n",
      "└───────┴─────────────────────┴──────────┴──────┴────────┴─────────────────────┴──────┘\n",
      "shape: (10, 7)\n",
      "┌───────┬─────────────────────┬──────────┬──────┬────────┬─────────────────────┬──────┐\n",
      "│ Row # ┆ AS_OF_DATE          ┆ CUSTOMER ┆ ITEM ┆ QTY    ┆ Amount in thousands ┆ ID   │\n",
      "│ ---   ┆ ---                 ┆ ---      ┆ ---  ┆ ---    ┆ ---                 ┆ ---  │\n",
      "│ u32   ┆ datetime[μs]        ┆ i64      ┆ i64  ┆ i64    ┆ f64                 ┆ i16  │\n",
      "╞═══════╪═════════════════════╪══════════╪══════╪════════╪═════════════════════╪══════╡\n",
      "│ 0     ┆ 2024-01-01 00:00:00 ┆ 150      ┆ 180  ┆ 966405 ┆ 835.603             ┆ 1000 │\n",
      "│ 1     ┆ 2024-01-01 00:00:00 ┆ 180      ┆ 150  ┆ 295337 ┆ 266.87              ┆ 1010 │\n",
      "│ 2     ┆ 2024-01-01 00:00:00 ┆ 120      ┆ 170  ┆ 962406 ┆ 997.515             ┆ 1020 │\n",
      "│ 3     ┆ 2024-01-01 00:00:00 ┆ 100      ┆ 240  ┆ 883691 ┆ 505.463             ┆ 1030 │\n",
      "│ …     ┆ …                   ┆ …        ┆ …    ┆ …      ┆ …                   ┆ …    │\n",
      "│ 6     ┆ 2024-01-01 00:00:00 ┆ 150      ┆ 160  ┆ 581636 ┆ 37.077              ┆ 1060 │\n",
      "│ 7     ┆ 2024-01-01 00:00:00 ┆ 140      ┆ 120  ┆ 253823 ┆ 928.636             ┆ 1070 │\n",
      "│ 8     ┆ 2024-01-01 00:00:00 ┆ 160      ┆ 200  ┆ 877957 ┆ 793.188             ┆ 1080 │\n",
      "│ 9     ┆ 2024-01-01 00:00:00 ┆ 220      ┆ 170  ┆ 660229 ┆ 512.122             ┆ 1090 │\n",
      "└───────┴─────────────────────┴──────────┴──────┴────────┴─────────────────────┴──────┘\n"
     ]
    }
   ],
   "source": [
    "# strings datatype casting\n",
    "\n",
    "# changing numeric fields to string\n",
    "out = dfOrder.select(\n",
    "    pl.all().exclude(\"ID\"),\n",
    "    pl.col(\"ID\").cast(pl.Utf8)\n",
    ")\n",
    "print(out)\n",
    "\n",
    "# change string back to numeric\n",
    "out = dfOrder.select(\n",
    "    pl.all().exclude(\"ID\"),\n",
    "    pl.col(\"ID\").cast(pl.Int16)\n",
    ")\n",
    "print(out)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "strict conversion from `str` to `f64` failed for column: strings_not_float, value(s) [\"not_a_number\"]; if you were trying to cast Utf8 to temporal dtypes, consider using `strptime`\n"
     ]
    }
   ],
   "source": [
    "df = pl.DataFrame({\"strings_not_float\": [\"4.0\", \"not_a_number\",\n",
    "                                          \"6.0\", \"7.0\", \"8.0\"]})\n",
    "try:\n",
    "    out = df.select(pl.col(\"strings_not_float\").cast(pl.Float64))\n",
    "    print(out)\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "\n",
    "# as per exception, it suggests to use strptime to convert string to numeric\n",
    "# this is only applicable and works well when string hold datatime values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (4, 7)\n",
      "┌───────┬──────┬─────────────────────┬──────────┬──────┬────────┬─────────────────────┐\n",
      "│ Row # ┆ ID   ┆ AS_OF_DATE          ┆ CUSTOMER ┆ ITEM ┆ QTY    ┆ Amount in thousands │\n",
      "│ ---   ┆ ---  ┆ ---                 ┆ ---      ┆ ---  ┆ ---    ┆ ---                 │\n",
      "│ u32   ┆ i64  ┆ datetime[μs]        ┆ i64      ┆ i64  ┆ i64    ┆ f64                 │\n",
      "╞═══════╪══════╪═════════════════════╪══════════╪══════╪════════╪═════════════════════╡\n",
      "│ 8     ┆ 1080 ┆ 2024-01-01 00:00:00 ┆ 160      ┆ 200  ┆ 877957 ┆ 793.188             │\n",
      "│ 1     ┆ 1010 ┆ 2024-01-01 00:00:00 ┆ 180      ┆ 150  ┆ 295337 ┆ 266.87              │\n",
      "│ 2     ┆ 1020 ┆ 2024-01-01 00:00:00 ┆ 120      ┆ 170  ┆ 962406 ┆ 997.515             │\n",
      "│ 9     ┆ 1090 ┆ 2024-01-01 00:00:00 ┆ 220      ┆ 170  ┆ 660229 ┆ 512.122             │\n",
      "└───────┴──────┴─────────────────────┴──────────┴──────┴────────┴─────────────────────┘\n",
      "shape: (10, 7)\n",
      "┌───────┬──────┬──────┬─────────────────────┬──────────┬──────┬─────────────────────┐\n",
      "│ Row # ┆ QTY  ┆ ID   ┆ AS_OF_DATE          ┆ CUSTOMER ┆ ITEM ┆ Amount in thousands │\n",
      "│ ---   ┆ ---  ┆ ---  ┆ ---                 ┆ ---      ┆ ---  ┆ ---                 │\n",
      "│ bool  ┆ bool ┆ i64  ┆ datetime[μs]        ┆ i64      ┆ i64  ┆ f64                 │\n",
      "╞═══════╪══════╪══════╪═════════════════════╪══════════╪══════╪═════════════════════╡\n",
      "│ false ┆ true ┆ 1000 ┆ 2024-01-01 00:00:00 ┆ 150      ┆ 180  ┆ 835.603             │\n",
      "│ true  ┆ true ┆ 1010 ┆ 2024-01-01 00:00:00 ┆ 180      ┆ 150  ┆ 266.87              │\n",
      "│ true  ┆ true ┆ 1020 ┆ 2024-01-01 00:00:00 ┆ 120      ┆ 170  ┆ 997.515             │\n",
      "│ true  ┆ true ┆ 1030 ┆ 2024-01-01 00:00:00 ┆ 100      ┆ 240  ┆ 505.463             │\n",
      "│ …     ┆ …    ┆ …    ┆ …                   ┆ …        ┆ …    ┆ …                   │\n",
      "│ true  ┆ true ┆ 1060 ┆ 2024-01-01 00:00:00 ┆ 150      ┆ 160  ┆ 37.077              │\n",
      "│ true  ┆ true ┆ 1070 ┆ 2024-01-01 00:00:00 ┆ 140      ┆ 120  ┆ 928.636             │\n",
      "│ true  ┆ true ┆ 1080 ┆ 2024-01-01 00:00:00 ┆ 160      ┆ 200  ┆ 793.188             │\n",
      "│ true  ┆ true ┆ 1090 ┆ 2024-01-01 00:00:00 ┆ 220      ┆ 170  ┆ 512.122             │\n",
      "└───────┴──────┴──────┴─────────────────────┴──────────┴──────┴─────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "# boolean\n",
    "# Boolean cast convert all non-zero numeric (int & floats) to true\n",
    "# Boolean cast convert all zero numeric (int & floats) to false\n",
    "\n",
    "print(dfOrder.sample(4))\n",
    "out = dfOrder.select(\n",
    "        pl.col(\"Row #\").cast(pl.Boolean),\n",
    "        pl.col(\"QTY\").cast(pl.Boolean),\n",
    "        pl.all().exclude(\"Row #\", \"QTY\"),\n",
    "        )\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (10, 3)\n",
      "┌─────────────────────┬──────────────────┬────────────────────────────┐\n",
      "│ AS_OF_DATE          ┆ DATE_as_Int      ┆ DATE_as_Str                │\n",
      "│ ---                 ┆ ---              ┆ ---                        │\n",
      "│ datetime[μs]        ┆ i64              ┆ str                        │\n",
      "╞═════════════════════╪══════════════════╪════════════════════════════╡\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ …                   ┆ …                ┆ …                          │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "└─────────────────────┴──────────────────┴────────────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "# dates\n",
    "out = dfOrder.select(\n",
    "                    pl.col(\"AS_OF_DATE\"),\n",
    "                    pl.col(\"AS_OF_DATE\").cast(pl.Int64).alias(\"DATE_as_Int\"),\n",
    "                    pl.col(\"AS_OF_DATE\").cast(pl.Utf8).alias(\"DATE_as_Str\")\n",
    "                    )\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (10, 3)\n",
      "┌─────────────────────┬──────────────────┬────────────────────────────┐\n",
      "│ AS_OF_DATE          ┆ DATE_as_Int      ┆ DATE_as_Str                │\n",
      "│ ---                 ┆ ---              ┆ ---                        │\n",
      "│ datetime[μs]        ┆ i64              ┆ str                        │\n",
      "╞═════════════════════╪══════════════════╪════════════════════════════╡\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ …                   ┆ …                ┆ …                          │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "│ 2024-01-01 00:00:00 ┆ 1704067200000000 ┆ 2024-01-01 00:00:00.000000 │\n",
      "└─────────────────────┴──────────────────┴────────────────────────────┘\n",
      "shape: (10, 3)\n",
      "┌─────────────────────┬────────────────────────┬──────────────┐\n",
      "│ AS_OF_DATE          ┆ AS_OF_DATE_as_strftime ┆ DATE_as_Str  │\n",
      "│ ---                 ┆ ---                    ┆ ---          │\n",
      "│ datetime[μs]        ┆ str                    ┆ datetime[μs] │\n",
      "╞═════════════════════╪════════════════════════╪══════════════╡\n",
      "│ 2024-01-01 00:00:00 ┆ 2024-01-01             ┆ null         │\n",
      "│ 2024-01-01 00:00:00 ┆ 2024-01-01             ┆ null         │\n",
      "│ 2024-01-01 00:00:00 ┆ 2024-01-01             ┆ null         │\n",
      "│ 2024-01-01 00:00:00 ┆ 2024-01-01             ┆ null         │\n",
      "│ …                   ┆ …                      ┆ …            │\n",
      "│ 2024-01-01 00:00:00 ┆ 2024-01-01             ┆ null         │\n",
      "│ 2024-01-01 00:00:00 ┆ 2024-01-01             ┆ null         │\n",
      "│ 2024-01-01 00:00:00 ┆ 2024-01-01             ┆ null         │\n",
      "│ 2024-01-01 00:00:00 ┆ 2024-01-01             ┆ null         │\n",
      "└─────────────────────┴────────────────────────┴──────────────┘\n"
     ]
    }
   ],
   "source": [
    "# To perform casting operations between strings and Dates/Datetimes, strftime and strptime are utilized.\n",
    "# Polars adopts the chrono format syntax for when formatting. \n",
    "# It's worth noting that strptime features additional options that support timezone functionality.\n",
    "# Refer to the API documentation for further information.\n",
    "\n",
    "dfOrderTmp = dfOrder.select(\n",
    "                    pl.col(\"AS_OF_DATE\"),\n",
    "                    pl.col(\"AS_OF_DATE\").cast(pl.Int64).alias(\"DATE_as_Int\"),\n",
    "                    pl.col(\"AS_OF_DATE\").cast(pl.Utf8).alias(\"DATE_as_Str\")\n",
    "                     )\n",
    "print(dfOrderTmp)\n",
    "\n",
    "out = dfOrderTmp.select(\n",
    "    pl.col(\"AS_OF_DATE\"),\n",
    "    pl.col(\"AS_OF_DATE\").dt.strftime(\"%Y-%m-%d\")\n",
    "    .alias(\"AS_OF_DATE_as_strftime\"),\n",
    "    pl.col(\"DATE_as_Str\").str.strptime(pl.Datetime, \"%Y-%m-%d\", strict=False)\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### working with Strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 9)\n",
      "┌─────┬─────────────────────┬─────────────┬───────────┬───┬─────────┬────────┬──────┬──────────┐\n",
      "│ ID  ┆ AS_OF_DATE          ┆ DESCRIPTION ┆ ADDRESS   ┆ … ┆ EMAIL   ┆ STATUS ┆ TYPE ┆ CATEGORY │\n",
      "│ --- ┆ ---                 ┆ ---         ┆ ---       ┆   ┆ ---     ┆ ---    ┆ ---  ┆ ---      │\n",
      "│ i64 ┆ datetime[μs]        ┆ str         ┆ str       ┆   ┆ str     ┆ str    ┆ str  ┆ i64      │\n",
      "╞═════╪═════════════════════╪═════════════╪═══════════╪═══╪═════════╪════════╪══════╪══════════╡\n",
      "│ 100 ┆ 2022-01-01 00:00:00 ┆ Customer 1  ┆ Address 1 ┆ … ┆ 1@email ┆ Active ┆ Corp ┆ 1600     │\n",
      "│ 230 ┆ 2022-01-01 00:00:00 ┆ Customer 4  ┆ Address 4 ┆ … ┆ 4@email ┆ Active ┆ Gov  ┆ 1600     │\n",
      "│ 160 ┆ 2022-01-01 00:00:00 ┆ Customer 2  ┆ Address 2 ┆ … ┆ 2@email ┆ Active ┆ Corp ┆ 1000     │\n",
      "│ 130 ┆ 2022-01-01 00:00:00 ┆ Customer 4  ┆ Address 4 ┆ … ┆ 4@email ┆ Active ┆ Corp ┆ 1200     │\n",
      "│ 140 ┆ 2022-01-01 00:00:00 ┆ Customer 5  ┆ Address 5 ┆ … ┆ 5@email ┆ Active ┆ Gov  ┆ 1200     │\n",
      "└─────┴─────────────────────┴─────────────┴───────────┴───┴─────────┴────────┴──────┴──────────┘\n",
      "shape: (15, 11)\n",
      "┌─────┬─────────────┬────────────┬───────────┬───┬────────────┬──────────┬────────────┬────────────┐\n",
      "│ ID  ┆ AS_OF_DATE  ┆ DESCRIPTIO ┆ ADDRESS   ┆ … ┆ TYPE       ┆ CATEGORY ┆ DESCRIPTIO ┆ ADDRESS_le │\n",
      "│ --- ┆ ---         ┆ N          ┆ ---       ┆   ┆ ---        ┆ ---      ┆ N_byte_cou ┆ tter_count │\n",
      "│ i64 ┆ datetime[μs ┆ ---        ┆ str       ┆   ┆ str        ┆ i64      ┆ nt         ┆ ---        │\n",
      "│     ┆ ]           ┆ str        ┆           ┆   ┆            ┆          ┆ ---        ┆ u32        │\n",
      "│     ┆             ┆            ┆           ┆   ┆            ┆          ┆ u32        ┆            │\n",
      "╞═════╪═════════════╪════════════╪═══════════╪═══╪════════════╪══════════╪════════════╪════════════╡\n",
      "│ 100 ┆ 2022-01-01  ┆ Customer 1 ┆ Address 1 ┆ … ┆ Corp       ┆ 1600     ┆ 10         ┆ 9          │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 110 ┆ 2022-01-01  ┆ Customer 2 ┆ Address 2 ┆ … ┆ Gov        ┆ 1200     ┆ 10         ┆ 9          │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 120 ┆ 2022-01-01  ┆ Customer 3 ┆ Address 3 ┆ … ┆ Individual ┆ 2400     ┆ 10         ┆ 9          │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 130 ┆ 2022-01-01  ┆ Customer 4 ┆ Address 4 ┆ … ┆ Corp       ┆ 1200     ┆ 10         ┆ 9          │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ …   ┆ …           ┆ …          ┆ …         ┆ … ┆ …          ┆ …        ┆ …          ┆ …          │\n",
      "│ 210 ┆ 2022-01-01  ┆ Customer 2 ┆ Address 2 ┆ … ┆ Individual ┆ 2400     ┆ 10         ┆ 9          │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 220 ┆ 2022-01-01  ┆ Customer 3 ┆ Address 3 ┆ … ┆ Corp       ┆ 1900     ┆ 10         ┆ 9          │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 230 ┆ 2022-01-01  ┆ Customer 4 ┆ Address 4 ┆ … ┆ Gov        ┆ 1600     ┆ 10         ┆ 9          │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 240 ┆ 2022-01-01  ┆ Customer 5 ┆ Address 5 ┆ … ┆ Individual ┆ 1900     ┆ 10         ┆ 9          │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "└─────┴─────────────┴────────────┴───────────┴───┴────────────┴──────────┴────────────┴────────────┘\n"
     ]
    }
   ],
   "source": [
    "# working with Strings\n",
    "# Polars store string as Utf8 strings\n",
    "# String processing functions are available in the str namespace.\n",
    "\n",
    "print(customer.sample(5))\n",
    "\n",
    "out = customer.select(\n",
    "    pl.all(),\n",
    "    pl.col(\"DESCRIPTION\").str.lengths().alias(\"DESCRIPTION_byte_count\"),\n",
    "    pl.col(\"ADDRESS\").str.n_chars().alias(\"ADDRESS_letter_count\"),\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (15, 5)\n",
      "┌───────────┬───────┬─────────┬─────────────┬───────────┐\n",
      "│ ADDRESS   ┆ regex ┆ literal ┆ starts_with ┆ ends_with │\n",
      "│ ---       ┆ ---   ┆ ---     ┆ ---         ┆ ---       │\n",
      "│ str       ┆ bool  ┆ bool    ┆ bool        ┆ bool      │\n",
      "╞═══════════╪═══════╪═════════╪═════════════╪═══════════╡\n",
      "│ Address 1 ┆ true  ┆ false   ┆ true        ┆ false     │\n",
      "│ Address 2 ┆ true  ┆ false   ┆ true        ┆ false     │\n",
      "│ Address 3 ┆ true  ┆ false   ┆ true        ┆ false     │\n",
      "│ Address 4 ┆ true  ┆ false   ┆ true        ┆ false     │\n",
      "│ …         ┆ …     ┆ …       ┆ …           ┆ …         │\n",
      "│ Address 2 ┆ true  ┆ false   ┆ true        ┆ false     │\n",
      "│ Address 3 ┆ true  ┆ false   ┆ true        ┆ false     │\n",
      "│ Address 4 ┆ true  ┆ false   ┆ true        ┆ false     │\n",
      "│ Address 5 ┆ true  ┆ false   ┆ true        ┆ false     │\n",
      "└───────────┴───────┴─────────┴─────────────┴───────────┘\n"
     ]
    }
   ],
   "source": [
    "# string parsing\n",
    "out = customer.select(\n",
    "    # pl.all(),\n",
    "    pl.col(\"ADDRESS\"),\n",
    "    pl.col(\"ADDRESS\").str.contains(\"aDD|ress\").alias(\"regex\"),\n",
    "    pl.col(\"ADDRESS\").str.contains(\"Add$\", literal=True).alias(\"literal\"),\n",
    "    pl.col(\"ADDRESS\").str.starts_with(\"Add\").alias(\"starts_with\"),\n",
    "    pl.col(\"ADDRESS\").str.ends_with(\"ress\").alias(\"ends_with\"),\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (3, 1)\n",
      "┌─────────┐\n",
      "│ a       │\n",
      "│ ---     │\n",
      "│ str     │\n",
      "╞═════════╡\n",
      "│ messi   │\n",
      "│ null    │\n",
      "│ ronaldo │\n",
      "└─────────┘\n",
      "shape: (15, 1)\n",
      "┌───────────────┐\n",
      "│ extracted_nrs │\n",
      "│ ---           │\n",
      "│ list[str]     │\n",
      "╞═══════════════╡\n",
      "│ [\"1\"]         │\n",
      "│ [\"2\"]         │\n",
      "│ [\"3\"]         │\n",
      "│ [\"4\"]         │\n",
      "│ …             │\n",
      "│ [\"2\"]         │\n",
      "│ [\"3\"]         │\n",
      "│ [\"4\"]         │\n",
      "│ [\"5\"]         │\n",
      "└───────────────┘\n"
     ]
    }
   ],
   "source": [
    "# extract a pettern\n",
    "df = pl.DataFrame(\n",
    "    {\n",
    "        \"a\": [\n",
    "            \"http://vote.com/ballon_dor?candidate=messi&ref=polars\",\n",
    "            \"http://vote.com/ballon_dor?candidat=jorginho&ref=polars\",\n",
    "            \"http://vote.com/ballon_dor?candidate=ronaldo&ref=polars\",\n",
    "        ]\n",
    "    }\n",
    ")\n",
    "out = df.select(\n",
    "    pl.col(\"a\").str.extract(r\"candidate=(\\w+)\", group_index=1),\n",
    ")\n",
    "print(out)\n",
    "\n",
    "# extract all\n",
    "df = pl.DataFrame({\"foo\": [\"123 bla 45 asd\", \"xyz 678 910t\"]})\n",
    "out = customer.select(\n",
    "    pl.col(\"ADDRESS\").str.extract_all(r\"(\\d+)\").alias(\"extracted_nrs\"),\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (15, 11)\n",
      "┌─────┬─────────────┬────────────┬───────────┬───┬────────────┬──────────┬────────────┬────────────┐\n",
      "│ ID  ┆ AS_OF_DATE  ┆ DESCRIPTIO ┆ ADDRESS   ┆ … ┆ TYPE       ┆ CATEGORY ┆ text_repla ┆ text_repla │\n",
      "│ --- ┆ ---         ┆ N          ┆ ---       ┆   ┆ ---        ┆ ---      ┆ ce         ┆ ce_all     │\n",
      "│ i64 ┆ datetime[μs ┆ ---        ┆ str       ┆   ┆ str        ┆ i64      ┆ ---        ┆ ---        │\n",
      "│     ┆ ]           ┆ str        ┆           ┆   ┆            ┆          ┆ str        ┆ str        │\n",
      "╞═════╪═════════════╪════════════╪═══════════╪═══╪════════════╪══════════╪════════════╪════════════╡\n",
      "│ 100 ┆ 2022-01-01  ┆ Customer 1 ┆ Address 1 ┆ … ┆ Corp       ┆ 1600     ┆ ADDRESS 1  ┆ AddRESS 1  │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 110 ┆ 2022-01-01  ┆ Customer 2 ┆ Address 2 ┆ … ┆ Gov        ┆ 1200     ┆ ADDRESS 2  ┆ AddRESS 2  │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 120 ┆ 2022-01-01  ┆ Customer 3 ┆ Address 3 ┆ … ┆ Individual ┆ 2400     ┆ ADDRESS 3  ┆ AddRESS 3  │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 130 ┆ 2022-01-01  ┆ Customer 4 ┆ Address 4 ┆ … ┆ Corp       ┆ 1200     ┆ ADDRESS 4  ┆ AddRESS 4  │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ …   ┆ …           ┆ …          ┆ …         ┆ … ┆ …          ┆ …        ┆ …          ┆ …          │\n",
      "│ 210 ┆ 2022-01-01  ┆ Customer 2 ┆ Address 2 ┆ … ┆ Individual ┆ 2400     ┆ ADDRESS 2  ┆ AddRESS 2  │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 220 ┆ 2022-01-01  ┆ Customer 3 ┆ Address 3 ┆ … ┆ Corp       ┆ 1900     ┆ ADDRESS 3  ┆ AddRESS 3  │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 230 ┆ 2022-01-01  ┆ Customer 4 ┆ Address 4 ┆ … ┆ Gov        ┆ 1600     ┆ ADDRESS 4  ┆ AddRESS 4  │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "│ 240 ┆ 2022-01-01  ┆ Customer 5 ┆ Address 5 ┆ … ┆ Individual ┆ 1900     ┆ ADDRESS 5  ┆ AddRESS 5  │\n",
      "│     ┆ 00:00:00    ┆            ┆           ┆   ┆            ┆          ┆            ┆            │\n",
      "└─────┴─────────────┴────────────┴───────────┴───┴────────────┴──────────┴────────────┴────────────┘\n"
     ]
    }
   ],
   "source": [
    "# replace | replace all\n",
    "df = pl.DataFrame({\"id\": [1, 2], \"text\": [\"123abc\", \"abc456\"]})\n",
    "out = customer.with_columns(\n",
    "    pl.col(\"ADDRESS\").str.replace(r\"Address\", \"ADDRESS\")\n",
    "    .alias(\"text_replace\"),\n",
    "    pl.col(\"ADDRESS\").str.replace_all(\"ress\", \"RESS\", literal=True)\n",
    "    .alias(\"text_replace_all\"),\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### using Expression to apply Aggregation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 9)\n",
      "┌───────┬─────────┬──────────┬─────────────┬───┬─────────┬──────┬──────────┬──────────────┐\n",
      "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL │\n",
      "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
      "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64     ┆ i64  ┆ i64      ┆ i64          │\n",
      "╞═══════╪═════════╪══════════╪═════════════╪═══╪═════════╪══════╪══════════╪══════════════╡\n",
      "│ 0     ┆ BUDGET  ┆ ABC Inc. ┆ 2021        ┆ … ┆ 20000   ┆ 1800 ┆ 12       ┆ 725180       │\n",
      "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 43000   ┆ 2100 ┆ 15       ┆ 873826       │\n",
      "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 38000   ┆ 1300 ┆ 18       ┆ 590613       │\n",
      "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 30000   ┆ 1800 ┆ 12       ┆ 308302       │\n",
      "│ 4     ┆ BUDGET  ┆ ABC Inc. ┆ 2020        ┆ … ┆ 13000   ┆ 1600 ┆ 22       ┆ 251211       │\n",
      "└───────┴─────────┴──────────┴─────────────┴───┴─────────┴──────┴──────────┴──────────────┘\n",
      "(200000, 8)\n"
     ]
    }
   ],
   "source": [
    "# most used functionality in EDA is to view data \n",
    "# aggregated by different criterias\n",
    "# Polars Data Frame literally shines in here and is the tool of choice\n",
    "# because aggregation requires speed, parallel optimized queries\n",
    "\n",
    "# let's review our Ledger Data Frame we created\n",
    "\n",
    "######################\n",
    "## LEDGER DataFrame ##\n",
    "######################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "sampleSize = 100_000\n",
    "org = \"ABC Inc.\"\n",
    "ledger_type = \"ACTUALS\" # BUDGET, STATS are other Ledger types\n",
    "fiscal_year_from = 2020\n",
    "fiscal_year_to = 2023\n",
    "random.seed(123)\n",
    "\n",
    "ledger = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, \n",
    "                                          fiscal_year_to+1, 1)),k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ACCOUNT\" : random.choices(accounts[\"ID\"], k=sampleSize),\n",
    "\t\"DEPT\" : random.choices(dept[\"ID\"], k=sampleSize),\n",
    "\t\"LOCATION\" : random.choices(location[\"ID\"], k=sampleSize),\n",
    "\t\"POSTED_TOTAL\": random.sample(range(1000000), sampleSize)\n",
    "})\n",
    "\n",
    "ledger_type = \"BUDGET\" # ACTUALS, STATS are other Ledger types\n",
    "ledger_budg = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, fiscal_year_to+1, 1))\n",
    "                               ,k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ACCOUNT\" : random.choices(accounts[\"ID\"], k=sampleSize),\n",
    "\t\"DEPT\" : random.choices(dept[\"ID\"], k=sampleSize),\n",
    "\t\"LOCATION\" : random.choices(location[\"ID\"], k=sampleSize),\n",
    "\t\"POSTED_TOTAL\": random.sample(range(1000000), sampleSize)\n",
    "})\n",
    "\n",
    "#########################################\n",
    "# combined ledger for Actuals and Budget\n",
    "#########################################\n",
    "dfLedger = pl.concat([ledger, ledger_budg], how=\"vertical\")\n",
    "print(dfLedger.sample(5).with_row_count(\"Row #\"))\n",
    "print(dfLedger.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 8)\n",
      "┌─────────┬──────────┬─────────────┬────────┬─────────┬──────┬──────────┬──────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
      "│ cat     ┆ cat      ┆ i64         ┆ i64    ┆ i64     ┆ i64  ┆ i64      ┆ f32          │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═════════╪══════╪══════════╪══════════════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 1      ┆ 16000   ┆ 2200 ┆ 19       ┆ 651.127991   │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2021        ┆ 7      ┆ 14000   ┆ 1900 ┆ 15       ┆ 933.062012   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 8      ┆ 23000   ┆ 1900 ┆ 19       ┆ 667.994995   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 4      ┆ 26000   ┆ 1100 ┆ 15       ┆ 70.503998    │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 7      ┆ 23000   ┆ 1000 ┆ 18       ┆ 155.164993   │\n",
      "└─────────┴──────────┴─────────────┴────────┴─────────┴──────┴──────────┴──────────────┘\n"
     ]
    }
   ],
   "source": [
    "# first let's change data type of few columns\n",
    "# to a categorical dtype\n",
    "out = dfLedger.with_columns(\n",
    "    pl.col(\"LEDGER\").cast(pl.Categorical),\n",
    "    pl.col(\"ORG\").cast(pl.Categorical),\n",
    "    (pl.col(\"POSTED_TOTAL\") / 1000).cast(pl.Float32)\n",
    ")\n",
    "print(out.sample(5))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 5)\n",
      "┌─────────┬─────────────┬───────┬──────────────┬────────────────────┐\n",
      "│ LEDGER  ┆ FISCAL_YEAR ┆ count ┆ PERIOD       ┆ TOTAL (in milions) │\n",
      "│ ---     ┆ ---         ┆ ---   ┆ ---          ┆ ---                │\n",
      "│ cat     ┆ i64         ┆ u32   ┆ list[i64]    ┆ f32                │\n",
      "╞═════════╪═════════════╪═══════╪══════════════╪════════════════════╡\n",
      "│ BUDGET  ┆ 2023        ┆ 25260 ┆ [1, 1, … 12] ┆ 12.646653          │\n",
      "│ ACTUALS ┆ 2023        ┆ 25136 ┆ [1, 1, … 12] ┆ 12.526934          │\n",
      "│ ACTUALS ┆ 2021        ┆ 25036 ┆ [1, 1, … 12] ┆ 12.506885          │\n",
      "│ ACTUALS ┆ 2020        ┆ 24957 ┆ [1, 1, … 12] ┆ 12.430145          │\n",
      "│ BUDGET  ┆ 2020        ┆ 24944 ┆ [1, 1, … 12] ┆ 12.488555          │\n",
      "└─────────┴─────────────┴───────┴──────────────┴────────────────────┘\n"
     ]
    }
   ],
   "source": [
    "q = (\n",
    "    out.lazy()\n",
    "    .group_by(\"LEDGER\", \"FISCAL_YEAR\")\n",
    "    .agg(\n",
    "        pl.count(),\n",
    "        pl.col(\"PERIOD\").sort(), # list of different periods\n",
    "        (pl.sum(\"POSTED_TOTAL\") / 1000000).cast(pl.Float32).alias(\"TOTAL (in milions)\")\n",
    "    )\n",
    "    .sort(\"count\", descending=True)\n",
    "    .limit(5)\n",
    ")\n",
    "\n",
    "df = q.collect()\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 8)\n",
      "┌─────────┬─────────────┬───────┬───────────┬───────────┬───────────┬───────────┬───────────┐\n",
      "│ LEDGER  ┆ FISCAL_YEAR ┆ count ┆ Q1 counts ┆ Q2 counts ┆ Q3 counts ┆ Q4 counts ┆ TOTAL (in │\n",
      "│ ---     ┆ ---         ┆ ---   ┆ ---       ┆ ---       ┆ ---       ┆ ---       ┆ milions)  │\n",
      "│ cat     ┆ i64         ┆ u32   ┆ i32       ┆ i32       ┆ i32       ┆ i32       ┆ ---       │\n",
      "│         ┆             ┆       ┆           ┆           ┆           ┆           ┆ f32       │\n",
      "╞═════════╪═════════════╪═══════╪═══════════╪═══════════╪═══════════╪═══════════╪═══════════╡\n",
      "│ ACTUALS ┆ 2023        ┆ 25136 ┆ 6339      ┆ 6377      ┆ 6222      ┆ 6198      ┆ 12.526934 │\n",
      "│ BUDGET  ┆ 2023        ┆ 25260 ┆ 6221      ┆ 6298      ┆ 6406      ┆ 6335      ┆ 12.646653 │\n",
      "│ ACTUALS ┆ 2022        ┆ 24871 ┆ 6284      ┆ 6138      ┆ 6242      ┆ 6207      ┆ 12.420824 │\n",
      "│ BUDGET  ┆ 2022        ┆ 24877 ┆ 6228      ┆ 6228      ┆ 6248      ┆ 6173      ┆ 12.404573 │\n",
      "│ ACTUALS ┆ 2021        ┆ 25036 ┆ 6305      ┆ 6203      ┆ 6187      ┆ 6341      ┆ 12.506885 │\n",
      "└─────────┴─────────────┴───────┴───────────┴───────────┴───────────┴───────────┴───────────┘\n"
     ]
    }
   ],
   "source": [
    "# conditional aggregation\n",
    "# Let's say we want to know counts by Quarters.\n",
    "# We could directly query that in the aggregation without the need of a lambda \n",
    "# or grooming the DataFrame.\n",
    "out = dfLedger.with_columns(\n",
    "    pl.col(\"LEDGER\").cast(pl.Categorical),\n",
    "    pl.col(\"ORG\").cast(pl.Categorical),\n",
    "    (pl.col(\"POSTED_TOTAL\") / 1000).cast(pl.Float32)\n",
    ")\n",
    "q = (\n",
    "    out.lazy()\n",
    "    .groupby(\"LEDGER\", \"FISCAL_YEAR\")\n",
    "    .agg(\n",
    "        pl.count(),\n",
    "        ((pl.col(\"PERIOD\") >= 1) & (pl.col(\"PERIOD\") <= 3)).sum().cast(pl.Int32).alias(\"Q1 counts\"),\n",
    "        ((pl.col(\"PERIOD\") >= 4) & (pl.col(\"PERIOD\") <= 6)).sum().cast(pl.Int32).alias(\"Q2 counts\"),\n",
    "        ((pl.col(\"PERIOD\") >= 7) & (pl.col(\"PERIOD\") <= 9)).sum().cast(pl.Int32).alias(\"Q3 counts\"),\n",
    "        ((pl.col(\"PERIOD\") >= 10) & (pl.col(\"PERIOD\") <= 12)).sum().cast(pl.Int32).alias(\"Q4 counts\"),\n",
    "        (pl.sum(\"POSTED_TOTAL\") / 1000000).cast(pl.Float32).alias(\"TOTAL (in milions)\")\n",
    "    )\n",
    "    .sort(\"FISCAL_YEAR\", descending=True)\n",
    "    .limit(5)\n",
    ")\n",
    "\n",
    "df = q.collect()\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 8)\n",
      "┌─────────┬─────────────┬───────┬─────────────┬─────────────┬────────────┬────────────┬────────────┐\n",
      "│ LEDGER  ┆ FISCAL_YEAR ┆ count ┆ avg 1-3     ┆ avg 4-6     ┆ avg 7-9    ┆ avg 10-12  ┆ TOTAL (in  │\n",
      "│ ---     ┆ ---         ┆ ---   ┆ Month       ┆ Month       ┆ Month      ┆ Month      ┆ milions)   │\n",
      "│ cat     ┆ i64         ┆ u32   ┆ ---         ┆ ---         ┆ ---        ┆ ---        ┆ ---        │\n",
      "│         ┆             ┆       ┆ f32         ┆ f32         ┆ f32        ┆ f32        ┆ f32        │\n",
      "╞═════════╪═════════════╪═══════╪═════════════╪═════════════╪════════════╪════════════╪════════════╡\n",
      "│ ACTUALS ┆ 2023        ┆ 25136 ┆ 489.695587  ┆ 497.554016  ┆ 504.889496 ┆ 501.516846 ┆ 12.526934  │\n",
      "│ BUDGET  ┆ 2023        ┆ 25260 ┆ 497.987671  ┆ 503.527313  ┆ 496.942291 ┆ 504.191254 ┆ 12.646653  │\n",
      "│ ACTUALS ┆ 2022        ┆ 24871 ┆ 500.281952  ┆ 505.403595  ┆ 494.125092 ┆ 497.914581 ┆ 12.420824  │\n",
      "│ BUDGET  ┆ 2022        ┆ 24877 ┆ 502.2229    ┆ 496.800751  ┆ 494.446503 ┆ 501.108185 ┆ 12.404573  │\n",
      "│ ACTUALS ┆ 2021        ┆ 25036 ┆ 499.123901  ┆ 503.51593   ┆ 497.496216 ┆ 498.117371 ┆ 12.506885  │\n",
      "└─────────┴─────────────┴───────┴─────────────┴─────────────┴────────────┴────────────┴────────────┘\n"
     ]
    }
   ],
   "source": [
    "# filtering\n",
    "# We can also filter the groups. Let's say we want to compute a mean per group,\n",
    "# but we don't want to include all values from that group, \n",
    "# and we also don't want to filter the rows from the DataFrame \n",
    "# (because we need those rows for another aggregation).\n",
    "\n",
    "# In the example below we show how this can be done.\n",
    "def avg_by_qtr(fromMonth: int, toMonth: int) -> pl.Expr:\n",
    "    return (\n",
    "        (pl.col(\"POSTED_TOTAL\")).filter(((pl.col(\"PERIOD\") >= fromMonth) & (pl.col(\"PERIOD\") <= toMonth)))\n",
    "        .mean()\n",
    "        .alias(f\"avg {fromMonth}-{toMonth} Month\")\n",
    "    )\n",
    "\n",
    "q = (\n",
    "    out.lazy()\n",
    "    .group_by(\"LEDGER\", \"FISCAL_YEAR\")\n",
    "    .agg(\n",
    "        pl.count(),\n",
    "        # ((pl.col(\"PERIOD\") >= 1) & (pl.col(\"PERIOD\") <= 3)).sum().cast(pl.Int32).alias(\"Q1 counts\"),\n",
    "        # ((pl.col(\"PERIOD\") >= 4) & (pl.col(\"PERIOD\") <= 6)).sum().cast(pl.Int32).alias(\"Q2 counts\"),\n",
    "        # ((pl.col(\"PERIOD\") >= 7) & (pl.col(\"PERIOD\") <= 9)).sum().cast(pl.Int32).alias(\"Q3 counts\"),\n",
    "        # ((pl.col(\"PERIOD\") >= 10) & (pl.col(\"PERIOD\") <= 12)).sum().cast(pl.Int32).alias(\"Q4 counts\"),\n",
    "        avg_by_qtr(1,3),\n",
    "        avg_by_qtr(4,6),\n",
    "        avg_by_qtr(7,9),\n",
    "        avg_by_qtr(10,12),\n",
    "        (pl.sum(\"POSTED_TOTAL\") / 1000000).cast(pl.Float32).alias(\"TOTAL (in milions)\")\n",
    "    )\n",
    "    .sort(\"FISCAL_YEAR\", descending=True)\n",
    "    .limit(5)\n",
    ")\n",
    "\n",
    "df = q.collect()\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do not kill parallelization\n",
    "\n",
    "# Python Users Only\n",
    "\n",
    "# The following section is specific to Python,\n",
    "# and doesn't apply to Rust. Within Rust, blocks and closures (lambdas) can,\n",
    "# and will, be executed concurrently.\n",
    "\n",
    "# We have all heard that Python is slow, and does \"not scale.\" \n",
    "# Besides the overhead of running \"slow\" bytecode, Python has to remain\n",
    "# within the constraints of the Global Interpreter Lock (GIL).\n",
    "# This means that if you were to use a lambda or a custom Python function to \n",
    "# apply during a parallelized phase, Polars speed is capped running Python code\n",
    "# preventing any multiple threads from executing the function.\n",
    "\n",
    "# This all feels terribly limiting, \n",
    "# especially because we often need those lambda functions in a .groupby() step, \n",
    "# for example. This approach is still supported by Polars,\n",
    "# but keeping in mind bytecode and the GIL costs have to be paid. \n",
    "# It is recommended to try to solve your queries using the expression syntax\n",
    "# before moving to lambdas.\n",
    "# If you want to learn more about using lambdas, go to the user defined functions section.\n",
    "\n",
    "# Conclusion\n",
    "\n",
    "# In the examples above we've seen that we can do a lot by combining expressions.\n",
    "# By doing so we delay the use of custom Python functions \n",
    "# that slow down the queries (by the slow nature of Python AND the GIL).\n",
    "\n",
    "# If we are missing a type expression let us know by opening a feature request!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### using Expression to handle missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 8)\n",
      "┌─────────┬──────────┬─────────────┬────────┬─────────┬──────┬──────────┬──────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
      "│ cat     ┆ cat      ┆ i64         ┆ i64    ┆ i64     ┆ i64  ┆ i64      ┆ f32          │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═════════╪══════╪══════════╪══════════════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 1      ┆ 10000   ┆ 1900 ┆ 14       ┆ 826.906006   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 1      ┆ 17000   ┆ 1400 ┆ 16       ┆ 725.926025   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 1      ┆ 29000   ┆ 1900 ┆ 18       ┆ 127.103996   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 1      ┆ 42000   ┆ 1400 ┆ 17       ┆ 198.975006   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 1      ┆ 39000   ┆ 1100 ┆ 22       ┆ 440.190002   │\n",
      "└─────────┴──────────┴─────────────┴────────┴─────────┴──────┴──────────┴──────────────┘\n",
      "Rows|Columns:  (38, 8)\n",
      "Rows|Columns:  shape: (5, 9)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬──────┬──────────┬──────────────┬─────────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL ┆ POSTED_TOTAL_re │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---  ┆ ---      ┆ ---          ┆ v               │\n",
      "│ cat     ┆ cat      ┆ i64         ┆ i64    ┆   ┆ i64  ┆ i64      ┆ f32          ┆ ---             │\n",
      "│         ┆          ┆             ┆        ┆   ┆      ┆          ┆              ┆ f32             │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪══════╪══════════╪══════════════╪═════════════════╡\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2022        ┆ 12     ┆ … ┆ 1700 ┆ 14       ┆ 869.401001   ┆ null            │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 12     ┆ … ┆ 2400 ┆ 14       ┆ 833.10498    ┆ null            │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 1300 ┆ 14       ┆ 963.809021   ┆ null            │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 12     ┆ … ┆ 1800 ┆ 14       ┆ 416.033997   ┆ null            │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2022        ┆ 12     ┆ … ┆ 1700 ┆ 14       ┆ 917.286987   ┆ null            │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴──────┴──────────┴──────────────┴─────────────────┘\n"
     ]
    }
   ],
   "source": [
    "# let's review two example scenarios\n",
    "\n",
    "# 1st case: one period/month of data is missing from dataset\n",
    "\n",
    "# another useful example, is user want to prepare budgets\n",
    "# based on actuals\n",
    "\n",
    "# 2nd use case: create datasets based on FILL strategy\n",
    "\n",
    "# check out few samples of Ledger Data Frame\n",
    "out = dfLedger.with_columns(\n",
    "    pl.col(\"LEDGER\").cast(pl.Categorical),\n",
    "    pl.col(\"ORG\").cast(pl.Categorical),\n",
    "    (pl.col(\"POSTED_TOTAL\") / 1000).cast(pl.Float32)\n",
    ").sort(\"FISCAL_YEAR\", \"PERIOD\")\n",
    "print(out.head(5))\n",
    "\n",
    "# count of rows for ACCOUNT = 10000, LOCATION = 14 & PERIOD = 12\n",
    "print(\"Rows|Columns: \", out.filter((pl.col(\"ACCOUNT\") == 10000) \n",
    "                 & (pl.col(\"LOCATION\") == 14) \n",
    "                 & (pl.col(\"PERIOD\") == 12)).shape)\n",
    "\n",
    "outRevised = dfLedger.with_columns(\n",
    "    pl.col(\"LEDGER\").cast(pl.Categorical),\n",
    "    pl.col(\"ORG\").cast(pl.Categorical),\n",
    "    (pl.col(\"POSTED_TOTAL\") / 1000).cast(pl.Float32),\n",
    "    pl.when(\n",
    "        (pl.col(\"ACCOUNT\") == 10000) \n",
    "                 & (pl.col(\"LOCATION\") == 14) \n",
    "                 & (pl.col(\"PERIOD\") == 12)\n",
    "        )\n",
    "    .then(None)\n",
    "    .otherwise((pl.col(\"POSTED_TOTAL\") / 1000).cast(pl.Float32))\n",
    "    .alias(\"POSTED_TOTAL_rev\")\n",
    ").sort(\"FISCAL_YEAR\", \"PERIOD\")\n",
    "\n",
    "print(\"Rows|Columns: \", outRevised.filter((pl.col(\"ACCOUNT\") == 10000) \n",
    "                 & (pl.col(\"LOCATION\") == 14) \n",
    "                 & (pl.col(\"PERIOD\") == 12)).sample(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (1, 9)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>LEDGER</th><th>ORG</th><th>FISCAL_YEAR</th><th>PERIOD</th><th>ACCOUNT</th><th>DEPT</th><th>LOCATION</th><th>POSTED_TOTAL</th><th>POSTED_TOTAL_rev</th></tr><tr><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td></tr></thead><tbody><tr><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>38</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (1, 9)\n",
       "┌────────┬─────┬─────────────┬────────┬───┬──────┬──────────┬──────────────┬──────────────────┐\n",
       "│ LEDGER ┆ ORG ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL ┆ POSTED_TOTAL_rev │\n",
       "│ ---    ┆ --- ┆ ---         ┆ ---    ┆   ┆ ---  ┆ ---      ┆ ---          ┆ ---              │\n",
       "│ u32    ┆ u32 ┆ u32         ┆ u32    ┆   ┆ u32  ┆ u32      ┆ u32          ┆ u32              │\n",
       "╞════════╪═════╪═════════════╪════════╪═══╪══════╪══════════╪══════════════╪══════════════════╡\n",
       "│ 0      ┆ 0   ┆ 0           ┆ 0      ┆ … ┆ 0    ┆ 0        ┆ 0            ┆ 38               │\n",
       "└────────┴─────┴─────────────┴────────┴───┴──────┴──────────┴──────────────┴──────────────────┘"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# display counts of Null rows\n",
    "outRevised.null_count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows|Columns:  shape: (5, 9)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬──────┬──────────┬──────────────┬─────────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL ┆ POSTED_TOTAL_re │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---  ┆ ---      ┆ ---          ┆ v               │\n",
      "│ cat     ┆ cat      ┆ i64         ┆ i64    ┆   ┆ i64  ┆ i64      ┆ f32          ┆ ---             │\n",
      "│         ┆          ┆             ┆        ┆   ┆      ┆          ┆              ┆ f32             │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪══════╪══════════╪══════════════╪═════════════════╡\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2023        ┆ 12     ┆ … ┆ 2400 ┆ 14       ┆ 658.724976   ┆ 0.0             │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2022        ┆ 12     ┆ … ┆ 1700 ┆ 14       ┆ 855.356995   ┆ 0.0             │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 1400 ┆ 14       ┆ 629.336975   ┆ 0.0             │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 2000 ┆ 14       ┆ 78.411003    ┆ 0.0             │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 12     ┆ … ┆ 1000 ┆ 14       ┆ 794.187012   ┆ 0.0             │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴──────┴──────────┴──────────────┴─────────────────┘\n"
     ]
    }
   ],
   "source": [
    "# Filling missing data\n",
    "\n",
    "# Fill with specified literal value\n",
    "outRevised_1 = outRevised.with_columns(\n",
    "        pl.col(\"POSTED_TOTAL_rev\").fill_null(\n",
    "            pl.lit(0.0),\n",
    "        ),\n",
    "    )\n",
    "print(\"Rows|Columns: \", outRevised_1.filter((pl.col(\"ACCOUNT\") == 10000) \n",
    "                 & (pl.col(\"LOCATION\") == 14) \n",
    "                 & (pl.col(\"PERIOD\") == 12)).sample(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows|Columns:  shape: (5, 9)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬──────┬──────────┬──────────────┬─────────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL ┆ POSTED_TOTAL_re │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---  ┆ ---      ┆ ---          ┆ v               │\n",
      "│ cat     ┆ cat      ┆ i64         ┆ i64    ┆   ┆ i64  ┆ i64      ┆ f32          ┆ ---             │\n",
      "│         ┆          ┆             ┆        ┆   ┆      ┆          ┆              ┆ f32             │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪══════╪══════════╪══════════════╪═════════════════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 12     ┆ … ┆ 1100 ┆ 14       ┆ 944.950012   ┆ 145.623993      │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2023        ┆ 12     ┆ … ┆ 1300 ┆ 14       ┆ 515.521973   ┆ 53.018002       │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 2200 ┆ 14       ┆ 112.551003   ┆ 829.671997      │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2020        ┆ 12     ┆ … ┆ 1500 ┆ 14       ┆ 142.300995   ┆ 119.074997      │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 2000 ┆ 14       ┆ 78.411003    ┆ 948.60199       │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴──────┴──────────┴──────────────┴─────────────────┘\n"
     ]
    }
   ],
   "source": [
    "# Filling missing data\n",
    "\n",
    "# Fill with a strategy\n",
    "outRevised_2 = outRevised.with_columns(\n",
    "        pl.col(\"POSTED_TOTAL_rev\").fill_null(\n",
    "            strategy=\"forward\"\n",
    "        ),\n",
    "    )\n",
    "print(\"Rows|Columns: \", outRevised_2.filter((pl.col(\"ACCOUNT\") == 10000) \n",
    "                 & (pl.col(\"LOCATION\") == 14) \n",
    "                 & (pl.col(\"PERIOD\") == 12)).sample(5))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows|Columns:  shape: (5, 9)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬──────┬──────────┬──────────────┬─────────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL ┆ POSTED_TOTAL_re │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---  ┆ ---      ┆ ---          ┆ v               │\n",
      "│ cat     ┆ cat      ┆ i64         ┆ i64    ┆   ┆ i64  ┆ i64      ┆ f32          ┆ ---             │\n",
      "│         ┆          ┆             ┆        ┆   ┆      ┆          ┆              ┆ f32             │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪══════╪══════════╪══════════════╪═════════════════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 12     ┆ … ┆ 1900 ┆ 14       ┆ 368.372009   ┆ 501.114014      │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 12     ┆ … ┆ 1100 ┆ 14       ┆ 637.629028   ┆ 501.114014      │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 2200 ┆ 14       ┆ 112.551003   ┆ 501.114014      │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2022        ┆ 12     ┆ … ┆ 1700 ┆ 14       ┆ 917.286987   ┆ 501.114014      │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 1000 ┆ 14       ┆ 79.495003    ┆ 501.114014      │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴──────┴──────────┴──────────────┴─────────────────┘\n"
     ]
    }
   ],
   "source": [
    "# Filling missing data\n",
    "\n",
    "# Fill with an expression\n",
    "outRevised_3 = outRevised.with_columns(\n",
    "        pl.col(\"POSTED_TOTAL_rev\").fill_null(\n",
    "            pl.median(\"POSTED_TOTAL_rev\")\n",
    "        ),\n",
    "    )\n",
    "print(\"Rows|Columns: \", outRevised_3.filter((pl.col(\"ACCOUNT\") == 10000) \n",
    "                 & (pl.col(\"LOCATION\") == 14) \n",
    "                 & (pl.col(\"PERIOD\") == 12)).sample(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows|Columns:  shape: (5, 9)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬──────┬──────────┬──────────────┬─────────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL ┆ POSTED_TOTAL_re │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---  ┆ ---      ┆ ---          ┆ v               │\n",
      "│ cat     ┆ cat      ┆ i64         ┆ i64    ┆   ┆ i64  ┆ i64      ┆ f32          ┆ ---             │\n",
      "│         ┆          ┆             ┆        ┆   ┆      ┆          ┆              ┆ f32             │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪══════╪══════════╪══════════════╪═════════════════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 2000 ┆ 14       ┆ 401.223999   ┆ 53.867001       │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2023        ┆ 12     ┆ … ┆ 1300 ┆ 14       ┆ 515.521973   ┆ 233.958008      │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 12     ┆ … ┆ 1100 ┆ 14       ┆ 886.544983   ┆ 592.49353       │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 2300 ┆ 14       ┆ 977.153015   ┆ 342.100006      │\n",
      "│ BUDGET  ┆ ABC Inc. ┆ 2023        ┆ 12     ┆ … ┆ 1400 ┆ 14       ┆ 809.125      ┆ 468.296997      │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴──────┴──────────┴──────────────┴─────────────────┘\n"
     ]
    }
   ],
   "source": [
    "# Filling missing data\n",
    "\n",
    "# Fill with interpolation\n",
    "outRevised_4 = outRevised.with_columns(\n",
    "        pl.col(\"POSTED_TOTAL_rev\").fill_null(\n",
    "            pl.col(\"POSTED_TOTAL_rev\").interpolate(),\n",
    "        ),\n",
    "    )\n",
    "print(\"Rows|Columns: \", outRevised_4.filter((pl.col(\"ACCOUNT\") == 10000) \n",
    "                 & (pl.col(\"LOCATION\") == 14) \n",
    "                 & (pl.col(\"PERIOD\") == 12)).sample(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### using Expression to apply Folds\n",
    "\n",
    "Polars provides expressions/methods for horizontal aggregations like sum,min, mean, etc. However, when you need a more complex aggregation the default methods Polars supplies may not be sufficient. That's when folds come in handy.\n",
    "\n",
    "The fold expression operates on columns for maximum speed. It utilizes the data layout very efficiently and often has vectorized execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (3, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>sum</th><th>mean</th><th>min</th><th>max</th></tr><tr><td>i64</td><td>f64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>11</td><td>2.0</td><td>1</td><td>3</td></tr><tr><td>22</td><td>2.0</td><td>1</td><td>3</td></tr><tr><td>33</td><td>2.0</td><td>1</td><td>3</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (3, 4)\n",
       "┌─────┬──────┬─────┬─────┐\n",
       "│ sum ┆ mean ┆ min ┆ max │\n",
       "│ --- ┆ ---  ┆ --- ┆ --- │\n",
       "│ i64 ┆ f64  ┆ i64 ┆ i64 │\n",
       "╞═════╪══════╪═════╪═════╡\n",
       "│ 11  ┆ 2.0  ┆ 1   ┆ 3   │\n",
       "│ 22  ┆ 2.0  ┆ 1   ┆ 3   │\n",
       "│ 33  ┆ 2.0  ┆ 1   ┆ 3   │\n",
       "└─────┴──────┴─────┴─────┘"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# basic calculations using Polars expressions\n",
    "df = pl.DataFrame(\n",
    "    {\n",
    "        \"a\": [1, 2, 3],\n",
    "        \"b\": [10, 20, 30],\n",
    "    }\n",
    ")\n",
    "\n",
    "out = df.select(\n",
    "    (pl.col(\"a\") + pl.col(\"b\")).alias(\"sum\"),\n",
    "    pl.col(\"a\").mean().alias(\"mean\"),\n",
    "    pl.col(\"a\").min().alias(\"min\"),\n",
    "    pl.col(\"a\").max().alias(\"max\")\n",
    "    )\n",
    "out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (3, 1)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>sum</th></tr><tr><td>i64</td></tr></thead><tbody><tr><td>11</td></tr><tr><td>22</td></tr><tr><td>33</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (3, 1)\n",
       "┌─────┐\n",
       "│ sum │\n",
       "│ --- │\n",
       "│ i64 │\n",
       "╞═════╡\n",
       "│ 11  │\n",
       "│ 22  │\n",
       "│ 33  │\n",
       "└─────┘"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using Polars Folds to perform complex algerbic aggregations\n",
    "out = df.select(\n",
    "    pl.fold(acc=pl.lit(0), function=lambda acc, x: acc + x, exprs=pl.all()).alias(\n",
    "        \"sum\"\n",
    "    ),\n",
    ")\n",
    "out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 9)\n",
      "┌───────┬─────────┬──────────┬─────────────┬───┬────────┬───────────┬─────────┬──────────┐\n",
      "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ASSETS ┆ LIABILITY ┆ REVENUE ┆ NETWORTH │\n",
      "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---    ┆ ---       ┆ ---     ┆ ---      │\n",
      "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64    ┆ i64       ┆ i64     ┆ i64      │\n",
      "╞═══════╪═════════╪══════════╪═════════════╪═══╪════════╪═══════════╪═════════╪══════════╡\n",
      "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 229177 ┆ 162079    ┆ 746304  ┆ 934536   │\n",
      "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 58165  ┆ 645852    ┆ 795922  ┆ 372016   │\n",
      "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 457983 ┆ 853620    ┆ 540738  ┆ 513516   │\n",
      "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 949507 ┆ 907875    ┆ 28191   ┆ 765534   │\n",
      "│ 4     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 848681 ┆ 419144    ┆ 732089  ┆ 511568   │\n",
      "└───────┴─────────┴──────────┴─────────────┴───┴────────┴───────────┴─────────┴──────────┘\n"
     ]
    }
   ],
   "source": [
    "# Let's use Polars Folds to perform a conditional Algebric operation\n",
    "# below sample DataFrame produces a Balance Sheet,\n",
    "# however, NETWORTH calculated is wrong\n",
    "\n",
    "\n",
    "############################\n",
    "## BalanceSheet DataFrame ##\n",
    "############################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "sampleSize = 100_000\n",
    "org = \"ABC Inc.\"\n",
    "ledger_type = \"ACTUALS\" # BUDGET, STATS are other Ledger types\n",
    "fiscal_year_from = 2020\n",
    "fiscal_year_to = 2023\n",
    "random.seed(123)\n",
    "\n",
    "balanceSheet = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, \n",
    "                                          fiscal_year_to+1, 1)),k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ASSETS\": random.sample(range(1000000), sampleSize),\n",
    "    \"LIABILITY\": random.sample(range(1000000), sampleSize),\n",
    "    \"REVENUE\": random.sample(range(1000000), sampleSize),\n",
    "    \"NETWORTH\": random.sample(range(1000000), sampleSize),\n",
    "})\n",
    "print(balanceSheet.sample(5).with_row_count(\"Row #\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<bound method DataFrame.head of shape: (100_000, 10)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬─────────┬──────────┬────────────┬────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ REVENUE ┆ NETWORTH ┆ NetWorth_c ┆ CostCenter │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---     ┆ ---      ┆ ---        ┆ ---        │\n",
      "│ str     ┆ str      ┆ i64         ┆ i64    ┆   ┆ i64     ┆ i64      ┆ i64        ┆ str        │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪═════════╪══════════╪════════════╪════════════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 10     ┆ … ┆ 825234  ┆ 630949   ┆ 358184     ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2020   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 1      ┆ … ┆ 401531  ┆ 307809   ┆ -289400    ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2020   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 777927  ┆ 536595   ┆ 597339     ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2021   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 3      ┆ … ┆ 445913  ┆ 400960   ┆ 398951     ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2020   │\n",
      "│ …       ┆ …        ┆ …           ┆ …      ┆ … ┆ …       ┆ …        ┆ …          ┆ …          │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 6      ┆ … ┆ 812724  ┆ 541937   ┆ 851037     ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2022   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ 8      ┆ … ┆ 188039  ┆ 876970   ┆ 301553     ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2023   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ 8      ┆ … ┆ 610809  ┆ 715951   ┆ -100056    ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2023   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 2      ┆ … ┆ 783855  ┆ 226190   ┆ 667217     ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2020   │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴─────────┴──────────┴────────────┴────────────┘>\n"
     ]
    }
   ],
   "source": [
    "# let's use Polars Folds to\n",
    "# A. calculate NetWorth_c = ASSETS + REVENUE - LIABILITY\n",
    "# B. generate a unique COSTCenter String = LEDGER+ORG+FISCAL_YEAR\n",
    "\n",
    "out = balanceSheet.with_columns(\n",
    "    pl.fold(acc=pl.lit(0),\n",
    "            function=lambda acc, x: acc + x, \n",
    "            exprs=(pl.col(\"ASSETS\",\"REVENUE\"), pl.col(\"LIABILITY\")*-1))\n",
    "            .alias(\"NetWorth_c\"),\n",
    "    pl.fold(acc=pl.lit(\"\"),\n",
    "            function=lambda acc, x: acc + x, \n",
    "            exprs=(pl.col(\"LEDGER\",\"ORG\",\"FISCAL_YEAR\"))\n",
    "            .alias(\"CostCenter\"),\n",
    "    )\n",
    ")\n",
    "print(out.head)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### using Expression to apply List & Arrays\n",
    "\n",
    "---\n",
    "Polars has first class support for List columns. Please note that, Polars List Column is different than Python list data type.\n",
    "\n",
    "For example, Python list data type can store data of any type, however, Polars List column contains each row as same data type.\n",
    "\n",
    "Polars can still keep different type in List column, but stores it as `Object` data type instead of list, hence List data type manipulations is not available.\n",
    "\n",
    "Polars also support Array data type which is analogous to numpy ndarray.\n",
    "\n",
    "Let's see these in action, so that it become more clear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ID</th><th>AS_OF_DATE</th><th>DESCRIPTION</th><th>ADDRESS</th><th>PHONE1</th><th>PHONE2</th><th>PHONE3</th><th>EMAIL</th><th>STATUS</th><th>TYPE</th><th>CATEGORY</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>str</td><td>list[str]</td><td>list[i64]</td><td>str</td><td>list[str]</td><td>str</td><td>str</td><td>str</td><td>i64</td></tr></thead><tbody><tr><td>0</td><td>100</td><td>2022-01-01 00:00:00</td><td>&quot;Customer 1&quot;</td><td>[&quot;Address 1a, Address 1b&quot;]</td><td>[100100, 100200]</td><td>&quot;100100 | 10020…</td><td>[&quot;100100 | 100200 | 100300 | 100400 | 100500&quot;]</td><td>&quot;1a@email 1b@em…</td><td>&quot;Active&quot;</td><td>&quot;Corp&quot;</td><td>1200</td></tr><tr><td>1</td><td>200</td><td>2022-01-01 00:00:00</td><td>&quot;Customer 1&quot;</td><td>[&quot;Address 1a, Address 1b&quot;]</td><td>[100100, 100200]</td><td>&quot;100100 | 10020…</td><td>[&quot;100100 | 100200 | 100300 | 100400 | 100500&quot;]</td><td>&quot;1a@email 1b@em…</td><td>&quot;Active&quot;</td><td>&quot;Gov&quot;</td><td>2000</td></tr><tr><td>2</td><td>210</td><td>2022-01-01 00:00:00</td><td>&quot;Customer 2&quot;</td><td>[&quot;Address 2a, Address 2b&quot;]</td><td>[200100, 200200]</td><td>&quot;200100 | 20020…</td><td>[&quot;200100 | 200200 | 200300 | 200400 | 200500&quot;]</td><td>&quot;2a@email 2b@em…</td><td>&quot;Active&quot;</td><td>&quot;Individual&quot;</td><td>1800</td></tr><tr><td>3</td><td>180</td><td>2022-01-01 00:00:00</td><td>&quot;Customer 4&quot;</td><td>[&quot;Address 4a, Address 4b&quot;]</td><td>[400100, 400200]</td><td>&quot;400100 | 40020…</td><td>[&quot;400100 | 300200 | 400300 | 400400 | 400500&quot;]</td><td>&quot;4a@email 4b@em…</td><td>&quot;Active&quot;</td><td>&quot;Individual&quot;</td><td>1200</td></tr><tr><td>4</td><td>140</td><td>2022-01-01 00:00:00</td><td>&quot;Customer 5&quot;</td><td>[&quot;Address 5a, Address 5b&quot;]</td><td>[500100, 500200]</td><td>&quot;500100 | 50020…</td><td>[&quot;500100 | 500200 | 500300 | 500400 | 500500&quot;]</td><td>&quot;5a@email 5b@em…</td><td>&quot;Active&quot;</td><td>&quot;Gov&quot;</td><td>1900</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 12)\n",
       "┌───────┬─────┬────────────────┬─────────────┬───┬────────────────┬────────┬────────────┬──────────┐\n",
       "│ Row # ┆ ID  ┆ AS_OF_DATE     ┆ DESCRIPTION ┆ … ┆ EMAIL          ┆ STATUS ┆ TYPE       ┆ CATEGORY │\n",
       "│ ---   ┆ --- ┆ ---            ┆ ---         ┆   ┆ ---            ┆ ---    ┆ ---        ┆ ---      │\n",
       "│ u32   ┆ i64 ┆ datetime[μs]   ┆ str         ┆   ┆ str            ┆ str    ┆ str        ┆ i64      │\n",
       "╞═══════╪═════╪════════════════╪═════════════╪═══╪════════════════╪════════╪════════════╪══════════╡\n",
       "│ 0     ┆ 100 ┆ 2022-01-01     ┆ Customer 1  ┆ … ┆ 1a@email       ┆ Active ┆ Corp       ┆ 1200     │\n",
       "│       ┆     ┆ 00:00:00       ┆             ┆   ┆ 1b@email       ┆        ┆            ┆          │\n",
       "│ 1     ┆ 200 ┆ 2022-01-01     ┆ Customer 1  ┆ … ┆ 1a@email       ┆ Active ┆ Gov        ┆ 2000     │\n",
       "│       ┆     ┆ 00:00:00       ┆             ┆   ┆ 1b@email       ┆        ┆            ┆          │\n",
       "│ 2     ┆ 210 ┆ 2022-01-01     ┆ Customer 2  ┆ … ┆ 2a@email       ┆ Active ┆ Individual ┆ 1800     │\n",
       "│       ┆     ┆ 00:00:00       ┆             ┆   ┆ 2b@email       ┆        ┆            ┆          │\n",
       "│ 3     ┆ 180 ┆ 2022-01-01     ┆ Customer 4  ┆ … ┆ 4a@email       ┆ Active ┆ Individual ┆ 1200     │\n",
       "│       ┆     ┆ 00:00:00       ┆             ┆   ┆ 4b@email       ┆        ┆            ┆          │\n",
       "│ 4     ┆ 140 ┆ 2022-01-01     ┆ Customer 5  ┆ … ┆ 5a@email       ┆ Active ┆ Gov        ┆ 1900     │\n",
       "│       ┆     ┆ 00:00:00       ┆             ┆   ┆ 5b@email donot ┆        ┆            ┆          │\n",
       "│       ┆     ┆                ┆             ┆   ┆ reply@ema…     ┆        ┆            ┆          │\n",
       "└───────┴─────┴────────────────┴─────────────┴───┴────────────────┴────────┴────────────┴──────────┘"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "########################\n",
    "## CUSTOMER DataFrame ##\n",
    "########################\n",
    "\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "customer = pl.DataFrame({\n",
    "    \"ID\":  list(range(100, 250, 10)),\n",
    "    \"AS_OF_DATE\" : datetime(2022, 1, 1),\n",
    "    \"DESCRIPTION\" : [\"Customer 1\",\"Customer 2\",\"Customer 3\",\"Customer 4\",\"Customer 5\"] * 3,\n",
    "    \"ADDRESS\" : [[\"Address 1a, Address 1b\"],[\"Address 2a, Address 2b\"],\n",
    "                 [\"Address 3a, Address 3b\"],[\"Address 4a, Address 4b\"],\n",
    "                 [\"Address 5a, Address 5b\"]] * 3,\n",
    "    \"PHONE1\" : [[100100,100200],[200100,200200],[300100,300200],\n",
    "                [400100,400200],[500100,500200]] * 3,\n",
    "    \"PHONE2\" : [\"100100 | 100200\",\"200100 | 200200\",\"300100 | 300200\",\n",
    "                \"400100 | 400200\",\"500100 | 500200\"] * 3,\n",
    "    \"PHONE3\" : [[\"100100 | 100200 | 100300 | 100400 | 100500\"],\n",
    "                [\"200100 | 200200 | 200300 | 200400 | 200500\"],\n",
    "                [\"300100 | 300200 | 300300 | 300400 | 300500\"],\n",
    "                [\"400100 | 300200 | 400300 | 400400 | 400500\"],\n",
    "                [\"500100 | 500200 | 500300 | 500400 | 500500\"],\n",
    "                ] \n",
    "                * 3,\n",
    "    \"EMAIL\" : [\"1a@email 1b@email\",\"2a@email 2b@email\",\"3a@email 3b@email\",\n",
    "               \"4a@email 4b@email\",\"5a@email 5b@email donotreply@email\"] * 3,\n",
    "    \"STATUS\" : \"Active\",\n",
    "    \"TYPE\" : [\"Corp\",\"Gov\",\"Individual\"] * 5,\n",
    "    \"CATEGORY\" : random.choices(category[\"ID\"], k=15),\n",
    "})\n",
    "customer.sample(5).with_row_count(\"Row #\")\n",
    "# Address, PHONE1, PHONE3, EMAIL are lists of [str, int, str, str]\n",
    "# PHONE2, EMAIL contains a list, but is saved as str"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (15, 8)\n",
      "┌─────┬─────────────┬─────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐\n",
      "│ ID  ┆ DESCRIPTION ┆ PHONE1      ┆ Phone1_Int ┆ PHONE2     ┆ PHONE2_new ┆ PHONE3     ┆ new        │\n",
      "│ --- ┆ ---         ┆ ---         ┆ _to-Str    ┆ ---        ┆ ---        ┆ ---        ┆ ---        │\n",
      "│ i64 ┆ str         ┆ list[i64]   ┆ ---        ┆ str        ┆ list[str]  ┆ list[str]  ┆ list[str]  │\n",
      "│     ┆             ┆             ┆ list[str]  ┆            ┆            ┆            ┆            │\n",
      "╞═════╪═════════════╪═════════════╪════════════╪════════════╪════════════╪════════════╪════════════╡\n",
      "│ 100 ┆ Customer 1  ┆ [100100,    ┆ [\"100100\", ┆ 100100 |   ┆ [\"100100\", ┆ [\"100100 | ┆ [\"100100\", │\n",
      "│     ┆             ┆ 100200]     ┆ \"100200\"]  ┆ 100200     ┆ \"100200\"]  ┆ 100200 |   ┆ \"100200\",  │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 100300 |   ┆ …          │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 100…       ┆ \"100500\"]  │\n",
      "│ 110 ┆ Customer 2  ┆ [200100,    ┆ [\"200100\", ┆ 200100 |   ┆ [\"200100\", ┆ [\"200100 | ┆ [\"200100\", │\n",
      "│     ┆             ┆ 200200]     ┆ \"200200\"]  ┆ 200200     ┆ \"200200\"]  ┆ 200200 |   ┆ \"200200\",  │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 200300 |   ┆ …          │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 200…       ┆ \"200500\"]  │\n",
      "│ 120 ┆ Customer 3  ┆ [300100,    ┆ [\"300100\", ┆ 300100 |   ┆ [\"300100\", ┆ [\"300100 | ┆ [\"300100\", │\n",
      "│     ┆             ┆ 300200]     ┆ \"300200\"]  ┆ 300200     ┆ \"300200\"]  ┆ 300200 |   ┆ \"300200\",  │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 300300 |   ┆ …          │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 300…       ┆ \"300500\"]  │\n",
      "│ 130 ┆ Customer 4  ┆ [400100,    ┆ [\"400100\", ┆ 400100 |   ┆ [\"400100\", ┆ [\"400100 | ┆ [\"400100\", │\n",
      "│     ┆             ┆ 400200]     ┆ \"400200\"]  ┆ 400200     ┆ \"400200\"]  ┆ 300200 |   ┆ \"300200\",  │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 400300 |   ┆ …          │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 400…       ┆ \"400500\"]  │\n",
      "│ …   ┆ …           ┆ …           ┆ …          ┆ …          ┆ …          ┆ …          ┆ …          │\n",
      "│ 210 ┆ Customer 2  ┆ [200100,    ┆ [\"200100\", ┆ 200100 |   ┆ [\"200100\", ┆ [\"200100 | ┆ [\"200100\", │\n",
      "│     ┆             ┆ 200200]     ┆ \"200200\"]  ┆ 200200     ┆ \"200200\"]  ┆ 200200 |   ┆ \"200200\",  │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 200300 |   ┆ …          │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 200…       ┆ \"200500\"]  │\n",
      "│ 220 ┆ Customer 3  ┆ [300100,    ┆ [\"300100\", ┆ 300100 |   ┆ [\"300100\", ┆ [\"300100 | ┆ [\"300100\", │\n",
      "│     ┆             ┆ 300200]     ┆ \"300200\"]  ┆ 300200     ┆ \"300200\"]  ┆ 300200 |   ┆ \"300200\",  │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 300300 |   ┆ …          │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 300…       ┆ \"300500\"]  │\n",
      "│ 230 ┆ Customer 4  ┆ [400100,    ┆ [\"400100\", ┆ 400100 |   ┆ [\"400100\", ┆ [\"400100 | ┆ [\"400100\", │\n",
      "│     ┆             ┆ 400200]     ┆ \"400200\"]  ┆ 400200     ┆ \"400200\"]  ┆ 300200 |   ┆ \"300200\",  │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 400300 |   ┆ …          │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 400…       ┆ \"400500\"]  │\n",
      "│ 240 ┆ Customer 5  ┆ [500100,    ┆ [\"500100\", ┆ 500100 |   ┆ [\"500100\", ┆ [\"500100 | ┆ [\"500100\", │\n",
      "│     ┆             ┆ 500200]     ┆ \"500200\"]  ┆ 500200     ┆ \"500200\"]  ┆ 500200 |   ┆ \"500200\",  │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 500300 |   ┆ …          │\n",
      "│     ┆             ┆             ┆            ┆            ┆            ┆ 500…       ┆ \"500500\"]  │\n",
      "└─────┴─────────────┴─────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘\n"
     ]
    }
   ],
   "source": [
    "# Polars provide powerful List manipulations methods\n",
    "\n",
    "# creating a list\n",
    "\n",
    "out = customer.select(\n",
    "    pl.col(\"ID\"),\n",
    "    pl.col(\"DESCRIPTION\"),\n",
    "    pl.col(\"PHONE1\"),\n",
    "    pl.col(\"PHONE1\").cast(pl.List(pl.Utf8)).alias(\"Phone1_Int_to-Str\"),\n",
    "    pl.col(\"PHONE2\"),\n",
    "    pl.col(\"PHONE2\").str.split(\" | \").alias(\"PHONE2_new\"),\n",
    "    pl.col(\"PHONE3\"),\n",
    "    pl.col(\"PHONE3\").explode().str.split(\" | \").alias(\"new\"),\n",
    "    )\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (30, 3)\n",
      "┌─────┬─────────────┬───────────────────┐\n",
      "│ ID  ┆ DESCRIPTION ┆ Phone1_Int_to-Str │\n",
      "│ --- ┆ ---         ┆ ---               │\n",
      "│ i64 ┆ str         ┆ str               │\n",
      "╞═════╪═════════════╪═══════════════════╡\n",
      "│ 100 ┆ Customer 1  ┆ 100100            │\n",
      "│ 100 ┆ Customer 1  ┆ 100200            │\n",
      "│ 110 ┆ Customer 2  ┆ 200100            │\n",
      "│ 110 ┆ Customer 2  ┆ 200200            │\n",
      "│ …   ┆ …           ┆ …                 │\n",
      "│ 230 ┆ Customer 4  ┆ 400100            │\n",
      "│ 230 ┆ Customer 4  ┆ 400200            │\n",
      "│ 240 ┆ Customer 5  ┆ 500100            │\n",
      "│ 240 ┆ Customer 5  ┆ 500200            │\n",
      "└─────┴─────────────┴───────────────────┘\n"
     ]
    }
   ],
   "source": [
    "# once you have a list to work with\n",
    "# use explode() to break list content by rows\n",
    "# say, list all phone #s by customers from PHONE1 column\n",
    "\n",
    "out = customer.select(\n",
    "    pl.col(\"ID\"),\n",
    "    pl.col(\"DESCRIPTION\"),\n",
    "    pl.col(\"PHONE1\").cast(pl.List(pl.Utf8)).alias(\"Phone1_Int_to-Str\"),\n",
    "    ).explode(\"Phone1_Int_to-Str\")\n",
    "print(out)\n",
    "\n",
    "# out = customer.select(\n",
    "#     pl.col(\"ID\"),\n",
    "#     pl.col(\"DESCRIPTION\"),\n",
    "#     pl.col(\"PHONE3\"),\n",
    "#     pl.col(\"PHONE3\").explode().str.split(\" | \").alias(\"new\"),\n",
    "#     ).explode(\"new\")\n",
    "# print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (15, 6)\n",
      "┌─────┬─────────────┬─────────────┬────────────────────────┬────────────────────────┬──────────────┐\n",
      "│ ID  ┆ DESCRIPTION ┆ new         ┆ top3                   ┆ bottom_3               ┆ observations │\n",
      "│ --- ┆ ---         ┆ ---         ┆ ---                    ┆ ---                    ┆ ---          │\n",
      "│ i64 ┆ str         ┆ list[str]   ┆ list[str]              ┆ list[str]              ┆ u32          │\n",
      "╞═════╪═════════════╪═════════════╪════════════════════════╪════════════════════════╪══════════════╡\n",
      "│ 100 ┆ Customer 1  ┆ [\"100100\",  ┆ [\"100100\", \"100200\",   ┆ [\"100300\", \"100400\",   ┆ 5            │\n",
      "│     ┆             ┆ \"100200\", … ┆ \"100300\"]              ┆ \"100500\"]              ┆              │\n",
      "│     ┆             ┆ \"100500\"]   ┆                        ┆                        ┆              │\n",
      "│ 110 ┆ Customer 2  ┆ [\"200100\",  ┆ [\"200100\", \"200200\",   ┆ [\"200300\", \"200400\",   ┆ 5            │\n",
      "│     ┆             ┆ \"200200\", … ┆ \"200300\"]              ┆ \"200500\"]              ┆              │\n",
      "│     ┆             ┆ \"200500\"]   ┆                        ┆                        ┆              │\n",
      "│ 120 ┆ Customer 3  ┆ [\"300100\",  ┆ [\"300100\", \"300200\",   ┆ [\"300300\", \"300400\",   ┆ 5            │\n",
      "│     ┆             ┆ \"300200\", … ┆ \"300300\"]              ┆ \"300500\"]              ┆              │\n",
      "│     ┆             ┆ \"300500\"]   ┆                        ┆                        ┆              │\n",
      "│ 130 ┆ Customer 4  ┆ [\"400100\",  ┆ [\"400100\", \"300200\",   ┆ [\"400300\", \"400400\",   ┆ 5            │\n",
      "│     ┆             ┆ \"300200\", … ┆ \"400300\"]              ┆ \"400500\"]              ┆              │\n",
      "│     ┆             ┆ \"400500\"]   ┆                        ┆                        ┆              │\n",
      "│ …   ┆ …           ┆ …           ┆ …                      ┆ …                      ┆ …            │\n",
      "│ 210 ┆ Customer 2  ┆ [\"200100\",  ┆ [\"200100\", \"200200\",   ┆ [\"200300\", \"200400\",   ┆ 5            │\n",
      "│     ┆             ┆ \"200200\", … ┆ \"200300\"]              ┆ \"200500\"]              ┆              │\n",
      "│     ┆             ┆ \"200500\"]   ┆                        ┆                        ┆              │\n",
      "│ 220 ┆ Customer 3  ┆ [\"300100\",  ┆ [\"300100\", \"300200\",   ┆ [\"300300\", \"300400\",   ┆ 5            │\n",
      "│     ┆             ┆ \"300200\", … ┆ \"300300\"]              ┆ \"300500\"]              ┆              │\n",
      "│     ┆             ┆ \"300500\"]   ┆                        ┆                        ┆              │\n",
      "│ 230 ┆ Customer 4  ┆ [\"400100\",  ┆ [\"400100\", \"300200\",   ┆ [\"400300\", \"400400\",   ┆ 5            │\n",
      "│     ┆             ┆ \"300200\", … ┆ \"400300\"]              ┆ \"400500\"]              ┆              │\n",
      "│     ┆             ┆ \"400500\"]   ┆                        ┆                        ┆              │\n",
      "│ 240 ┆ Customer 5  ┆ [\"500100\",  ┆ [\"500100\", \"500200\",   ┆ [\"500300\", \"500400\",   ┆ 5            │\n",
      "│     ┆             ┆ \"500200\", … ┆ \"500300\"]              ┆ \"500500\"]              ┆              │\n",
      "│     ┆             ┆ \"500500\"]   ┆                        ┆                        ┆              │\n",
      "└─────┴─────────────┴─────────────┴────────────────────────┴────────────────────────┴──────────────┘\n"
     ]
    }
   ],
   "source": [
    "# operating on list columns\n",
    "# using list methods inside columns to view slice of data\n",
    "# for example , PHONE3 column has 5 values in list\n",
    "# use head(), tail(), slice(), length()\n",
    "\n",
    "out = customer.select(\n",
    "    pl.col(\"ID\"),\n",
    "    pl.col(\"DESCRIPTION\"),\n",
    "    # pl.col(\"PHONE3\"),\n",
    "    pl.col(\"PHONE3\").explode().str.split(\" | \").alias(\"new\"),\n",
    "    ).with_columns(\n",
    "    pl.col(\"new\").list.head(3).alias(\"top3\"),\n",
    "    pl.col(\"new\").list.slice(-3, 3).alias(\"bottom_3\"),\n",
    "    pl.col(\"new\").list.lengths().alias(\"observations\"),\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (15, 3)\n",
      "┌─────┬─────────────┬───────────────────────────────────┐\n",
      "│ ID  ┆ DESCRIPTION ┆ EMAIL                             │\n",
      "│ --- ┆ ---         ┆ ---                               │\n",
      "│ i64 ┆ str         ┆ str                               │\n",
      "╞═════╪═════════════╪═══════════════════════════════════╡\n",
      "│ 100 ┆ Customer 1  ┆ 1a@email 1b@email                 │\n",
      "│ 110 ┆ Customer 2  ┆ 2a@email 2b@email                 │\n",
      "│ 120 ┆ Customer 3  ┆ 3a@email 3b@email                 │\n",
      "│ 130 ┆ Customer 4  ┆ 4a@email 4b@email                 │\n",
      "│ …   ┆ …           ┆ …                                 │\n",
      "│ 210 ┆ Customer 2  ┆ 2a@email 2b@email                 │\n",
      "│ 220 ┆ Customer 3  ┆ 3a@email 3b@email                 │\n",
      "│ 230 ┆ Customer 4  ┆ 4a@email 4b@email                 │\n",
      "│ 240 ┆ Customer 5  ┆ 5a@email 5b@email donotreply@ema… │\n",
      "└─────┴─────────────┴───────────────────────────────────┘\n",
      "shape: (15, 4)\n",
      "┌─────┬─────────────┬───────────────────────────────────┬───────────────┐\n",
      "│ ID  ┆ DESCRIPTION ┆ EMAIL                             ┆ InvalidEMAILs │\n",
      "│ --- ┆ ---         ┆ ---                               ┆ ---           │\n",
      "│ i64 ┆ str         ┆ str                               ┆ u32           │\n",
      "╞═════╪═════════════╪═══════════════════════════════════╪═══════════════╡\n",
      "│ 100 ┆ Customer 1  ┆ 1a@email 1b@email                 ┆ 0             │\n",
      "│ 110 ┆ Customer 2  ┆ 2a@email 2b@email                 ┆ 0             │\n",
      "│ 120 ┆ Customer 3  ┆ 3a@email 3b@email                 ┆ 0             │\n",
      "│ 130 ┆ Customer 4  ┆ 4a@email 4b@email                 ┆ 0             │\n",
      "│ …   ┆ …           ┆ …                                 ┆ …             │\n",
      "│ 210 ┆ Customer 2  ┆ 2a@email 2b@email                 ┆ 0             │\n",
      "│ 220 ┆ Customer 3  ┆ 3a@email 3b@email                 ┆ 0             │\n",
      "│ 230 ┆ Customer 4  ┆ 4a@email 4b@email                 ┆ 0             │\n",
      "│ 240 ┆ Customer 5  ┆ 5a@email 5b@email donotreply@ema… ┆ 1             │\n",
      "└─────┴─────────────┴───────────────────────────────────┴───────────────┘\n"
     ]
    }
   ],
   "source": [
    "# working with List elements\n",
    "# let;s say, we want to list customers with\n",
    "# number of invalid emailIDs\n",
    "\n",
    "out = customer.select(\n",
    "    pl.col(\"ID\"),\n",
    "    pl.col(\"DESCRIPTION\"),\n",
    "    pl.col(\"EMAIL\"),\n",
    ")\n",
    "print(out)\n",
    "# for example, Customer 5 has one email ID = donotreply\n",
    "\n",
    "out = out.with_columns(\n",
    "    pl.col(\"EMAIL\").str.split(\" \").list.eval(pl.element().str.contains(\"donotreply\"))\n",
    "    .list.sum().alias(\"InvalidEMAILs\")\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "# row wise computation #\n",
    "\n",
    "# If you ask me, this is the most important feature of Polars\n",
    "# I highly recommend practicing row wise computation using Polars\n",
    "# as this is very beneficial and used extensively in data preparation \n",
    "# for Machine Learning models training\n",
    "\n",
    "##########################\n",
    "# FROM POLARS user guide #\n",
    "# We can apply any Polars operations on the elements of the list\n",
    "# with the list.eval (list().eval in Rust) expression! \n",
    "# These expressions run entirely on Polars' query engine and can run in parallel,\n",
    "# so will be well optimized.\n",
    "# Let's say we have another set of weather data across three days, for different stations:\n",
    "##########################\n",
    "\n",
    "# for example calculate RANK in below list of numbers\n",
    "# Number    RANK\n",
    "# 7         5\n",
    "# 3.5       3   # when duplicate, next RANK will skip 1 by each duplicate\n",
    "# 3.5       3\n",
    "# 1         1\n",
    "# 2         2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (4, 2)\n",
      "┌─────┬───────────────────┐\n",
      "│ ID  ┆ NUMBER            │\n",
      "│ --- ┆ ---               │\n",
      "│ i64 ┆ list[f64]         │\n",
      "╞═════╪═══════════════════╡\n",
      "│ 1   ┆ [7.0, 3.5, … 2.0] │\n",
      "│ 2   ┆ [6.0, 2.5, … 4.0] │\n",
      "│ 3   ┆ [3.0, 3.5, … 5.0] │\n",
      "│ 4   ┆ [2.0, 3.5, … 3.0] │\n",
      "└─────┴───────────────────┘\n",
      "shape: (4, 3)\n",
      "┌─────┬───────────────────┬───────────────────┐\n",
      "│ ID  ┆ NUMBER            ┆ NUMBERS_RANK      │\n",
      "│ --- ┆ ---               ┆ ---               │\n",
      "│ i64 ┆ list[f64]         ┆ list[f64]         │\n",
      "╞═════╪═══════════════════╪═══════════════════╡\n",
      "│ 1   ┆ [7.0, 3.5, … 2.0] ┆ [0.2, 0.5, … 0.8] │\n",
      "│ 2   ┆ [6.0, 2.5, … 4.0] ┆ [0.2, 0.8, … 0.4] │\n",
      "│ 3   ┆ [3.0, 3.5, … 5.0] ┆ [1.0, 0.7, … 0.3] │\n",
      "│ 4   ┆ [2.0, 3.5, … 3.0] ┆ [1.0, 0.5, … 0.8] │\n",
      "└─────┴───────────────────┴───────────────────┘\n"
     ]
    }
   ],
   "source": [
    "import polars as pl\n",
    "sampleDF = pl.DataFrame(\n",
    "    {\n",
    "        \"ID\": list(range(1,5)),\n",
    "        \"NUMBER\": [[7, 3.5, 3.5, 1, 2],\n",
    "                   [6, 2.5, 3.5, 1, 4],\n",
    "                   [3, 3.5, 3.5, 5, 5],\n",
    "                   [2, 3.5, 3.5, 4, 3]]\n",
    "    }\n",
    ")\n",
    "print(sampleDF)\n",
    "\n",
    "rank_pct = (pl.element().rank(descending=True) / pl.col(\"*\").count()).round(2)\n",
    "\n",
    "out = sampleDF.with_columns(\n",
    "    # create the list of homogeneous data\n",
    "    pl.concat_list(pl.all().exclude(\"ID\")).alias(\"all_numbers\")\n",
    ").select(\n",
    "    # select all columns except the intermediate list\n",
    "    pl.all().exclude(\"all_numbers\"),\n",
    "    # compute the rank by calling `list.eval`\n",
    "    pl.col(\"all_numbers\").list.eval(rank_pct, parallel=True).alias(\"NUMBERS_RANK\"),\n",
    ")\n",
    "\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 9)\n",
      "┌───────┬─────────┬──────────┬─────────────┬───┬────────┬───────────┬─────────┬──────────┐\n",
      "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ASSETS ┆ LIABILITY ┆ REVENUE ┆ NETWORTH │\n",
      "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---    ┆ ---       ┆ ---     ┆ ---      │\n",
      "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64    ┆ i64       ┆ i64     ┆ i64      │\n",
      "╞═══════╪═════════╪══════════╪═════════════╪═══╪════════╪═══════════╪═════════╪══════════╡\n",
      "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 229177 ┆ 162079    ┆ 746304  ┆ 934536   │\n",
      "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 58165  ┆ 645852    ┆ 795922  ┆ 372016   │\n",
      "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 457983 ┆ 853620    ┆ 540738  ┆ 513516   │\n",
      "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 949507 ┆ 907875    ┆ 28191   ┆ 765534   │\n",
      "│ 4     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 848681 ┆ 419144    ┆ 732089  ┆ 511568   │\n",
      "└───────┴─────────┴──────────┴─────────────┴───┴────────┴───────────┴─────────┴──────────┘\n",
      "shape: (5, 9)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬───────────┬─────────┬──────────┬─────────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ LIABILITY ┆ REVENUE ┆ NETWORTH ┆ cf_RANK         │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---       ┆ ---     ┆ ---      ┆ ---             │\n",
      "│ str     ┆ str      ┆ i64         ┆ i64    ┆   ┆ i64       ┆ i64     ┆ i64      ┆ list[f64]       │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪═══════════╪═════════╪══════════╪═════════════════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ 10     ┆ … ┆ 118334    ┆ 110193  ┆ 182505   ┆ [0.25, 0.75, …  │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 0.5]            │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 4      ┆ … ┆ 100528    ┆ 316989  ┆ 431846   ┆ [0.25, 1.0, …   │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 0.5]            │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 251313    ┆ 8225    ┆ 386441   ┆ [0.25, 0.75, …  │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 0.5]            │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ 2      ┆ … ┆ 436892    ┆ 38548   ┆ 684450   ┆ [0.75, 0.5, …   │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 0.25]           │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 5      ┆ … ┆ 671804    ┆ 643440  ┆ 473534   ┆ [1.0, 0.25, …   │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 0.75]           │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴───────────┴─────────┴──────────┴─────────────────┘\n"
     ]
    }
   ],
   "source": [
    "# applying RANK | eval functions on BALANCE SHEET data\n",
    "# for example\n",
    "\n",
    "############################\n",
    "## BalanceSheet DataFrame ##\n",
    "############################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "sampleSize = 100_000\n",
    "org = \"ABC Inc.\"\n",
    "ledger_type = \"ACTUALS\" # BUDGET, STATS are other Ledger types\n",
    "fiscal_year_from = 2020\n",
    "fiscal_year_to = 2023\n",
    "random.seed(123)\n",
    "\n",
    "balanceSheet = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, \n",
    "                                          fiscal_year_to+1, 1)),k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ASSETS\": random.sample(range(1000000), sampleSize),\n",
    "    \"LIABILITY\": random.sample(range(1000000), sampleSize),\n",
    "    \"REVENUE\": random.sample(range(1000000), sampleSize),\n",
    "    \"NETWORTH\": random.sample(range(1000000), sampleSize),\n",
    "})\n",
    "print(balanceSheet.sample(5).with_row_count(\"Row #\"))\n",
    "\n",
    "rank_pct = (pl.element().rank(descending=True) / pl.col(\"*\").count()).round(2)\n",
    "\n",
    "out = balanceSheet.with_columns(\n",
    "    # create the list of homogeneous data\n",
    "    pl.concat_list(pl.all().exclude(\"LEDGER\",\"ORG\",\"FISCAL_YEAR\",\"PERIOD\")).alias(\"all_CASHFLOW\")\n",
    ").select(\n",
    "    # select all columns except the intermediate list\n",
    "    pl.all().exclude(\"all_CASHFLOW\"),\n",
    "    # compute the rank by calling `list.eval`\n",
    "    pl.col(\"all_CASHFLOW\").list.eval(rank_pct, parallel=True).alias(\"cf_RANK\"),\n",
    ")\n",
    "\n",
    "print(out.sample(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (2, 2)\n",
      "┌───────────────┬───────────────┐\n",
      "│ Array_1       ┆ Array_2       │\n",
      "│ ---           ┆ ---           │\n",
      "│ array[i64, 2] ┆ array[i64, 3] │\n",
      "╞═══════════════╪═══════════════╡\n",
      "│ [1, 3]        ┆ [1, 7, 3]     │\n",
      "│ [2, 5]        ┆ [8, 1, 0]     │\n",
      "└───────────────┴───────────────┘\n",
      "shape: (2, 2)\n",
      "┌─────────────┬─────────────┐\n",
      "│ Array_1_min ┆ Array_2_sum │\n",
      "│ ---         ┆ ---         │\n",
      "│ i64         ┆ i64         │\n",
      "╞═════════════╪═════════════╡\n",
      "│ 1           ┆ 11          │\n",
      "│ 2           ┆ 9           │\n",
      "└─────────────┴─────────────┘\n"
     ]
    }
   ],
   "source": [
    "# Polars Array datatype\n",
    "# Arrays are a new data type that was recently introduced, \n",
    "# and are still pretty nascent in features that it offers. \n",
    "\n",
    "# The major difference between a List and an Array is that the latter\n",
    "# is limited to having the same number of elements per row,\n",
    "# while a List can have a variable number of elements. \n",
    "\n",
    "# Both still require that each element's data type is the same.\n",
    "\n",
    "array_df = pl.DataFrame(\n",
    "    [\n",
    "        pl.Series(\"Array_1\", [[1, 3], [2, 5]]),\n",
    "        pl.Series(\"Array_2\", [[1, 7, 3], [8, 1, 0]]),\n",
    "    ],\n",
    "    schema={\"Array_1\": pl.Array(2, pl.Int64), \"Array_2\": pl.Array(3, pl.Int64)},\n",
    ")\n",
    "print(array_df)\n",
    "\n",
    "out = array_df.select(\n",
    "    pl.col(\"Array_1\").arr.min().suffix(\"_min\"),\n",
    "    pl.col(\"Array_2\").arr.sum().suffix(\"_sum\"),\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### using Expression to apply Structs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 9)\n",
      "┌───────┬─────────┬──────────┬─────────────┬───┬────────┬───────────┬─────────┬──────────┐\n",
      "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ASSETS ┆ LIABILITY ┆ REVENUE ┆ NETWORTH │\n",
      "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---    ┆ ---       ┆ ---     ┆ ---      │\n",
      "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64    ┆ i64       ┆ i64     ┆ i64      │\n",
      "╞═══════╪═════════╪══════════╪═════════════╪═══╪════════╪═══════════╪═════════╪══════════╡\n",
      "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 229177 ┆ 162079    ┆ 746304  ┆ 934536   │\n",
      "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 58165  ┆ 645852    ┆ 795922  ┆ 372016   │\n",
      "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 457983 ┆ 853620    ┆ 540738  ┆ 513516   │\n",
      "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 949507 ┆ 907875    ┆ 28191   ┆ 765534   │\n",
      "│ 4     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 848681 ┆ 419144    ┆ 732089  ┆ 511568   │\n",
      "└───────┴─────────┴──────────┴─────────────┴───┴────────┴───────────┴─────────┴──────────┘\n",
      "shape: (4, 1)\n",
      "┌──────────────┐\n",
      "│ FISCAL_YEAR  │\n",
      "│ ---          │\n",
      "│ struct[2]    │\n",
      "╞══════════════╡\n",
      "│ {2023,25136} │\n",
      "│ {2021,25036} │\n",
      "│ {2020,24957} │\n",
      "│ {2022,24871} │\n",
      "└──────────────┘\n"
     ]
    }
   ],
   "source": [
    "# like Polars List and Arrays, which allows users to work with data in sets\n",
    "# Polars provide one more datatype Struct\n",
    "# Struct data type is different than List and Arrays\n",
    "# as Struct allows users to work with multiple columns without copying actual data\n",
    "\n",
    "# Struct\n",
    "############################\n",
    "## BalanceSheet DataFrame ##\n",
    "############################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "sampleSize = 100_000\n",
    "org = \"ABC Inc.\"\n",
    "ledger_type = \"ACTUALS\" # BUDGET, STATS are other Ledger types\n",
    "fiscal_year_from = 2020\n",
    "fiscal_year_to = 2023\n",
    "random.seed(123)\n",
    "\n",
    "balanceSheet = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, \n",
    "                                          fiscal_year_to+1, 1)),k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ASSETS\": random.sample(range(1000000), sampleSize),\n",
    "    \"LIABILITY\": random.sample(range(1000000), sampleSize),\n",
    "    \"REVENUE\": random.sample(range(1000000), sampleSize),\n",
    "    \"NETWORTH\": random.sample(range(1000000), sampleSize),\n",
    "})\n",
    "print(balanceSheet.sample(5).with_row_count(\"Row #\"))\n",
    "\n",
    "out = balanceSheet.select(pl.col(\"FISCAL_YEAR\").value_counts(sort=True))\n",
    "print(out)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (4, 2)\n",
      "┌─────────────┬────────┐\n",
      "│ FISCAL_YEAR ┆ counts │\n",
      "│ ---         ┆ ---    │\n",
      "│ i64         ┆ u32    │\n",
      "╞═════════════╪════════╡\n",
      "│ 2023        ┆ 25136  │\n",
      "│ 2021        ┆ 25036  │\n",
      "│ 2020        ┆ 24957  │\n",
      "│ 2022        ┆ 24871  │\n",
      "└─────────────┴────────┘\n"
     ]
    }
   ],
   "source": [
    "# using unnest will convert Struct to regular data frame\n",
    "out = balanceSheet.select(pl.col(\"FISCAL_YEAR\").value_counts(sort=True)).unnest(\"FISCAL_YEAR\")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (0, 8)\n",
      "┌────────┬─────┬─────────────┬────────┬────────┬───────────┬─────────┬──────────┐\n",
      "│ LEDGER ┆ ORG ┆ FISCAL_YEAR ┆ PERIOD ┆ ASSETS ┆ LIABILITY ┆ REVENUE ┆ NETWORTH │\n",
      "│ ---    ┆ --- ┆ ---         ┆ ---    ┆ ---    ┆ ---       ┆ ---     ┆ ---      │\n",
      "│ str    ┆ str ┆ i64         ┆ i64    ┆ i64    ┆ i64       ┆ i64     ┆ i64      │\n",
      "╞════════╪═════╪═════════════╪════════╪════════╪═══════════╪═════════╪══════════╡\n",
      "└────────┴─────┴─────────────┴────────┴────────┴───────────┴─────────┴──────────┘\n"
     ]
    }
   ],
   "source": [
    "# using Struct to identify duplicates\n",
    "out = balanceSheet.filter(pl.struct(\"LEDGER\",\"ORG\",\"FISCAL_YEAR\",\"PERIOD\",\"ASSETS\").is_duplicated())\n",
    "# is_duplicated() is same as is_unique()\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (100_000, 9)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬───────────┬─────────┬──────────┬──────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ LIABILITY ┆ REVENUE ┆ NETWORTH ┆ Rank │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---       ┆ ---     ┆ ---      ┆ ---  │\n",
      "│ str     ┆ str      ┆ i64         ┆ i64    ┆   ┆ i64       ┆ i64     ┆ i64      ┆ u32  │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪═══════════╪═════════╪══════════╪══════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 10     ┆ … ┆ 535987    ┆ 825234  ┆ 630949   ┆ 1    │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 1      ┆ … ┆ 708385    ┆ 401531  ┆ 307809   ┆ 1    │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 202378    ┆ 777927  ┆ 536595   ┆ 1    │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 3      ┆ … ┆ 378510    ┆ 445913  ┆ 400960   ┆ 1    │\n",
      "│ …       ┆ …        ┆ …           ┆ …      ┆ … ┆ …         ┆ …       ┆ …        ┆ …    │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 6      ┆ … ┆ 274014    ┆ 812724  ┆ 541937   ┆ 1    │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ 8      ┆ … ┆ 804535    ┆ 188039  ┆ 876970   ┆ 1    │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ 8      ┆ … ┆ 752404    ┆ 610809  ┆ 715951   ┆ 1    │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 2      ┆ … ┆ 489051    ┆ 783855  ┆ 226190   ┆ 1    │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴───────────┴─────────┴──────────┴──────┘\n"
     ]
    }
   ],
   "source": [
    "# using Struct for multi column Ranking\n",
    "# This is another very useful feature of Polars\n",
    "\n",
    "# consider a situation, when you are working with TimeSeries data\n",
    "# and want to fill in missing data based on certain RANK\n",
    "\n",
    "out = balanceSheet.with_columns(\n",
    "    pl.struct(\"FISCAL_YEAR\", \"PERIOD\")\n",
    "    .rank(\"dense\", descending=True)\n",
    "    .over(\"LEDGER\", \"ORG\", \"ASSETS\")\n",
    "    .alias(\"Rank\")\n",
    ")\n",
    "# .filter(pl.struct(\"LEDGER\", \"ORG\", \"ASSETS\").is_duplicated())\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use Struct in multi-column apply\n",
    "# we will cover this in USD (user defined functions) section below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Numpy conversion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (3, 2)\n",
      "┌──────────┬──────────┐\n",
      "│ a_log    ┆ b_log    │\n",
      "│ ---      ┆ ---      │\n",
      "│ f64      ┆ f64      │\n",
      "╞══════════╪══════════╡\n",
      "│ 0.0      ┆ 1.386294 │\n",
      "│ 0.693147 ┆ 1.609438 │\n",
      "│ 1.098612 ┆ 1.791759 │\n",
      "└──────────┴──────────┘\n"
     ]
    }
   ],
   "source": [
    "# Polars expressions support NumPy ufuncs\n",
    "# This means that if a function is not provided by Polars\n",
    "# we can use NumPy and we still have fast columnar operation through the NumPy API.\n",
    "\n",
    "import polars as pl\n",
    "import numpy as np\n",
    "\n",
    "df = pl.DataFrame({\"a\": [1, 2, 3], \"b\": [4, 5, 6]})\n",
    "\n",
    "out = df.select(np.log(pl.all()).suffix(\"_log\"))\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions, User defined functions and Windows function\n",
    "---\n",
    "\n",
    "Polars provide large number of built-in functions through Polars Expressions.\n",
    "Here are few examples using built-in expressions as functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (3, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>sum</th><th>mean</th><th>min</th><th>max</th></tr><tr><td>i64</td><td>f64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>11</td><td>2.0</td><td>1</td><td>3</td></tr><tr><td>22</td><td>2.0</td><td>1</td><td>3</td></tr><tr><td>33</td><td>2.0</td><td>1</td><td>3</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (3, 4)\n",
       "┌─────┬──────┬─────┬─────┐\n",
       "│ sum ┆ mean ┆ min ┆ max │\n",
       "│ --- ┆ ---  ┆ --- ┆ --- │\n",
       "│ i64 ┆ f64  ┆ i64 ┆ i64 │\n",
       "╞═════╪══════╪═════╪═════╡\n",
       "│ 11  ┆ 2.0  ┆ 1   ┆ 3   │\n",
       "│ 22  ┆ 2.0  ┆ 1   ┆ 3   │\n",
       "│ 33  ┆ 2.0  ┆ 1   ┆ 3   │\n",
       "└─────┴──────┴─────┴─────┘"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# basic calculations using Polars expressions\n",
    "df = pl.DataFrame(\n",
    "    {\n",
    "        \"a\": [1, 2, 3],\n",
    "        \"b\": [10, 20, 30],\n",
    "    }\n",
    ")\n",
    "\n",
    "out = df.select(\n",
    "    (pl.col(\"a\") + pl.col(\"b\")).alias(\"sum\"),\n",
    "    pl.col(\"a\").mean().alias(\"mean\"),\n",
    "    pl.col(\"a\").min().alias(\"min\"),\n",
    "    pl.col(\"a\").max().alias(\"max\")\n",
    "    )\n",
    "out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 9)\n",
      "┌───────┬─────────┬──────────┬─────────────┬───┬────────┬───────────┬─────────┬──────────┐\n",
      "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ASSETS ┆ LIABILITY ┆ REVENUE ┆ NETWORTH │\n",
      "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---    ┆ ---       ┆ ---     ┆ ---      │\n",
      "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64    ┆ i64       ┆ i64     ┆ i64      │\n",
      "╞═══════╪═════════╪══════════╪═════════════╪═══╪════════╪═══════════╪═════════╪══════════╡\n",
      "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 229177 ┆ 162079    ┆ 746304  ┆ 934536   │\n",
      "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 58165  ┆ 645852    ┆ 795922  ┆ 372016   │\n",
      "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 457983 ┆ 853620    ┆ 540738  ┆ 513516   │\n",
      "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 949507 ┆ 907875    ┆ 28191   ┆ 765534   │\n",
      "│ 4     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 848681 ┆ 419144    ┆ 732089  ┆ 511568   │\n",
      "└───────┴─────────┴──────────┴─────────────┴───┴────────┴───────────┴─────────┴──────────┘\n"
     ]
    }
   ],
   "source": [
    "# Let's use Polars expression built-in functions\n",
    "# to perform a conditional Algebric operation\n",
    "\n",
    "# below sample DataFrame produces a Balance Sheet,\n",
    "# however, NETWORTH calculated is wrong\n",
    "\n",
    "############################\n",
    "## BalanceSheet DataFrame ##\n",
    "############################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "sampleSize = 100_000\n",
    "org = \"ABC Inc.\"\n",
    "ledger_type = \"ACTUALS\" # BUDGET, STATS are other Ledger types\n",
    "fiscal_year_from = 2020\n",
    "fiscal_year_to = 2023\n",
    "random.seed(123)\n",
    "\n",
    "balanceSheet = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, \n",
    "                                          fiscal_year_to+1, 1)),k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ASSETS\": random.sample(range(1000000), sampleSize),\n",
    "    \"LIABILITY\": random.sample(range(1000000), sampleSize),\n",
    "    \"REVENUE\": random.sample(range(1000000), sampleSize),\n",
    "    \"NETWORTH\": random.sample(range(1000000), sampleSize),\n",
    "})\n",
    "print(balanceSheet.sample(5).with_row_count(\"Row #\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Polars Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 10)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬─────────┬──────────┬────────────┬────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ REVENUE ┆ NETWORTH ┆ NetWorth_c ┆ CostCenter │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---     ┆ ---      ┆ ---        ┆ ---        │\n",
      "│ str     ┆ str      ┆ i64         ┆ i64    ┆   ┆ i64     ┆ i64      ┆ i64        ┆ str        │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪═════════╪══════════╪════════════╪════════════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 10     ┆ … ┆ 825234  ┆ 630949   ┆ 358184     ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2020   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 1      ┆ … ┆ 401531  ┆ 307809   ┆ -289400    ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2020   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 777927  ┆ 536595   ┆ 597339     ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2021   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 3      ┆ … ┆ 445913  ┆ 400960   ┆ 398951     ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2020   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ 9      ┆ … ┆ 339738  ┆ 116069   ┆ -72121     ┆ ACTUALSABC │\n",
      "│         ┆          ┆             ┆        ┆   ┆         ┆          ┆            ┆ Inc.2023   │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴─────────┴──────────┴────────────┴────────────┘\n"
     ]
    }
   ],
   "source": [
    "# let's use Polars expression as functions to\n",
    "# A. calculate NetWorth_c = ASSETS + REVENUE - LIABILITY\n",
    "# B. generate a unique COSTCenter String = LEDGER+ORG+FISCAL_YEAR\n",
    "\n",
    "out = balanceSheet.with_columns(\n",
    "        (pl.col(\"ASSETS\") + pl.col(\"REVENUE\") -  pl.col(\"LIABILITY\"))\n",
    "            .alias(\"NetWorth_c\"),\n",
    "        (pl.col(\"LEDGER\") + pl.col(\"ORG\") + pl.col(\"FISCAL_YEAR\").cast(pl.Utf8))\n",
    "            .alias(\"CostCenter\"),\n",
    "    )\n",
    "print(out.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### user defined function\n",
    "\n",
    "let's pretend, we have a function which is more complex in that case, a user defined function is required\n",
    "\n",
    "for example, ORG incur interest on Liability, which is calculated as following and substracted from NetWorth\n",
    "\n",
    "interest paid on Liability Accounts\n",
    "```math\n",
    "    i = P (1 + r/n)^n*T\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# user defined function\n",
    "\n",
    "# let's first setup OOPs for Liability class\n",
    "import random\n",
    "class Liability:\n",
    "\n",
    "    def __init__(self, P, n, r, t):\n",
    "        self.principalAmount = P\n",
    "        # self.rate = r/100\n",
    "        self.rate = r/100\n",
    "        self.compound = n # in case of simple interest, n = 1\n",
    "        self.time = t/12\n",
    "\n",
    "    def getLoanInterest(self):\n",
    "    # compound rate interest deposit\n",
    "        # returns a tuple of interest and Total\n",
    "        return round(self.principalAmount \n",
    "                     * (1 + self.rate / self.compound)**(self.compound * self.time) \n",
    "                     - self.principalAmount, ndigits=2), round(self.principalAmount, ndigits=2)\n",
    "    \n",
    "def getInterest(amount):\n",
    "    compound=1.0\n",
    "    rate=2.875\n",
    "    t=60\n",
    "    d = Liability(amount, compound, rate, t)\n",
    "    return d.getLoanInterest()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (100_000, 9)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬───────────┬─────────┬──────────┬─────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ LIABILITY ┆ REVENUE ┆ NETWORTH ┆ int_on_liab │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---       ┆ ---     ┆ ---      ┆ ---         │\n",
      "│ str     ┆ str      ┆ i64         ┆ i64    ┆   ┆ i64       ┆ i64     ┆ i64      ┆ list[f64]   │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪═══════════╪═════════╪══════════╪═════════════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 10     ┆ … ┆ 535987    ┆ 825234  ┆ 630949   ┆ [81607.61,  │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 535987.0]   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 1      ┆ … ┆ 708385    ┆ 401531  ┆ 307809   ┆ [107856.36, │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 708385.0]   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 202378    ┆ 777927  ┆ 536595   ┆ [30813.41,  │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 202378.0]   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 3      ┆ … ┆ 378510    ┆ 445913  ┆ 400960   ┆ [57630.68,  │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 378510.0]   │\n",
      "│ …       ┆ …        ┆ …           ┆ …      ┆ … ┆ …         ┆ …       ┆ …        ┆ …           │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 6      ┆ … ┆ 274014    ┆ 812724  ┆ 541937   ┆ [41720.47,  │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 274014.0]   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ 8      ┆ … ┆ 804535    ┆ 188039  ┆ 876970   ┆ [122495.84, │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 804535.0]   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ 8      ┆ … ┆ 752404    ┆ 610809  ┆ 715951   ┆ [114558.55, │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 752404.0]   │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 2      ┆ … ┆ 489051    ┆ 783855  ┆ 226190   ┆ [74461.29,  │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ 489051.0]   │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴───────────┴─────────┴──────────┴─────────────┘\n"
     ]
    }
   ],
   "source": [
    "out = balanceSheet.with_columns(\n",
    "    [\n",
    "        pl.col(\"LIABILITY\").map_elements(getInterest).alias(\"int_on_liab\")\n",
    "    ]\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (100_000, 9)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬───────────┬─────────┬──────────┬─────────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ LIABILITY ┆ REVENUE ┆ NETWORTH ┆ liability_new_C │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---       ┆ ---     ┆ ---      ┆ alc             │\n",
      "│ str     ┆ str      ┆ i64         ┆ i64    ┆   ┆ i64       ┆ i64     ┆ i64      ┆ ---             │\n",
      "│         ┆          ┆             ┆        ┆   ┆           ┆         ┆          ┆ f64             │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪═══════════╪═════════╪══════════╪═════════════════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 10     ┆ … ┆ 535987    ┆ 825234  ┆ 630949   ┆ 1.0604e41       │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 1      ┆ … ┆ 708385    ┆ 401531  ┆ 307809   ┆ 1.4014e41       │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ 202378    ┆ 777927  ┆ 536595   ┆ 4.0037e40       │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 3      ┆ … ┆ 378510    ┆ 445913  ┆ 400960   ┆ 7.4882e40       │\n",
      "│ …       ┆ …        ┆ …           ┆ …      ┆ … ┆ …         ┆ …       ┆ …        ┆ …               │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 6      ┆ … ┆ 274014    ┆ 812724  ┆ 541937   ┆ 5.4209e40       │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ 8      ┆ … ┆ 804535    ┆ 188039  ┆ 876970   ┆ 1.5916e41       │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ 8      ┆ … ┆ 752404    ┆ 610809  ┆ 715951   ┆ 1.4885e41       │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 2      ┆ … ┆ 489051    ┆ 783855  ┆ 226190   ┆ 9.6751e40       │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴───────────┴─────────┴──────────┴─────────────────┘\n"
     ]
    }
   ],
   "source": [
    "## ideal way of doing it, use Polars built in Expressions\n",
    "# compound=1.0\n",
    "# rate=2.875\n",
    "# t=60\n",
    "\n",
    "out = balanceSheet.with_columns(\n",
    "    # create the list of homogeneous data\n",
    "    pl.concat_list(pl.all()\n",
    "                   .exclude([\"LEDGER\",\"ORG\",\"FISCAL_YEAR\",\n",
    "                            \"PERIOD\",\"ASSETS\",\"REVENUE\",\"NETWORTH\"]))\n",
    "                            .alias(\"liab\")\n",
    ").select([\n",
    "    # select all columns except the intermediate list\n",
    "    pl.all().exclude(\"liab\"),\n",
    "    # compute the rank by calling `arr.eval`\n",
    "    pl.col(\"LIABILITY\").map_elements(lambda x: \n",
    "                                     round(x\n",
    "                     * (1 + 2.875 / 1.0)**(1.0 * 60) \n",
    "                     - x, ndigits=2)\n",
    "                             )\n",
    "    .alias(\"liability_new_Calc\")\n",
    "])\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# keep in mind,\n",
    "# wherver possible, use Polars Expresssion than user defined function\n",
    "# as Polars Expressions outperform USD or any other methods in speed\n",
    "# above USD can we re-written as this Polars Expression\n",
    "\n",
    "# using Polars custom function\n",
    "# using map or apply\n",
    "# be mindful, apply or map when call Python function will be slow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### windows function\n",
    "\n",
    "Window functions are expressions with superpowers. They allow you to perform aggregations on groups in the select context.\n",
    "\n",
    "use multiple group by operations in parallel, using a single query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 9)\n",
      "┌───────┬─────────┬──────────┬─────────────┬───┬────────┬───────────┬─────────┬──────────┐\n",
      "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ASSETS ┆ LIABILITY ┆ REVENUE ┆ NETWORTH │\n",
      "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---    ┆ ---       ┆ ---     ┆ ---      │\n",
      "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64    ┆ i64       ┆ i64     ┆ i64      │\n",
      "╞═══════╪═════════╪══════════╪═════════════╪═══╪════════╪═══════════╪═════════╪══════════╡\n",
      "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 677730 ┆ 118334    ┆ 110193  ┆ 182505   │\n",
      "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 981373 ┆ 100528    ┆ 316989  ┆ 431846   │\n",
      "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 431052 ┆ 251313    ┆ 8225    ┆ 386441   │\n",
      "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 259722 ┆ 436892    ┆ 38548   ┆ 684450   │\n",
      "│ 4     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 13388  ┆ 671804    ┆ 643440  ┆ 473534   │\n",
      "└───────┴─────────┴──────────┴─────────────┴───┴────────┴───────────┴─────────┴──────────┘\n"
     ]
    }
   ],
   "source": [
    "print(balanceSheet.sample(5).with_row_count(\"Row #\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (100_000, 5)\n",
      "┌─────────┬─────────────┬────────────────┬─────────────────┬──────────────┐\n",
      "│ LEDGER  ┆ FISCAL_YEAR ┆ networth_by_FY ┆ avg_rev_by_FYAP ┆ avg_assets   │\n",
      "│ ---     ┆ ---         ┆ ---            ┆ ---             ┆ ---          │\n",
      "│ str     ┆ i64         ┆ f64            ┆ f64             ┆ f64          │\n",
      "╞═════════╪═════════════╪════════════════╪═════════════════╪══════════════╡\n",
      "│ ACTUALS ┆ 2020        ┆ 498528.963417  ┆ 500178.734118   ┆ 500706.79386 │\n",
      "│ ACTUALS ┆ 2020        ┆ 498528.963417  ┆ 504533.509133   ┆ 500706.79386 │\n",
      "│ ACTUALS ┆ 2021        ┆ 499910.464251  ┆ 502578.376791   ┆ 500706.79386 │\n",
      "│ ACTUALS ┆ 2020        ┆ 498528.963417  ┆ 500137.570938   ┆ 500706.79386 │\n",
      "│ …       ┆ …           ┆ …              ┆ …               ┆ …            │\n",
      "│ ACTUALS ┆ 2022        ┆ 501173.793897  ┆ 504434.734793   ┆ 500706.79386 │\n",
      "│ ACTUALS ┆ 2023        ┆ 497007.493555  ┆ 495342.956584   ┆ 500706.79386 │\n",
      "│ ACTUALS ┆ 2023        ┆ 497007.493555  ┆ 495342.956584   ┆ 500706.79386 │\n",
      "│ ACTUALS ┆ 2020        ┆ 498528.963417  ┆ 509464.849498   ┆ 500706.79386 │\n",
      "└─────────┴─────────────┴────────────────┴─────────────────┴──────────────┘\n"
     ]
    }
   ],
   "source": [
    "out = balanceSheet.select(\n",
    "    \"LEDGER\",\n",
    "    \"FISCAL_YEAR\",\n",
    "    pl.col(\"NETWORTH\").mean().over(\"FISCAL_YEAR\").alias(\"networth_by_FY\"),\n",
    "    pl.col(\"REVENUE\")\n",
    "    .mean()\n",
    "    .over([\"FISCAL_YEAR\", \"PERIOD\"])\n",
    "    .alias(\"avg_rev_by_FYAP\"),\n",
    "    pl.col(\"ASSETS\").mean().alias(\"avg_assets\"),\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (100_000, 3)\n",
      "┌─────────────┬────────┬──────────┐\n",
      "│ FISCAL_YEAR ┆ PERIOD ┆ NETWORTH │\n",
      "│ ---         ┆ ---    ┆ ---      │\n",
      "│ i64         ┆ i64    ┆ i64      │\n",
      "╞═════════════╪════════╪══════════╡\n",
      "│ 2020        ┆ 10     ┆ 630949   │\n",
      "│ 2020        ┆ 1      ┆ 307809   │\n",
      "│ 2021        ┆ 12     ┆ 536595   │\n",
      "│ 2020        ┆ 3      ┆ 400960   │\n",
      "│ …           ┆ …      ┆ …        │\n",
      "│ 2022        ┆ 6      ┆ 541937   │\n",
      "│ 2023        ┆ 8      ┆ 876970   │\n",
      "│ 2023        ┆ 8      ┆ 715951   │\n",
      "│ 2020        ┆ 2      ┆ 226190   │\n",
      "└─────────────┴────────┴──────────┘\n"
     ]
    }
   ],
   "source": [
    "filtered = balanceSheet.filter(pl.col(\"LEDGER\") == \"ACTUALS\").select(\n",
    "    \"FISCAL_YEAR\",\n",
    "    \"PERIOD\",\n",
    "    \"NETWORTH\",\n",
    ")\n",
    "print(filtered)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (100_000, 3)\n",
      "┌─────────────┬────────┬──────────┐\n",
      "│ FISCAL_YEAR ┆ PERIOD ┆ NETWORTH │\n",
      "│ ---         ┆ ---    ┆ ---      │\n",
      "│ i64         ┆ i64    ┆ i64      │\n",
      "╞═════════════╪════════╪══════════╡\n",
      "│ 2020        ┆ 1      ┆ 307809   │\n",
      "│ 2020        ┆ 1      ┆ 272713   │\n",
      "│ 2021        ┆ 1      ┆ 617923   │\n",
      "│ 2020        ┆ 1      ┆ 34682    │\n",
      "│ …           ┆ …      ┆ …        │\n",
      "│ 2022        ┆ 12     ┆ 225945   │\n",
      "│ 2023        ┆ 12     ┆ 230376   │\n",
      "│ 2023        ┆ 12     ┆ 586496   │\n",
      "│ 2020        ┆ 12     ┆ 858653   │\n",
      "└─────────────┴────────┴──────────┘\n"
     ]
    }
   ],
   "source": [
    "out = filtered.with_columns(\n",
    "    pl.col([\"FISCAL_YEAR\",\"PERIOD\",\"NETWORTH\",]).sort_by(\"PERIOD\", descending=False).over(\"FISCAL_YEAR\"),\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import note about user defined funtions\n",
    "\n",
    "# Do not kill parallelization\n",
    "\n",
    "# Python Users Only\n",
    "\n",
    "# The following section is specific to Python, and doesn't apply to Rust. \n",
    "# Within Rust, blocks and closures (lambdas) can, and will, be executed concurrently.\n",
    "\n",
    "# We have all heard that Python is slow, and does \"not scale.\" \n",
    "# Besides the overhead of running \"slow\" bytecode, \n",
    "# Python has to remain within the constraints of the Global Interpreter Lock (GIL).\n",
    "# This means that if you were to use a lambda or a custom Python function to\n",
    "# apply during a parallelized phase, Polars speed is capped running Python code\n",
    "# preventing any multiple threads from executing the function.\n",
    "\n",
    "# This all feels terribly limiting, especially because we often need \n",
    "# those lambda functions in a .groupby() step, for example. \n",
    "# This approach is still supported by Polars, \n",
    "# but keeping in mind bytecode and the GIL costs have to be paid.\n",
    "# It is recommended to try to solve your queries using the expression syntax\n",
    "# before moving to lambdas. \n",
    "#If you want to learn more about using lambdas, go to the user defined functions section. \n",
    "\n",
    "# Conclusion\n",
    "\n",
    "# In the examples above we've seen that we can do a lot by combining expressions.\n",
    "# By doing so we delay the use of custom Python functions \n",
    "# that slow down the queries (by the slow nature of Python AND the GIL).\n",
    "\n",
    "# If we are missing a type expression let us know by opening a feature request!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Polars Data Transformation\n",
    "---\n",
    "\n",
    "joins, concat, pivot and melts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### joins\n",
    "Polars Dataframe allows two data sets join by one or more columns.\n",
    "Polars supports the following join strategies by specifying the strategy argument:\n",
    "\n",
    "- inner join:\tReturns row with matching keys in both frames. Non-matching rows in either the left or right frame are discarded.\n",
    "- left join:\tReturns all rows in the left dataframe, whether or not a match in the right-frame is found. Non-matching rows have their right columns null-filled.\n",
    "- outer join: \tReturns all rows from both the left and right dataframe. If no match is found in one frame, columns from the other frame are null-filled.\n",
    "- cross join:\tReturns the Cartesian product of all rows from the left frame with all rows from the right frame. Duplicates rows are retained; the table length of A cross-joined with B is always len(A) × len(B).\n",
    "- asof join\tA left-join in which the match is performed on the nearest key rather than on equal keys.\n",
    "- semi join:\tReturns all rows from the left frame in which the join key is also present in the right frame.\n",
    "- anti join:\tReturns all rows from the left frame in which the join key is not present in the right frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 9)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>ACCOUNT</th><th>AS_OF_DATE</th><th>DESCRIPTION</th><th>REGION</th><th>TYPE</th><th>STATUS</th><th>CLASSIFICATION</th><th>CATEGORY</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>0</td><td>30000</td><td>2022-01-01 00:00:00</td><td>&quot;Revenue&quot;</td><td>&quot;Region A&quot;</td><td>&quot;R&quot;</td><td>&quot;Active&quot;</td><td>&quot;REVENUE&quot;</td><td>&quot;Cash&quot;</td></tr><tr><td>1</td><td>37000</td><td>2022-01-01 00:00:00</td><td>&quot;Revenue&quot;</td><td>&quot;Region C&quot;</td><td>&quot;R&quot;</td><td>&quot;Active&quot;</td><td>&quot;REVENUE&quot;</td><td>&quot;Non_Members&quot;</td></tr><tr><td>2</td><td>43000</td><td>2022-01-01 00:00:00</td><td>&quot;Statistical Ac…</td><td>&quot;Region D&quot;</td><td>&quot;S&quot;</td><td>&quot;Active&quot;</td><td>&quot;STATISTICS&quot;</td><td>&quot;Consulting&quot;</td></tr><tr><td>3</td><td>13000</td><td>2022-01-01 00:00:00</td><td>&quot;Liabilities&quot;</td><td>&quot;Region D&quot;</td><td>&quot;L&quot;</td><td>&quot;Active&quot;</td><td>&quot;LIABILITIES&quot;</td><td>&quot;Allowance&quot;</td></tr><tr><td>4</td><td>44000</td><td>2022-01-01 00:00:00</td><td>&quot;Revenue&quot;</td><td>&quot;Region E&quot;</td><td>&quot;R&quot;</td><td>&quot;Active&quot;</td><td>&quot;REVENUE&quot;</td><td>&quot;Subscriptions&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 9)\n",
       "┌───────┬─────────┬───────────────┬──────────────┬───┬──────┬────────┬──────────────┬──────────────┐\n",
       "│ Row # ┆ ACCOUNT ┆ AS_OF_DATE    ┆ DESCRIPTION  ┆ … ┆ TYPE ┆ STATUS ┆ CLASSIFICATI ┆ CATEGORY     │\n",
       "│ ---   ┆ ---     ┆ ---           ┆ ---          ┆   ┆ ---  ┆ ---    ┆ ON           ┆ ---          │\n",
       "│ u32   ┆ i64     ┆ datetime[μs]  ┆ str          ┆   ┆ str  ┆ str    ┆ ---          ┆ str          │\n",
       "│       ┆         ┆               ┆              ┆   ┆      ┆        ┆ str          ┆              │\n",
       "╞═══════╪═════════╪═══════════════╪══════════════╪═══╪══════╪════════╪══════════════╪══════════════╡\n",
       "│ 0     ┆ 30000   ┆ 2022-01-01    ┆ Revenue      ┆ … ┆ R    ┆ Active ┆ REVENUE      ┆ Cash         │\n",
       "│       ┆         ┆ 00:00:00      ┆              ┆   ┆      ┆        ┆              ┆              │\n",
       "│ 1     ┆ 37000   ┆ 2022-01-01    ┆ Revenue      ┆ … ┆ R    ┆ Active ┆ REVENUE      ┆ Non_Members  │\n",
       "│       ┆         ┆ 00:00:00      ┆              ┆   ┆      ┆        ┆              ┆              │\n",
       "│ 2     ┆ 43000   ┆ 2022-01-01    ┆ Statistical  ┆ … ┆ S    ┆ Active ┆ STATISTICS   ┆ Consulting   │\n",
       "│       ┆         ┆ 00:00:00      ┆ Accounts     ┆   ┆      ┆        ┆              ┆              │\n",
       "│ 3     ┆ 13000   ┆ 2022-01-01    ┆ Liabilities  ┆ … ┆ L    ┆ Active ┆ LIABILITIES  ┆ Allowance    │\n",
       "│       ┆         ┆ 00:00:00      ┆              ┆   ┆      ┆        ┆              ┆              │\n",
       "│ 4     ┆ 44000   ┆ 2022-01-01    ┆ Revenue      ┆ … ┆ R    ┆ Active ┆ REVENUE      ┆ Subscription │\n",
       "│       ┆         ┆ 00:00:00      ┆              ┆   ┆      ┆        ┆              ┆ s            │\n",
       "└───────┴─────────┴───────────────┴──────────────┴───┴──────┴────────┴──────────────┴──────────────┘"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "########################\n",
    "## ACCOUNTS DataFrame ##\n",
    "########################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "accounts = pl.DataFrame({\n",
    "    \"ACCOUNT\":  list(range(10000, 45000, 1000)),\n",
    "    \"AS_OF_DATE\" : datetime(2022, 1, 1),\n",
    "    \"DESCRIPTION\" : [\"Operating Expenses\",\"Non Operating Expenses\",\"Assets\",\n",
    "                     \"Liabilities\",\"Net worth accounts\", \"Statistical Accounts\",\n",
    "                     \"Revenue\"] * 5,\n",
    "    \"REGION\": [\"Region A\",\"Region B\",\"Region C\",\"Region D\", \"Region E\"] * 7,\n",
    "    \"TYPE\" : [\"E\",\"E\",\"A\",\"L\",\"N\",\"S\",\"R\"] * 5,\n",
    "    \"STATUS\" : \"Active\",\n",
    "    \"CLASSIFICATION\" : [\"OPERATING_EXPENSES\",\"NON-OPERATING_EXPENSES\", \n",
    "                        \"ASSETS\",\"LIABILITIES\",\"NET_WORTH\",\"STATISTICS\",\n",
    "                        \"REVENUE\"] * 5,\n",
    "    \"CATEGORY\" : [\n",
    "       \t\t\"Travel\",\"Payroll\",\"non-Payroll\",\"Allowance\",\"Cash\",\n",
    "       \t\t\"Facility\",\"Supply\",\"Services\",\"Investment\",\"Misc.\",\n",
    "       \t\t\"Depreciation\",\"Gain\",\"Service\",\"Retired\",\"Fault.\",\n",
    "       \t\t\"Receipt\",\"Accrual\",\"Return\",\"Credit\",\"ROI\",\n",
    "       \t\t\"Cash\",\"Funds\",\"Invest\",\"Transfer\",\"Roll-over\",\n",
    "       \t\t\"FTE\",\"Members\",\"Non_Members\",\"Temp\",\"Contractors\",\n",
    "       \t\t\"Sales\",\"Merchant\",\"Service\",\"Consulting\",\"Subscriptions\"\n",
    "       \t],\n",
    "})\n",
    "accounts.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 9)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>DEPT</th><th>AS_OF_DATE</th><th>DESCRIPTION</th><th>REGION</th><th>STATUS</th><th>CLASSIFICATION</th><th>TYPE</th><th>CATEGORY</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>0</td><td>1000</td><td>2022-01-01 00:00:00</td><td>&quot;Sales &amp; Market…</td><td>&quot;Region A&quot;</td><td>&quot;Active&quot;</td><td>&quot;SALES&quot;</td><td>&quot;S&quot;</td><td>&quot;sales&quot;</td></tr><tr><td>1</td><td>2200</td><td>2022-01-01 00:00:00</td><td>&quot;Information Te…</td><td>&quot;Region A&quot;</td><td>&quot;Active&quot;</td><td>&quot;IT&quot;</td><td>&quot;I&quot;</td><td>&quot;IT_Staff&quot;</td></tr><tr><td>2</td><td>1500</td><td>2022-01-01 00:00:00</td><td>&quot;Sales &amp; Market…</td><td>&quot;Region C&quot;</td><td>&quot;Active&quot;</td><td>&quot;SALES&quot;</td><td>&quot;S&quot;</td><td>&quot;sales&quot;</td></tr><tr><td>3</td><td>1800</td><td>2022-01-01 00:00:00</td><td>&quot;Business leade…</td><td>&quot;Region C&quot;</td><td>&quot;Active&quot;</td><td>&quot;BUSINESS&quot;</td><td>&quot;B&quot;</td><td>&quot;business&quot;</td></tr><tr><td>4</td><td>1400</td><td>2022-01-01 00:00:00</td><td>&quot;other temp&quot;</td><td>&quot;Region B&quot;</td><td>&quot;Active&quot;</td><td>&quot;OTHERS&quot;</td><td>&quot;O&quot;</td><td>&quot;others&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 9)\n",
       "┌───────┬──────┬──────────────┬────────────────────┬───┬────────┬────────────────┬──────┬──────────┐\n",
       "│ Row # ┆ DEPT ┆ AS_OF_DATE   ┆ DESCRIPTION        ┆ … ┆ STATUS ┆ CLASSIFICATION ┆ TYPE ┆ CATEGORY │\n",
       "│ ---   ┆ ---  ┆ ---          ┆ ---                ┆   ┆ ---    ┆ ---            ┆ ---  ┆ ---      │\n",
       "│ u32   ┆ i64  ┆ datetime[μs] ┆ str                ┆   ┆ str    ┆ str            ┆ str  ┆ str      │\n",
       "╞═══════╪══════╪══════════════╪════════════════════╪═══╪════════╪════════════════╪══════╪══════════╡\n",
       "│ 0     ┆ 1000 ┆ 2022-01-01   ┆ Sales & Marketing  ┆ … ┆ Active ┆ SALES          ┆ S    ┆ sales    │\n",
       "│       ┆      ┆ 00:00:00     ┆                    ┆   ┆        ┆                ┆      ┆          │\n",
       "│ 1     ┆ 2200 ┆ 2022-01-01   ┆ Information        ┆ … ┆ Active ┆ IT             ┆ I    ┆ IT_Staff │\n",
       "│       ┆      ┆ 00:00:00     ┆ Technology         ┆   ┆        ┆                ┆      ┆          │\n",
       "│ 2     ┆ 1500 ┆ 2022-01-01   ┆ Sales & Marketing  ┆ … ┆ Active ┆ SALES          ┆ S    ┆ sales    │\n",
       "│       ┆      ┆ 00:00:00     ┆                    ┆   ┆        ┆                ┆      ┆          │\n",
       "│ 3     ┆ 1800 ┆ 2022-01-01   ┆ Business leaders   ┆ … ┆ Active ┆ BUSINESS       ┆ B    ┆ business │\n",
       "│       ┆      ┆ 00:00:00     ┆                    ┆   ┆        ┆                ┆      ┆          │\n",
       "│ 4     ┆ 1400 ┆ 2022-01-01   ┆ other temp         ┆ … ┆ Active ┆ OTHERS         ┆ O    ┆ others   │\n",
       "│       ┆      ┆ 00:00:00     ┆                    ┆   ┆        ┆                ┆      ┆          │\n",
       "└───────┴──────┴──────────────┴────────────────────┴───┴────────┴────────────────┴──────┴──────────┘"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "##########################\n",
    "## DEPARTMENT DataFrame ##\n",
    "##########################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "dept = pl.DataFrame({\n",
    "    \"DEPT\":  list(range(1000, 2500, 100)),\n",
    "    \"AS_OF_DATE\" : datetime(2022, 1, 1),\n",
    "    \"DESCRIPTION\" : [\"Sales & Marketing\",\"Human Resource\",\n",
    "                     \"Information Technology\",\"Business leaders\",\"other temp\"] * 3,\n",
    "    \"REGION\": [\"Region A\",\"Region B\",\"Region C\"] * 5,\n",
    "    \"STATUS\" : \"Active\",\n",
    "    \"CLASSIFICATION\" : [\"SALES\",\"HR\", \"IT\",\"BUSINESS\",\"OTHERS\"] * 3,\n",
    "    \"TYPE\" : [\"S\",\"H\",\"I\",\"B\",\"O\"] * 3,\n",
    "    \"CATEGORY\" : [\"sales\",\"human_resource\",\"IT_Staff\",\"business\",\"others\"] * 3,\n",
    "})\n",
    "dept.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>LOCATION</th><th>AS_OF_DATE</th><th>DESCRIPTION</th><th>REGION</th><th>TYPE</th><th>CATEGORY</th></tr><tr><td>u32</td><td>i64</td><td>datetime[μs]</td><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>0</td><td>16</td><td>2022-01-01 00:00:00</td><td>&quot;Atlanta&quot;</td><td>&quot;Region B&quot;</td><td>&quot;Physical&quot;</td><td>&quot;Mfg&quot;</td></tr><tr><td>1</td><td>22</td><td>2022-01-01 00:00:00</td><td>&quot;San Francisco&quot;</td><td>&quot;Region D&quot;</td><td>&quot;Physical&quot;</td><td>&quot;Mfg&quot;</td></tr><tr><td>2</td><td>13</td><td>2022-01-01 00:00:00</td><td>&quot;Philadelphia&quot;</td><td>&quot;Region C&quot;</td><td>&quot;Physical&quot;</td><td>&quot;Mfg&quot;</td></tr><tr><td>3</td><td>20</td><td>2022-01-01 00:00:00</td><td>&quot;Kansas City&quot;</td><td>&quot;Region B&quot;</td><td>&quot;Physical&quot;</td><td>&quot;Ship&quot;</td></tr><tr><td>4</td><td>15</td><td>2022-01-01 00:00:00</td><td>&quot;Richmond&quot;</td><td>&quot;Region A&quot;</td><td>&quot;Physical&quot;</td><td>&quot;Recv&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 7)\n",
       "┌───────┬──────────┬─────────────────────┬───────────────┬──────────┬──────────┬──────────┐\n",
       "│ Row # ┆ LOCATION ┆ AS_OF_DATE          ┆ DESCRIPTION   ┆ REGION   ┆ TYPE     ┆ CATEGORY │\n",
       "│ ---   ┆ ---      ┆ ---                 ┆ ---           ┆ ---      ┆ ---      ┆ ---      │\n",
       "│ u32   ┆ i64      ┆ datetime[μs]        ┆ str           ┆ str      ┆ str      ┆ str      │\n",
       "╞═══════╪══════════╪═════════════════════╪═══════════════╪══════════╪══════════╪══════════╡\n",
       "│ 0     ┆ 16       ┆ 2022-01-01 00:00:00 ┆ Atlanta       ┆ Region B ┆ Physical ┆ Mfg      │\n",
       "│ 1     ┆ 22       ┆ 2022-01-01 00:00:00 ┆ San Francisco ┆ Region D ┆ Physical ┆ Mfg      │\n",
       "│ 2     ┆ 13       ┆ 2022-01-01 00:00:00 ┆ Philadelphia  ┆ Region C ┆ Physical ┆ Mfg      │\n",
       "│ 3     ┆ 20       ┆ 2022-01-01 00:00:00 ┆ Kansas City   ┆ Region B ┆ Physical ┆ Ship     │\n",
       "│ 4     ┆ 15       ┆ 2022-01-01 00:00:00 ┆ Richmond      ┆ Region A ┆ Physical ┆ Recv     │\n",
       "└───────┴──────────┴─────────────────────┴───────────────┴──────────┴──────────┴──────────┘"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "########################\n",
    "## LOCATION DataFrame ##\n",
    "########################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "location = pl.DataFrame({\n",
    "    \"LOCATION\":  list(range(11, 23)),\n",
    "    \"AS_OF_DATE\" : datetime(2022, 1, 1),\n",
    "    \"DESCRIPTION\" : [\"Boston\",\"New York\",\"Philadelphia\",\"Cleveland\",\"Richmond\",\n",
    "                     \"Atlanta\",\"Chicago\",\"St. Louis\",\"Minneapolis\",\"Kansas City\",\n",
    "                     \"Dallas\",\"San Francisco\"],\n",
    "    \"REGION\": [\"Region A\",\"Region B\",\"Region C\",\"Region D\"] * 3,\n",
    "    \"TYPE\" : \"Physical\",\n",
    "    \"CATEGORY\" : [\"Ship\",\"Recv\",\"Mfg\"] * 4\n",
    "})\n",
    "location.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 9)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>LEDGER</th><th>ORG</th><th>FISCAL_YEAR</th><th>PERIOD</th><th>ACCOUNT</th><th>DEPT</th><th>LOCATION</th><th>POSTED_TOTAL</th></tr><tr><td>u32</td><td>str</td><td>str</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>0</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>1</td><td>26000</td><td>1700</td><td>14</td><td>635256</td></tr><tr><td>1</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2020</td><td>8</td><td>31000</td><td>2300</td><td>21</td><td>502139</td></tr><tr><td>2</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2020</td><td>9</td><td>31000</td><td>1300</td><td>22</td><td>729446</td></tr><tr><td>3</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2020</td><td>6</td><td>15000</td><td>2400</td><td>22</td><td>769327</td></tr><tr><td>4</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>8</td><td>31000</td><td>1700</td><td>17</td><td>49468</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 9)\n",
       "┌───────┬─────────┬──────────┬─────────────┬───┬─────────┬──────┬──────────┬──────────────┐\n",
       "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL │\n",
       "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
       "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64     ┆ i64  ┆ i64      ┆ i64          │\n",
       "╞═══════╪═════════╪══════════╪═════════════╪═══╪═════════╪══════╪══════════╪══════════════╡\n",
       "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 26000   ┆ 1700 ┆ 14       ┆ 635256       │\n",
       "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 31000   ┆ 2300 ┆ 21       ┆ 502139       │\n",
       "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 31000   ┆ 1300 ┆ 22       ┆ 729446       │\n",
       "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 15000   ┆ 2400 ┆ 22       ┆ 769327       │\n",
       "│ 4     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 31000   ┆ 1700 ┆ 17       ┆ 49468        │\n",
       "└───────┴─────────┴──────────┴─────────────┴───┴─────────┴──────┴──────────┴──────────────┘"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "######################\n",
    "## LEDGER DataFrame ##\n",
    "######################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "sampleSize = 100_000\n",
    "org = \"ABC Inc.\"\n",
    "ledger_type = \"ACTUALS\" # BUDGET, STATS are other Ledger types\n",
    "fiscal_year_from = 2020\n",
    "fiscal_year_to = 2023\n",
    "random.seed(123)\n",
    "\n",
    "ledger = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, \n",
    "                                          fiscal_year_to+1, 1)),k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ACCOUNT\" : random.choices(accounts[\"ACCOUNT\"], k=sampleSize),\n",
    "\t\"DEPT\" : random.choices(dept[\"DEPT\"], k=sampleSize),\n",
    "\t\"LOCATION\" : random.choices(location[\"LOCATION\"], k=sampleSize),\n",
    "\t\"POSTED_TOTAL\": random.sample(range(1000000), sampleSize)\n",
    "})\n",
    "ledger.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (100_000, 15)\n",
      "┌─────────┬──────────┬─────────────┬────────┬───┬──────┬────────┬────────────────────┬─────────────┐\n",
      "│ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ PERIOD ┆ … ┆ TYPE ┆ STATUS ┆ CLASSIFICATION     ┆ CATEGORY    │\n",
      "│ ---     ┆ ---      ┆ ---         ┆ ---    ┆   ┆ ---  ┆ ---    ┆ ---                ┆ ---         │\n",
      "│ str     ┆ str      ┆ i64         ┆ i64    ┆   ┆ str  ┆ str    ┆ str                ┆ str         │\n",
      "╞═════════╪══════════╪═════════════╪════════╪═══╪══════╪════════╪════════════════════╪═════════════╡\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 10     ┆ … ┆ A    ┆ Active ┆ ASSETS             ┆ non-Payroll │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 1      ┆ … ┆ E    ┆ Active ┆ OPERATING_EXPENSES ┆ Travel      │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ 12     ┆ … ┆ N    ┆ Active ┆ NET_WORTH          ┆ Gain        │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 3      ┆ … ┆ L    ┆ Active ┆ LIABILITIES        ┆ Roll-over   │\n",
      "│ …       ┆ …        ┆ …           ┆ …      ┆ … ┆ …    ┆ …      ┆ …                  ┆ …           │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ 6      ┆ … ┆ L    ┆ Active ┆ LIABILITIES        ┆ Merchant    │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ 8      ┆ … ┆ A    ┆ Active ┆ ASSETS             ┆ Transfer    │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2023        ┆ 8      ┆ … ┆ N    ┆ Active ┆ NET_WORTH          ┆ Cash        │\n",
      "│ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ 2      ┆ … ┆ E    ┆ Active ┆ NON-OPERATING_EXPE ┆ Payroll     │\n",
      "│         ┆          ┆             ┆        ┆   ┆      ┆        ┆ NSES               ┆             │\n",
      "└─────────┴──────────┴─────────────┴────────┴───┴──────┴────────┴────────────────────┴─────────────┘\n"
     ]
    }
   ],
   "source": [
    "## using joins\n",
    "\n",
    "df_inner_ledger_join = ledger.join(accounts, on=\"ACCOUNT\", how=\"inner\")\n",
    "print(df_inner_ledger_join)\n",
    "\n",
    "# df_inner_ledger_join = ledger.join(dept, on=\"DEPT\", how=\"inner\")\n",
    "# print(df_inner_ledger_join)\n",
    "\n",
    "# df_inner_ledger_join = ledger.join(location, on=\"LOCATION\", how=\"inner\")\n",
    "# print(df_inner_ledger_join)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### concatenation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a number of ways to concatenate data from separate DataFrames:\n",
    "\n",
    "- Vertical: two dataframes with the same columns can be vertically concatenated to make a longer dataframe\n",
    "- Horizontal: two dataframes with the same number of rows and non-overlapping columns can be horizontally concatenated to make a wider dataframe\n",
    "- Diagonal: two dataframes with different numbers of rows and columns can be diagonally concatenated to make a dataframe which might be longer and/ or wider. Where column names overlap values will be vertically concatenated. Where column names do not overlap new rows and columns will be added. Missing values will be set as null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((35, 8), (15, 8), (12, 6), (100000, 8))"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accounts.shape, dept.shape, location.shape, ledger.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 9)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>LEDGER</th><th>ORG</th><th>FISCAL_YEAR</th><th>PERIOD</th><th>ACCOUNT</th><th>DEPT</th><th>LOCATION</th><th>POSTED_TOTAL</th></tr><tr><td>u32</td><td>str</td><td>str</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>0</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>1</td><td>26000</td><td>1700</td><td>14</td><td>635256</td></tr><tr><td>1</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2020</td><td>8</td><td>31000</td><td>2300</td><td>21</td><td>502139</td></tr><tr><td>2</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2020</td><td>9</td><td>31000</td><td>1300</td><td>22</td><td>729446</td></tr><tr><td>3</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2020</td><td>6</td><td>15000</td><td>2400</td><td>22</td><td>769327</td></tr><tr><td>4</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>8</td><td>31000</td><td>1700</td><td>17</td><td>49468</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 9)\n",
       "┌───────┬─────────┬──────────┬─────────────┬───┬─────────┬──────┬──────────┬──────────────┐\n",
       "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL │\n",
       "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
       "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64     ┆ i64  ┆ i64      ┆ i64          │\n",
       "╞═══════╪═════════╪══════════╪═════════════╪═══╪═════════╪══════╪══════════╪══════════════╡\n",
       "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 26000   ┆ 1700 ┆ 14       ┆ 635256       │\n",
       "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 31000   ┆ 2300 ┆ 21       ┆ 502139       │\n",
       "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 31000   ┆ 1300 ┆ 22       ┆ 729446       │\n",
       "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 15000   ┆ 2400 ┆ 22       ┆ 769327       │\n",
       "│ 4     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 31000   ┆ 1700 ┆ 17       ┆ 49468        │\n",
       "└───────┴─────────┴──────────┴─────────────┴───┴─────────┴──────┴──────────┴──────────────┘"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "######################\n",
    "## LEDGER DataFrame ##\n",
    "######################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "sampleSize = 100_000\n",
    "org = \"ABC Inc.\"\n",
    "ledger_type = \"ACTUALS\" # BUDGET, STATS are other Ledger types\n",
    "fiscal_year_from = 2020\n",
    "fiscal_year_to = 2023\n",
    "random.seed(123)\n",
    "\n",
    "ledger = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, \n",
    "                                          fiscal_year_to+1, 1)),k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ACCOUNT\" : random.choices(accounts[\"ID\"], k=sampleSize),\n",
    "\t\"DEPT\" : random.choices(dept[\"ID\"], k=sampleSize),\n",
    "\t\"LOCATION\" : random.choices(location[\"ID\"], k=sampleSize),\n",
    "\t\"POSTED_TOTAL\": random.sample(range(1000000), sampleSize)\n",
    "})\n",
    "ledger.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr > th,\n",
       ".dataframe > tbody > tr > td {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 9)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Row #</th><th>LEDGER</th><th>ORG</th><th>FISCAL_YEAR</th><th>PERIOD</th><th>ACCOUNT</th><th>DEPT</th><th>LOCATION</th><th>POSTED_TOTAL</th></tr><tr><td>u32</td><td>str</td><td>str</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>0</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>11</td><td>30000</td><td>1100</td><td>11</td><td>74976</td></tr><tr><td>1</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>9</td><td>42000</td><td>2300</td><td>22</td><td>115846</td></tr><tr><td>2</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2022</td><td>10</td><td>17000</td><td>1300</td><td>16</td><td>156533</td></tr><tr><td>3</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>8</td><td>20000</td><td>1100</td><td>14</td><td>212062</td></tr><tr><td>4</td><td>&quot;ACTUALS&quot;</td><td>&quot;ABC Inc.&quot;</td><td>2021</td><td>5</td><td>38000</td><td>1600</td><td>21</td><td>204052</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 9)\n",
       "┌───────┬─────────┬──────────┬─────────────┬───┬─────────┬──────┬──────────┬──────────────┐\n",
       "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ACCOUNT ┆ DEPT ┆ LOCATION ┆ POSTED_TOTAL │\n",
       "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---     ┆ ---  ┆ ---      ┆ ---          │\n",
       "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64     ┆ i64  ┆ i64      ┆ i64          │\n",
       "╞═══════╪═════════╪══════════╪═════════════╪═══╪═════════╪══════╪══════════╪══════════════╡\n",
       "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 30000   ┆ 1100 ┆ 11       ┆ 74976        │\n",
       "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 42000   ┆ 2300 ┆ 22       ┆ 115846       │\n",
       "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 17000   ┆ 1300 ┆ 16       ┆ 156533       │\n",
       "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 20000   ┆ 1100 ┆ 14       ┆ 212062       │\n",
       "│ 4     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 38000   ┆ 1600 ┆ 21       ┆ 204052       │\n",
       "└───────┴─────────┴──────────┴─────────────┴───┴─────────┴──────┴──────────┴──────────────┘"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ledger_budg = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, fiscal_year_to+1, 1))\n",
    "                               ,k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ACCOUNT\" : random.choices(accounts[\"ID\"], k=sampleSize),\n",
    "\t\"DEPT\" : random.choices(dept[\"ID\"], k=sampleSize),\n",
    "\t\"LOCATION\" : random.choices(location[\"ID\"], k=sampleSize),\n",
    "\t\"POSTED_TOTAL\": random.sample(range(1000000), sampleSize)\n",
    "})\n",
    "ledger_budg.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#########################################\n",
    "# combined ledger for Actuals and Budget\n",
    "#########################################\n",
    "dfLedger = pl.concat([ledger, ledger_budg], how=\"vertical\")\n",
    "dfLedger.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(200000, 8)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfLedger.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Horizontal concatenation fails when dataframes have overlapping columns or a different number of rows.\n",
    "# make sure, number of rows are same and column names are not same\n",
    "\n",
    "# dfLedger = pl.concat([ledger, ledger_budg], how=\"horizontal\")\n",
    "# dfLedger.sample(5).with_row_count(\"Row #\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### pivot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pivot a column in a DataFrame and perform one of the following aggregations:\n",
    "\n",
    "- first\n",
    "- sum\n",
    "- min\n",
    "- max\n",
    "- mean\n",
    "- median\n",
    "\n",
    "#### Eager Pivot\n",
    "The pivot operation consists of a group by one, or multiple columns (these will be the new y-axis), the column that will be pivoted (this will be the new x-axis) and an aggregation.\n",
    "\n",
    "#### Lazy Pivot\n",
    "A Polars LazyFrame always need to know the schema of a computation statically (before collecting the query). As a pivot's output schema depends on the data, and it is therefore impossible to determine the schema without running the query.\n",
    "\n",
    "Polars could have abstracted this fact for you just like Spark does, but we don't want you to shoot yourself in the foot with a shotgun. The cost should be clear upfront."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 3)\n",
      "┌─────┬─────┬─────┐\n",
      "│ foo ┆ N   ┆ bar │\n",
      "│ --- ┆ --- ┆ --- │\n",
      "│ str ┆ i64 ┆ str │\n",
      "╞═════╪═════╪═════╡\n",
      "│ A   ┆ 1   ┆ k   │\n",
      "│ A   ┆ 2   ┆ l   │\n",
      "│ B   ┆ 2   ┆ m   │\n",
      "│ B   ┆ 4   ┆ n   │\n",
      "│ C   ┆ 2   ┆ o   │\n",
      "└─────┴─────┴─────┘\n",
      "shape: (3, 6)\n",
      "┌─────┬──────┬──────┬──────┬──────┬──────┐\n",
      "│ foo ┆ k    ┆ l    ┆ m    ┆ n    ┆ o    │\n",
      "│ --- ┆ ---  ┆ ---  ┆ ---  ┆ ---  ┆ ---  │\n",
      "│ str ┆ i64  ┆ i64  ┆ i64  ┆ i64  ┆ i64  │\n",
      "╞═════╪══════╪══════╪══════╪══════╪══════╡\n",
      "│ A   ┆ 1    ┆ 2    ┆ null ┆ null ┆ null │\n",
      "│ B   ┆ null ┆ null ┆ 2    ┆ 4    ┆ null │\n",
      "│ C   ┆ null ┆ null ┆ null ┆ null ┆ 2    │\n",
      "└─────┴──────┴──────┴──────┴──────┴──────┘\n"
     ]
    }
   ],
   "source": [
    "df = pl.DataFrame(\n",
    "    {\n",
    "        \"foo\": [\"A\", \"A\", \"B\", \"B\", \"C\"],\n",
    "        \"N\": [1, 2, 2, 4, 2],\n",
    "        \"bar\": [\"k\", \"l\", \"m\", \"n\", \"o\"],\n",
    "    }\n",
    ")\n",
    "print(df)\n",
    "\n",
    "out = df.pivot(index=\"foo\", columns=\"bar\", values=\"N\", aggregate_function=\"sum\")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 9)\n",
      "┌───────┬─────────┬──────────┬─────────────┬───┬────────┬───────────┬─────────┬──────────┐\n",
      "│ Row # ┆ LEDGER  ┆ ORG      ┆ FISCAL_YEAR ┆ … ┆ ASSETS ┆ LIABILITY ┆ REVENUE ┆ NETWORTH │\n",
      "│ ---   ┆ ---     ┆ ---      ┆ ---         ┆   ┆ ---    ┆ ---       ┆ ---     ┆ ---      │\n",
      "│ u32   ┆ str     ┆ str      ┆ i64         ┆   ┆ i64    ┆ i64       ┆ i64     ┆ i64      │\n",
      "╞═══════╪═════════╪══════════╪═════════════╪═══╪════════╪═══════════╪═════════╪══════════╡\n",
      "│ 0     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 229177 ┆ 162079    ┆ 746304  ┆ 934536   │\n",
      "│ 1     ┆ ACTUALS ┆ ABC Inc. ┆ 2022        ┆ … ┆ 58165  ┆ 645852    ┆ 795922  ┆ 372016   │\n",
      "│ 2     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 457983 ┆ 853620    ┆ 540738  ┆ 513516   │\n",
      "│ 3     ┆ ACTUALS ┆ ABC Inc. ┆ 2021        ┆ … ┆ 949507 ┆ 907875    ┆ 28191   ┆ 765534   │\n",
      "│ 4     ┆ ACTUALS ┆ ABC Inc. ┆ 2020        ┆ … ┆ 848681 ┆ 419144    ┆ 732089  ┆ 511568   │\n",
      "└───────┴─────────┴──────────┴─────────────┴───┴────────┴───────────┴─────────┴──────────┘\n"
     ]
    }
   ],
   "source": [
    "############################\n",
    "## BalanceSheet DataFrame ##\n",
    "############################\n",
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "sampleSize = 100_000\n",
    "org = \"ABC Inc.\"\n",
    "ledger_type = \"ACTUALS\" # BUDGET, STATS are other Ledger types\n",
    "fiscal_year_from = 2020\n",
    "fiscal_year_to = 2023\n",
    "random.seed(123)\n",
    "\n",
    "balanceSheet = pl.DataFrame({\n",
    "\t\"LEDGER\" : ledger_type,\n",
    "\t\"ORG\" : org,\n",
    "\t\"FISCAL_YEAR\": random.choices(list(range(fiscal_year_from, \n",
    "                                          fiscal_year_to+1, 1)),k=sampleSize),\n",
    "\t\"PERIOD\": random.choices(list(range(1, 12+1, 1)),k=sampleSize),\n",
    "\t\"ASSETS\": random.sample(range(1000000), sampleSize),\n",
    "    \"LIABILITY\": random.sample(range(1000000), sampleSize),\n",
    "    \"REVENUE\": random.sample(range(1000000), sampleSize),\n",
    "    \"NETWORTH\": random.sample(range(1000000), sampleSize),\n",
    "})\n",
    "print(balanceSheet.sample(5).with_row_count(\"Row #\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (4, 13)\n",
      "┌─────────────┬──────────┬──────────┬──────────┬───┬──────────┬──────────┬──────────┬──────────┐\n",
      "│ FISCAL_YEAR ┆ 10       ┆ 1        ┆ 12       ┆ … ┆ 4        ┆ 8        ┆ 2        ┆ 7        │\n",
      "│ ---         ┆ ---      ┆ ---      ┆ ---      ┆   ┆ ---      ┆ ---      ┆ ---      ┆ ---      │\n",
      "│ i64         ┆ f64      ┆ f64      ┆ f64      ┆   ┆ f64      ┆ f64      ┆ f64      ┆ f64      │\n",
      "╞═════════════╪══════════╪══════════╪══════════╪═══╪══════════╪══════════╪══════════╪══════════╡\n",
      "│ 2020        ┆ 519604.0 ┆ 512152.0 ┆ 491098.5 ┆ … ┆ 487557.0 ┆ 494666.0 ┆ 491307.0 ┆ 494794.0 │\n",
      "│ 2021        ┆ 514325.0 ┆ 493920.5 ┆ 502130.5 ┆ … ┆ 483349.0 ┆ 493417.0 ┆ 503360.5 ┆ 514359.0 │\n",
      "│ 2023        ┆ 490111.5 ┆ 500121.0 ┆ 497847.5 ┆ … ┆ 484256.0 ┆ 506810.0 ┆ 486628.5 ┆ 487821.0 │\n",
      "│ 2022        ┆ 490166.0 ┆ 503240.0 ┆ 506036.0 ┆ … ┆ 509159.0 ┆ 487072.5 ┆ 495781.5 ┆ 507723.5 │\n",
      "└─────────────┴──────────┴──────────┴──────────┴───┴──────────┴──────────┴──────────┴──────────┘\n"
     ]
    }
   ],
   "source": [
    "out = balanceSheet.pivot(index=\"FISCAL_YEAR\", columns=\"PERIOD\", values=\"NETWORTH\", aggregate_function=\"median\")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (4, 13)\n",
      "┌───────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐\n",
      "│ FISCAL_YE ┆ 10        ┆ 1         ┆ 12        ┆ … ┆ 4         ┆ 8         ┆ 2         ┆ 7        │\n",
      "│ AR        ┆ ---       ┆ ---       ┆ ---       ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---      │\n",
      "│ ---       ┆ i64       ┆ i64       ┆ i64       ┆   ┆ i64       ┆ i64       ┆ i64       ┆ i64      │\n",
      "│ i64       ┆           ┆           ┆           ┆   ┆           ┆           ┆           ┆          │\n",
      "╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡\n",
      "│ 2020      ┆ 108224941 ┆ 108038900 ┆ 104273367 ┆ … ┆ 972685995 ┆ 101209961 ┆ 103032396 ┆ 10381162 │\n",
      "│           ┆ 5         ┆ 1         ┆ 8         ┆   ┆           ┆ 2         ┆ 6         ┆ 31       │\n",
      "│ 2021      ┆ 108108676 ┆ 104743057 ┆ 104492372 ┆ … ┆ 100276120 ┆ 101698492 ┆ 104962643 ┆ 10383994 │\n",
      "│           ┆ 5         ┆ 7         ┆ 9         ┆   ┆ 2         ┆ 2         ┆ 3         ┆ 32       │\n",
      "│ 2023      ┆ 102746210 ┆ 105297297 ┆ 103340523 ┆ … ┆ 103721463 ┆ 106394484 ┆ 104096539 ┆ 10014053 │\n",
      "│           ┆ 5         ┆ 9         ┆ 5         ┆   ┆ 5         ┆ 2         ┆ 8         ┆ 61       │\n",
      "│ 2022      ┆ 102879002 ┆ 103223842 ┆ 100630852 ┆ … ┆ 105154873 ┆ 105099774 ┆ 105418639 ┆ 10246101 │\n",
      "│           ┆ 9         ┆ 6         ┆ 5         ┆   ┆ 5         ┆ 6         ┆ 1         ┆ 72       │\n",
      "└───────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘\n"
     ]
    }
   ],
   "source": [
    "# lazy execution\n",
    "q = (balanceSheet.lazy()\n",
    "     .collect()\n",
    "     .pivot(index=\"FISCAL_YEAR\", columns=\"PERIOD\", values=\"NETWORTH\", aggregate_function=\"sum\")\n",
    "     .lazy()\n",
    ")\n",
    "\n",
    "out = q.collect()\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### melts\n",
    "\n",
    "Melt operations unpivot a DataFrame from wide format to long format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (3, 4)\n",
      "┌─────┬─────┬─────┬─────┐\n",
      "│ A   ┆ B   ┆ C   ┆ D   │\n",
      "│ --- ┆ --- ┆ --- ┆ --- │\n",
      "│ str ┆ i64 ┆ i64 ┆ i64 │\n",
      "╞═════╪═════╪═════╪═════╡\n",
      "│ a   ┆ 1   ┆ 10  ┆ 2   │\n",
      "│ b   ┆ 3   ┆ 11  ┆ 4   │\n",
      "│ a   ┆ 5   ┆ 12  ┆ 6   │\n",
      "└─────┴─────┴─────┴─────┘\n",
      "shape: (6, 4)\n",
      "┌─────┬─────┬──────────┬───────┐\n",
      "│ A   ┆ B   ┆ variable ┆ value │\n",
      "│ --- ┆ --- ┆ ---      ┆ ---   │\n",
      "│ str ┆ i64 ┆ str      ┆ i64   │\n",
      "╞═════╪═════╪══════════╪═══════╡\n",
      "│ a   ┆ 1   ┆ C        ┆ 10    │\n",
      "│ b   ┆ 3   ┆ C        ┆ 11    │\n",
      "│ a   ┆ 5   ┆ C        ┆ 12    │\n",
      "│ a   ┆ 1   ┆ D        ┆ 2     │\n",
      "│ b   ┆ 3   ┆ D        ┆ 4     │\n",
      "│ a   ┆ 5   ┆ D        ┆ 6     │\n",
      "└─────┴─────┴──────────┴───────┘\n"
     ]
    }
   ],
   "source": [
    "df = pl.DataFrame(\n",
    "    {\n",
    "        \"A\": [\"a\", \"b\", \"a\"],\n",
    "        \"B\": [1, 3, 5],\n",
    "        \"C\": [10, 11, 12],\n",
    "        \"D\": [2, 4, 6],\n",
    "    }\n",
    ")\n",
    "print(df)\n",
    "\n",
    "out = df.melt(id_vars=[\"A\", \"B\"], value_vars=[\"C\", \"D\"])\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (100_000, 4)\n",
      "┌─────────────┬────────┬──────────┬────────┐\n",
      "│ FISCAL_YEAR ┆ PERIOD ┆ variable ┆ value  │\n",
      "│ ---         ┆ ---    ┆ ---      ┆ ---    │\n",
      "│ i64         ┆ i64    ┆ str      ┆ i64    │\n",
      "╞═════════════╪════════╪══════════╪════════╡\n",
      "│ 2020        ┆ 10     ┆ NETWORTH ┆ 630949 │\n",
      "│ 2020        ┆ 1      ┆ NETWORTH ┆ 307809 │\n",
      "│ 2021        ┆ 12     ┆ NETWORTH ┆ 536595 │\n",
      "│ 2020        ┆ 3      ┆ NETWORTH ┆ 400960 │\n",
      "│ …           ┆ …      ┆ …        ┆ …      │\n",
      "│ 2022        ┆ 6      ┆ NETWORTH ┆ 541937 │\n",
      "│ 2023        ┆ 8      ┆ NETWORTH ┆ 876970 │\n",
      "│ 2023        ┆ 8      ┆ NETWORTH ┆ 715951 │\n",
      "│ 2020        ┆ 2      ┆ NETWORTH ┆ 226190 │\n",
      "└─────────────┴────────┴──────────┴────────┘\n"
     ]
    }
   ],
   "source": [
    "out = balanceSheet.melt(id_vars=[\"FISCAL_YEAR\", \"PERIOD\"], value_vars=[\"NETWORTH\"])\n",
    "print(out)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Polars SQL\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## how to use Lazy API\n",
    "\n",
    "---\n",
    "\n",
    "- the lazy API allows Polars to apply automatic query optimization with the query optimizer\n",
    "- the lazy API allows you to work with larger than memory datasets using streaming\n",
    "- the lazy API can catch schema errors before processing the data\n",
    "\n",
    "In the ideal case we use the lazy API right from a file as the query optimizer may help us to reduce the amount of data we read from the file.\n",
    "- scan_csv or scan_parquet or scan_xxx\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "import polars as pl\n",
    "\n",
    "from ..paths import DATA_DIR\n",
    "\n",
    "q1 = (\n",
    "    pl.scan_csv(f\"{DATA_DIR}/reddit.csv\")\n",
    "    .with_columns(pl.col(\"name\").str.to_uppercase())\n",
    "    .filter(pl.col(\"comment_karma\") > 0)\n",
    ")\n",
    "```\n",
    "\n",
    "If we were to run the code above on the Reddit CSV the query would not be evaluated. \n",
    "Instead Polars takes each line of code, adds it to the internal query graph and optimizes the query graph.\n",
    "\n",
    "```\n",
    "import polars as pl\n",
    "\n",
    "from ..paths import DATA_DIR\n",
    "\n",
    "q4 = (\n",
    "    pl.scan_csv(f\"{DATA_DIR}/reddit.csv\")\n",
    "    .with_columns(pl.col(\"name\").str.to_uppercase())\n",
    "    .filter(pl.col(\"comment_karma\") > 0)\n",
    "    .collect()\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Execution on larger-than-memory (RAM) data analytics\n",
    "\n",
    "If your data requires more memory than you have available Polars may be able to process the data in batches using streaming mode. To use streaming mode you simply pass the streaming=True argument to collect\n",
    "\n",
    "```\n",
    "import polars as pl\n",
    "\n",
    "from ..paths import DATA_DIR\n",
    "\n",
    "q5 = (\n",
    "    pl.scan_csv(f\"{DATA_DIR}/reddit.csv\")\n",
    "    .with_columns(pl.col(\"name\").str.to_uppercase())\n",
    "    .filter(pl.col(\"comment_karma\") > 0)\n",
    "    .collect(streaming=True)\n",
    ")\n",
    "```\n",
    "\n",
    "Execution on a partial dataset\n",
    "\n",
    "While you're writing, optimizing or checking your query on a large dataset, querying all available data may lead to a slow development process.\n",
    "\n",
    "You can instead execute the query with the .fetch method. The .fetch method takes a parameter n_rows and tries to 'fetch' that number of rows at the data source. The number of rows cannot be guaranteed, however, as the lazy API does not count how many rows there are at each stage of the query.\n",
    "\n",
    "Here we \"fetch\" 100 rows from the source file and apply the predicates.\n",
    "\n",
    "```\n",
    "import polars as pl\n",
    "\n",
    "from ..paths import DATA_DIR\n",
    "\n",
    "q9 = (\n",
    "    pl.scan_csv(f\"{DATA_DIR}/reddit.csv\")\n",
    "    .with_columns(pl.col(\"name\").str.to_uppercase())\n",
    "    .filter(pl.col(\"comment_karma\") > 0)\n",
    "    .fetch(n_rows=int(100))\n",
    ")\n",
    "```\n",
    "\n",
    "- TODO: cover streaming topic\n",
    "- TODO: cover sinking to a a file\n",
    "- TODO: all topics from Lazy API Chapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Query Optimization\n",
    "import polars as pl\n",
    "q3 = pl.DataFrame({\"foo\": [\"a\", \"b\", \"c\"], \"bar\": [0, 1, 2]}).lazy()\n",
    "\n",
    "print(q3.schema)\n",
    "\n",
    "q3.describe_optimized_plan()\n",
    "\n",
    "## query example to show schema\n",
    "lazy_eager_query = (\n",
    "    pl.DataFrame(\n",
    "        {\n",
    "            \"id\": [\"a\", \"b\", \"c\"],\n",
    "            \"month\": [\"jan\", \"feb\", \"mar\"],\n",
    "            \"values\": [0, 1, 2],\n",
    "        }\n",
    "    )\n",
    "    .lazy()\n",
    "    .with_columns((2 * pl.col(\"values\")).alias(\"double_values\"))\n",
    "    .collect()\n",
    "    .pivot(\n",
    "        index=\"id\", columns=\"month\", values=\"double_values\", aggregate_function=\"first\"\n",
    "    )\n",
    "    .lazy()\n",
    "    .filter(pl.col(\"mar\").is_null())\n",
    "    .collect()\n",
    ")\n",
    "print(lazy_eager_query)\n",
    "q3.show_graph(optimized=False)\n",
    "q3.explain(optimized=False)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "RPA",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}