No .lineapy folder found. Creating a new folder in /Users/sangyoonpark directory. \n", "\n" ], "text/plain": [ "No .lineapy folder found. Creating a new folder in \u001b[35m/Users/\u001b[0m\u001b[95msangyoonpark\u001b[0m directory. \n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# NBVAL_IGNORE_OUTPUT\n", "\n", "import os\n", "import lineapy\n", "import pandas as pd\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "id": "cQTeIJjVLE9M", "metadata": { "id": "cQTeIJjVLE9M" }, "source": [ "## Exploring data" ] }, { "cell_type": "markdown", "id": "7e1eb514-ad0a-48fa-89d1-3adff56b6c64", "metadata": { "id": "7e1eb514-ad0a-48fa-89d1-3adff56b6c64" }, "source": [ "Let's load the toy data to use." ] }, { "cell_type": "code", "execution_count": 2, "id": "410e0e61-9135-4067-8fb0-3538757ac9b7", "metadata": { "id": "410e0e61-9135-4067-8fb0-3538757ac9b7" }, "outputs": [], "source": [ "# NBVAL_IGNORE_OUTPUT\n", "\n", "# Load data\n", "df = pd.read_csv(\"https://raw.githubusercontent.com/LineaLabs/lineapy/main/examples/tutorials/data/iris.csv\")" ] }, { "cell_type": "code", "execution_count": 3, "id": "7cc4e268-e4d4-48e3-8ae2-7f25723be3e2", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 423 }, "id": "7cc4e268-e4d4-48e3-8ae2-7f25723be3e2", "outputId": "c4167cab-3e33-4b30-ad4b-40c401086390" }, "outputs": [ { "data": { "text/html": [ "
| \n", " | sepal.length | \n", "sepal.width | \n", "petal.length | \n", "petal.width | \n", "variety | \n", "
|---|---|---|---|---|---|
| 0 | \n", "5.1 | \n", "3.5 | \n", "1.4 | \n", "0.2 | \n", "Setosa | \n", "
| 1 | \n", "4.9 | \n", "3.0 | \n", "1.4 | \n", "0.2 | \n", "Setosa | \n", "
| 2 | \n", "4.7 | \n", "3.2 | \n", "1.3 | \n", "0.2 | \n", "Setosa | \n", "
| 3 | \n", "4.6 | \n", "3.1 | \n", "1.5 | \n", "0.2 | \n", "Setosa | \n", "
| 4 | \n", "5.0 | \n", "3.6 | \n", "1.4 | \n", "0.2 | \n", "Setosa | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 145 | \n", "6.7 | \n", "3.0 | \n", "5.2 | \n", "2.3 | \n", "Virginica | \n", "
| 146 | \n", "6.3 | \n", "2.5 | \n", "5.0 | \n", "1.9 | \n", "Virginica | \n", "
| 147 | \n", "6.5 | \n", "3.0 | \n", "5.2 | \n", "2.0 | \n", "Virginica | \n", "
| 148 | \n", "6.2 | \n", "3.4 | \n", "5.4 | \n", "2.3 | \n", "Virginica | \n", "
| 149 | \n", "5.9 | \n", "3.0 | \n", "5.1 | \n", "1.8 | \n", "Virginica | \n", "
150 rows × 5 columns
\n", "Pipeline source generated in the directory: output/00_api_basics/demo_pipeline \n",
"\n"
],
"text/plain": [
"Pipeline source generated in the directory: output/00_api_basics/demo_pipeline \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Generated python module demo_pipeline.py \n",
"\n"
],
"text/plain": [
"Generated python module demo_pipeline.py \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Generated Dockerfile demo_pipeline_Dockerfile \n",
"\n"
],
"text/plain": [
"Generated Dockerfile demo_pipeline_Dockerfile \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Generated requirements file demo_pipeline_requirements.txt \n",
"\n"
],
"text/plain": [
"Generated requirements file demo_pipeline_requirements.txt \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Added Airflow DAG named demo_pipeline_dag. Start a run from the Airflow UI or CLI. \n",
"\n"
],
"text/plain": [
"Added Airflow DAG named demo_pipeline_dag. Start a run from the Airflow UI or CLI. \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"PosixPath('output/00_api_basics/demo_pipeline')"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# NBVAL_IGNORE_OUTPUT\n",
"\n",
"# Build an Airflow pipeline with both length and width artifacts\n",
"lineapy.to_pipeline(\n",
" artifacts=[length_artifact.name, width_artifact.name],\n",
" pipeline_name=\"demo_pipeline\",\n",
" framework=\"AIRFLOW\",\n",
" output_dir=\"output/00_api_basics/demo_pipeline/\",\n",
")"
]
},
{
"cell_type": "markdown",
"id": "d8ba315d",
"metadata": {
"id": "d8ba315d"
},
"source": [
"where\n",
"\n",
"- `artifacts` is the list of artifact names to be used for the pipeline\n",
"- `pipeline_name` is the name of the pipeline\n",
"- `output_dir` is the location to put the files for running the pipeline\n",
"- `framework` is the name of orchestration framework to use (currently supports `SCRIPTS` and `AIRFLOW`)"
]
},
{
"cell_type": "markdown",
"id": "3b9b4d97",
"metadata": {
"id": "3b9b4d97"
},
"source": [
"And we see the following files have been generated:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "168dbbe5",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "168dbbe5",
"outputId": "a97f7787-3d8c-4c26-9474-c480f17b07f8"
},
"outputs": [
{
"data": {
"text/plain": [
"['demo_pipeline_requirements.txt',\n",
" 'demo_pipeline_Dockerfile',\n",
" 'demo_pipeline_dag.py',\n",
" 'demo_pipeline.py']"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# NBVAL_IGNORE_OUTPUT\n",
"\n",
"# Check the generated files for running the pipeline\n",
"os.listdir(\"output/00_api_basics/demo_pipeline/\")"
]
},
{
"cell_type": "markdown",
"id": "31e7e390",
"metadata": {
"id": "31e7e390"
},
"source": [
"where\n",
"\n",
"- `[PIPELINE-NAME].py` contains the artifact’s cleaned-up code packaged as a function\n",
"- `[PIPELINE-NAME]_dag.py` uses the packaged function to define the pipeline\n",
"- `[PIPELINE-NAME]_requirements.txt` lists dependencies for running the pipeline\n",
"- `[PIPELINE-NAME]_Dockerfile` contains commands to set up the environment to run the pipeline"
]
},
{
"cell_type": "markdown",
"id": "zXbqIPBntkMM",
"metadata": {
"id": "zXbqIPBntkMM"
},
"source": [
"Specifically, we have `demo_pipeline.py` looking as follows:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "JYwfiWXX1bDW",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "JYwfiWXX1bDW",
"outputId": "6c83ee8e-136b-44b2-9eb7-2a663bf015d3"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"import pickle\r\n",
"\r\n",
"\r\n",
"def iris_diff_avg_length():\r\n",
"\r\n",
" import pandas as pd\r\n",
"\r\n",
" df = pd.read_csv(\r\n",
" \"https://raw.githubusercontent.com/LineaLabs/lineapy/main/examples/tutorials/data/iris.csv\"\r\n",
" )\r\n",
" avg_length_setosa = df.query(\"variety == 'Setosa'\")[\"petal.length\"].mean()\r\n",
" avg_length_virginica = df.query(\"variety == 'Virginica'\")[\"petal.length\"].mean()\r\n",
" diff_avg_length = avg_length_setosa - avg_length_virginica\r\n",
" pickle.dump(\r\n",
" diff_avg_length,\r\n",
" open(\"/Users/sangyoonpark/.lineapy/linea_pickles/CazbEiW\", \"wb\"),\r\n",
" )\r\n",
"\r\n",
"\r\n",
"def iris_diff_avg_width():\r\n",
"\r\n",
" import pandas as pd\r\n",
"\r\n",
" df = pd.read_csv(\r\n",
" \"https://raw.githubusercontent.com/LineaLabs/lineapy/main/examples/tutorials/data/iris.csv\"\r\n",
" )\r\n",
" avg_width_setosa = df.query(\"variety == 'Setosa'\")[\"petal.width\"].mean()\r\n",
" avg_width_virginica = df.query(\"variety == 'Virginica'\")[\"petal.width\"].mean()\r\n",
" diff_avg_width = avg_width_setosa - avg_width_virginica\r\n",
" width_artifact = pickle.dump(\r\n",
" diff_avg_width, open(\"/Users/sangyoonpark/.lineapy/linea_pickles/l5PX1bp\", \"wb\")\r\n",
" )\r\n"
]
}
],
"source": [
"# NBVAL_IGNORE_OUTPUT\n",
"\n",
"%cat output/00_api_basics/demo_pipeline/demo_pipeline.py"
]
},
{
"cell_type": "markdown",
"id": "jOvakSzWuNZx",
"metadata": {
"id": "jOvakSzWuNZx"
},
"source": [
"We can see that LineaPy used artifacts to automatically 1) clean up their code to retain only essential operations and 2) package the cleaned-up code into importable functions.\n",
"\n",
"And we see `demo_pipeline_dag.py` automatically composing an Airflow DAG with these functions:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "N4r32Qjg1hYW",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "N4r32Qjg1hYW",
"outputId": "002e89e0-b71d-4da2-88b9-059798d3c0b0"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"import os\r\n",
"\r\n",
"import demo_pipeline\r\n",
"from airflow import DAG\r\n",
"from airflow.operators.python_operator import PythonOperator\r\n",
"from airflow.utils.dates import days_ago\r\n",
"\r\n",
"default_dag_args = {\"owner\": \"airflow\", \"retries\": 2, \"start_date\": days_ago(1)}\r\n",
"\r\n",
"dag = DAG(\r\n",
" dag_id=\"demo_pipeline_dag\",\r\n",
" schedule_interval=\"*/15 * * * *\",\r\n",
" max_active_runs=1,\r\n",
" catchup=False,\r\n",
" default_args=default_dag_args,\r\n",
")\r\n",
"\r\n",
"\r\n",
"iris_diff_avg_length = PythonOperator(\r\n",
" dag=dag,\r\n",
" task_id=\"iris_diff_avg_length_task\",\r\n",
" python_callable=demo_pipeline.iris_diff_avg_length,\r\n",
")\r\n",
"\r\n",
"iris_diff_avg_width = PythonOperator(\r\n",
" dag=dag,\r\n",
" task_id=\"iris_diff_avg_width_task\",\r\n",
" python_callable=demo_pipeline.iris_diff_avg_width,\r\n",
")\r\n"
]
}
],
"source": [
"# NBVAL_IGNORE_OUTPUT\n",
"\n",
"%cat output/00_api_basics/demo_pipeline/demo_pipeline_dag.py"
]
},
{
"cell_type": "markdown",
"id": "d9dda73d",
"metadata": {
"id": "d9dda73d"
},
"source": [
"These files, once placed in the location that Airflow expects (usually `dag/` under Airflow’s home directory), should let us immediately execute the pipeline from the UI or CLI.\n",
"\n",
"For a more detailed illustration of pipeline building, please check [this](https://docs.lineapy.org/en/latest/guide/build_pipelines/index.html#) tutorial."
]
},
{
"cell_type": "markdown",
"id": "3e6feba4",
"metadata": {},
"source": [
"## Recap\n",
"\n",
"In this tutorial, we learned basic functionalities of LineaPy including how to save, browse, and retrieve an artifact. We saw that a LineaPy artifact stores not only the value of a variable but also its full development code. This then helps to automate time-consuming, manual steps in a data science workflow such as code cleanup and pipeline building, hence helping data scientists move faster towards productionization and impact."
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "00_api_basics.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}