Importing library in Amazon FinSpace
Important
Amazon FinSpace Dataset Browser will be discontinued on March 26,
2025. Starting November 29, 2023, FinSpace will no longer accept the creation of new Dataset Browser
environments. Customers using Amazon FinSpace with Managed Kdb Insights
You can install notebook-scoped libraries on a running Amazon FinSpace cluster directly via a FinSpace notebook. This capability is useful in scenarios in which you do not have access to a PyPI repository but need to analyze and visualize a dataset.
Notebook-scoped libraries provide you the following benefits:
-
Runtime installation – You can import Python libraries from PyPI repositories and install them on your remote cluster on the fly when you need them. These libraries are instantly available to your Spark runtime environment. There is no need to restart the notebook session or recreate your cluster.
-
Dependency isolation – The libraries you install using FinSpace notebooks are isolated to your notebook session and don't interfere with bootstrapped cluster libraries or libraries installed from other notebook sessions. These notebook-scoped libraries take precedence over bootstrapped libraries. Multiple notebook users can import their preferred version of the library and use it without dependency clashes on the same cluster.
-
Portable library environment – The library package installation happens from your notebook file. This allows you to recreate the library environment when you switch the notebook to a different cluster by re-executing the notebook code. At the end of the notebook session, the libraries you install through FinSpace notebooks are automatically removed from the hosting cluster.
The following example code shows how to install pandas and matplotlib from the PiPY repository.
sc.install_pypi_package("pandas==0.25.1") #Install pandas version 0.25.1
sc.install_pypi_package("matplotlib", "https://pypi.org/simple") #Install matplotlib from given PyPI repositoryYou can uninstall packages using the uninstall_package PySpark API.
sc.uninstall_package('pandas')