{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Load And Dump Arrays\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The LArray library provides methods and functions to load and dump Array, Session, Axis Group objects to several formats such as Excel, CSV and HDF5. The HDF5 file format is designed to store and organize large amounts of data. It allows to read and write data much faster than when working with CSV and Excel files. \n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# first of all, import the LArray library\n", "from larray import *" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading Arrays - Basic Usage (CSV, Excel, HDF5)\n", "\n", "To read an array from a CSV file, you must use the ``read_csv`` function:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "csv_dir = get_example_filepath('examples')\n", "\n", "# read the array population from the file 'population.csv'.\n", "# The data of the array below is derived from a subset of the demo_pjan table from Eurostat\n", "population = read_csv(csv_dir / 'population.csv')\n", "population" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To read an array from a sheet of an Excel file, you can use the ``read_excel`` function:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "filepath_excel = get_example_filepath('examples.xlsx')\n", "\n", "# read the array from the sheet 'births' of the Excel file 'examples.xlsx'\n", "# The data of the array below is derived from a subset of the demo_fasec table from Eurostat\n", "births = read_excel(filepath_excel, 'births')\n", "births" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ``open_excel`` function in combination with the ``load`` method allows you to load several arrays from the same Workbook without opening and closing it several times:\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "# open the Excel file 'population.xlsx' and let it opened as long as you keep the indent.\n", "# The Python keyword ``with`` ensures that the Excel file is properly closed even if an error occurs\n", "with open_excel(filepath_excel) as wb:\n", " # load the array 'population' from the sheet 'population' \n", " population = wb['population'].load()\n", " # load the array 'births' from the sheet 'births'\n", " births = wb['births'].load()\n", " # load the array 'deaths' from the sheet 'deaths'\n", " deaths = wb['deaths'].load()\n", "\n", "# the Workbook is automatically closed when getting out the block defined by the with statement\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " **Warning:** `open_excel` requires to work on Windows and to have the library ``xlwings`` installed.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `HDF5` file format is specifically designed to store and organize large amounts of data. \n", "Reading and writing data in this file format is much faster than with CSV or Excel. \n", "An HDF5 file can contain multiple arrays, each array being associated with a key.\n", "To read an array from an HDF5 file, you must use the ``read_hdf`` function and provide the key associated with the array:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "filepath_hdf = get_example_filepath('examples.h5')\n", "\n", "# read the array from the file 'examples.h5' associated with the key 'deaths'\n", "# The data of the array below is derived from a subset of the demo_magec table from Eurostat\n", "deaths = read_hdf(filepath_hdf, 'deaths')\n", "deaths" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Dumping Arrays - Basic Usage (CSV, Excel, HDF5)\n", "\n", "To write an array in a CSV file, you must use the ``to_csv`` method:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# save the array population in the file 'population.csv'\n", "population.to_csv('population.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To write an array to a sheet of an Excel file, you can use the ``to_excel`` method:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# save the array population in the sheet 'population' of the Excel file 'population.xlsx' \n", "population.to_excel('population.xlsx', 'population')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that ``to_excel`` create a new Excel file if it does not exist yet. \n", "If the file already exists, a new sheet is added after the existing ones if that sheet does not already exists:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# add a new sheet 'births' to the file 'population.xlsx' and save the array births in it\n", "births.to_excel('population.xlsx', 'births')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To reset an Excel file, you simply need to set the `overwrite_file` argument as True:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 1. reset the file 'population.xlsx' (all sheets are removed)\n", "# 2. create a sheet 'population' and save the array population in it\n", "population.to_excel('population.xlsx', 'population', overwrite_file=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ``open_excel`` function in combination with the ``dump()`` method allows you to open a Workbook and to export several arrays at once. If the Excel file doesn't exist, the ``overwrite_file`` argument must be set to True.\n", "\n", "
\n", " **Warning:** The ``save`` method must be called at the end of the block defined by the *with* statement to actually write data in the Excel file, otherwise you will end up with an empty file.\n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "# to create a new Excel file, argument overwrite_file must be set to True\n", "with open_excel('population.xlsx', overwrite_file=True) as wb:\n", " # add a new sheet 'population' and dump the array population in it \n", " wb['population'] = population.dump()\n", " # add a new sheet 'births' and dump the array births in it \n", " wb['births'] = births.dump()\n", " # add a new sheet 'deaths' and dump the array deaths in it \n", " wb['deaths'] = deaths.dump()\n", " # actually write data in the Workbook\n", " wb.save()\n", " \n", "# the Workbook is automatically closed when getting out the block defined by the with statement\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To write an array in an HDF5 file, you must use the ``to_hdf`` function and provide the key that will be associated with the array:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# save the array population in the file 'population.h5' and associate it with the key 'population'\n", "population.to_hdf('population.h5', 'population')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Specifying Wide VS Narrow format (CSV, Excel)\n", "\n", "By default, all reading functions assume that arrays are stored in the ``wide`` format, meaning that their last axis is represented horizontally:\n", "\n", "| country \\\\ time | 2013 | 2014 | 2015 |\n", "| --------------- | -------- | -------- | -------- |\n", "| Belgium | 11137974 | 11180840 | 11237274 |\n", "| France | 65600350 | 65942267 | 66456279 |\n", "\n", "By setting the ``wide`` argument to False, reading functions will assume instead that arrays are stored in the ``narrow`` format, i.e. one column per axis plus one value column:\n", "\n", "| country | time | value |\n", "| ------- | ---- | -------- |\n", "| Belgium | 2013 | 11137974 |\n", "| Belgium | 2014 | 11180840 |\n", "| Belgium | 2015 | 11237274 |\n", "| France | 2013 | 65600350 |\n", "| France | 2014 | 65942267 |\n", "| France | 2015 | 66456279 |\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# set 'wide' argument to False to indicate that the array is stored in the 'narrow' format\n", "population_BE_FR = read_csv(csv_dir / 'population_narrow_format.csv', wide=False)\n", "population_BE_FR" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# same for the read_excel function\n", "population_BE_FR = read_excel(filepath_excel, sheet='population_narrow_format', wide=False)\n", "population_BE_FR" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default, writing functions will set the name of the column containing the data to 'value'. You can choose the name of this column by using the ``value_name`` argument. For example, using ``value_name='population'`` you can export the previous array as:\n", "\n", "| country | time | population |\n", "| ------- | ---- | ---------- |\n", "| Belgium | 2013 | 11137974 |\n", "| Belgium | 2014 | 11180840 |\n", "| Belgium | 2015 | 11237274 |\n", "| France | 2013 | 65600350 |\n", "| France | 2014 | 65942267 |\n", "| France | 2015 | 66456279 |\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# dump the array population_BE_FR in a narrow format (one column per axis plus one value column).\n", "# By default, the name of the column containing data is set to 'value'\n", "population_BE_FR.to_csv('population_narrow_format.csv', wide=False)\n", "\n", "# same but replace 'value' by 'population'\n", "population_BE_FR.to_csv('population_narrow_format.csv', wide=False, value_name='population')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# same for the to_excel method\n", "population_BE_FR.to_excel('population.xlsx', 'population_narrow_format', wide=False, value_name='population')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Like with the ``to_excel`` method, it is possible to export arrays in a ``narrow`` format using ``open_excel``. \n", "To do so, you must set the ``wide`` argument of the ``dump`` method to False:\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "with open_excel('population.xlsx') as wb:\n", " # dump the array population_BE_FR in a narrow format: \n", " # one column per axis plus one value column.\n", " # Argument value_name can be used to change the name of the \n", " # column containing the data (default name is 'value')\n", " wb['population_narrow_format'] = population_BE_FR.dump(wide=False, value_name='population')\n", " # don't forget to call save()\n", " wb.save()\n", "\n", "# in the sheet 'population_narrow_format', data is written as:\n", "# | country | time | population |\n", "# | ------- | ---- | ---------- |\n", "# | Belgium | 2013 | 11137974 |\n", "# | Belgium | 2014 | 11180840 |\n", "# | Belgium | 2015 | 11237274 |\n", "# | France | 2013 | 65600350 |\n", "# | France | 2014 | 65942267 |\n", "# | France | 2015 | 66456279 |\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Specifying Position in Sheet (Excel)\n", "\n", "If you want to read an array from an Excel sheet which does not start at cell `A1` (when there is more than one array stored in the same sheet for example), you will need to use the ``range`` argument. \n", "\n", "
\n", " **Warning:** Note that the ``range`` argument is only available if you have the library ``xlwings`` installed (Windows).\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "# the 'range' argument must be used to load data not starting at cell A1.\n", "# This is useful when there is several arrays stored in the same sheet\n", "births = read_excel(filepath_excel, sheet='population_births_deaths', range='A9:E15')\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using ``open_excel``, ranges are passed in brackets:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "with open_excel(filepath_excel) as wb:\n", " # store sheet 'population_births_deaths' in a temporary variable sh\n", " sh = wb['population_births_deaths']\n", " # load the array population from range A1:E7\n", " population = sh['A1:E7'].load()\n", " # load the array births from range A9:E15\n", " births = sh['A9:E15'].load()\n", " # load the array deaths from range A17:E23\n", " deaths = sh['A17:E23'].load()\n", "\n", "# the Workbook is automatically closed when getting out the block defined by the with statement\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When exporting arrays to Excel files, data is written starting at cell `A1` by default. Using the ``position`` argument of the ``to_excel`` method, it is possible to specify the top left cell of the dumped data. This can be useful when you want to export several arrays in the same sheet for example\n", "\n", "
\n", " **Warning:** Note that the ``position`` argument is only available if you have the library ``xlwings`` installed (Windows).\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "filename = 'population.xlsx'\n", "sheetname = 'population_births_deaths'\n", "\n", "# save the arrays population, births and deaths in the same sheet 'population_births_and_deaths'.\n", "# The 'position' argument is used to shift the location of the second and third arrays to be dumped\n", "population.to_excel(filename, sheetname)\n", "births.to_excel(filename, sheetname, position='A9')\n", "deaths.to_excel(filename, sheetname, position='A17')\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using ``open_excel``, the position is passed in brackets (this allows you to also add extra informations): \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "with open_excel('population.xlsx') as wb:\n", " # add a new sheet 'population_births_deaths' and write 'population' in the first cell\n", " # note: you can use wb['new_sheet_name'] = '' to create an empty sheet\n", " wb['population_births_deaths'] = 'population'\n", " # store sheet 'population_births_deaths' in a temporary variable sh\n", " sh = wb['population_births_deaths']\n", " # dump the array population in sheet 'population_births_deaths' starting at cell A2\n", " sh['A2'] = population.dump()\n", " # add 'births' in cell A10\n", " sh['A10'] = 'births'\n", " # dump the array births in sheet 'population_births_deaths' starting at cell A11 \n", " sh['A11'] = births.dump()\n", " # add 'deaths' in cell A19\n", " sh['A19'] = 'deaths'\n", " # dump the array deaths in sheet 'population_births_deaths' starting at cell A20\n", " sh['A20'] = deaths.dump()\n", " # don't forget to call save()\n", " wb.save()\n", " \n", "# the Workbook is automatically closed when getting out the block defined by the with statement\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exporting data without headers (Excel)\n", "\n", "For some reasons, you may want to export only the data of an array without axes. For example, you may want to insert a new column containing extra information. As an exercise, let us consider we want to add the capital city for each country present in the array containing the total population by country:\n", "\n", "| country | capital city | 2013 | 2014 | 2015 |\n", "| ------- | ------------ | -------- | -------- | -------- |\n", "| Belgium | Brussels | 11137974 | 11180840 | 11237274 |\n", "| France | Paris | 65600350 | 65942267 | 66456279 |\n", "| Germany | Berlin | 80523746 | 80767463 | 81197537 |\n", "\n", "Assuming you have prepared an excel sheet as below: \n", "\n", "| country | capital city | 2013 | 2014 | 2015 |\n", "| ------- | ------------ | -------- | -------- | -------- |\n", "| Belgium | Brussels | | | |\n", "| France | Paris | | | |\n", "| Germany | Berlin | | | ||\n", "\n", "you can then dump the data at right place by setting the ``header`` argument of ``to_excel`` to False and specifying the position of the data in sheet:\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "population_by_country = population.sum('gender')\n", "\n", "# export only the data of the array population_by_country starting at cell C2\n", "population_by_country.to_excel('population.xlsx', 'population_by_country', header=False, position='C2')\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using ``open_excel``, you can easily prepare the sheet and then export only data at the right place by either setting the ``header`` argument of the ``dump`` method to False or avoiding to call ``dump``:\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "with open_excel('population.xlsx') as wb:\n", " # create new empty sheet 'population_by_country'\n", " wb['population_by_country'] = ''\n", " # store sheet 'population_by_country' in a temporary variable sh\n", " sh = wb['population_by_country']\n", " # write extra information (description)\n", " sh['A1'] = 'Population at 1st January by country'\n", " # export column names\n", " sh['A2'] = ['country', 'capital city']\n", " sh['C2'] = population_by_country.time.labels\n", " # export countries as first column\n", " sh['A3'].options(transpose=True).value = population_by_country.country.labels\n", " # export capital cities as second column\n", " sh['B3'].options(transpose=True).value = ['Brussels', 'Paris', 'Berlin']\n", " # export only data of population_by_country\n", " sh['C3'] = population_by_country.dump(header=False)\n", " # or equivalently\n", " sh['C3'] = population_by_country\n", " # don't forget to call save()\n", " wb.save()\n", " \n", "# the Workbook is automatically closed when getting out the block defined by the with statement\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Specifying the Number of Axes at Reading (CSV, Excel)\n", "\n", "By default, ``read_csv`` and ``read_excel`` will search the position of the first cell containing the special character ``\\`` in the header line in order to determine the number of axes of the array to read. The special character ``\\`` is used to separate the name of the two last axes. If there is no special character ``\\``, ``read_csv`` and ``read_excel`` will consider that the array to read has only one dimension. For an array stored as:\n", "\n", "| country | gender \\\\ time | 2013 | 2014 | 2015 |\n", "| ------- | -------------- | -------- | -------- | -------- |\n", "| Belgium | Male | 5472856 | 5493792 | 5524068 |\n", "| Belgium | Female | 5665118 | 5687048 | 5713206 |\n", "| France | Male | 31772665 | 31936596 | 32175328 |\n", "| France | Female | 33827685 | 34005671 | 34280951 |\n", "| Germany | Male | 39380976 | 39556923 | 39835457 |\n", "| Germany | Female | 41142770 | 41210540 | 41362080 |\n", "\n", "``read_csv`` and ``read_excel`` will find the special character ``\\`` in the second cell meaning it expects three axes (country, gender and time). \n", "\n", "Sometimes, you need to read an array for which the name of the last axis is implicit: \n", "\n", "| country | gender | 2013 | 2014 | 2015 |\n", "| ------- | ------ | -------- | -------- | -------- |\n", "| Belgium | Male | 5472856 | 5493792 | 5524068 |\n", "| Belgium | Female | 5665118 | 5687048 | 5713206 |\n", "| France | Male | 31772665 | 31936596 | 32175328 |\n", "| France | Female | 33827685 | 34005671 | 34280951 |\n", "| Germany | Male | 39380976 | 39556923 | 39835457 |\n", "| Germany | Female | 41142770 | 41210540 | 41362080 |\n", "\n", "For such case, you will have to inform ``read_csv`` and ``read_excel`` of the number of axes of the output array by setting the ``nb_axes`` argument:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# read the 3 x 2 x 3 array stored in the file 'population_missing_axis_name.csv' wihout using 'nb_axes' argument.\n", "population = read_csv(csv_dir / 'population_missing_axis_name.csv')\n", "# shape and data type of the output array are not what we expected\n", "population.info" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# by setting the 'nb_axes' argument, you can indicate to read_csv the number of axes of the output array\n", "population = read_csv(csv_dir / 'population_missing_axis_name.csv', nb_axes=3)\n", "\n", "# give a name to the last axis\n", "population = population.rename(-1, 'time')\n", "\n", "# shape and data type of the output array are what we expected\n", "population.info" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# same for the read_excel function\n", "population = read_excel(filepath_excel, sheet='population_missing_axis_name', nb_axes=3)\n", "population = population.rename(-1, 'time')\n", "population.info" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## NaNs and Missing Data Handling at Reading (CSV, Excel)\n", "\n", "Sometimes, there is no data available for some label combinations. In the example below, the rows corresponding to `France - Male` and `Germany - Female` are missing:\n", "\n", "| country | gender \\\\ time | 2013 | 2014 | 2015 |\n", "| ------- | -------------- | -------- | -------- | -------- |\n", "| Belgium | Male | 5472856 | 5493792 | 5524068 |\n", "| Belgium | Female | 5665118 | 5687048 | 5713206 |\n", "| France | Female | 33827685 | 34005671 | 34280951 |\n", "| Germany | Male | 39380976 | 39556923 | 39835457 |\n", "\n", "By default, ``read_csv`` and ``read_excel`` will fill cells associated with missing label combinations with nans. \n", "Be aware that, in that case, an int array will be converted to a float array." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# by default, cells associated will missing label combinations are filled with nans.\n", "# In that case, the output array is converted to a float array\n", "read_csv(csv_dir / 'population_missing_values.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, it is possible to choose which value to use to fill missing cells using the ``fill_value`` argument:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "read_csv(csv_dir / 'population_missing_values.csv', fill_value=0)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# same for the read_excel function\n", "read_excel(filepath_excel, sheet='population_missing_values', fill_value=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sorting Axes at Reading (CSV, Excel, HDF5)\n", "\n", "The ``sort_rows`` and ``sort_columns`` arguments of the reading functions allows you to sort rows and columns alphabetically:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# sort labels at reading --> Male and Female labels are inverted\n", "read_csv(csv_dir / 'population.csv', sort_rows=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "read_excel(filepath_excel, sheet='births', sort_rows=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "read_hdf(filepath_hdf, key='deaths').sort_labels()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Metadata (HDF5)\n", "\n", "It is possible to add metadata to arrays:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "population.meta.title = 'Population at 1st January'\n", "population.meta.origin = 'Table demo_jpan from Eurostat'\n", "\n", "population.info" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These metadata are automatically saved and loaded when working with the HDF5 file format: " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "population.to_hdf('population.h5', 'population')\n", "\n", "new_population = read_hdf('population.h5', 'population')\n", "new_population.info" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " **Warning:** Currently, metadata associated with arrays cannot be saved and loaded when working with CSV and Excel files.\n", " This restriction does not apply however to metadata associated with sessions.\n", "
" ] } ], "metadata": { "celltoolbar": "Edit Metadata", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.15" }, "livereveal": { "autolaunch": false, "scroll": true } }, "nbformat": 4, "nbformat_minor": 2 }