RemoteReaderBase

class csvio.remote.remote_base.RemoteReaderBase(options: Dict[str, Any], processors: Optional[List[csvio.processors.processor_base.ProcessorBase]] = None, fieldnames: List[str] = [], open_kwargs: Dict[str, Any] = {}, csv_kwargs: Dict[str, Any] = {}, max_retries: int = 3, timeout: int = 5, increment_timeout: int = 10)

Bases: csvio.csvreader.CSVReader, abc.ABC

Base class for remote readers.

All remote readers are derived from this class and should implement the _csv_downloader() abstract method at a minimum.

Parameters

options (required) – Options that are to be used by the downloader child class that is derived from this base class

property csv_kwargs: Dict[str, Any]
Returns

A dictionary of key, value pairs that should be passed to the DictReader constructor within this class.

delete(missing_ok: bool = False) bool

Delete the file at the path provided in the filename parameter

Parameters

missing_ok (optional) – Parameter to pass to the pathlib.Path.unlink() method.

Returns

True If file is deleted successfully.

False On failure.

property fieldnames: List[str]
Returns

List of column headings

property file_ext: str
Returns

Extension suffix of the file without parent directory and file name.

property filedir: str
Returns

Parent directory path of the file (excluding the name of the file)

property filename: str
Returns

File name without the parent directory path.

property filename_no_ext: str
Returns

File name without parent directory and file extension.

property filepath: str
Returns

Complete file path including the parent directory, file name and extension

get_temp_path() str
Returns

Full path to a randomly named csv file in the system’s temporary directory.

property num_rows: int
Returns

The total number of rows in the CSV (excluding column headings)

property open_kwargs: Dict[str, Any]
Returns

A dictionary of key, value pairs that should be passed to the open method within this class.

property path_obj: pathlib.Path
Returns

pathlib.Path object representing filename.

property rows: List[Dict[str, Any]]
Returns

A list of dictionaries where each item in it represents a row in the CSV file. Each dictionary in the list maps the column heading (fieldname) to the corresponding value for it from the CSV.

rows_from_column_key(column_name: str, rows: Optional[List[Dict[str, Any]]] = None) Dict[str, List[Dict[str, Any]]]

Collect all the rows in the rows parameter that have the same values for the column defined in the column_name parameter, and construct a dictionary with the column_name value as the key and the corresponding rows as a list of dictionaries, as the value of this key.

Parameters
  • column_name (required) – Name of the column that is to be used as the key under which all the rows having the samee value of this column will be collected.

  • rows (optional. If not provided self.rows will be used.) – List of dictionaries representing the rows that will be separated and collected under a the common value of the column name provided in column_name parameter.

Returns

A dictionary constructed using the logic as explained above.

rows_to_nested_dicts(column_order: List[str], rows: Optional[List[Dict[str, Any]]] = None) Dict[str, Any]

Collect all values of columns that are the same and construct a nested dictionary that has the common values as the keys, in the same order of hierarchy as provided in the column_order parameter.

The value of the last column name in the column_order list

Parameters
  • column_order (required) – An ordered list of column names, to be used for constructing the dictionary

  • rows (optional. If not provided self.rows will be used.) – List of dictionaries representing the rows that will be transformed to the output Dictionary.

Returns

A dictionary with same column values collected under a common key in a hierarchical order.

Example:

CSV Contents: fruit_stock.csv

Supplier,Fruit,Origin,Quantity
Big Apples,Apple,Spain,1
Big Melons,Melons,Italy,2
Long Mangoes,Mango,India,3
Small Strawberries,Strawberry,France,4
Short Mangoes,Mango,France,5
Sweet Strawberries,Strawberry,Spain,6
Square Apples,Apple,Italy,7
Small Melons,Melons,Italy,8
Dark Berries,Strawberry,Australia,9
Sweet Berries,Blackcurrant,Australia,10

Create dictionary with hierarchy {"Fruit": [rows]}

from csvio.csvreader import CSVReader
from json import dumps

reader = CSVReader("fruit_stock.csv")

col_order = ["Fruit"]

dict_tree= reader.rows_to_nested_dicts(col_order)

print(dumps(dict_tree, indent=4))

Output:

{
    "Apple": [
        {
            "Supplier": "Big Apples",
            "Fruit": "Apple",
            "Origin": "Spain",
            "Quantity": "1"
        },
        {
            "Supplier": "Square Apples",
            "Fruit": "Apple",
            "Origin": "Italy",
            "Quantity": "7"
        }
    ],
    "Melons": [
        {
            "Supplier": "Big Melons",
            "Fruit": "Melons",
            "Origin": "Italy",
            "Quantity": "2"
        },
        {
            "Supplier": "Small Melons",
            "Fruit": "Melons",
            "Origin": "Italy",
            "Quantity": "8"
        }
    ],
    "Mango": [
        {
            "Supplier": "Long Mangoes",
            "Fruit": "Mango",
            "Origin": "India",
            "Quantity": "3"
        },
        {
            "Supplier": "Short Mangoes",
            "Fruit": "Mango",
            "Origin": "France",
            "Quantity": "5"
        }
    ],
    "Strawberry": [
        {
            "Supplier": "Small Strawberries",
            "Fruit": "Strawberry",
            "Origin": "France",
            "Quantity": "4"
        },
        {
            "Supplier": "Sweet Strawberries",
            "Fruit": "Strawberry",
            "Origin": "Spain",
            "Quantity": "6"
        },
        {
            "Supplier": "Dark Berries",
            "Fruit": "Strawberry",
            "Origin": "Australia",
            "Quantity": "9"
        }
    ],
    "Blackcurrant": [
        {
            "Supplier": "Sweet Berries",
            "Fruit": "Blackcurrant",
            "Origin": "Australia",
            "Quantity": "10"
        }
    ]
}

Create dictionary with hierarchy {"Fruit": "Origin" : [rows]}

from csvio.csvreader import CSVReader
from json import dumps

reader = CSVReader("fruit_stock.csv")

col_order = ["Fruit", "Origin"]

dict_tree= reader.rows_to_nested_dicts(col_order)

print(dumps(dict_tree, indent=4))

Output:

{
    "Apple": {
        "Spain": [
            {
                "Supplier": "Big Apples",
                "Fruit": "Apple",
                "Origin": "Spain",
                "Quantity": "1"
            }
        ],
        "Italy": [
            {
                "Supplier": "Square Apples",
                "Fruit": "Apple",
                "Origin": "Italy",
                "Quantity": "7"
            }
        ]
    },
    "Melons": {
        "Italy": [
            {
                "Supplier": "Big Melons",
                "Fruit": "Melons",
                "Origin": "Italy",
                "Quantity": "2"
            },
            {
                "Supplier": "Small Melons",
                "Fruit": "Melons",
                "Origin": "Italy",
                "Quantity": "8"
            }
        ]
    },
    "Mango": {
        "India": [
            {
                "Supplier": "Long Mangoes",
                "Fruit": "Mango",
                "Origin": "India",
                "Quantity": "3"
            }
        ],
        "France": [
            {
                "Supplier": "Short Mangoes",
                "Fruit": "Mango",
                "Origin": "France",
                "Quantity": "5"
            }
        ]
    },
    "Strawberry": {
        "France": [
            {
                "Supplier": "Small Strawberries",
                "Fruit": "Strawberry",
                "Origin": "France",
                "Quantity": "4"
            }
        ],
        "Spain": [
            {
                "Supplier": "Sweet Strawberries",
                "Fruit": "Strawberry",
                "Origin": "Spain",
                "Quantity": "6"
            }
        ],
        "Australia": [
            {
                "Supplier": "Dark Berries",
                "Fruit": "Strawberry",
                "Origin": "Australia",
                "Quantity": "9"
            }
        ]
    },
    "Blackcurrant": {
        "Australia": [
            {
                "Supplier": "Sweet Berries",
                "Fruit": "Blackcurrant",
                "Origin": "Australia",
                "Quantity": "10"
            }
        ]
    }
}

Construct a dictionary with number of rows for each unique Origin

from csvio.csvreader import CSVReader
from json import dumps

reader = CSVReader("fruit_stock.csv")

col_order = ["Origin"]

origin_fruit_count = {}
dict_tree = reader.rows_to_nested_dicts(col_order)

for origin in dict_tree:
    origin_fruit_count.setdefault(origin, len(dict_tree[origin]))

print(dumps(origin_fruit_count, indent=4))

Output:

{
    "Spain": 2,
    "Italy": 3,
    "India": 1,
    "France": 2,
    "Australia": 2
}
touch(exist_ok: bool = False) bool

Create a blank file at the path provided in the filename parameter.

Parameters

exist_ok (optional) – Parameter to pass to the pathlib.Path.touch() method.

Returns

True If blank file is created successfully.

False On failure.