FTPReader
- class csvio.remote.FTPReader(options: Dict[str, Any], processors: Optional[List[csvio.processors.processor_base.ProcessorBase]] = None, fieldnames: List[str] = [], open_kwargs: Dict[str, Any] = {}, csv_kwargs: Dict[str, Any] = {}, max_retries: int = 3, timeout: int = 5, increment_timeout: int = 10)
Bases:
csvio.remote.remote_base.RemoteReaderBase
Download CSV file from a FTP location.
- Parameters
options (required) – Options that are to be used by the downloader child class that is derived from this base class
The
options
parameter to the constructor of this class accepts a dictionary with the options to download a CSV file from a FTP locationExample FTP options dictionary:
{ "hostname": "localhost", "username": "ftp_user", "password": "ftp_pass", "remote_dir": "data/csv_files" "remote_file": "products.csv" }
Example CSV download from FTP:
from csvio.remote import FTPReader from json import dumps ftp_details = { "hostname": "localhost", "username": "admin", "password": "admin", "remote_dir": "data", "remote_file": "fruit_stock.csv" } ftpreader = FTPReader(ftp_details) print(ftpreader.filepath) print() print(ftpreader.fieldnames) print() print(dumps(ftpreader.rows, indent=4))
Output:
C:\Users\User\AppData\Local\Temp\tmpn7oxqdu0.csv ['Supplier', 'Fruit', 'Origin', 'Quantity'] [ { "Supplier": "Big Apples", "Fruit": "Apple", "Origin": "Spain", "Quantity": "1" }, { "Supplier": "Big Melons", "Fruit": "Melons", "Origin": "Italy", "Quantity": "2" }, { "Supplier": "Long Mangoes", "Fruit": "Mango", "Origin": "India", "Quantity": "3" }, { "Supplier": "Small Strawberries", "Fruit": "Strawberry", "Origin": "France", "Quantity": "4" }, { "Supplier": "Short Mangoes", "Fruit": "Mango", "Origin": "France", "Quantity": "5" }, { "Supplier": "Sweet Strawberries", "Fruit": "Strawberry", "Origin": "Spain", "Quantity": "6" }, { "Supplier": "Square Apples", "Fruit": "Apple", "Origin": "Italy", "Quantity": "7" }, { "Supplier": "Small Melons", "Fruit": "Melons", "Origin": "Italy", "Quantity": "8" }, { "Supplier": "Dark Berries", "Fruit": "Strawberry", "Origin": "Australia", "Quantity": "9" }, { "Supplier": "Sweet Berries", "Fruit": "Blackcurrant", "Origin": "Australia", "Quantity": "10" } ]
- property csv_kwargs: Dict[str, Any]
- Returns
A dictionary of key, value pairs that should be passed to the DictReader constructor within this class.
- delete(missing_ok: bool = False) bool
Delete the file at the path provided in the filename parameter
- Parameters
missing_ok (optional) – Parameter to pass to the
pathlib.Path.unlink()
method.- Returns
True
If file is deleted successfully.False
On failure.
- property fieldnames: List[str]
- Returns
List of column headings
- property file_ext: str
- Returns
Extension suffix of the file without parent directory and file name.
- property filedir: str
- Returns
Parent directory path of the file (excluding the name of the file)
- property filename: str
- Returns
File name without the parent directory path.
- property filename_no_ext: str
- Returns
File name without parent directory and file extension.
- property filepath: str
- Returns
Complete file path including the parent directory, file name and extension
- get_temp_path() str
- Returns
Full path to a randomly named csv file in the system’s temporary directory.
- property num_rows: int
- Returns
The total number of rows in the CSV (excluding column headings)
- property open_kwargs: Dict[str, Any]
- Returns
A dictionary of key, value pairs that should be passed to the open method within this class.
- property path_obj: pathlib.Path
- Returns
pathlib.Path
object representing filename.
- property rows: List[Dict[str, Any]]
- Returns
A list of dictionaries where each item in it represents a row in the CSV file. Each dictionary in the list maps the column heading (fieldname) to the corresponding value for it from the CSV.
- rows_from_column_key(column_name: str, rows: Optional[List[Dict[str, Any]]] = None) Dict[str, List[Dict[str, Any]]]
Collect all the rows in the
rows
parameter that have the same values for the column defined in thecolumn_name
parameter, and construct a dictionary with thecolumn_name
value as the key and the corresponding rows as a list of dictionaries, as the value of this key.- Parameters
column_name (required) – Name of the column that is to be used as the key under which all the rows having the samee value of this column will be collected.
rows (optional. If not provided
self.rows
will be used.) – List of dictionaries representing the rows that will be separated and collected under a the common value of the column name provided incolumn_name
parameter.
- Returns
A dictionary constructed using the logic as explained above.
- rows_to_nested_dicts(column_order: List[str], rows: Optional[List[Dict[str, Any]]] = None) Dict[str, Any]
Collect all values of columns that are the same and construct a nested dictionary that has the common values as the keys, in the same order of hierarchy as provided in the column_order parameter.
The value of the last column name in the column_order list
- Parameters
column_order (required) – An ordered list of column names, to be used for constructing the dictionary
rows (optional. If not provided
self.rows
will be used.) – List of dictionaries representing the rows that will be transformed to the output Dictionary.
- Returns
A dictionary with same column values collected under a common key in a hierarchical order.
Example:
CSV Contents: fruit_stock.csv
Supplier,Fruit,Origin,Quantity Big Apples,Apple,Spain,1 Big Melons,Melons,Italy,2 Long Mangoes,Mango,India,3 Small Strawberries,Strawberry,France,4 Short Mangoes,Mango,France,5 Sweet Strawberries,Strawberry,Spain,6 Square Apples,Apple,Italy,7 Small Melons,Melons,Italy,8 Dark Berries,Strawberry,Australia,9 Sweet Berries,Blackcurrant,Australia,10
Create dictionary with hierarchy
{"Fruit": [rows]}
from csvio.csvreader import CSVReader from json import dumps reader = CSVReader("fruit_stock.csv") col_order = ["Fruit"] dict_tree= reader.rows_to_nested_dicts(col_order) print(dumps(dict_tree, indent=4))
Output:
{ "Apple": [ { "Supplier": "Big Apples", "Fruit": "Apple", "Origin": "Spain", "Quantity": "1" }, { "Supplier": "Square Apples", "Fruit": "Apple", "Origin": "Italy", "Quantity": "7" } ], "Melons": [ { "Supplier": "Big Melons", "Fruit": "Melons", "Origin": "Italy", "Quantity": "2" }, { "Supplier": "Small Melons", "Fruit": "Melons", "Origin": "Italy", "Quantity": "8" } ], "Mango": [ { "Supplier": "Long Mangoes", "Fruit": "Mango", "Origin": "India", "Quantity": "3" }, { "Supplier": "Short Mangoes", "Fruit": "Mango", "Origin": "France", "Quantity": "5" } ], "Strawberry": [ { "Supplier": "Small Strawberries", "Fruit": "Strawberry", "Origin": "France", "Quantity": "4" }, { "Supplier": "Sweet Strawberries", "Fruit": "Strawberry", "Origin": "Spain", "Quantity": "6" }, { "Supplier": "Dark Berries", "Fruit": "Strawberry", "Origin": "Australia", "Quantity": "9" } ], "Blackcurrant": [ { "Supplier": "Sweet Berries", "Fruit": "Blackcurrant", "Origin": "Australia", "Quantity": "10" } ] }
Create dictionary with hierarchy
{"Fruit": "Origin" : [rows]}
from csvio.csvreader import CSVReader from json import dumps reader = CSVReader("fruit_stock.csv") col_order = ["Fruit", "Origin"] dict_tree= reader.rows_to_nested_dicts(col_order) print(dumps(dict_tree, indent=4))
Output:
{ "Apple": { "Spain": [ { "Supplier": "Big Apples", "Fruit": "Apple", "Origin": "Spain", "Quantity": "1" } ], "Italy": [ { "Supplier": "Square Apples", "Fruit": "Apple", "Origin": "Italy", "Quantity": "7" } ] }, "Melons": { "Italy": [ { "Supplier": "Big Melons", "Fruit": "Melons", "Origin": "Italy", "Quantity": "2" }, { "Supplier": "Small Melons", "Fruit": "Melons", "Origin": "Italy", "Quantity": "8" } ] }, "Mango": { "India": [ { "Supplier": "Long Mangoes", "Fruit": "Mango", "Origin": "India", "Quantity": "3" } ], "France": [ { "Supplier": "Short Mangoes", "Fruit": "Mango", "Origin": "France", "Quantity": "5" } ] }, "Strawberry": { "France": [ { "Supplier": "Small Strawberries", "Fruit": "Strawberry", "Origin": "France", "Quantity": "4" } ], "Spain": [ { "Supplier": "Sweet Strawberries", "Fruit": "Strawberry", "Origin": "Spain", "Quantity": "6" } ], "Australia": [ { "Supplier": "Dark Berries", "Fruit": "Strawberry", "Origin": "Australia", "Quantity": "9" } ] }, "Blackcurrant": { "Australia": [ { "Supplier": "Sweet Berries", "Fruit": "Blackcurrant", "Origin": "Australia", "Quantity": "10" } ] } }
Construct a dictionary with number of rows for each unique
Origin
from csvio.csvreader import CSVReader from json import dumps reader = CSVReader("fruit_stock.csv") col_order = ["Origin"] origin_fruit_count = {} dict_tree = reader.rows_to_nested_dicts(col_order) for origin in dict_tree: origin_fruit_count.setdefault(origin, len(dict_tree[origin])) print(dumps(origin_fruit_count, indent=4))
Output:
{ "Spain": 2, "Italy": 3, "India": 1, "France": 2, "Australia": 2 }
- touch(exist_ok: bool = False) bool
Create a blank file at the path provided in the filename parameter.
- Parameters
exist_ok (optional) – Parameter to pass to the
pathlib.Path.touch()
method.- Returns
True
If blank file is created successfully.False
On failure.