helpers package¶
Submodules¶
helpers.connections module¶
-
helpers.connections.
execute_sql
(pg_str, sql)¶ Execute a sql query with psycopg2.
- Args:
pg_str: connection string using helper function psycopg_connection_string, returning:
host= port= user= dbname= password=
sql: SQL string in triple quotes:
```CREATE TABLE foo (bar text)```
- Returns:
- Executed sql with conn.cursor().execute(sql)
-
helpers.connections.
get_config
(full_path)¶ Get config file with all login credentials, port numbers, etc.
- Args:
- full_path: provide the full path to the config.ini file, for example authentication/config.ini
- Returns:
- The entire configuration file to use them with
config.get(config_name, 'AUTHURL')
-
helpers.connections.
objectstore_connection
(config_full_path, config_name, print_config_vars=None)¶ Get an objectsctore connection.
- Args:
- config_full_path: /path_to_config/config.ini or config.ini if in root.
- config_name: objectstore
- print_config_vars: if set to True: print all variables from the config file
- Returns:
- An objectstore connection session.
-
helpers.connections.
postgres_engine_pandas
(config_full_path, db_config_name)¶ Pandas uses SQLalchemy, this is the config wrapper to insert config parameters in to_sql queries.
- Args:
- config_full_path: location of the config.ini file including the name of the file, for example authentication/config.ini
- db_config_name: dev or docker to get the ip user/password and port values.
- Returns:
- The postgres pandas engine to do sql queries with.
-
helpers.connections.
psycopg_connection_string
(config_full_path, db_config_name)¶ Postgres connection string for psycopg2.
- Args:
- config_full_path: location of the config.ini file including the name of the file, for example authentication/config.ini
- db_config_name: dev or docker to get the ip user/password and port values.
- Returns:
- Returns the psycopg required connection string: ‘PG:host= port= user= dbname= password=’
helpers.files module¶
-
helpers.files.
create_dir_if_not_exists
(directory)¶ Create directory if it does not yet exists.
- Args:
- Specify the name of directory, for example: dir/anotherdir
- Returns:
- Creates the directory if it does not exists, of return the error message.
-
helpers.files.
save_file
(data, output_folder, filename)¶ save_file currently works with: csv, txt, geojson and json as suffixes. It reads the filename suffix and saves the file as the appropriate type.
- Args:
- data: list of flattened dictionary objects for example: [{id:1, attr:value, attr2:value}, {id:2, attr:value, attr2:value}]
- filename: data_output.csv or data_output.json
- output_folder: dir/anotherdir
- Returns:
- Saved the list of objects to the given geojson or csv type.
-
helpers.files.
unzip
(path, filename_as_folder=False)¶ Find all zip files and unzip in root.
- Args:
- path: set the folder to check for zip files.
- filename_as_folder:Set it to True to unzip to subfolders with name of zipfile instead of in the root folder.
- Returns:
- Unzipped files in the path directory or in the path/name of the zip file.
helpers.json_dict_handlers module¶
-
helpers.json_dict_handlers.
clean_dict
(dictionary, key_name)¶ Remove a field from a dict based on key name. Args:
- dictionary: {id:1, dates:2018-12-02}
- key_name: ‘dates’
- Returns:
- {id:1}
-
helpers.json_dict_handlers.
flatten_json
(json_object)¶ Flatten nested json Object. Args:
1 json_object, for example: {“key”: “subkey”: { “subsubkey”:”value” }}- Returns:
- {“key.subkey.subsubkey”:”value”}
- Source:
- https://towardsdatascience.com/flattening-json-objects-in-python-f5343c794b10
-
helpers.json_dict_handlers.
joinByKeyNames
(geojson, dataset, key1, key2)¶ Insert data from dataset to geojson where key1 from dataset matches key2 from geojson
-
helpers.json_dict_handlers.
jsonPoints2geojson
(df, latColumn, lonColumn)¶ Convert JSON with lat/lon columns to geojson. https://gis.stackexchange.com/questions/220997/pandas-to-geojson-multiples-points-features-with-python
-
helpers.json_dict_handlers.
openJsonArrayKeyDict2FlattenedJson
(fileName)¶ Open json and return array of objects without object value name. For example: [{‘container’:{…}}, {‘container’:{…}}] returns now as [{…},{…}])
helpers.logging module¶
-
helpers.logging.
logger
()¶ Setup basic logging for console.
- Usage:
- Initialize the logger by adding the code at the top of your script:
logger = logger()
TODO: add log file export