outputting module

Author:Dominic Hunt
class outputting.LoggerWriter(writer)[source]

Bases: object

Fake file-like stream object that redirects writes to a logger instance. Taken from https://stackoverflow.com/a/51612402

Parameters:writer (logging function) –
flush()[source]
write(message)[source]
class outputting.Saving(label=None, output_path=None, config=None, config_file=None, pickle_store=False, min_log_level='INFO', numpy_error_level='log')[source]

Bases: object

Creates the folder structure for the saved data and created the log file as log.txt

Parameters:
  • label (string, optional) – The label for the simulation. Default None will mean no data is saved to files.
  • output_path (string, optional) – The path that will be used for the run output. Default None
  • config (dict, optional) – The parameters of the running simulation/fitting. This is used to create a YAML configuration file. Default None
  • config_file (string, optional) – The file name and path of a .yaml configuration file. Default None
  • pickle_store (bool, optional) – If true the data for each model, task and participant is recorded. Default is False
  • min_log_level (str, optional) – Defines the level of the log from (DEBUG, INFO, WARNING, ERROR, CRITICAL). Default INFO See https://docs.python.org/3/library/logging.html#levels
  • numpy_error_level ({'log', 'raise'}) – Defines the response to numpy errors. Default log. See numpy.seterr
Returns:

file_name_gen – Creates a new file with the name <handle> and the extension <extension>. It takes two string parameters: (handle, extension) and returns one fileName string
Return type:

function

See also

folderSetup
creates the folders
outputting.date()[source]

Calculate today’s date as a string in the form <year>-<month>-<day> and returns it

Returns:date_today – The current date in the format <year>-<month>-<day>
Return type:str
outputting.dictKeyGen(store, maxListLen=None, returnList=False, abridge=False)[source]

Identifies the columns necessary to convert a dictionary into a table

Parameters:
  • store (dict) – The dictionary to be broken down into keys
  • maxListLen (int or float with no decimal places or None, optional) – The length of the longest expected list. Only useful if returnList is True. Default None
  • returnList (bool, optional) – Defines if the lists will be broken into 1D lists or values. Default False, lists will be broken into values
  • abridge (bool, optional) – Defines if the final dataset will be a summary or the whole lot. If it is a summary, lists of more than 10 elements are removed. Default False, not abridged
Returns:

  • keySet (dict with values of dict, list or None) – The dictionary of keys to be extracted
  • maxListLen (int or float with no decimal places or None, optional) – If returnList is True this should be the length of the longest list. If returnList is False this should return its original value

Examples

>>> store = {'string': 'string'}
>>> dictKeyGen(store)
({'string': None}, 1)
>>> store = {'num': 23.6}
>>> dictKeyGen(store)
({'num': None}, 1)
>>> store = {'array': np.array([[1, 2, 3, 4, 5, 6], [7, 8, 9, 10, 11, 12]])}
>>> dictKeyGen(store, returnList=True, abridge=True)
({'array': array([[0],
       [1]])}, 6)
>>> store = {'dict': {1: "a", 2: "b"}}
>>> dictKeyGen(store, maxListLen=7, returnList=True, abridge=True)
({'dict': {1: None, 2: None}}, 7)
outputting.fancy_logger(log_file=None, log_level=10, numpy_error_level='log')[source]

Sets up the style of logging for all the simulations

Parameters:
  • log_file (string, optional) – Provides the path the log will be written to. Default “./log.txt”
  • log_level ({logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL}) – Defines the level of the log. Default logging.INFO
  • numpy_error_level ({'log', 'raise'}) – Defines the response to numpy errors. Default log. See numpy.seterr
Returns:

close_loggers – Closes the logging systems that have been set up
Return type:

function

See also

logging()
The Python standard logging library
numpy.seterr()
The function npErrResp is passed to for defining the response to numpy errors
outputting.file_name_generator(output_folder=None)[source]

Keeps track of filenames that have been used and generates the next unused one

Parameters:output_folder (string, optional) – The folder into which the new file will be placed. Default is the current working directory
Returns:new_file_name – Creates a new file with the name <handle> and the extension <extension>. It takes two string parameters: (handle, extension) and returns one fileName string
Return type:function

Examples

>>> file_name_gen = file_name_generator("./")
>>> file_name_gen("a", "b")
'./a.b'
>>> file_name_gen("a", "b")
'./a_1.b'
>>> file_name_gen("", "")
'./'
>>> file_name_gen = file_name_generator()
>>> fileName = file_name_gen("", "")
>>> fileName == os.getcwd()
False
outputting.flatDictKeySet(store, selectKeys=None)[source]

Generates a dictionary of keys and identifiers for the new dictionary, including only the keys in the keys list. Any keys with lists will be split into a set of keys, one for each element in the original key.

These are named <key><location>

Parameters:
  • store (list of dicts) – The dictionaries would be expected to have many of the same keys. Any dictionary keys containing lists in the input have been split into multiple numbered keys
  • selectKeys (list of strings, optional) – The keys whose data will be included in the return dictionary. Default None, which results in all keys being returned
Returns:

keySet – The dictionary of keys to be extracted
Return type:

dict with values of dict, list or None

See also

reframeListDicts(), newFlatDict()

outputting.folder_path_cleaning(folder)[source]

Modifies string file names from Windows format to Unix format if necessary and makes sure there is a / at the end.

Parameters:folder (string) – The folder path
Returns:folder_path – The folder path
Return type:str
outputting.folder_setup(label, date_string, pickle_data=False, base_path=None)[source]

Identifies and creates the folder the data will be stored in

Folder will be created as “./Outputs/<sim_label>_<date>/”. If that had previously been created then it is created as “./Outputs/<sim_label>_<date>_no_<#>/”, where “<#>” is the first available integer.

A subfolder is also created with the name Pickle if pickle is true.

Parameters:
  • label (str) – The label for the simulation
  • date_string (str) – The date identifier
  • pickle_data (bool, optional) – If true the data for each model, task and participant is recorded. Default is False
  • base_path (str, optional) – The path into which the new folder will be placed. Default is current working directory
Returns:

folder_name – The folder path that has just been created
Return type:

string

See also

newFile()
Creates a new file
saving()
Creates the log system
outputting.listKeyGen(data, maxListLen=None, returnList=False, abridge=False)[source]

Identifies the columns necessary to convert a list into a table

Parameters:
  • data (numpy.ndarray or list) – The list to be broken down
  • maxListLen (int or float with no decimal places or None, optional) – The length of the longest expected list. Only useful if returnList is True. Default None
  • returnList (bool, optional) – Defines if the lists will be broken into 1D lists or values. Default False, lists will be broken into values
  • abridge (bool, optional) – Defines if the final dataset will be a summary or the whole lot. If it is a summary, lists of more than 10 elements are removed. Default False, not abridged
Returns:

  • returnList (None or list of tuples of ints or ints) – The list of co-ordinates for the elements to be extracted from the data. If None the list is used as-is.
  • maxListLen (int or float with no decimal places or None, optional) – If returnList is True this should be the length of the longest list. If returnList is False this should return its original value

Examples

>>> listKeyGen([[1, 2, 3, 4, 5, 6], [4, 5, 6, 7, 8, 9]], maxListLen=None, returnList=False, abridge=False)
(array([[0, 0], [1, 0], [0, 1], [1, 1], [0, 2], [1, 2], [0, 3], [1, 3], [0, 4], [1, 4], [0, 5], [1, 5]]), 1)
>>> listKeyGen([[1, 2, 3, 4, 5, 6], [4, 5, 6, 7, 8, 9]], maxListLen=None, returnList=False, abridge=True)
(None, None)
>>> listKeyGen([[1, 2, 3, 4, 5, 6], [4, 5, 6, 7, 8, 9]], maxListLen=None, returnList=True, abridge=True)
(array([[0],
       [1]]), 6)
outputting.listSelection(data, loc)[source]

Allows numpy array-like referencing of lists

Parameters:
  • data (list) – The data to be referenced
  • loc (tuple of integers) – The location to be referenced
Returns:

selection – The referenced subset
Return type:

list

Examples

>>> listSelection([1, 2, 3], (0,))
1
>>> listSelection([[1, 2, 3], [4, 5, 6]], (0,))
[1, 2, 3]
>>> listSelection([[1, 2, 3], [4, 5, 6]], (0, 2))
3
outputting.newFlatDict(store, selectKeys=None, labelPrefix='')[source]

Takes a list of dictionaries and returns a dictionary of 1D lists.

If a dictionary did not have that key or list element, then ‘None’ is put in its place

Parameters:
  • store (list of dicts) – The dictionaries would be expected to have many of the same keys. Any dictionary keys containing lists in the input have been split into multiple numbered keys
  • selectKeys (list of strings, optional) – The keys whose data will be included in the return dictionary. Default None, which results in all keys being returned
  • labelPrefix (string) – An identifier to be added to the beginning of each key string.
Returns:

newStore – The new dictionary with the keys from the keySet and the values as 1D lists with ‘None’ if the keys, value pair was not found in the store.
Return type:

dict

Examples

>>> store = [{'list': [1, 2, 3, 4, 5, 6]}]
>>> newFlatDict(store)
{'list_[0]': [1], 'list_[1]': [2], 'list_[2]': [3], 'list_[3]': [4], 'list_[4]': [5], 'list_[5]': [6]}
>>> store = [{'string': 'string'}]
>>> newFlatDict(store)
{'string': ["'string'"]}
>>> store = [{'dict': {1: {3: "a"}, 2: "b"}}]
>>> newFlatDict(store)
{'dict_1_3': ["'a'"], 'dict_2': ["'b'"]}
outputting.newListDict(store, labelPrefix='', maxListLen=0)[source]

Takes a dictionary of numbers, strings, lists and arrays and returns a dictionary of 1D arrays.

If there is a single value, then a list is created with that value repeated

Parameters:
  • store (dict) – A dictionary of numbers, strings, lists, dictionaries and arrays
  • labelPrefix (string) – An identifier to be added to the beginning of each key string. Default empty string
Returns:

newStore – The new dictionary with the keys from the keySet and the values as 1D lists.
Return type:

dict

Examples

>>> store = {'list': [1, 2, 3, 4, 5, 6]}
>>> newListDict(store)
{'list': [1, 2, 3, 4, 5, 6]}
>>> store = {'string': 'string'}
>>> newListDict(store)
{'string': ['string']}
>>> store = {'dict': {1: {3: "a"}, 2: "b"}}
>>> newListDict(store)
{'dict_1_3': ['a'], 'dict_2': ['b']}
outputting.pad(values, maxListLen)[source]

Pads a list with None

Parameters:
  • values (list) – The list to be extended
  • maxListLen (int) – The number of elements the list needs to have
outputting.pickleLog(results, file_name_gen, label='')[source]

Stores the data in the appropriate pickle file in a Pickle subfolder of the outputting folder

Parameters:
  • results (dict) – The data to be stored
  • file_name_gen (function) – Creates a new file with the name <handle> and the extension <extension>. It takes two string parameters: (handle, extension) and returns one fileName string
  • label (string, optional) – A label for the results file
outputting.pickle_write(data, handle, file_name_gen)[source]

Writes the data to a pickle file

Parameters:
  • data (object) – Data to be written to the file
  • handle (string) – The name of the file
  • file_name_gen (function) – Creates a new file with the name <handle> and the extension <extension>. It takes two string parameters: (handle, extension) and returns one fileName string