utils module

Author:Dominic Hunt

exception utils.ClassNameError

Bases: Exception

exception utils.FunctionNameError

Bases: Exception

utils.argProcess(**kwargs)

utils.callableDetails(item)

Takes a callable item and extracts the details.

Currently only extracts things stored in item.Name and item.Params

Parameters:item (callable item) – Returns:details – Contains the properties of the Return type:tuple pair with string and dictionary of strings

Examples

>>> from utils import callableDetails
>>> def foo(): print("foo")
>>> foo.Name = "boo"
>>> callableDetails(foo)
('boo', None)
>>> foo.Params = {1: 2, 2: 3}
>>> callableDetails(foo)
('boo', {'1': '2', '2': '3'})

utils.callableDetailsString(item)

Takes a callable item and returns a string detailing the function.

Currently only extracts things stored in item.Name and item.Params

Parameters:item (callable item) – Returns:description – Contains the properties and name of the callable Return type:string

Examples

>>> from utils import callableDetailsString
>>> def foo(): print("foo")
>>> foo.Name = "boo"
>>> callableDetailsString(foo)
'boo'
>>> foo.Params = {1: 2, 2: 3}
>>> callableDetailsString(foo)
'boo with 1 : 2, 2 : 3'

utils.date()

Provides a string of today’s date

Returns:date – The string is of the form [year]-[month]-[day] Return type:string

utils.discountAverage(data, discount)

An accumulating mean

Parameters:

• data (list or 1-D array of floats) – The set of values to be averaged
• discount (float) – The value by which each previous value is discounted

Returns:

results – The values from the moving average

Return type:

ndArray of length data

Examples

>>> discountAverage([1, 2, 3, 4], 1)
array([1. , 1.5, 2. , 2.5])
>>> discountAverage([1, 2, 3, 4], 0.25)
array([1.        , 1.8       , 2.71428571, 3.68235294])

utils.errorResp()

Takes an error that has been caught and returns the details as a string

Returns:description – Contains the description of the error Return type:string

utils.find_class(class_name, class_folder, inherited_class, excluded_files=None)

Finds and imports a class from a given folder. Does not look in subfolders

Parameters:

• class_name (string) – The name of the class to be used
• class_folder (str) – The path where the class is likely to be found
• inherited_class (class) – The class that the searched for class inherits from
• excluded_files (list, optional) – A list of modules to be excluded from the search. Can be described using portions of file names.

Returns:

sought_class – The uninstansiated class sought

Return type:

inherited_class

utils.find_function(function_name, function_folder, excluded_files=None)

Finds and imports a function from a given folder. Does not look in subfolders

Parameters:

• function_name (string) – The name of the function to be used
• function_folder (str) – The path where the function is likely to be found
• excluded_files (list, optional) – A list of modules to be excluded from the search. Can be described using portions of file names.

Returns:

sought_class – The uninstansiated class sought

Return type:

inherited_class

utils.flatten(data)

Yields the elements in order from any N dimensional iterable

Parameters:data (iterable) – Yields:ID ((string,list)) – A pair containing the value at each location and the co-ordinates used to access them.

Examples

>>> a = [[1, 2, 3],[4, 5, 6]]
>>> for i, loc in flatten(a): print(i,loc)
1 [0, 0]
2 [0, 1]
3 [0, 2]
4 [1, 0]
5 [1, 1]
6 [1, 2]

utils.get_class_args(inspected_class, arg_ignore=['self'])

Finds the arguments that could be passed into the specified class

utils.get_class_attributes(inspected_class, ignore=['self'])

Finds the public attributes of the specified class

utils.get_function_args(inspected_function)

Finds the arguments that could be passed into the specified function

Parameters:inspected_function – Returns:

utils.kendalw(data, ranked=False)

Calculates Kendall’s W for a n*m array with n items and m ‘judges’.

Parameters:

• data (list or np.ndarray) – The data in the form of an n*m array with n items and m ‘judges’
• ranked (bool, optional) – If the data has already been ranked or not. Default False

Returns:

w – The Kendall’s W

Return type:

float

Notes

Based on Legendre, P. (2010). Coefficient of Concordance. In Encyclopedia of Research Design (pp. 164–169). 2455 Teller Road, Thousand Oaks California 91320 United States: SAGE Publications, Inc. http://doi.org/10.4135/9781412961288.n55

Examples

>>> data = np.array([[2., 0., 5., 1.], [3., 3., 3., 4.], [1., 5., 3., 5.], [1., 1., 4., 2.], [2., 4., 5., 1.], [1., 0., 0., 2.]])

>>> kendalw(data)
0.22857142857142856

>>> data = np.array([[1, 1, 1, 1], [2, 2, 2, 2], [3, 3, 3, 3], [4, 4, 4, 4], [5, 5, 5, 5], [6, 6, 6, 6]])

>>> kendalw(data)
1.0

utils.kendalwt(data, ranked=False)

Calculates Kendall’s W for a n*m array with n items and m ‘judges’. Corrects for ties.

Parameters:

• data (list or np.ndarray) – The data in the form of an n*m array with n items and m ‘judges’
• ranked (bool, optional) – If the data has already been ranked or not. Default False

Returns:

w – The Kendall’s W

Return type:

float

Notes

Based on Legendre, P. (2010). Coefficient of Concordance. In Encyclopedia of Research Design (pp. 164–169). 2455 Teller Road, Thousand Oaks California 91320 United States: SAGE Publications, Inc. http://doi.org/10.4135/9781412961288.n55

Examples

>>> data = np.array([[2., 0., 5., 1.], [3., 3., 3., 4.], [1., 5., 3., 5.], [1., 1., 4., 2.], [2., 4., 5., 1.], [1., 0., 0., 2.]])
>>> kendalwt(data)
0.24615384615384617

>>> data = np.array([[1, 1, 1, 1], [2, 2, 2, 2], [3, 3, 3, 3], [4, 4, 4, 4], [5, 5, 5, 5], [6, 6, 6, 6]])
>>> kendalwt(data)
1.0

utils.kendalwts(data, ranked=False)

Calculates Kendall’s W for a n*m array with n items and m ‘judges’. Corrects for ties.

Parameters:

• data (list or np.ndarray) – The data in the form of an n*m array with n items and m ‘judges’
• ranked (bool, optional) – If the data has already been ranked or not. Default False

Returns:

w – The Kendall’s W

Return type:

float

Notes

Based on Legendre, P. (2010). Coefficient of Concordance. In Encyclopedia of Research Design (pp. 164–169). 2455 Teller Road, Thousand Oaks California 91320 United States: SAGE Publications, Inc. http://doi.org/10.4135/9781412961288.n55

Examples

>>> data = np.array([[2., 0., 5., 1.], [3., 3., 3., 4.], [1., 5., 3., 5.], [1., 1., 4., 2.], [2., 4., 5., 1.], [1., 0., 0., 2.]])
>>> kendalwts(data)
0.24615384615384617

>>> data = np.array([[1, 1, 1, 1], [2, 2, 2, 2], [3, 3, 3, 3], [4, 4, 4, 4], [5, 5, 5, 5], [6, 6, 6, 6]])
>>> kendalwts(data)
1.0

utils.kldivergence(m0, m1, c0, c1)

Calculates the Kullback–Leibler divergence between two distributions using the means and covariances

Parameters:

• m0 (array of N floats) – The means of distribution 0
• m1 (array of N floats) – The means of distribution 1
• c0 (NxN array of floats) – The covariance matrix for distribution 0
• c1 (NxN array of floats) – The covariance matrix for distribution 1

Returns:

kl – The Kullback–Leibler divergence

Return type:

float

utils.listMerge(*args)

For merging lists with objects that are not solely numbers

Parameters:args (list of lists) – A list of 1D lists of objects Returns:combinations – An np.array with len(args) columns and a row for each combination Return type:np.array

Examples

>>> listMerge([1, 2, 3], [5, 6, 7]).T
array([[1, 2, 3, 1, 2, 3, 1, 2, 3],
       [5, 5, 5, 6, 6, 6, 7, 7, 7]])

utils.listMergeGen(*args)

Fast merging of lists of numbers

Parameters:args (list of lists of numbers) – A list of 1D lists of numbers Yields:combination (numpy.array of 1 x len(args)) – Array of all combinations

Examples

>>> for i in listMergeGen(0.7): print(repr(i))
array([0.7])
>>> for i in listMergeGen([0.7, 0.1]): print(repr(i))
array([0.7])
array([0.1])
>>> for i in listMergeGen([0.7, 0.1], [0.6]): print(repr(i))
array([0.7,  0.6])
array([0.1,  0.6])
>>> for i in listMergeGen([0.7, 0.1], []): print(repr(i))

>>> for i in listMergeGen([0.7, 0.1], 0.6): print(repr(i))
array([0.7,  0.6])
array([0.1,  0.6])

utils.listMergeNP(*args)

Fast merging of lists of numbers

Parameters:args (list of lists of numbers) – A list of 1D lists of numbers Returns:combinations – An np.array with len(args) columns and a row for each combination Return type:np.array

Examples

>>> listMergeNP([1, 2, 3], [5, 6, 7]).T
array([[1, 2, 3, 1, 2, 3, 1, 2, 3], [5, 5, 5, 6, 6, 6, 7, 7, 7]])

utils.list_all_equal(data)

Checks if all of the elements of a list are the same.

Parameters:data (list of 1D) – The list of elements to compare Returns:equivalence – True if the elements are all the same Return type:bool

Notes

Based on https://stackoverflow.com/questions/3844801

utils.mergeDatasetRepr(data, dataLabel='')

Take a list of dictionaries and turn it into a dictionary of lists of strings

Parameters:

• data (list of dicts containing strings, lists or numbers) –
• dataLabel (string, optional) – This string will be appended to the front of each key in the new dataset Default blank

Returns:

newStore – For each key a list will be formed of the string representations of each of the former key values.

Return type:

dictionary of lists of strings

utils.mergeDatasets(data, extend=False)

Take a list of dictionaries and turn it into a dictionary of lists of objects

Parameters:

• data (list of dicts containing strings, lists or numbers) –
• extend (bool, optional) – If lists should be extended rather than appended. Default False

Returns:

newStore – For each key a list will be formed of the former key values. If a data set did not contain a key a value of None will be entered for it.

Return type:

dictionary of lists of objects

utils.mergeDicts(*args)

Merges any number of dictionaries with different keys into a new dict

Precedence goes to key value pairs in latter dicts

Parameters:args (list of dictionaries) – Returns:mergedDict Return type:dictionary

utils.mergeTwoDicts(x, y)

Given two dicts, merge them into a new dict as a shallow copy

Assumes different keys in both dictionaries

Parameters:

• x (dictionary) –
• y (dictionary) –

Returns:

mergedDict

Return type:

dictionary

utils.movingaverage(data, windowSize, edgeCorrection=False)

Average over an array

Parameters:

• data (list of floats) – The data to average
• windowSize (int) – The size of the window
• edgeCorrection (bool) – If True the edges are repaired so that there is no unusual dropoff

Returns:

convolution

Return type:

array

Examples

>>> movingaverage([1, 1, 1, 1, 1], 3)
array([0.66666667, 1.        , 1.        , 1.        , 0.66666667])
>>> movingaverage([1, 1, 1, 1, 1, 1, 1, 1], 4)
array([0.5 , 0.75, 1.  , 1.  , 1.  , 1.  , 1.  , 0.75])
>>> movingaverage([1, 1, 1, 1, 1], 3, edgeCorrection=True)
array([1., 1., 1., 1., 1.])

utils.runningAverage(data)

An accumulating mean

Parameters:data (list or 1-D array of floats) – The set of values to be averaged Returns:results – The values from the moving average Return type:ndArray of length data

Examples

>>> runningAverage([1,2,3,4])
array([1. ,  1.5,  2. ,  2.5])

utils.runningMean(oldMean, newValue, numValues)

A running mean

Parameters:

• oldMean (float) – The old running average mean
• newValue (float) – The new value to be added to the mean
• numValues (int) – The number of values in the new running mean once this value is included

Returns:

newMean – The new running average mean

Return type:

float

Notes

Based on Donald Knuth’s Art of Computer Programming, Vol 2, page 232, 3rd edition and taken from https://www.johndcook.com/blog/standard_deviation/

Examples

>>> runningMean(1, 2, 2)
1.5
>>> runningMean(1.5, 3, 3)
2.0

utils.runningSTD(oldSTD, oldMean, newMean, newValue)

Parameters:

• oldSTD (float) – The old running average standard deviation
• oldMean (float) – The old running average mean
• newMean (float) – The new running average mean
• newValue (float) – The new value to be added to the mean

Returns:

newSTD – The new running average standard deviation

Return type:

float

Notes

Based on Donald Knuth’s Art of Computer Programming, Vol 2, page 232, 3rd edition (which is based on B. P. Welford (2012) Note on a Method for Calculating Corrected Sums of Squares and Products, Technometrics, 4:3, 419-420, DOI: 10.1080/00401706.1962.10490022 This version is taken from https://www.johndcook.com/blog/standard_deviation/

Examples

>>> runningSTD(0, 1, 1.5, 2)
0.5

>>> runningSTD(0.5, 1.5, 2.0, 3)
2.0

utils.unique(seq, idfun=None)

Finds the unique items in a list and returns them in order found.

Inspired by discussion on http://www.peterbe.com/plog/uniqifiers-benchmark Notably f10 Andrew Dalke and f8 by Dave Kirby

Parameters:

• seq (an iterable object) – The sequence from which the unique list will be compiled
• idfun (function, optional) – A hashing function for transforming the items into the form that is to be compared. Default is the None

Returns:

result – The list of unique items

Return type:

list

Examples

>>> a=list('ABeeE')
>>> unique(a)
['A', 'B', 'e', 'E']

>>> unique(a, lambda x: x.lower())
['A', 'B', 'e']

Note

Unless order is needed it is best to use list(set(seq))

utils.varyingParams(intObjects, params)

Takes a list of models or tasks and returns a dictionary with only the parameters which vary and their values