Utilpy Packages¶

utilpy.utils¶

utilpy.utils.chunks(l, n)[source]¶

Yield successive n-sized chunks from l.

Args:

l:list: list of data
n:int: n-sized

Return:

list of list: [[]]

utilpy.utils.clean_str(string)[source]¶

Tokenization/string cleaning for all datasets except for SST. Original taken from https://github.com/yoonkim/CNN_sentence/blob/master/process_data.py

Args:

string:str: string of sentences

Returns:

Lowercase cleaned string:str

utilpy.utils.download_file(url, save_dir='tmp/', filename=None)[source]¶

download and save the file

Args:

url:str: url of file
save_dir:str: directory to save
filename:str: Name of file to save as

utilpy.utils.download_files(urls, save_dir='tmp/')[source]¶

download all files from urls

Args:

urls:list: urls of file
save_dir:str: directory to save

utilpy.utils.format_filename(s)[source]¶

Take a string and return a valid filename constructed from the string. Uses a whitelist approach: any characters not present in valid_chars are removed. Also spaces are replaced with underscores.

Args:

s:str: string

Returns:

Valid filename : str

utilpy.utils.logger(logger_name='utilpy.utils', filename='log.log', level=10)[source]¶

logger for logging

Args:

logger_name:str: name of logger
filename:str: path and file name
level:str: logging level

Return:

logger funcation for logging:logger

utilpy.utils.page_soup(url)[source]¶

Make BeautifulSoup of page

Args:

url:str: url of page

Returns:

BeautifulSoup of page:soup

utilpy.utils.read_config(filename)[source]¶

Parse config yaml file

Args:

filename:str: path of config file

utilpy.utils.thread_pool(func, args, threads)[source]¶

Run programe in thread

Args:

func:object: function to run in thread
args:list: paramters to pass if multiple please use zip(args_list2, args_list2)
threads: int: number of thread

Return:

list of result: dict

utilpy.utils.top_n(lis, n)[source]¶

Get index of top n item from list with value

Args:

lis:list: list
n:int: number of top value

Return:

dict of top value and index: dict

utilpy.utils.walk_directory(dir_path, files_only=True)[source]¶

Get list of all file in directory recursive

Args:

dir_path:str: parent directory to start
files_only:bool: get list of files only other

Return:

If file only than list of all files otherwise files and list of directories

utilpy.decorator¶

utilpy.decorator.profile(fnc)[source]¶: A decorator that uses cProfile to profile a function And print the result

utilpy.decorator.threading_d(func)[source]¶

A decorator to run function in background on thread

Args:

func:function

Function with args

Return:

background_thread: Thread

utilpy.decorator.timeout(seconds=10, error_message='Timer expired')[source]¶

Decorator to throw timeout error, if function doesnt complete in certain time

Args:

seconds:int: No of seconds to wait
error_message:str: Error message

utilpy.log¶

Logger function that print time required in each session of logger

class utilpy.log.Logger[source]¶

Bases: object

Print the log message with time required between each session New session should start with “<” and end with “>” session ends with “</>”

log(message)[source]¶: log message new session starts with < and ends with > session end with </>

utilspy.files¶

utilpy.files.delete_dirs(folder_names)[source]¶

delete folder and all its contents

Args: `

folder_names:str` or list: string or list of folders

utilpy.files.make_dirs(folder_names)[source]¶

make folder if it doesnot exist

Args:

folder_names:str or list: string or list of folders

utilpy.files.move_file(filename, out_dir)[source]¶

move file/dir

Args:

filename:str: filename of file to be moved or name of dir to be moved
out_dir:str: output directory

utilpy.files.parse_path(path, fn_only=False, ext=False, al=True)[source]¶

get the directory from filename,

Args:

path:str: path of file
fn_only:bol: get the file name only from path
ext:bol: split file name into name and extension
al: bol: get list of dir and file name

Returns:

list of value: list

utilpy.files.unzip_file(filename, save_dir='data')[source]¶

unzip all file

Args:

filename:str: filename of zip file
sav_dir:str: parent directory to zip

Warning

for rar file type, install rar and unrar

apt install rar && apt install unrar

utilpy.files.zip_folder(dir_path, filename=None, exclude=None, include=None)[source]¶

zip all file in folder,

Args:

dir_path:str: parent directory to zip
filename:str: filename of zip file
exclude:list: list of file to exclude
include:list: list of file to include