pyhacks 1.0.20

Ease developers to use queue/threads functions to handle big amount of data

pyhacks

Python module to ease writing scripts go over big amount of data in order to perform the same actions. A simple preconfigured threads and queue management and more hacking utils

Will elaborate more when it will get mature enough :)

capabilities

PyHacks

Using all below capabilities to easily manage threads and queue to perform operations on Item objects.

from pyhacks import PyHacks, Net

if __name__ == "__main__":
	net = Net()
	
	def handle_item(item):
		host = item.get("host")
		url = item.get("url")
		replied_to_ping = net.reply_to_ping(host)
		ip = net.resolve(host)
		if ip == None:
			return False
		item.set("Reply To Ping", replied_to_ping)
		item.set("ip", ip)
		hacks.logger.green("{} {} {}".format(host, replied_to_ping, ip))
		hacks.exporter.put(item)
		return True
	
	hacks = PyHacks(handle_item, 25, "output.csv")
	hosts = hacks.parse.csv("ips.csv")
	
	for host in hosts:
		hacks.qt.put(host)
	
	hacks.finish()

    

QueueThreads

QueueThreads class allows you to manage queue and threads easily.

qt = QueueThreads(handle_function, num_worker_threads, logger)
qt.put({"counter":1, "key1":"value1"})
qt.finish() # waiting until job is finished

parameters

handle_function - a function recevies item parameter from queue, will be used in all threads defined, return value has to be bool

def handle_item(item):
		host = item.get("host")
		url = item.get("url")
		replied_to_ping = net.reply_to_ping(host)
		ip = net.resolve(host)
		if ip == None:
			return False
		item.set("Reply To Ping", replied_to_ping)
		item.set("ip", ip)
		hacks.logger.green("{} {} {}".format(host, replied_to_ping, ip))
		hacks.exporter.put(item)
		return True

num_worker_threads - numbers of threads used to run handle_functions on queue items logger - a Logger instance to be used for automatic logging

Parser

Parse files and returns list of Item objects

parse = Parser()
items = parse.csv("ips.csv")
for item in items:
	qt.put(item)

Logger

Easy logging mechanism + writing into files under logs directory, used by QueueThreads for autoamtic logging

logger = Logger(verbose=False)
logger.green("text")
logger.red("text")
logger.yellow("text")

Exporter

Exporting Item objects into a csv/txt file

export = Exporter(output.csv)
export.put({"counter":1,"key1":"value"})
export.finish()