Skip to main content

Ease developers to use queue/threads functions to handle big amount of data

Project description

pyhacks

Python module to ease writing scripts go over big amount of data in order to perform the same actions. A simple preconfigured threads and queue management and more hacking utils

Will elaborate more when it will get mature enough :)

capabilities

PyHacks

Using all below capabilities to easily manage threads and queue to perform operations on Item objects.

from pyhacks import PyHacks, Net

if __name__ == "__main__":
	net = Net()
	
	def handle_item(item):
		host = item.get("host")
		url = item.get("url")
		replied_to_ping = net.reply_to_ping(host)
		ip = net.resolve(host)
		if ip == None:
			return False
		item.set("Reply To Ping", replied_to_ping)
		item.set("ip", ip)
		hacks.logger.green("{} {} {}".format(host, replied_to_ping, ip))
		hacks.exporter.put(item)
		return True
	
	hacks = PyHacks(handle_item, 25, "output.csv")
	hosts = hacks.parse.csv("ips.csv")
	
	for host in hosts:
		hacks.qt.put(host)
	
	hacks.finish()

    
QueueThreads

QueueThreads class allows you to manage queue and threads easily.

qt = QueueThreads(handle_function, num_worker_threads, logger)
qt.put({"counter":1, "key1":"value1"})
qt.finish() # waiting until job is finished
parameters

handle_function - a function recevies item parameter from queue, will be used in all threads defined, return value has to be bool

def handle_item(item):
		host = item.get("host")
		url = item.get("url")
		replied_to_ping = net.reply_to_ping(host)
		ip = net.resolve(host)
		if ip == None:
			return False
		item.set("Reply To Ping", replied_to_ping)
		item.set("ip", ip)
		hacks.logger.green("{} {} {}".format(host, replied_to_ping, ip))
		hacks.exporter.put(item)
		return True

num_worker_threads - numbers of threads used to run handle_functions on queue items logger - a Logger instance to be used for automatic logging

Parser

Parse files and returns list of Item objects

parse = Parser()
items = parse.csv("ips.csv")
for item in items:
	qt.put(item)
Logger

Easy logging mechanism + writing into files under logs directory, used by QueueThreads for autoamtic logging

logger = Logger(verbose=False)
logger.green("text")
logger.red("text")
logger.yellow("text")
Exporter

Exporting Item objects into a csv/txt file

export = Exporter(output.csv)
export.put({"counter":1,"key1":"value"})
export.finish()

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pyhacks, version 1.0.20
Filename, size File type Python version Upload date Hashes
Filename, size pyhacks-1.0.20.tar.gz (8.1 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page