Skip to main content

Utilities for the analysis of the GMANE email list database

Project description

This project delivers helper classes for the analysis of the GMANE email database. Install with:

$ pip install gmane


$ python install

For greater control of customization (and debugging), clone the repo and install with pip with -e:

$ git clone

$ pip install -e <path_to_repo>

This install method is especially useful with reload function from IPython.lib.deepreload and the standard importlib.

Functionalities are based on physics articles on interaction networks: [1] Stability in human interaction networks: primitive typology of vertex, prominence of measures and activity statistics: [2] A connective differentiation of textual production in interaction networks: [3] Versinus: a visualization method for graphs in evolution:

With core concepts of 1) analysis of topological structure; 2) analysis of textual production; 3) visualization of evolving structures. Activity distribution along time and among participants are also approached through specific routines and indirectly through 1), 2) and 3).

Ideally, this package should ease: - Downloading GMANE email list data. - Building elementary data structures with downloaded data. - Analysis of data through complex networks and NLP criteria. - Visualization through diverse layout methods.

PS. Implemented measures of symmetry in network agents activity by hand (not found in network and numeric packages) according to [1].

Usage example

Download messages from one GMANE list:

import gmane as g
dl=g.DownloadGmaneData() # saves into ~/.gmane/
dl.downloadListsIDS() # acquires all GMANE list_ids
dl.cleanDownloadedLists() # remove empty messages for coherence
dl.downloadedStats() # creates ~/.gmane/stats.txt

# to load message contents to Python objects:
# load 10 messages from list with list_id gmane.ietf.rfc822

# or access the structures downloaded to your filesystem
# and download all messages from 5 lists
for list_id in dl.downloaded_lists[:5]:

# to load first three lists with the greated number
# of downloaded messages:
dl.downloadedStats() # might take a while
for list_stat in dl.lists[:3]:

# to make basic datastructures of a list with
# greatest number of messages:
print("first: ", mm[ids[0]][2], "last:", mm[ids[-1]][2])

# circular (directional) statistics for activity along time
# (hours of the day, days of the week, days of the month, etc):
# mean_vec, mean_angle, size_mean_vec, circular_mean,
# circular_variance, circular dispersion
# and histograms
print("made overall circular activity statistics along time")

# make latex tables to observe distributions within bins of interest

print("made overall activity statistics among participants")

# build the interaction network of the messages:

print("number of nodes: {}, number of edges: {}".format(
nw.g.number_of_nodes(), nw.g.number_of_edges()))

nm=g.NetworkMeasures(nw) # take measures, including symmetry related measures
np=g.NetworkPartitioning(nm) # partition in primitive typology
sa=np.sectorialized_agents # get members of each sector
print("{} agents in periphery, {} are intermediary and {} hubs".format(sa[0],sa[1],sa[2]))
sa=np.sectorialized_agents__ # smoothed histogram for classification
print("{} agents in periphery, {} are intermediary and {} hubs".format(sa[0],sa[1],sa[2]))

# draw
print("drawer started")
print("gave (x,y) for each author with 5-15-80")
print("drawer two started")
print("gave (x,y) for each author with \
sectors by comparison with Erdos-Renyi")
nd.drawNetwork( iN,nm ,"test.png")
nd2.drawNetwork( iN,nm,"test2.png")

# make basic PCA plots of network measures:
# Plot PCA with a colored primitive sectors

# Evolves network with measures, partitions,
# PCA, principal components and Versinus plots saved to disk
lm=lms[0] # loaded messages from list with most messages
# ne.makeVideo() use this to avoid evolving again just to make video
# see or g.NetworkEvolution to make movies:

# after making network evolution measurements and video,
# you can both make music:
print("music is done")
# avconv -i mixY.wav -i evo[..<depends on the evolution done>..].avi final.avi
# delivers you the final.avi animation with a soundtrack relative to network measures
# currently it is the 'four hubs dance' by default:

# and further analysis of measures and Erdos sectors:
print("Written png files with network measures along evolution timeline")

# Enjoy!

Further documentation is in tests/ folder and object docstrings.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gmane-0.1.dev25.tar.gz (26.7 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page