Skip to main content

An API for working with IP addresses in Apache Spark.

Project description



An API for working with IP addresses in Apache Spark. Built on top of ipaddress.


  • pip install PySparkIP
  • from PySparkIP import *


This project is licensed under the Apache License. Please see LICENSE file for more details.



Before using in SparkSQL, initialize PySparkIP by passing spark to PySparkIP, then define IPAddressUDT() in the schema.
Optionally pass the log level as well (if left unspecified, PySparkIP resets the log level to "WARN" and gives a warning message).
NOTE: Values that are unsuccessfully converted to IP addresses will be converted to "::"

from PySparkIP import *

# Initialize for SparkSQL use (not needed for pure PySpark)
# or PySparkIP(spark, "DEBUG"), PySparkIP(spark, "FATAL"), etc if specifying a log level

schema = StructType([StructField("IPAddress", IPAddressUDT())])
ipDF ="ipFile.json", schema=schema)


Cast StringType() to IPAddressUDT()

# PySpark
ipDF ='ip_string'))
# SparkSQL
spark.sql("SELECT to_ip(ip_string) FROM IPAddresses")

Check address type

# Multicast'*').withColumn("IPColumn", isMulticast("IPAddress"))
spark.sql("SELECT * FROM IPAddresses WHERE isMulticast(IPAddress)")

Other address types:
    isPrivate, isGlobal, isUnspecified, isReserved, 
    isLoopback, isLinkLocal, isIPv4Mapped, is6to4, 
    isTeredo, isIPv4, isIPv6

Output address in different formats

# Exploded
spark.sql("SELECT explodedIP(IPAddress) FROM IPAddresses")"IPAddress"))

# Compressed
spark.sql("SELECT compressedIP(IPAddress) FROM IPAddresses")"IPAddress"))

Sort IP Addresses

# SparkSQL doesn't support values > LONG_MAX
# To sort IPv6 addresses, use ipAsBinary
# To sort IPv4 addresses, use either ipv4AsNum or ipAsBinary, but ipv4AsNum is more efficient

# Sort IPv4 and IPv6
spark.sql("SELECT * FROM IPAddresses SORT BY ipAsBinary(IPAddress)")'*').sort(ipAsBinary("IPAddress"))

# Sort ONLY IPv4
spark.sql("SELECT * FROM IPv4 SORT BY ipv4AsNum(IPAddress)")'*').sort(ipv4AsNum("IPAddress"))

IP network functions

# Network contains
spark.sql("SELECT * FROM IPAddresses WHERE networkContains(IPAddress, '')")'*').filter("networkContains(IPAddress, '')")'*').withColumn("netCol", networkContains("")("IPAddress"))

# Or use ipaddress.ip_network objects
net1 = ipaddress.ip_network('::/10')'*').filter(networkContains(net1)("IPAddress"))

IP Set

Create IP Sets (Note: This functionality also works with add and remove):

# Strings
ipStr = ''
netStr = ''
# Tuples, lists, or sets
ip_net_mix = ('::5', '', '')
# ipaddress objects
ipAddr = ipaddress.ip_address('::')
# Dataframes
ipMulticastDF = spark.sql("SELECT IPAddress FROM IPAddresses WHERE isMulticast(IPAddress)")

Or use our predefined networks (multicastIPs, privateIPs, 
 publicIPs, reservedIPs, unspecifiedIPs, linkLocalIPs, 
 loopBackIPs, ipv4MappedIPs, ipv4TranslatedIPs, ipv4ipv6TranslatedIPs,
 teredoIPs, sixToFourIPs, or siteLocalIPs)

# Mix them together
ipSet = IPSet(ipStr, '::/16', '2001::', netStr, ip_net_mix, privateIPs)
ipSet2 = IPSet("6::", "", ipAddr, ipMulticastDF)
# Use other IPSets
ipSet3 = IPSet(ipSet, ipSet2)
# Or just make an empty set
ipSet4 = IPSet()

Use IP Sets:

# Initialize an IP Set
setOfIPs = {"", "5422:6622:1dc6:366a:e728:84d4:257e:655a", "::"}
ipSet = IPSet(setOfIPs)

# Use it!'*').filter("setContains(IPAddress, 'ipSet')")'*').withColumn("setCol", setContains(ipSet)("IPAddress"))

Register IP Sets for use in SparkSQL:

Before using IP Sets in SparkSQL, register it by passing it to PySparkIPSets

ipSet = IPSet('::')
ipSet2 = IPSet()

# Pass the set, then the set name
PySparkIPSets.add(ipSet, 'ipSet')
PySparkIPSets.add(ipSet2, 'ipSet2')

Remove IP Sets from registered sets in SparkSQL:

PySparkIPSets.remove('ipSet', 'ipSet2')

Use IP Sets in SparkSQL:

# Note you have to pass the variable name using SparkSQL, not the actual variable

# Initialize an IP Set
setOfIPs = {"", "5422:6622:1dc6:366a:e728:84d4:257e:655a", "::"}
ipSet = IPSet(setOfIPs)

# Register it
PySparkIPSets.add(ipSet, 'ipSet')

#Use it!
# Set Contains
spark.sql("SELECT * FROM IPAddresses WHERE setContains(IPAddress, 'ipSet')")

# Show sets available to use

# Remove a set

# Clear sets available

IP Set functions (outside Spark):

ipSet = IPSet()

# Add
ipSet.add('', '::/16')

# Remove

# Contains

# Clear

# Show all

# Union
ipSet2 = ('2001::', '::33', 'ffff::f')

# Intersection

# Diff

# Show All

# Return All

# Is empty

# Compare IPSets
ipSet2 = ('2001::', '::33', 'ffff::f')
ipSet == ipSet2
ipSet != ipSet2

# Return the # of elements in the set

Other operations (outside Spark):

# Nets intersect
net1 = ''
net2 = ''
# or ipaddress.ip_network('')
netsIntersect(net1, net2)

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

PySparkIP-1.2.4.tar.gz (12.6 kB view hashes)

Uploaded source

Built Distribution

PySparkIP-1.2.4-py3-none-any.whl (9.6 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page