Skip to main content

A Hy library that provides a Lispy functional interface by wrapping Python's popular data libraries, such as Pandas and Matplotlib.

Project description

HyFive

What Is HyFive?

HyFive is a Hy library that provides a Lispy functional interface by wrapping Python's popular data libraries, such as Pandas and Matplotlib.

HyFive vs. Vanilla Pandas

Pandas DataFrame has its own quirks. It ranges from not having a filter method to having a different definiton of join to that of SQL. This is evident from the bewildering comparison between Pandas and SQL. HyFive aims to provide a Lispy interface that is as close as possible to that of Spark's SQL and DataFrame.

From a functional programming perspective, Pandas interfaces are oddly difficult to compose unlike Spark SQL's method-chaining convention. Due to this difficulty, Pandas often perversely incentivises short names for dataframes in favour of creating intermediate variables, which litters the namespace.

HyFive utilises Hy's threading macros to mimic Spark DataFrame's method chaining convention, whilst staying with the familiar Pandas dataframe. Consider the following HyFive snippet:

(setv DATAFRAME
  (-> NAME-REGISTRY
      (hf.with-column "variant"
        (let [mod-res-id (hf.mod "resident_id" 3)]
          (hf.cond-col [(hf.eq? mod-res-id 1) (hf.lit "a")]
                       [(hf.eq? mod-res-id 2) (hf.lit "b")]
                       [:else                 (hf.lit "c")])))
      (hf.filter (hf.is-in "variant" ["a" "b"]))
      (hf.join AGE-REGISTRY :on "resident_id")
      (hf.group-by "variant")
      (hf.agg {"min_age"  (hf.min "age")
               "mean_age" (hf.mean "age")
               "std_age"  (hf.std "age")
               "max_age"  (hf.max "age")})
      (hf.order-by "min_age" :desc True)))

Here, we carry out simple operations of adding a column, filtering rows, joining tables, aggregating groups and sorting rows. Apart from the Lispy cond, these operations would have a one-to-one translation to Spark dataframe or SQL.

In contrast, one would have to work a bit harder in pure Pandas:

mod_res_id = NAME_REGISTRY.resident_id % 3
variant = np.where(mod_res_id == 1, 'a', np.where(mod_res_id == 2, 'b', 'c'))
select_ix = np.isin(variant, ['a', 'b'])
dataframe = (NAME_REGISTRY
                .assign(variant=variant)
                [select_ix]
                .merge(AGE_REGISTRY, on='resident_id')
                .groupby('variant')
                .apply(lambda df: pd.Series({
                    'min_age': df.age.min(),
                    'mean_age': df.age.mean(),
                    'std_age': df.age.std(),
                    'max_age': df.age.max()
                }))
                .reset_index()
                .sort_values(by='min_age', ascending=False)
                .reset_index(drop=True))

The Pandas version is less readable and we lose the one-to-one translation to Spark dataframe or SQL.

Trying HyFive

Clone the repository, and on the root directory of the enter the following command on terminal:

./run build-docker

Run the unit tests with the following command:

./run unit-tests .

Invoke the Hy REPL by running:

./run repl

And finally, import HyFive using:

(import [hyfive :as hf])

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dev-aa-test-1, version 0.1.0
Filename, size File type Python version Upload date Hashes
Filename, size dev_aa_test_1-0.1.0-py3-none-any.whl (15.3 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size dev_aa_test_1-0.1.0.tar.gz (3.1 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page