HyFive

What Is HyFive?

HyFive is a Hy library that provides a Lispy functional interface by wrapping Python's popular data libraries, such as Pandas and Matplotlib.

HyFive vs. Vanilla Pandas

Pandas DataFrame has its own quirks. It ranges from not having a filter method to having a different definiton of join to that of SQL. This is evident from the bewildering comparison between Pandas and SQL. HyFive aims to provide a Lispy interface that is as close as possible to that of Spark's SQL and DataFrame.

From a functional programming perspective, Pandas interfaces are oddly difficult to compose unlike Spark SQL's method-chaining convention. Due to this difficulty, Pandas often perversely incentivises short names for dataframes in favour of creating intermediate variables, which litters the namespace.

HyFive utilises Hy's threading macros to mimic Spark DataFrame's method chaining convention, whilst staying with the familiar Pandas dataframe. Consider the following HyFive snippet:

(setv DATAFRAME
  (-> NAME-REGISTRY
      (hf.with-column "variant"
        (let [mod-res-id (hf.mod "resident_id" 3)]
          (hf.cond-col [(hf.eq? mod-res-id 1) (hf.lit "a")]
                       [(hf.eq? mod-res-id 2) (hf.lit "b")]
                       [:else                 (hf.lit "c")])))
      (hf.filter (hf.is-in "variant" ["a" "b"]))
      (hf.join AGE-REGISTRY :on "resident_id")
      (hf.group-by "variant")
      (hf.agg {"min_age"  (hf.min "age")
               "mean_age" (hf.mean "age")
               "std_age"  (hf.std "age")
               "max_age"  (hf.max "age")})
      (hf.order-by "min_age" :desc True)))

Here, we carry out simple operations of adding a column, filtering rows, joining tables, aggregating groups and sorting rows. Apart from the Lispy cond, these operations would have a one-to-one translation to Spark dataframe or SQL.

In contrast, one would have to work a bit harder in pure Pandas:

mod_res_id = NAME_REGISTRY.resident_id % 3
variant = np.where(mod_res_id == 1, 'a', np.where(mod_res_id == 2, 'b', 'c'))
select_ix = np.isin(variant, ['a', 'b'])
dataframe = (NAME_REGISTRY
                .assign(variant=variant)
                [select_ix]
                .merge(AGE_REGISTRY, on='resident_id')
                .groupby('variant')
                .apply(lambda df: pd.Series({
                    'min_age': df.age.min(),
                    'mean_age': df.age.mean(),
                    'std_age': df.age.std(),
                    'max_age': df.age.max()
                }))
                .reset_index()
                .sort_values(by='min_age', ascending=False)
                .reset_index(drop=True))

The Pandas version is less readable and we lose the one-to-one translation to Spark dataframe or SQL.

Trying HyFive

Clone the repository, and on the root directory of the enter the following command on terminal:

./run build-docker

Run the unit tests with the following command:

./run unit-tests .

Invoke the Hy REPL by running:

./run repl

And finally, import HyFive using:

(import [hyfive :as hf])