Skip to main content

A Hy library that provides a Lispy functional interface by wrapping Python's popular data libraries, such as Pandas and Matplotlib.

Project description


What Is HyFive?

HyFive is a Hy library that provides a Lispy functional interface by wrapping Python's popular data libraries, such as Pandas and Matplotlib.

HyFive vs. Vanilla Pandas

Pandas DataFrame has its own quirks. It ranges from not having a filter method to having a different definiton of join to that of SQL. This is evident from the bewildering comparison between Pandas and SQL. HyFive aims to provide a Lispy interface that is as close as possible to that of Spark's SQL and DataFrame.

From a functional programming perspective, Pandas interfaces are oddly difficult to compose unlike Spark SQL's method-chaining convention. Due to this difficulty, Pandas often perversely incentivises short names for dataframes in favour of creating intermediate variables, which litters the namespace.

HyFive utilises Hy's threading macros to mimic Spark DataFrame's method chaining convention, whilst staying with the familiar Pandas dataframe. Consider the following HyFive snippet:

      (hf.with-column "variant"
        (let [mod-res-id (hf.mod "resident_id" 3)]
          (hf.cond-col [(hf.eq? mod-res-id 1) (hf.lit "a")]
                       [(hf.eq? mod-res-id 2) (hf.lit "b")]
                       [:else                 (hf.lit "c")])))
      (hf.filter ( "variant" ["a" "b"]))
      (hf.join AGE-REGISTRY :on "resident_id")
      ( "variant")
      (hf.agg {"min_age"  (hf.min "age")
               "mean_age" (hf.mean "age")
               "std_age"  (hf.std "age")
               "max_age"  (hf.max "age")})
      (hf.order-by "min_age" :desc True)))

Here, we carry out simple operations of adding a column, filtering rows, joining tables, aggregating groups and sorting rows. Apart from the Lispy cond, these operations would have a one-to-one translation to Spark dataframe or SQL.

In contrast, one would have to work a bit harder in pure Pandas:

mod_res_id = NAME_REGISTRY.resident_id % 3
variant = np.where(mod_res_id == 1, 'a', np.where(mod_res_id == 2, 'b', 'c'))
select_ix = np.isin(variant, ['a', 'b'])
dataframe = (NAME_REGISTRY
                .merge(AGE_REGISTRY, on='resident_id')
                .apply(lambda df: pd.Series({
                    'min_age': df.age.min(),
                    'mean_age': df.age.mean(),
                    'std_age': df.age.std(),
                    'max_age': df.age.max()
                .sort_values(by='min_age', ascending=False)

The Pandas version is less readable and we lose the one-to-one translation to Spark dataframe or SQL.

Trying HyFive

Clone the repository, and on the root directory of the enter the following command on terminal:

./run build-docker

Run the unit tests with the following command:

./run unit-tests .

Invoke the Hy REPL by running:

./run repl

And finally, import HyFive using:

(import [hyfive :as hf])

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dev_aa_test_1-0.1.0.tar.gz (3.1 kB view hashes)

Uploaded source

Built Distribution

dev_aa_test_1-0.1.0-py3-none-any.whl (15.3 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page