Skip to main content

Aligning Human & Machine vision

Project description

Harmonizing the object recognition strategies of deep neural networks with humans


Thomas Fel*, Ivan Felipe Rodriguez*, Drew Linsley*, Thomas Serre

Read the official paper »
Explore results . Documentation . Models zoo . Tutorials · Click-me paper

Paper summary

The many successes of deep neural networks (DNNs) over the past decade have largely been driven by computational scale rather than insights from biological intelligence. Here, we explore if these trends have also carried concomitant improvements in explaining visual strategies underlying human object recognition. We do this by comparing two related but distinct properties of visual strategies in humans and DNNs: where they believe important visual features are in images and how they use those features to categorize objects. Across 85 different DNNs and three independent datasets measuring human visual strategies on ImageNet, we find a trade-off between DNN top-1 categorization accuracy and their alignment with humans. State-of-the-art DNNs are progressively becoming less aligned with humans. We rectify this growing issue by introducing the harmonization procedure: a general-purpose training routine that aligns DNN and human visual strategies while improving object classification performance.

Aligning the Gradients

Human and DNNs rely on different features to recognize objects. In contrast, our neural harmonizer aligns DNN feature importance with humans. Gradients are smoothed from both humans and DNNs with a Gaussian kernel to improve visualization.

Breaking the trade-off between performance and alignment

The trade-off between DNN performance and alignment with human feature importance from the ClickMe dataset. Human feature alignment is the mean Spearman correlation between human and DNN feature importance maps, normalized by the average inter-rater alignment of humans. The grey-shaded region illustrates the convex hull of the trade-off between ImageNet accuracy and human feature alignment. All the models trained with the harmonization procedure are more accurate and aligned than versions of those models trained only for classification. Arrows denote a shift in performance after training with the harmonization procedure.

🗞️ Citation

If you use or build on our work as part of your workflow in a scientific publication, please consider citing the official paper:

@article{fel2022aligning,
  title={Harmonizing the object recognition strategies of deep neural networks with humans},
  author={Fel, Thomas and Felipe, Ivan and Linsley, Drew and Serre, Thomas},
  journal={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2022}
}

Moreover, this paper relies heavily on previous work from the Lab, notably Learning What and Where to Attend where the ambitious ClickMe dataset was collected.

@article{linsley2018learning,
  title={Learning what and where to attend},
  author={Linsley, Drew and Shiebler, Dan and Eberhardt, Sven and Serre, Thomas},
  journal={International Conference on Learning Representations (ICLR)},
  year={2019}
}

Tutorials

Evaluate your own model (pytorch and tensorflow)

Open In Colab Open In Colab

📝 License

The package is released under MIT license.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Harmonization-0.0.9.tar.gz (13.7 kB view details)

Uploaded Source

Built Distribution

Harmonization-0.0.9-py3-none-any.whl (21.1 kB view details)

Uploaded Python 3

File details

Details for the file Harmonization-0.0.9.tar.gz.

File metadata

  • Download URL: Harmonization-0.0.9.tar.gz
  • Upload date:
  • Size: 13.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.2

File hashes

Hashes for Harmonization-0.0.9.tar.gz
Algorithm Hash digest
SHA256 66e3ea8187824ef6db1136896d72417303770f9a640351f9c505279147b7ec72
MD5 32db2cb6602dcde9bcb3a0f2c327b0fb
BLAKE2b-256 56773eab8b0c7bfd006c1023e12bed2111c241c85788a035265faf12ef98e8c1

See more details on using hashes here.

File details

Details for the file Harmonization-0.0.9-py3-none-any.whl.

File metadata

File hashes

Hashes for Harmonization-0.0.9-py3-none-any.whl
Algorithm Hash digest
SHA256 74b4f269778e1ca010718234ac8a108b2b5ed2da1bafeec89e0d11732bcd5698
MD5 117c1294cf12673f3e7d2214f1480b5f
BLAKE2b-256 fdba8a8655264c67a824b382dbd07af43df598a8b7672cfbfa2c3fa882b93c5f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page