Fast operations on lists and other iterables

Each of the try_* functions have a map option causes the function to accept an iterable of items to convert and returns a list. Using try_float as an example, the following are all functionally equivalent.

>>> from fastnumbers import try_float
>>> iterable = ["5", "4.5", "34567.6", "32"]
>>> try_float(iterable, map=list) == list(map(try_float, iterable))
True
>>> try_float(iterable, map=list) == [try_float(x) for x in iterable]
True
>>> try_float(iterable, map=list) == list(try_float(iterable, map=True))
True

The difference is that the map option is 2x the speed of the list comprehension method, and 1.5x the speed of the map method. The reason is that it avoids Python function call overhead on each iteration. Note that True causes the function to return an iterator, and list causes it to return a list. In practice the performance of these are similar (see Timing for raw data).

If you need to store your output in a numpy array, you can use try_array to do this conversion directly. This function has some additional handling for overflow that is not present in the other fastnumbers functions that may come in handy when dealing with numpy arrays.

>>> from fastnumbers import try_array
>>> import numpy as np
>>> iterable = ["5", "4.5", "34567.6", "32"]
>>> np.array_equal(np.array(try_float(iterable, map=list), dtype=np.float64), try_array(iterable))
True

You will see about a 2x speedup of doing this in one step over converting to a list then converting that list to an array.

About the on_fail option

The on_fail option is a way for you to do anything in the event that the given input cannot be converted to a number. It can

• return given object as-is if set to fastnumbers.INPUT (this is the default)

• raise a ValueError if set to fastnumbers.RAISE

• return a default value if given any non-callable object

• call a function with the given object if given a single-argument callable

Below are a couple of ideas to get you thinking.

NOTE:: There is also an on_type_error option that behaves the same as on_fail except that a) it is triggered when the given object is of an invalid type and b) the default value is fastnumbers.RAISE, not fastnumbers.INPUT.

>>> from fastnumbers import INPUT, RAISE, try_float
>>> # You want to convert strings that can be converted to numbers, but
>>> # leave the rest as strings. Use fastnumbers.INPUT (the default)
>>> try_float('45.6')
45.6
>>> try_float('invalid input')
'invalid input'
>>> try_float('invalid input', on_fail=INPUT)
'invalid input'
>>>
>>>
>>>
>>> # You want to convert any invalid string to NaN
>>> try_float('45.6', on_fail=float('nan'))
45.6
>>> try_float('invalid input', on_fail=float('nan'))
nan
>>>
>>>
>>>
>>> # Simple callable case, send the input through some function to generate a number.
>>> try_float('invalid input', on_fail=lambda x: float(x.count('i')))  # count the 'i's
3.0
>>>
>>>
>>>
>>> # Suppose we know that our input could either be a number, or if not
>>> # then we know we just have to strip off parens to get to the number
>>> # e.g. the input could be '45' or '(45)'. Also, suppose that if it
>>> # still cannot be converted to a number we want to raise an exception.
>>> def strip_parens_and_try_again(x):
...     return try_float(x.strip('()'), on_fail=RAISE)
...
>>> try_float('45', on_fail=strip_parens_and_try_again)
45.0
>>> try_float('(45)', on_fail=strip_parens_and_try_again)
45.0
>>> try_float('invalid input', on_fail=strip_parens_and_try_again) #doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
  ...
ValueError: invalid literal for float(): invalid input
>>>
>>>
>>>
>>> # Suppose that whenever an invalid input is given, it needs to be
>>> # logged and then a default value is returned.
>>> def log_and_default(x, log_method=print, default=0.0):
...     log_method("The input {!r} is not valid!".format(x))
...     return default
...
>>> try_float('45', on_fail=log_and_default)
45.0
>>> try_float('invalid input', on_fail=log_and_default)
The input 'invalid input' is not valid!
0.0
>>> try_float('invalid input', on_fail=lambda x: log_and_default(x, default=float('nan')))
The input 'invalid input' is not valid!
nan

About the denoise option

The denoise option is available on the try_real and try_forceint options. To best understand its usage, consider the following native Python behavior:

>>> int(3.453e21)
3452999999999999737856
>>> int(float("3.453e21"))
3452999999999999737856
>>> # Most users would likely expect this result from decimal.Decimal
>>> import decimal
>>> int(decimal.Decimal("3.453e21"))
3453000000000000000000
>>> # But watch out, even decimal.Decimal doesn't help for float input
>>> import decimal
>>> int(decimal.Decimal(3.453e21))
3452999999999999737856

Because the conversion of a float to an int goes through the C double data type which has inherent limitations on accuracy (See this Stack Overflow question for examples) the resulting int result has “noise” digits that are not part of the original float representation.

For functions where this makes sense, fastnumbers provides the denoise option to give you the results that decimal.Decimal would give for strings containing floats.

>>> from fastnumbers import try_real
>>> try_real(3.453e21)
3452999999999999737856
>>> try_real("3.453e21")
3452999999999999737856
>>> try_real(3.453e21, denoise=True)
3453000000000000000000
>>> try_real("3.453e21", denoise=True)
3453000000000000000000

Two things to keep in mind:

1. The denoise option adds additional overhead to the conversion calculation, so please consider the trade-offs between speed and accuracy when determining whether or not to use it. It is significantly faster than using decimal.Decimal, but much slower than not using it at all.

2. For string input, denoise will return results identical to decimal.Decimal. For float input, denoise will return results that are accurate to about 15 digits (C double can only store 16 decimal digits, so this means that only the last possible digit may not be accurate).