python SettingWithCopyWarning 原理及解决办法

悠悠 2022-12-18 02:58 137阅读 0赞

Regardless of how long you worked with `pandas`, be it a day or a year, sooner or later you are likely to run into the infamous `SettingWithCopyWarning`. In this article, I explain what causes the problem and how to properly address the issue.

# Warning not an error #

Before I dive into the technicalities, I want to highlight that `SettingWithCopyWarning` is — as the name suggests — a warning, not an error. So the code we are executing will most likely not break and produce the end result. However, the end result might not be the one we actually intended to obtain.

The reason why I wanted to highlight the distinction is that we might be tempted to ignore the warning when we see that the code actually succeeds in returning a result. And as a matter of fact, the result might be correct! The best practice is to be extra careful and actually understand the underlying principles. This way, we can often save a lot of time trying to identify an obscure bug, which we could have avoided in the first place.

# Views versus copies #

The key concepts that are connected to the`SettingWithCopyWarning` are views and copies. Some operations in `pandas` (and `numpy` as well) will return views of the original data, while other copies.

To put it very simply, a **view** is a subset of the original object (`DataFrame` or `Series`) linked to the original source, while a **copy** is an entirely new object. A consequence of the distinction is that when we modify the view, we modify the original object as well. That does not happen with copies, as they are not connected to the original objects.

Having described the difference, `SettingWithCopyWarning` is actually letting us know that the code we have written might have done one thing, when in fact we wanted to do another. I will illustrate this with a real-life example. Imagine having a large `DataFrame`. For some analysis, you filter (slice) the `DataFrame` to only contain a subset of the full data, for example, users from a certain country. Then, you might want to modify some values in the extracted `DataFrame`, let’s say cap the maximum value of a feature at 100. This is the typical case in which you could encounter the infamous warning — you wanted to only modify the extracted frame, while you ended up modifying the source data. You can easily imagine that this is not something you wanted to do and can lead to potential problems later on.

**Note**: To understand whether a frame is a view of a copy, you can use the internal `_is_view` and `_is_copy` methods of a `pandas.DataFrame`. The first one returns a boolean, while the second either a `weakref` to the original `DataFrame`, or `None`.

# Common occurrences of the warning #

In this section, I go over the most common cases when the `SettingWithCopyWarning` occurs in practice. I will illustrate the cases using a small custom `DataFrame`, as it is more than enough to understand the logic.

To prepare the data I run the following code:

Running the code prints out the small `DataFrame`:

![Image for post][]

![Image for post][Image for post 1]

In order to clearly understand what is happening, for each of the cases below we will be starting with a clean slate — the result of running the `get_data` function.

For future reference, this article was written using `pandas` version 1.0.3.

## 1. Chained Assignment ##

To explain the concept of the chained assignment, we will sequentially go over the building blocks. The assignment operation (also knows as the **set operation**) simply sets the value of an object. We can illustrate this by creating a list:

x_list = [1, 2, 3, 4, 5]

Even though the first example is based on lists, the same principles apply to arrays, `Series`, and `DataFrames` (as we will see in a minute). The second type of operation is called the **get operation** and is used for accessing and returning the value of an object. **Indexing** is a type of a *get* operation and we can index the list by running `x_list[0:3]`, which returns

[1, 2, 3]

The last building block is called **chaining** and essentially refers to chaining multiple indexing operations, such as `x_list[0:3][1]`, which returns 2.

Having described all the individual pieces, by **chained assignment** we mean a combination of chaining and assignment. It is time to refer to our toy`DataFrame`. First, we slice the `DataFrame` to display observations with the value of the `B` feature higher than 12.

X = get_data()
    X[X['B'] > 12]

![Image for post][Image for post 2]

![Image for post][Image for post 3]

There are only 2 rows fulfilling that criterion. Let’s replace the values of the `C` feature with 999.

X[X['B'] > 12]['C'] = 999
    X[X['B'] > 12]['C']

Running the line above results in the infamous warning:

SettingWithCopyWarning: 
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

In the generated output, we see that the values were not replaced!

![Image for post][Image for post 4]

![Image for post][Image for post 5]

We saw the warning because we chained two indexing operations. In the background, these chained operations are executed independently. The first part is the *get* operation, which returns a `DataFrame` containing all the rows in which the value of `B` is higher than 12. The second part is the *set*operation, and is carried out on the new `DataFrame` created by the *get*operation. So we are not modifying the original `DataFrame`!

This is quite obvious when we used two square brackets in a row, however, the same would happen by using `loc`, `iloc` or the dot method of accessing the column. For example, running `X.loc[X[‘B’] > 12][‘C’] = 999` would give the same incorrect result.

To properly replace the values within the `DataFrame`, we need to use `loc` in the following way:

X.loc[X['B'] > 12, 'C'] = 999
    X[X['B'] > 12]['C']

![Image for post][Image for post 6]

![Image for post][Image for post 7]

We can see that the values were successfully replaced in the original `DataFrame`.

## 2. Hidden Chaining ##

Hidden chaining can be quite a tricky problem to debug, as it is often not that quite obvious where exactly the problem lies. We will go over an example. First, let’s load the data and using the knowledge from the previous case, create a `DataFrame` that is a subset of the original one. We filter out all the rows with the value of the `C` feature larger than 101.

![Image for post][Image for post 8]

![Image for post][Image for post 9]

It often happens that we then explore and further process the new `DataFrame`. Let’s imagine running a few lines of code to further inspect the `temp` object, such as the `shape`, `describe`, `plot` methods of a `pandas.DataFrame`.

We are not actually printing the output here, as this is not the important part. Now, after running a few more lines of code, let’s replace the value of the `C` feature in the first row of `temp` with 999:

temp.loc[2, 'C'] = 999

While doing so, we meet our old friend:

SettingWithCopyWarning: 
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

Let’s inspect the values of the`C` in both the original and extracted `DataFrames`:

print(f"New DataFrame: {temp.loc[2, 'C']}")
    print(f"Original DataFrame: {X.loc[2, 'C']}")# New DataFrame: 999
    # Original DataFrame: 102

So we actually happened? The reason for the warning lies in the fact that chained indexing can occur in two lines, not only within one. When we created the new `DataFrame`, we used the output of the *get* operation. That might have been a copy of the original `DataFrame` or it might have not. There was no way to know until we checked. Quoting the `pandas`documentation on the chained indexing:

> Outside of simple cases, it’s very hard to predict whether it \[chained indexing\] will return a view or a copy (it depends on the memory layout of the array, about which pandas makes no guarantees),…

So when we indexed `temp` to assign the new value, we actually used chained indexing. As a result, we might have also modified `X`, while modifying `temp`.

The tricky part is that in an actual codebase, the two lines of code that are responsible for the hidden chained assignment can be separated by dozens of lines, which makes identifying the potential problem quite difficult.

To solve that problem, we can directly instruct `pandas` to create a copy of the original `DataFrame` by using the `copy` method. Below, we use that approach to avoid the hidden chaining:

We see that running the code generates the correct output.

## 3. An example of a false negative ##

Lastly, we go over the case of a false negative (`pandas` not informing us about `SettingWithCopyWarning`, when it actually should have) mentioned in \[3\]. It happens in cases when we are using chained indexing while slicing multiple columns of the `DataFrame`. Let’s look at two simple cases.

X = get_data()
    X.loc[X['A'] > 2, ['A', 'B']]['A'] = 999
    X

and

X = get_data()
    X[['A', 'B', 'C']]['A'] = 999
    X

both produce the following outcome, without actually showing the `SettingWithCopyWarning`.

![Image for post][Image for post 10]

![Image for post][Image for post 11]

As we can see from the image above, the values were not modified as we wanted. Due to including multiple columns in the indexing operation, the warning was not displayed. We can easily verify that this does not happen in the case of a single column — running `X[[‘C’]][‘C’] = 999`  
produces the warning and does not modify `X`.

# Some deeper context on the origin of the warning #

We can say that `pandas` inherited the concept of views and copies from `numpy`. Under the hood, `pandas` uses `numpy` for efficient data storage and manipulation. In `numpy`, the views and copies follow a certain set of rules and are returned in a predictable manner (see \[5\] for more information). So why that is not the case with `pandas`? The problem lies in the fact that `numpy` arrays are limited to a single data type. And as we know, that is not the case for `pandas`.

In practice, indexing (*get* operation) on a multi dtype `DataFrame` will always return a copy of the frame. The same operation on a single type frame will almost always return a view based on a single `numpy` array, which is the most efficient way of approaching the problem. However, as we already saw in the quote from the documentation, returning the view depends on the memory layout of the object and is unfortunately not guaranteed.

Summing up, `pandas` does its best to combine its versatile approach to indexing (thanks to which it is vastly popular and basically a prerequisite to doing data science in Python) and the efficiency of using the underlying `numpy` arrays. This results in some small nuisances, however, the trade-off is definitely worth it and the problems can be overcome with the proper understanding of how `pandas` works under the hood.

# Conclusions #

In this article, I explained the difference between copies and views in pandas and how they are related to the infamous `SettingWithCopyWarning`. The main idea narrows down to being aware of what chained indexing is and how to successfully avoid it. The general rules are:

*  if you want to change the original `DataFrame`, use the single assignment.
 *  if you want to make a copy of the `DataFrame`, do so explicitly using the `copy` method.

Following these two rules can save you a lot of time on debugging some weird cases, especially in a lengthy codebase.

It is also worth mentioning that the `SettingWithCopyWarning` occurs only when we are using the *set* operations (assigning). However, it is best to avoid chained indexing for the *get* operations as well. That is because chained operations are generally slower and could cause issues in case you later decide to add the *set* operations to the code.

You can find the code used for this article on my [GitHub][]. As always, any constructive feedback is welcome. You can reach out to me on [Twitter][] or in the comments.

[Image for post]: /images/20221122/68c652f834d04399bdf7ced53a231cf3.png
[Image for post 1]: /images/20221122/ef417ec4252f46a7b3696684c041134e.png
[Image for post 2]: /images/20221122/beb431f48aeb49b89421fddcd3d820ed.png
[Image for post 3]: /images/20221122/1e92b1d7163e4507bd52eb778a4ce669.png
[Image for post 4]: /images/20221122/61e60a863eb34451b8f768ccd75d3337.png
[Image for post 5]: /images/20221122/c270d948a187417281e8041242ece9b9.png
[Image for post 6]: /images/20221122/d54381089f21475ca3e69be431129936.png
[Image for post 7]: /images/20221122/e89c2b160e454ee4a6e0c6b7a2986731.png
[Image for post 8]: /images/20221122/0042edf864d74ee8886cd722be3fe919.png
[Image for post 9]: /images/20221122/93c9115eff444193bccc76eb939ddd68.png
[Image for post 10]: /images/20221122/4ae53c240cc4406ab2c938b11c32c8aa.png
[Image for post 11]: /images/20221122/b8fe64bc77fb4d4f86ac39b13350f04b.png
[GitHub]: https://github.com/erykml/medium_articles/blob/master/Data%20Wrangling/SettingwithCopyWarning_explained.ipynb
[Twitter]: https://twitter.com/erykml1?source=post_page---------------------------