Python之pyarrow：pyarrow的简介、安装、使用方法之详细攻略

爱被打了一巴掌 2022-11-20 08:24 542阅读 0赞

**目录**

pyarrow的简介

1、项目的主要组成部分包括:

2、Arrow特点

pyarrow的安装

pyarrow的使用方法

--------------------

# pyarrow的简介 #

用于Apache Arrow的Python库。这个库为Arrow c++库提供的功能提供了Python API，以及用于与panda、NumPy和Python生态系统中的其他软件进行箭头集成和互操作性的工具。Apache Arrow是一个用于内存分析的开发平台。它包含一组技术，使大数据系统能够快速处理和移动数据。

## 1、项目的主要组成部分包括: ##

*  [The Arrow Columnar In-Memory Format][]: a standard and efficient in-memory representation of various datatypes, plain or nested
 *  [The Arrow IPC Format][]: an efficient serialization of the Arrow format and associated metadata, for communication between processes and heterogeneous environments
 *  [The Arrow Flight RPC protocol][]: based on the Arrow IPC format, a building block for remote services exchanging Arrow data with application-defined semantics (for example a storage server or a database)
 *  [C++ libraries][C_ libraries]
 *  [C bindings using GLib][]
 *  [C\# .NET libraries][C_ .NET libraries]
 *  [Gandiva][]: an [LLVM][]\-based Arrow expression compiler, part of the C++ codebase
 *  [Go libraries][]
 *  [Java libraries][]
 *  [JavaScript libraries][]
 *  [Plasma Object Store][]: a shared-memory blob store, part of the C++ codebase
 *  [Python libraries][]
 *  [R libraries][]
 *  [Ruby libraries][]
 *  [Rust libraries][]

## 2、Arrow特点 ##

*  **Format**： Apache Arrow为平面和层次数据定义了一种与语言无关的柱状内存格式，这种格式的组织用于在现代硬件(如cpu和gpu)上进行高效的分析操作。Arrow内存格式还支持零拷贝读取，用于不需要序列化开销的闪电般的数据访问。
 *  **Libraries**： Arrow的库实现了这种格式，并为一系列用例提供了构建块，包括高性能分析。许多流行的项目使用箭头有效地发送柱状数据，或者作为分析引擎的基础。库可用于C、c++、c\#、Go、Java、JavaScript、MATLAB、Python、R、Ruby和Rust。请参阅如何安装和启动。
 *  **Ecosystem**：Apache Arrow是由开发人员社区创建并为其服务的软件。我们致力于开放、友好的沟通和协商一致的决策。我们的提交者来自不同的组织和背景，我们欢迎所有人与我们一起参与。

# pyarrow的安装 #

pip install --user -i https://pypi.tuna.tsinghua.edu.cn/simple pyarrow

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxMTg1ODY4_size_16_color_FFFFFF_t_70][]

# pyarrow的使用方法 #

Join the mailing list: send an email to dev-subscribe@arrow.apache.org. Share your ideas and use cases for the project.
    Follow our activity on JIRA
    Learn the format
    Contribute code to one of the reference implementations

[The Arrow Columnar In-Memory Format]: https://github.com/apache/arrow/blob/master/docs/source/format/Columnar.rst
[The Arrow IPC Format]: https://github.com/apache/arrow/blob/master/docs/source/format/Columnar.rst#serialization-and-interprocess-communication-ipc
[The Arrow Flight RPC protocol]: https://github.com/apache/arrow/tree/master/format/Flight.proto
[C_ libraries]: https://github.com/apache/arrow/tree/master/cpp
[C bindings using GLib]: https://github.com/apache/arrow/tree/master/c_glib
[C_ .NET libraries]: https://github.com/apache/arrow/tree/master/csharp
[Gandiva]: https://github.com/apache/arrow/tree/master/cpp/src/gandiva
[LLVM]: https://llvm.org/
[Go libraries]: https://github.com/apache/arrow/tree/master/go
[Java libraries]: https://github.com/apache/arrow/tree/master/java
[JavaScript libraries]: https://github.com/apache/arrow/tree/master/js
[Plasma Object Store]: https://github.com/apache/arrow/tree/master/cpp/src/plasma
[Python libraries]: https://github.com/apache/arrow/tree/master/python
[R libraries]: https://github.com/apache/arrow/tree/master/r
[Ruby libraries]: https://github.com/apache/arrow/tree/master/ruby
[Rust libraries]: https://github.com/apache/arrow/tree/master/rust
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxMTg1ODY4_size_16_color_FFFFFF_t_70]: /images/20221120/75586945c0d54a98a2d2aede057d0bae.png