Intel微处理器Uncore架构简介

╰+哭是因爲堅強的太久メ 2022-12-21 09:24 629阅读 0赞

uncore一词，是英特尔用来描述微处理器中，功能上为非处理器核心（Core）所负担，但是对处理器性能的发挥和维持有必不可少的作用的组成部分。处理器核心（Core）包含的处理器组件都涉及处理器命令的运行，包括算术逻辑单元（ALU）、浮点运算单元（FPU）、一级缓存（L1 Cache）、二级缓存（L2 Cache）。Uncore的功能包括QPI控制器、三级缓存（L3 Cache）、内存一致性监测（snoop agent pipeline）、内存控制器，以及Thunderbolt控制器。至于其余的总线控制器，像是PCI-E、SPI等，则是属于芯片组的一部分。

英特尔Uncore设计根源，来自于北桥芯片。Uncroe的设计是，将对于处理器核心有关键作用的功能重新组合编排，从物理上使它们更靠近核心（集成至处理器芯片上，而它们有部分原来是位于北桥上），以降低它们的访问延时。而北桥上余下的和处理器核心并无太大关键作用的功能，像是PCI-E控制器或者是电源控制单元（PCU），并没有集成至Uncore部分，而是继续作为芯片组的一部分。

具体而言，微架构中的uncore是被细分为数个模块单元的。uncore连接至处理器核心是通过一个叫Cache Box（CBox）的接口实现的，CBox也是末级缓存（Last Level Cache，LLC）的连接接口，同时负责管理缓存一致性。复合的内部与外部QPI链接由物理层单元（Physical Layer units）管理，称为PBox。PBox、CBox以及一个或更多的内置存储器控制器（iMC，作MBox）的连接由系统配置控制器（System Config Controller，作UBox）和路由器（Router，作RBox）负责管理。

从uncore部分移出列表总线控制器，可以更好地促进性能的提升，通过允许uncore的时钟频率（UCLK）运作于基准的2.66GHz，提升至超过超频限制值的3.44GHz，实现性能提升。这种时脉提升使得核心访问关键功能部件（像是存储器控制器）时的延时值更低（典型情况下处理器核心访问DRAM的时间可降低10纳秒或更多）。

Linux会把识别出的Uncore设备列在sysfs中：

# ls -d /sys/devices/uncore_*
    /sys/devices/uncore_cha_0   /sys/devices/uncore_imc_1
    /sys/devices/uncore_cha_1   /sys/devices/uncore_imc_2
    /sys/devices/uncore_cha_10  /sys/devices/uncore_imc_3
    /sys/devices/uncore_cha_11  /sys/devices/uncore_imc_4
    /sys/devices/uncore_cha_2   /sys/devices/uncore_imc_5
    /sys/devices/uncore_cha_3   /sys/devices/uncore_irp_0
    /sys/devices/uncore_cha_4   /sys/devices/uncore_irp_1
    /sys/devices/uncore_cha_5   /sys/devices/uncore_irp_2
    /sys/devices/uncore_cha_6   /sys/devices/uncore_irp_3
    /sys/devices/uncore_cha_7   /sys/devices/uncore_irp_4
    /sys/devices/uncore_cha_8   /sys/devices/uncore_irp_5
    /sys/devices/uncore_cha_9   /sys/devices/uncore_m2m_0
    /sys/devices/uncore_iio_0   /sys/devices/uncore_m2m_1
    /sys/devices/uncore_iio_1   /sys/devices/uncore_m3upi_0
    /sys/devices/uncore_iio_2   /sys/devices/uncore_m3upi_1
    /sys/devices/uncore_iio_3   /sys/devices/uncore_pcu
    /sys/devices/uncore_iio_4   /sys/devices/uncore_ubox
    /sys/devices/uncore_iio_5   /sys/devices/uncore_upi_0
    /sys/devices/uncore_imc_0   /sys/devices/uncore_upi_1

PMU Tools  
开源项目pmu-tools集成了ucevent(uncore event)的工具，可以查看并跟踪uncore事件。使用ucevnent.py列出支持的uncore events：

cd pmu-tools/ucevent
    ./ucevent.py
    
    CHA (Home Agent) CACHE Events
      CHA.LLC_DRD_MISS_PCT           LLC DRd Miss Percentage
    
    CHA (Home Agent) HA (Home Agent) REQUEST Events
      CHA.PCT_RD_REQUESTS            Percent Read Requests
      CHA.PCT_WR_REQUESTS            Percent Write Requests
    
    CHA (Home Agent) INGRESS Events
      CHA.AVG_INGRESS_DEPTH          Average Ingress (from CMS) Depth
      CHA.AVG_INGRESS_LATENCY        Average Ingress (from CMS) Latency
      CHA.AVG_INGRESS_LATENCY_WHEN_NE Average Latency in Non-Empty Ingress (from CMS)
      CHA.CYC_INGRESS_BLOCKED        Cycles Ingress (from CMS) Blocked
      CHA.INGRESS_REJ_V_INS          Ingress (from CMS) Rejects vs. Inserts
    
    CHA (Home Agent) TOR (Table of Requests, pending transactions) Events
      CHA.AVG_CRD_MISS_LATENCY       Average Code Read Latency
      CHA.AVG_DEMAND_RD_HIT_LATENCY  Average Data Read Hit Latency
      CHA.AVG_DEMAND_RD_MISS_LOCAL_LATENCY Average Data Read Local Miss Latency
      CHA.AVG_DRD_MISS_LATENCY       Average Data Read Miss Latency
      CHA.AVG_IA_CRD_LLC_HIT_LATENCY Average Code Read Latency
      CHA.AVG_RFO_MISS_LATENCY       Average RFO Latency
      CHA.AVG_TOR_DRDS_MISS_WHEN_NE  Average Data Read Misses in Non-Empty TOR
      CHA.AVG_TOR_DRDS_WHEN_NE       Average Data Reads in Non-Empty TOR
      CHA.FAST_STR_LLC_HIT           Fast String operations
      CHA.FAST_STR_LLC_MISS          Fast String misses
      CHA.LLC_CRD_MISS_TO_LOC_MEM    LLC Code Read Misses to Local Memory
      CHA.LLC_CRD_MISS_TO_REM_MEM    LLC Code Read Misses to Remote Memory
      CHA.LLC_DRD_MISS_TO_LOC_MEM    LLC Data Read Misses to Local Memory
      CHA.LLC_DRD_MISS_TO_REM_MEM    LLC Data Read Misses to Remote Memory
      CHA.LLC_DRD_PREFETCH_HITS      DRd Prefetches that Hit the LLC
      ...

比如跟踪10秒内不同Socket上的PCI带宽：

# ucevent.py -I 2000 CBO.PCIE_DATA_BYTES sleep 10
    S0-CBO.PCIE_DATA_BYTES
    |      S1-CBO.PCIE_DATA_BYTES
    384.00 256.00
    0.00   256.00
    0.00   0.00
    0.00   0.00

再比如跟踪某个应用在内存控制器(iMC)上统计获得的缺页次数，一般情况下，这个指标可以用来衡量内存延时:`# ucevent.py iMC.PCT_REQUESTS_PAGE_MISS my-workload`

## PCM (Processor Counter Monitor) ##

PCM提供了一些监控LLC, PCI，QPI的工具。其中，pcm.x可以监控core, cache miss/hit和各个socket的upi带宽：

./pcm.x
    ...
    
    Detected Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz "Intel(r) microarchitecture codename Skylake-SP" stepping 4 microcode level 0x2000043
    
     EXEC  : instructions per nominal CPU cycle
     IPC   : instructions per CPU cycle
     FREQ  : relation to nominal CPU frequency='unhalted clock ticks'/'invariant timer ticks' (includes Intel Turbo Boost)
     AFREQ : relation to nominal CPU frequency while in active state (not in power-saving C state)='unhalted clock ticks'/'invariant timer ticks while in C0-state'  (includes Intel Turbo Boost)
     L3MISS: L3 (read) cache misses
     L2MISS: L2 (read) cache misses (including other core's L2 cache *hits*)
     L3HIT : L3 (read) cache hit ratio (0.00-1.00)
     L2HIT : L2 cache hit ratio (0.00-1.00)
     L3MPI : number of L3 (read) cache misses per instruction
     L2MPI : number of L2 (read) cache misses per instruction
     READ  : bytes read from main memory controller (in GBytes)
     WRITE : bytes written to main memory controller (in GBytes)
     L3OCC : L3 occupancy (in KBytes)
     TEMP  : Temperature reading in 1 degree Celsius relative to the TjMax temperature (thermal headroom): 0 corresponds to the max temperature
     energy: Energy in Joules
    
    
     Core (SKT) | EXEC | IPC  | FREQ  | AFREQ | L3MISS | L2MISS | L3HIT | L2HIT | L3MPI | L2MPI |   L3OCC | TEMP
    
       0    0     0.01   0.37   0.04    1.17      36 K    103 K    0.54    0.63    0.00    0.00      576     47
       1    1     0.00   0.88   0.00    1.17    4602       13 K    0.64    0.68    0.00    0.00      432     56
       2    0     0.01   0.82   0.01    1.17      20 K     61 K    0.57    0.70    0.00    0.00      384     48
       3    1     0.01   0.87   0.01    1.17      16 K     41 K    0.56    0.72    0.00    0.00     2016     57
       4    0     0.01   0.78   0.01    1.17      18 K     66 K    0.67    0.66    0.00    0.00      672     49
       5    1     0.00   0.51   0.01    1.17      14 K     31 K    0.51    0.62    0.00    0.00     1152     58
       6    0     0.01   0.65   0.01    1.17      21 K     72 K    0.65    0.61    0.00    0.00     1968     50
       7    1     0.00   0.82   0.00    1.17    6567       13 K    0.45    0.60    0.00    0.00      480     58
    ...
    
    Intel(r) UPI data traffic estimation in bytes (data traffic coming to CPU/socket through UPI links):
    
                   UPI0     UPI1    |  UPI0   UPI1
    ---------------------------------------------------------------------------------------------------------------
     SKT    0       43 M     43 M   |    0%     0%
     SKT    1       32 M     31 M   |    0%     0%
    ---------------------------------------------------------------------------------------------------------------
    ...

pcm.memory.x可以用来监控各个内存通道的读写带宽：

# ./pcm-memory.x
    
     Processor Counter Monitor: Memory Bandwidth Monitoring Utility  ($Format:%ci ID=%h$)
    
     This utility measures memory bandwidth per channel or per DIMM rank in real-time
    
    Number of physical cores: 24
    Number of logical cores: 48
    Number of online logical cores: 48
    Threads (logical cores) per physical core: 2
    Num sockets: 2
    Physical cores per socket: 12
    Core PMU (perfmon) version: 4
    Number of core PMU generic (programmable) counters: 4
    Width of generic (programmable) counters: 48 bits
    Number of core PMU fixed counters: 3
    Width of fixed counters: 48 bits
    Nominal core frequency: 2300000000 Hz
    Package thermal spec power: 105 Watt; Package minimum power: 64 Watt; Package maximum power: 231 Watt;
    Socket 0: 2 memory controllers detected with total number of 6 channels. 2 QPI ports detected.
    Socket 1: 2 memory controllers detected with total number of 6 channels. 2 QPI ports detected.
    
    Detected Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz "Intel(r) microarchitecture codename Skylake-SP" stepping 4 microcode level 0x2000043
    Update every 1 seconds
    |---------------------------------------||---------------------------------------|
    |--             Socket  0             --||--             Socket  1             --|
    |---------------------------------------||---------------------------------------|
    |--     Memory Channel Monitoring     --||--     Memory Channel Monitoring     --|
    |---------------------------------------||---------------------------------------|
    |-- Mem Ch  0: Reads (MB/s):    18.91 --||-- Mem Ch  0: Reads (MB/s):    16.60 --|
    |--            Writes(MB/s):    23.22 --||--            Writes(MB/s):    18.96 --|
    |-- Mem Ch  1: Reads (MB/s):    18.98 --||-- Mem Ch  1: Reads (MB/s):    16.80 --|
    |--            Writes(MB/s):    23.39 --||--            Writes(MB/s):    19.15 --|
    |-- Mem Ch  2: Reads (MB/s):    18.77 --||-- Mem Ch  2: Reads (MB/s):    16.35 --|
    |--            Writes(MB/s):    23.05 --||--            Writes(MB/s):    18.71 --|
    |-- Mem Ch  3: Reads (MB/s):    17.89 --||-- Mem Ch  3: Reads (MB/s):    15.75 --|
    |--            Writes(MB/s):    22.94 --||--            Writes(MB/s):    18.44 --|
    |-- Mem Ch  4: Reads (MB/s):    17.87 --||-- Mem Ch  4: Reads (MB/s):    15.88 --|
    |--            Writes(MB/s):    22.78 --||--            Writes(MB/s):    18.55 --|
    |-- Mem Ch  5: Reads (MB/s):    18.04 --||-- Mem Ch  5: Reads (MB/s):    16.80 --|
    |--            Writes(MB/s):    22.93 --||--            Writes(MB/s):    19.64 --|
    |-- NODE 0 Mem Read (MB/s) :   110.47 --||-- NODE 1 Mem Read (MB/s) :    98.18 --|
    |-- NODE 0 Mem Write(MB/s) :   138.31 --||-- NODE 1 Mem Write(MB/s) :   113.46 --|
    |-- NODE 0 P. Write (T/s):      18728 --||-- NODE 1 P. Write (T/s):      18732 --|
    |-- NODE 0 Memory (MB/s):      248.78 --||-- NODE 1 Memory (MB/s):      211.63 --|
    |---------------------------------------||---------------------------------------|
    |---------------------------------------||---------------------------------------|
    |--                 System Read Throughput(MB/s):        208.65                --|
    |--                System Write Throughput(MB/s):        251.77                --|
    |--               System Memory Throughput(MB/s):        460.42                --|
    |---------------------------------------||---------------------------------------|

numa带宽监控

# ./pcm-numa.x #

…  
Detected Intel® Xeon® Gold 5118 CPU @ 2.30GHz “Intel® microarchitecture codename Skylake-SP” stepping 4 microcode level 0x2000043  
Update every 1.0 seconds  
Time elapsed: 1000 ms  
Core | IPC | Instructions | Cycles | Local DRAM accesses | Remote DRAM Accesses  
0 0.37 30 M 81 M 41 K 90 K  
1 0.71 10 M 14 M 90 K 20 K  
2 0.64 14 M 22 M 20 K 59 K  
3 1.10 79 M 71 M 7264 21 K  
4 0.82 26 M 31 M 33 K 48 K  
5 0.95 12 M 13 M 8131 27 K  
6 0.37 13 M 35 M 306 K 111 K  
7 1.00 7417 K 7432 K 9991 9136  
8 0.76 12 M 15 M 13 K 6833  
9 1.03 14 M 13 M 10 K 19 K  
10 0.56 6269 K 11 M 42 K 50 K  
11 0.92 26 M 29 M 60 K 85 K  
12 0.87 14 M 16 M 17 K 10 K  
13 0.64 4130 K 6504 K 5615 40 K  
14 0.88 32 M 36 M 57 K 51 K  
15 0.88 10 M 11 M 8689 16 K  
…