TrueNAS Scale の監視を考える

Summary ¶

TrueNAS Scale を本番投入するために、監視を確認しておく。
弊宅では Zabbix を活用してほぼ全ての機器を監視しているので、できればそれに合わせたい。

Alerts ¶

TrueNAS Scale には Alerts 機能が標準搭載されている

アラートの送信先として E-Mail, InfluxDB, Mattermost, OpsGenie, PagerDuty, Slack, SNMP Trap, Telegram or VictorOPS が利用できる。

アラートの種類としては、下記となっているこれを調べた時点で Zabbix でアラート通知の仕組みを構築しなくても、ことが足りそうな雰囲気があるためアラート通知は TrueNAS Scale の機能で組んで、どうしても痒い pools 使用量や Datasets の統計などはグラフとしてほしいため Zabbix に実装することにした API を使って監視を作る計画をしていた所だが、煩雑になること後1ヶ月ぐらいで 24.04 が出てくることを鑑みて控えめに対応することに

TrueNAS の Level 判定は下記で、調べた限りでは指定レベルより高いのは通知が来る様子のためたいていは Warning を登録しておけば間違えなさそうである

1
2
3
4
5
6
7
8
9
↑ 低レベル
Info
Notice
Warning
Error
Critical
Alert
Emergency
↓ ヤブァい

Applications ¶

Description	Set Warning Level	Set Frequency
Application Update Available	INFO	IMMEDIATELY
Catalog Not Healthy	WARNING	IMMEDIATELY
Unable to Configure Applications	CRITICAL	IMMEDIATELY
Unable to Start Applications	CRITICAL	IMMEDIATELY
Unable to Sync Catalog	CRITICAL	IMMEDIATELY

Certificates ¶

Description	Set Warning Level	Set Frequency
Certificate Has Expired	CRITICAL	IMMEDIATELY
Certificate Is Expiring	NOTICE	IMMEDIATELY
Certificate Is Expiring Soon	WARNING	IMMEDIATELY
Certificate Parsing Failed	WARNING	IMMEDIATELY
Certificate Revoked	CRITICAL	IMMEDIATELY
Web UI HTTPS Certificate Setup Failed	CRITICAL	IMMEDIATELY

Clustering ¶

Description	Set Warning Level	Set Frequency
CTDB service initialization failed	WARNING	IMMEDIATELY
Clustered account name(s) collide with local non-clustered accounts	CRITICAL	IMMEDIATELY
Clustered configuration volume lacks adequate redundancy	WARNING	IMMEDIATELY
Clustered service start failed	WARNING	IMMEDIATELY
Clustered time consistency check failed	WARNING	IMMEDIATELY
Glusterd UUID changed	CRITICAL	IMMEDIATELY
Glusterd peer information is unavailable	CRITICAL	IMMEDIATELY

Directory Service ¶

Description	Set Warning Level	Set Frequency
Active Directory Bind Is Not Healthy	WARNING	IMMEDIATELY
Active Directory Domain Validation Failed	WARNING	IMMEDIATELY
LDAP Bind Is Not Healthy	WARNING	IMMEDIATELY

Hardware ¶

Description	Set Warning Level	Set Frequency
Critical IPMI Firmware Update Available	CRITICAL	IMMEDIATELY
Disk(s) Are Formatted With Data Integrity Feature (DIF).	CRITICAL	IMMEDIATELY
IPMI System Event	WARNING	IMMEDIATELY
IPMI System Event Log Low Space Left	WARNING	IMMEDIATELY
S.M.A.R.T. Error	CRITICAL	IMMEDIATELY
smartd Is Not Running	WARNING	IMMEDIATELY

Network ¶

Description	Set Warning Level	Set Frequency
Ports are Not ACTIVE on LAGG Interface	CRITICAL	IMMEDIATELY
There are No ACTIVE Ports on LAGG Interface	CRITICAL	IMMEDIATELY

Reporting ¶

Description	Set Warning Level	Set Frequency
syslog-ng Is Not Running	WARNING	IMMEDIATELY

Description	Set Warning Level	Set Frequency
Deprecated Service Configuration Detected	WARNING	IMMEDIATELY
Deprecated Service is Running	WARNING	IMMEDIATELY
IP Addresses Bound to an iSCSI Portal Were Not Found	WARNING	IMMEDIATELY
NFS Services Could Not Bind to Specific IP Addresses, Using 0.0.0.0	WARNING	IMMEDIATELY
NFS shares reference hosts that could not be resolved	WARNING	IMMEDIATELY
NFS start is blocked by entries in /etc/exports.d	ERROR	IMMEDIATELY
NTLMv1 authentication has been attempted in the last 24 hours	WARNING	IMMEDIATELY
SMB share path has unresolvable issues	CRITICAL	IMMEDIATELY
SMB1 connections to TrueNAS server have been performed in last 24 hours	NOTICE	IMMEDIATELY
Share Is Unavailable Because It Uses A Locked Dataset	WARNING	IMMEDIATELY

Storage ¶

Description	Set Warning Level	Set Frequency
Critical Quota Exceeded on Dataset	CRITICAL	IMMEDIATELY
New Feature Flags Are Available for Pool	NOTICE	IMMEDIATELY
Pool Space Usage Is Above 70%	NOTICE	IMMEDIATELY
Pool Space Usage Is Above 80%	WARNING	IMMEDIATELY
Pool Space Usage Is Above 90%	CRITICAL	IMMEDIATELY
Pool Status Is Not Healthy	CRITICAL	IMMEDIATELY
Pool consuming USB disks	WARNING	IMMEDIATELY
Quota Exceeded on Dataset	WARNING	IMMEDIATELY
Scrub Is Paused	WARNING	IMMEDIATELY
Too Many Snapshots Exist	WARNING	IMMEDIATELY
Too Many Snapshots Exist For Dataset	WARNING	IMMEDIATELY

System ¶

Description	Set Warning Level	Set Frequency
Admin User Is Overridden	WARNING	IMMEDIATELY
Boot Pool Is Not Healthy	CRITICAL	IMMEDIATELY
Core Files Detected	WARNING	IMMEDIATELY
Failover Event Caused System Reboot	WARNING	IMMEDIATELY
Fenced Caused System Reboot	WARNING	IMMEDIATELY
NTP Health Check Failed	WARNING	IMMEDIATELY
Pending Confirmation From iX Portal for TrueCommand API Key	INFO	IMMEDIATELY
SSH Login Failures	WARNING	IMMEDIATELY
System Not Ready For Kdump	WARNING	IMMEDIATELY
The Web Interface Сould Not Bind to Configured Address	WARNING	IMMEDIATELY
TrueCommand API Key Disabled by iX Portal	CRITICAL	IMMEDIATELY
TrueCommand Container Failed Scheduled Health Check	CRITICAL	IMMEDIATELY
TrueCommand Service Failed Scheduled Health Check	CRITICAL	IMMEDIATELY
Unencrypted datasets detected within encrypted datasets	WARNING	IMMEDIATELY
Update Failed	CRITICAL	IMMEDIATELY

Tasks ¶

Description	Set Warning Level	Set Frequency
Cloud Backup Task Failed	ERROR	IMMEDIATELY
Cloud Sync Task Failed	ERROR	IMMEDIATELY
Creating VMWare Snapshot Failed	WARNING	IMMEDIATELY
Replication Failed	CRITICAL	IMMEDIATELY
Replication Succeeded	INFO	IMMEDIATELY
Rsync Task Failed	CRITICAL	IMMEDIATELY
Rsync Task Succeeded	INFO	IMMEDIATELY
Scrub Failed to Start	WARNING	IMMEDIATELY
Scrub Finished	INFO	IMMEDIATELY
Scrub Started	INFO	IMMEDIATELY
Snapshot Task Failed	CRITICAL	IMMEDIATELY
Task Is Unavailable Because It Uses A Locked Dataset	WARNING	IMMEDIATELY
VMWare Login Failed	WARNING	IMMEDIATELY
VMWare Snapshot Deletion Failed	WARNING	IMMEDIATELY

UPS ¶

Description	Set Warning Level	Set Frequency
UPS Battery LOW	ALERT	IMMEDIATELY
UPS Battery Needs Replacement	CRITICAL	IMMEDIATELY
UPS Communication Established	INFO	IMMEDIATELY
UPS Communication Lost	CRITICAL	IMMEDIATELY
UPS On Battery	CRITICAL	IMMEDIATELY
UPS On Line Power	INFO	IMMEDIATELY

Disks ¶

TrueNAS Scale には Disk の温度で通知する仕組みがある
これを正しく設定しておくことで製品仕様の温度外になりそうな時に警告を受け取れるようにしておく
仕組みの記載がドキュメントになく単に Informational, Difference, Critical の温度が設定できるだけしかわからないため各製品の対応温度で弊宅は設定しておくことにした。複数ある場合は一括では設定できないので苦行である

Vendor	Model	dev name	Informational	Difference	Critical
WD	Red SN700 500GB (WDS500G1R0C)	nvme0n1, nvme1n1		60	70
Buffalo	SSD-PSTU3A	sda	`-`	`-`	`-`
TOSHIBA	MG08 (MG08ACA16TE)	sdc		50	55
WDC	Ultrastar DC HC550 (WUH721816ALE6L4)	sdb, sdd	50	55	60
Samsung	Samsung SSD 870 EVO 1TB (MZ-77E1T0)	sde,sdf	50	60	70

SSD-PSTU3A は温度を取れなかった

Red SN700 500GB (WDS500G1R0C)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.1.74-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       WD Red SN700 500GB
Serial Number:                      XXXXXXXXXXXXX
Firmware Version:                   111150WD
PCI Vendor/Subsystem ID:            0x15b7
IEEE OUI Identifier:                0x001b44
Total NVM Capacity:                 500,107,862,016 [500 GB]
Unallocated NVM Capacity:           0
Controller ID:                      8215
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          500,107,862,016 [500 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            001b44 8b4a4349b3
Local Time is:                      Sat Mar 23 01:12:09 2024 JST
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0e):         Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size:         128 Pages
Warning  Comp. Temp. Threshold:     84 Celsius
Critical Comp. Temp. Threshold:     88 Celsius
Namespace 1 Features (0x02):        NA_Fields

Toshiba MG08ACA16TE

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.1.74-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba MG08ACA... Enterprise Capacity HDD
Device Model:     TOSHIBA MG08ACA16TE
Serial Number:    XXXXXXXXXXXX
LU WWN Device Id: 5 000039 cc8c808ca
Firmware Version: 0103
User Capacity:    16,000,900,661,248 bytes [16.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Mar 23 01:23:15 2024 JST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   050    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0027   100   100   001    Pre-fail  Always       -       7981
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       423
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   050    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   050    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0032   097   097   000    Old_age   Always       -       1273
 10 Spin_Retry_Count        0x0033   100   100   030    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       3
 23 Helium_Condition_Lower  0x0023   100   100   075    Pre-fail  Always       -       0
 24 Helium_Condition_Upper  0x0023   100   100   075    Pre-fail  Always       -       0
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       1
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       424
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       42 (Min/Max 15/44)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
220 Disk_Shift              0x0002   100   100   000    Old_age   Always       -       85721089
222 Loaded_Hours            0x0032   100   100   000    Old_age   Always       -       39
223 Load_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
224 Load_Friction           0x0022   100   100   000    Old_age   Always       -       0
226 Load-in_Time            0x0026   100   100   000    Old_age   Always       -       531
240 Head_Flying_Hours       0x0001   100   100   001    Pre-fail  Offline      -       0

Ultrastar DC HC550 (WUH721816ALE6L4)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.1.74-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Ultrastar DC HC550
Device Model:     WDC  WUH721816ALE6L4
Serial Number:    XXXXXXXX
LU WWN Device Id: 5 000cca 2aaccbe88
Firmware Version: PCGNW680
User Capacity:    16,000,900,661,248 bytes [16.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-4 published, ANSI INCITS 529-2018
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Mar 23 01:13:21 2024 JST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   001    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   148   148   054    Pre-fail  Offline      -       48
  3 Spin_Up_Time            0x0007   084   084   001    Pre-fail  Always       -       330 (Average 329)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       32
  5 Reallocated_Sector_Ct   0x0033   100   100   001    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   001    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   128   128   020    Pre-fail  Offline      -       18
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       6149
 10 Spin_Retry_Count        0x0013   100   100   001    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       28
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       290
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       290
194 Temperature_Celsius     0x0002   051   051   000    Old_age   Always       -       42 (Min/Max 19/53)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   100   100   000    Old_age   Always       -       0

Samsung SSD 870 EVO 1TB (MZ-77E1T0)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.1.74-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 870 EVO 1TB
Serial Number:    XXXXXXXXXXXXXXX
LU WWN Device Id: 5 002538 f33523259
Firmware Version: SVT03B6Q
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Mar 23 01:15:38 2024 JST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       3473
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       20
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always       -       15
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   062   038   000    Old_age   Always       -       38
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       5
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       16822188654
252 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0

通知先 ¶

ということで弊宅では通知先として Discord を利用している Slack を長年使ってきたが履歴が90日で消えるようになったりしてしまったため乗り換えた。

今回 TrueNAS Scale から Discord に通知を出したいのだが、標準では Discord に対応していない。Discord のドキュメントを確認すると Discord Developer Portal — Documentation — Webhook というのがある。これは Slack 互換の Webhook で Discord が受けてくれる機能のようなのでこれを使う

Discord の連携画面で Webhook を発行すると https://discordapp.com/api/webhooks/{webhook.id}/{webhook.token} の形式で発行されるのでこれをドキュメントにある https://discordapp.com/api/webhooks/{webhook.id}/{webhook.token}/slack にする

TureNAS Scale ¶

あとは Slack として登録すればよい

System Settings > Alert Settings > Alert Services で追加する
「Send Test Alert」を押すとテストメッセージが送信できるので試して最後に「Save」を押して保存する

System Settings > Alert Settings > Alert Services

テスト通知の様子

参考情報 ¶

SNMP OID と名前の対比 ¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
.1.3.6.1.4.1.50536.1.1.1.1.1    TRUENAS-MIB::zpoolIndex.1 = INTEGER: 1
.1.3.6.1.4.1.50536.1.1.1.1.2    TRUENAS-MIB::zpoolName.1 = STRING: "DataVol1"
.1.3.6.1.4.1.50536.1.1.1.1.3    TRUENAS-MIB::zpoolHealth.1 = STRING: "ONLINE"
.1.3.6.1.4.1.50536.1.1.1.1.4    TRUENAS-MIB::zpoolReadOps.1 = Counter64: 1595
.1.3.6.1.4.1.50536.1.1.1.1.5    TRUENAS-MIB::zpoolWriteOps.1 = Counter64: 170122
.1.3.6.1.4.1.50536.1.1.1.1.6    TRUENAS-MIB::zpoolReadBytes.1 = Counter64: 11812864
.1.3.6.1.4.1.50536.1.1.1.1.7    TRUENAS-MIB::zpoolWriteBytes.1 = Counter64: 1846116352

.1.3.6.1.4.1.50536.1.2.1.1.1    TRUENAS-MIB::zvolIndex.1 = INTEGER: 1
.1.3.6.1.4.1.50536.1.2.1.1.2    TRUENAS-MIB::zvolDescr.1 = STRING: "DataVol1"
.1.3.6.1.4.1.50536.1.2.1.1.3    TRUENAS-MIB::zvolUsedBytes.1 = Counter64: 53440512
.1.3.6.1.4.1.50536.1.2.1.1.4    TRUENAS-MIB::zvolAvailableBytes.1 = Counter64: 15856965816320
.1.3.6.1.4.1.50536.1.2.1.1.5    TRUENAS-MIB::zvolReferencedBytes.1 = Counter64: 253952

.1.3.6.1.4.1.50536.1.3.1        TRUENAS-MIB::zfsArcSize.0 = Gauge32: 527517
.1.3.6.1.4.1.50536.1.3.2        TRUENAS-MIB::zfsArcMeta.0 = Gauge32: 203840
.1.3.6.1.4.1.50536.1.3.3        TRUENAS-MIB::zfsArcData.0 = Gauge32: 319793
.1.3.6.1.4.1.50536.1.3.4        TRUENAS-MIB::zfsArcHits.0 = Gauge32: 101678769
.1.3.6.1.4.1.50536.1.3.5        TRUENAS-MIB::zfsArcMisses.0 = Gauge32: 30550

.1.3.6.1.4.1.50536.1.3.6        TRUENAS-MIB::zfsArcC.0 = Gauge32: 1019010
.1.3.6.1.4.1.50536.1.3.8        TRUENAS-MIB::zfsArcMissPercent.0 = STRING: "0.030036579047390433"
.1.3.6.1.4.1.50536.1.3.9        TRUENAS-MIB::zfsArcCacheHitRatio.0 = STRING: "99.97"
.1.3.6.1.4.1.50536.1.3.10       TRUENAS-MIB::zfsArcCacheMissRatio.0 = STRING: "0.03"

.1.3.6.1.4.1.50536.1.4.1        TRUENAS-MIB::zfsL2ArcHits.0 = Counter32: 0
.1.3.6.1.4.1.50536.1.4.2        TRUENAS-MIB::zfsL2ArcMisses.0 = Counter32: 0
.1.3.6.1.4.1.50536.1.4.3        TRUENAS-MIB::zfsL2ArcRead.0 = Counter32: 0
.1.3.6.1.4.1.50536.1.4.4        TRUENAS-MIB::zfsL2ArcWrite.0 = Counter32: 0
.1.3.6.1.4.1.50536.1.4.5        TRUENAS-MIB::zfsL2ArcSize.0 = Gauge32: 0

.1.3.6.1.4.1.50536.1.5.1        TRUENAS-MIB::zfsZilstatOps1sec.0 = Counter64: 0
.1.3.6.1.4.1.50536.1.5.2        TRUENAS-MIB::zfsZilstatOps5sec.0 = Counter64: 0
.1.3.6.1.4.1.50536.1.5.3        TRUENAS-MIB::zfsZilstatOps10sec.0 = Counter64: 0

.1.3.6.1.4.1.50536.3.1.2        TRUENAS-MIB::hddTempDevice.1 = STRING: "sdc"
.1.3.6.1.4.1.50536.3.1.3        TRUENAS-MIB::hddTempValue.1 = Gauge32: 42000