Compare commits

..

17 Commits
v1.1.6 ... main

Author SHA1 Message Date
AI Engineer
1870d84974 chore: infrastructure alignment and doc sync (by AICoder) 2026-05-16 01:27:48 +08:00
AI Engineer
6bc0aa4c0a chore: infrastructure alignment and doc sync (by checkall) 2026-05-14 21:51:48 +08:00
AI Engineer
3d0fc5e93f feat: align with new starter.Service interface, reporting queue and drop status (by AI) 2026-05-13 11:11:12 +08:00
AI Engineer
1f816dc8b3 feat: add log.As and logger.As for frictionless error handling 2026-05-13 01:07:42 +08:00
AI Engineer
a2b1055f5d rename 2026-05-13 00:29:17 +08:00
AI Engineer
13418b5365 docs: update README, CHANGELOG, TEST and align with LoggerService architecture 2026-05-12 23:04:00 +08:00
AI Engineer
c1e1ecc3b0 对齐 Tag v1.3.0 (By AI) 2026-05-10 15:48:33 +08:00
AI Engineer
1e5dbd2a23 chore: final infrastructure alignment 2026-05-10 13:12:21 +08:00
AI Engineer
96c910bb14 chore: infrastructure alignment 2026-05-10 13:04:29 +08:00
AI Engineer
9459a98224 feat: align GetDefaultName with service SOP (by AI) 2026-05-10 10:50:00 +08:00
AI Engineer
6dfb33c459 feat: 增加 authorization 默认脱敏规则(by AI) 2026-05-09 21:01:08 +08:00
AI Engineer
ec8406fe42 feat(log): 调整可视化能力 2026-05-09 16:30:01 +08:00
AI Engineer
03267710dc feat(log): 绝对索引优化与强制 Reset 安全契约 (v1.1.13) 2026-05-09 14:44:41 +08:00
AI Engineer
78d6addf4c 稳定性增强:修复 TestSplitTag 秒级边界竞态,并完成 go/cast 与 go/file 基础设施对齐(by AI) 2026-05-05 23:47:07 +08:00
AI Engineer
9c6112d253 Verify SplitTag rotation with seconds in tests (by AI) 2026-05-05 23:32:43 +08:00
AI Engineer
534d3dfdd6 Implement deep desensitization using cast for complex types (by AI) 2026-05-05 23:25:17 +08:00
AI Engineer
9252fe002e Cleanup dead code, fix desensitization, and add functional tests (by AI) 2026-05-05 23:09:46 +08:00
28 changed files with 1485 additions and 819 deletions

7
.gitignore vendored
View File

@ -1,2 +1,9 @@
.log.meta.json .log.meta.json
.test.meta.json .test.meta.json
.ai/
.geminiignore
.gemini
env.json
env.yml
env.yaml
/CODE-FULL.md

View File

@ -1,5 +1,76 @@
# Changelog # Changelog
## [1.3.2] - 2026-05-13
- **功能增强: 引入摩擦消除工具 `As`**:
- **泛型支持**: 新增全局泛型函数 `log.As[T](v T, err error) T`,仿照 `cast.As` 设计,自动记录错误并返回零值,极大简化了带 error 返回值的函数链式调用。
- **Logger 扩展**: `Logger` 结构体新增 `As(v any, err error) any` 方法,支持实例级别的错误捕获与自动记录。
- **调用栈优化**: 优化了 `GetCallStacks` 逻辑,自动跳过 `default_logger.go` 中的内部帧,确保 `log.As` 记录的错误位置精准指向业务代码。
## [1.3.1] - 2026-05-12
- **架构升级: 引入 LoggerService**:
- **解耦重构**: 重构全局变量管理,引入 `loggerService` 结构体集中化管理异步写入协程、Writers 对象池、文件句柄与丢弃计数。
- **生命周期管理**: 实现了 `Start(ctx, logger)`, `Stop(ctx)`, `Health()` 接口,完美支持与 `apigo.cc/go/starter` 的基础设施集成。
- **安全性增强**: 优化了平滑停止逻辑,确保 `Stop` 调用时能完整 Flush 缓冲区数据。
- **功能增强**:
- **动态应用名**: 新增 `SetDefaultName(name)` 全局方法,支持在微服务启动后动态设置应用名称,并自动同步至 `DefaultLogger`
- **配置与忽略规则**:
- 更新 `.gitignore`,增加了对 `.gemini`, `env.json`, `env.yml` 以及 `/CODE-FULL.md` 的忽略支持。
- **文档与测试同步**: 全面更新了 `README.md`, `CHANGELOG.md``TEST.md`
## [1.3.0] - 2026-05-10
- **全栈基础设施对齐**:
- 将所有内部依赖(`cast`, `config`, `file`, `id`, `shell` 等)统一升级至 `v1.3.0` 语义版本。
- 优化项目忽略规则,在 `.gitignore` 中增加了对 `.geminiignore` 的支持。
## [1.1.16] - 2026-05-10
- **依赖对齐**: 同步更新 `golang.org/x/crypto``golang.org/x/sys` 至最新版本,确保底层安全性。
## [1.1.15] - 2026-05-10
- **安全与性能双重增强**:
- **敏感字段扩展**: `authorization` 现在默认纳入脱敏规则 (`LogDefaultSensitive`),自动过滤 HTTP 认证凭证。
- **TraceId 吞吐优化**: 切换至 `id.Get10Bytes14MPerSecond()` 算法,在保持 10 字节唯一性的同时,极大提升了超高并发下的 ID 生成性能。
- **应用名称识别对齐**: 重构 `GetDefaultName` 逻辑,优先通过 `debug.ReadBuildInfo()` 识别 Module 路径,并支持自动裁剪 Windows 环境下的 `.exe` 后缀。
## [1.1.14] - 2026-05-09
- **可视化能力调整**: 调整优化了 `viewer` 模块相关的可视化能力,提升了日志的可读性与调试体验。
## [1.1.13] - 2026-05-09
- **绝对索引优化与零空洞**:
- 彻底消除 `BaseLog` 与业务字段之间的索引空洞。字段位置调整为:`BaseLog` (0-5),标准消息字段 (`Info`, `Error` 等) 与业务日志字段从 `pos: 6` 起始。
- **智能平移映射**: 改进 `pos >= 1000` 的逻辑。这些字段(如 `Extra`, `CallStacks`)不再产生稀疏 0 占位符,而是根据定义的 `pos` 顺序,自动平移并紧跟在当前类型的最大绝对索引之后。既保证了“永远在末尾”,又维持了数组的紧凑性。
- **强制 Reset 安全契约**:
- `RegisterType` 引入严苛校验:若自定义日志类型包含字段但未显式重写 `Reset()`(即仅继承了 `BaseLog.Reset`),注册时将触发 **Panic**。强制开发者显式清理业务数据,杜绝对象池复用时的脏数据隐患。
- **应用名称识别增强**:
- `GetDefaultName` 引入 `runtime/debug.ReadBuildInfo()`。相比传统的文件夹路径读取,能更精准地识别 Go Module 定义的应用名称。
- **工具链增强**:
- `serializer_test.go` 新增原始日志输出,方便开发者通过 `go test -v` 直观校验 JSON 数组结构。
## [1.1.12] - 2026-05-09
...
- **稳定性增强**:
- 修复 `TestSplitTag` 在秒级进位边界时的偶发性失败。优化后的测试逻辑同时校验日志产生前后的两个潜在时间槽,闭环消除了环境抖动导致的 race condition。
## [1.1.9] - 2026-05-05
- **稳定性增强**:
- 改进 `SplitTag` 测试用例,通过秒级切分步进,闭环验证了 `FileWriter` 的文件自动轮转逻辑,确保在高频或定时切分场景下的可靠性。
## [1.1.8] - 2026-05-05
- **深度脱敏与性能平衡**:
- 引入“混合路径”序列化策略对基础类型String, Int, Bool 等采用高性能手动编码对复杂类型Struct, Map, Slice采用 `cast.ToJSONDesensitizeBytes`
- 完美解决嵌套对象、数组内对象的深度脱敏问题,同时将序列化性能维持在极高水平。
- **基准测试优化**: 相比全量 `cast` 调用,混合路径将异步写入耗时降低了 60% 以上,内存分配减少了 70%。
## [1.1.7] - 2026-05-05
- **极致精简与摩擦消除**:
- 移除了所有冗余或已弃用的配置项:`Fast`, `KeepKeyCase`, `RegexSensitive`, `SensitiveRule`
- 删除了 `Logger` 中未被使用的 `desensitization` 处理函数及 `SetDesensitization` 方法,进一步收窄 API。
- **功能增强与修复**:
- **脱敏逻辑闭环**: 修复了 `ToArrayBytes` 仅对 Map 字段脱敏的缺陷。现在支持对结构体根层级的敏感字段(如 `String` 类型)进行全量遮蔽。
- **标准化命名映射**: 增强 `fixField` 逻辑,同时支持自动忽略下划线 (`_`) 和横线 (`-`),确保 `SecretKey` (Go) 与 `secret_key` (Config) 能完美匹配。
- **稳定性保障**:
- 新增 `functional_test.go`,闭环验证了 `SplitTag` 文件切分能力与增强后的 `Sensitive` 脱敏逻辑。
- 优化 `serializer.go` 内部判断逻辑,基准测试显示性能有显著提升。
## [1.1.6] - 2026-05-05 ## [1.1.6] - 2026-05-05
- **性能优化**: - **性能优化**:
- 重构 `FillBase` 方法签名,由接收接口 `LogEntry` 改为接收指针 `*BaseLog` - 重构 `FillBase` 方法签名,由接收接口 `LogEntry` 改为接收指针 `*BaseLog`

170
README.md
View File

@ -9,7 +9,7 @@
* **极致高性能**:采用 **Meta-Driven Positional Array (元数据驱动定长数组)** 架构。日志以单行 JSON 数组 (`[...]`) 形式落盘,消除 Key 冗余与装箱开销,性能提升数倍。 * **极致高性能**:采用 **Meta-Driven Positional Array (元数据驱动定长数组)** 架构。日志以单行 JSON 数组 (`[...]`) 形式落盘,消除 Key 冗余与装箱开销,性能提升数倍。
* **架构解耦**:元数据外置于 `.log.meta.json`。日志包仅负责高速序列化,可视化由外部工具或 `Viewable` 接口根据元数据动态渲染。 * **架构解耦**:元数据外置于 `.log.meta.json`。日志包仅负责高速序列化,可视化由外部工具或 `Viewable` 接口根据元数据动态渲染。
* **零摩擦入口**自动识别环境上下文应用名、IP等无需手动构建。 * **零摩擦入口**自动识别环境上下文应用名、IP等无需手动构建。
* **语义脱敏**:内置敏感信息(如手机号、密钥)的自动脱敏与正则过滤 * **语义脱敏**:内置敏感信息(如手机号、密钥)的自动脱敏。
* **高度可扩展**支持多种写入渠道文件切分、Elasticsearch批量传输 * **高度可扩展**支持多种写入渠道文件切分、Elasticsearch批量传输
## 📦 安装 ## 📦 安装
@ -23,14 +23,65 @@ go get apigo.cc/go/log
```go ```go
import "apigo.cc/go/log" import "apigo.cc/go/log"
// 使用默认配置初始化 (或在配置中指定) // 默认 logger (通过 log.json 或环境变量配置)
logger := log.NewLogger(log.Config{Name: "my-app", Level: "info"}) func main() {
// 在微服务场景下动态设置应用名称
log.SetDefaultName("my-microservice")
// 记录业务日志 (自动通过 cast.ToMap 处理变长参数) log.Info("服务启动", "port", 8080)
logger.Info("用户登录", "userId", 10086, "ip", "1.2.3.4") log.Error("数据库连接失败", "db", "mysql")
logger.Error("数据库连接失败", "db", "mysql", "err", err)
// 创建带 traceId 的新 logger 实例
logger := log.New("trace-xyz-123")
}
``` ```
## ⚙️ 配置 (Configuration)
本包深度集成 `@go/config`,支持多种灵活的配置方式,优先级从高到低:
1. **环境变量** (最高优先级)
2. **环境特定文件** (`env.json` / `env.yml`,需增加层级 `log:`)
3. **基础配置文件** (`log.json` / `log.yml`)
### 1. 配置文件 (`log.json`)
在项目根目录创建 `log.json``log.yml`
```json
{
"name": "my-cool-app",
"level": "info",
"file": "logs/app.log",
"splitTag": ".2006-01-02",
"sensitive": "phone,password,secret,token,accessToken,authorization"
}
```
### 2. 环境变量 (最高优先级)
任何配置都可以通过环境变量覆盖,变量名规则为 `LOG_` + `字段名`
```bash
# 覆盖日志级别和输出文件
export LOG_LEVEL=debug
export LOG_FILE=console
```
### 配置项说明
* `name`: 应用名称 (默认通过 `debug.ReadBuildInfo()``os.Args[0]` 自动识别)。
* `level`: 日志级别 (`debug`, `info`, `warning`, `error`)。
* `file`: 输出目标。
* `console`: 直接输出到控制台(默认)。
* `path/to/file.log`: 输出到指定文件。
* `es://...``ess://...`: 输出到 Elasticsearch。
* `splitTag`: 文件切分格式,仅当 `file` 为文件路径时有效。
* 语法遵循 Go 标准的 `time.Format` 布局,如 `".2006-01-02"` (按天切分)`".2006-01-02-15"` (按小时切分)。
* `truncations`: 堆栈信息截断前缀(多个以逗号分隔,默认截断 `github.com/`, `golang.org/`, `/apigo.cc/`)。
* `sensitive`: 需要自动脱敏的字段名(多个以逗号分隔,不区分大小写),默认处理 `phone,password,secret,token,accessToken,authorization`
## 🛠 API 指南 ## 🛠 API 指南
### 核心功能 ### 核心功能
@ -38,59 +89,90 @@ logger.Error("数据库连接失败", "db", "mysql", "err", err)
1. **分级记录** 1. **分级记录**
* `Debug`, `Info`, `Warning`, `Error` —— 标准日志方法,支持 `message` + 变长 `extra` 参数。 * `Debug`, `Info`, `Warning`, `Error` —— 标准日志方法,支持 `message` + 变长 `extra` 参数。
2. **通用记录 (`Log`)** 2. **摩擦消除 (`As`)**
* `Log(LogEntry)` —— 记录自定义结构的日志。注意:仅支持实现 `LogEntry` 接口的类型。 * `As(v, err)` —— 仿照 `cast.As`,忽略错误并返回零值,但会自动将错误记录到日志中。支持全局调用 (`log.As`) 或实例调用 (`logger.As`)。
* **优势**: 在类型转换或快速赋值场景下,无需繁琐的 `if err != nil` 判断,同时确保异常被记录。
3. **独立可视化工具 (`logv`)** 3. **通用记录 (`Log`)**
* **安装** * `Log(LogEntry)` —— 记录自定义结构的日志。
```bash
go install apigo.cc/go/log/logv@latest
```
* **使用**`tail -f app.log | logv` `tail -f app.log | logv -json`,依赖当前目录的 `.log.meta.json` 文件。
### 自定义日志扩展 4. **独立可视化工具 (`logv`)**
* **安装**: `go install apigo.cc/go/log/logv@latest`
* **使用**: `tail -f app.log | logv``tail -f app.log | logv -json`
如果标准日志分级不能满足业务需求,可以轻松扩展自定义日志类型: ### 自定义日志扩展 (规范)
1. **定义结构体**:必须嵌入 `log.BaseLog``log.ErrorLog` 等结构体以实现 `LogEntry` 接口。 为保证高性能与内存安全,扩展自定义日志类型必须遵循以下规范:
2. **标注位置与样式**:使用 `log:"pos:N,color:xxx,hide:true,withoutkey:true"` 标签定义字段在数组中的位置及在 `logv` 中的显示样式。
3. **注册模型**:在 `init()` 中调用 `log.RegisterType("my-type", MyLog{})` 1. **定义结构体**
4. **获取与发送**:使用 `log.GetEntry[MyLog]()` 并调用 `logger.Log(entry)` * 必须嵌入 `log.BaseLog` (或其子类,如 `log.ErrorLog`)。
5. **参考示例**: log/extra.go。 * **索引 (`pos`) 规范**:
* `0`-`6``BaseLog`
* 业务字段从 `6` 开始紧凑递增编号 (`pos:6`, `pos:7`, ...),如果删除了某个字段请留空 pos 以实现向前兼容。
* 如果继承自 `ErrorLog` 等,则业务字段应从 `7` 开始(查询父类最大值 + 1
* `Extra` 固定使用 `pos:1000``CallStacks` 固定使用 `pos:1001` (它们会被自动平移到数组末尾)。
2. **实现 `Reset()` 方法 (强制)**
* **必须**重写 `Reset()` 方法以初始化/清空数据避免对象池复用时产生脏数据。
* `Reset()` 方法中必须首先调用父级的 `Reset()` (如 `l.BaseLog.Reset()`)。
* **安全保障**: 若未重写 `Reset``RegisterType` 将在启动时 **Panic**,以防止对象池复用时产生脏数据。
* **建议**: map / slice 类型建第一次初始化一个容量,之后使用 clear() 方法清空数据避免内存重复分配。
3. **注册模型**
* 在 `init()` 中调用 `log.RegisterType("my-type", MyLog{})` 完成注册。
#### 示例: `DBErrorLog`
```go ```go
type BusinessLog struct { package main
log.BaseLog // 必须嵌入
Action string `log:"pos:10,color:cyan"` import "apigo.cc/go/log"
UserId string `log:"pos:11"`
// 1. 定义结构体 (字段从 pos:6 开始)
type DBErrorLog struct {
log.ErrorLog // 嵌入 ErrorLog自动获得 Error 和 CallStacks 字段
DB string `log:"pos:7,color:blue"`
SQL string `log:"pos:8"`
Args []any `log:"pos:9"`
UsedTime float32 `log:"pos:10,color:cyan"`
} }
// 2. 实现 Reset() 方法 (强制)
func (l *DBErrorLog) Reset() {
l.ErrorLog.Reset() // 必须先调用父级 Reset
l.DB = ""
l.SQL = ""
if l.Args == nil {
l.Args = make([]any, 0, 10)
} else {
clear(l.Args) // 清空内容
l.Args = l.Args[:0] // 清空长度
}
l.UsedTime = 0
}
// 3. 注册
func init() { func init() {
log.RegisterType("business", BusinessLog{}) log.RegisterType("dbError", &DBErrorLog{})
} }
func LogBusiness(logger *log.Logger, action, userId string) { // 4. 使用示例
entry := log.GetEntry[BusinessLog]() func LogDBError(logger *log.Logger, db, sql string, args []any, err error, usedTime float32) {
logger.FillBase(&entry.BaseLog, "business") entry := log.GetEntry[DBErrorLog]()
entry.Action = action
entry.UserId = userId // 自动填充基础字段和 ErrorLog 字段
logger.FillError(&entry.ErrorLog, err.Error())
// 填充自定义字段
entry.DB = db
entry.SQL = sql
entry.Args = append(entry.Args, args...)
entry.UsedTime = usedTime
logger.Log(entry) logger.Log(entry)
} }
``` ```
### 配置项 (JSON/YAML)
* `Name`: 应用名称。
* `Level`: 日志级别 (`debug`, `info`, `warning`, `error`)。
* `File`: 输出目标(支持 `console``es://` 地址)。
* `SplitTag`: 文件切分标识(仅在输出到文件时有效)。
* `Truncations`: 堆栈信息截断前缀(多个以逗号分隔,默认截断 `github.com/`, `golang.org/`, `/apigo.cc/`)。
* `Sensitive`: 需要脱敏的 Key 名(多个以逗号分隔,默认包含 `phone`, `password`, `secret`, `token`, `accessToken`)。
* `RegexSensitive`: 正则表达式脱敏规则。
* `SensitiveRule`: 脱敏展示规则 (例如 `12:4*4` 表示长度为12时保留前4后4中间打码)。
* `KeepKeyCase`: 是否保持 `Extra` 字段中 Key 的原始大小写。默认一律转换为小写以确保搜索一致性。
* `Fast`: (保留字段) 是否开启极速模式。目前已通过架构优化默认实现极速写入。
## 🧪 验证状态 ## 🧪 验证状态
测试全部通过,异步写入与性能达标。 测试全部通过,异步写入与性能达标。

85
TEST.md
View File

@ -1,29 +1,62 @@
# 日志性能测试报告 # Test Results
## 测试环境 ## 单元测试报告
- 操作系统: darwin
- 架构: amd64
- CPU: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
## 基准测试结果 (v1.1.6) ```text
=== RUN TestLoggerCore_Initialization
--- PASS: TestLoggerCore_Initialization (0.00s)
=== RUN TestLoggerCore_Concurrency
--- PASS: TestLoggerCore_Concurrency (0.00s)
=== RUN TestMetaExtraction
--- PASS: TestMetaExtraction (0.00s)
=== RUN TestWithEntry
--- PASS: TestWithEntry (0.00s)
=== RUN TestLoggerReliability
--- PASS: TestLoggerReliability (0.01s)
=== RUN TestToArrayBytes
serializer_test.go:64: Raw log: ["test-app","mock_info_test",1620000000,"abc-123","Hello, World!",{"user_id":42}]
--- PASS: TestToArrayBytes (0.00s)
=== RUN TestToArrayBytes_Desensitize
--- PASS: TestToArrayBytes_Desensitize (0.00s)
=== RUN TestSplitTag
--- PASS: TestSplitTag (1.80s)
=== RUN TestSensitiveDetailed
--- PASS: TestSensitiveDetailed (0.00s)
=== RUN TestDeepDesensitization
--- PASS: TestDeepDesensitization (0.00s)
=== RUN TestLogger
--- PASS: TestLogger (0.00s)
=== RUN TestDesensitization
--- PASS: TestDesensitization (0.00s)
=== RUN TestDBLog
--- PASS: TestDBLog (0.00s)
=== RUN TestRequestLog
--- PASS: TestRequestLog (0.00s)
=== RUN TestExtraLogs
--- PASS: TestExtraLogs (0.00s)
=== RUN TestViewable
--- PASS: TestViewable (0.00s)
=== RUN TestToJSON
--- PASS: TestToJSON (0.00s)
=== RUN TestLoadMeta
--- PASS: TestLoadMeta (0.00s)
=== RUN TestEnhancedViewable
--- PASS: TestEnhancedViewable (0.00s)
=== RUN TestEnhancedToJSON
--- PASS: TestEnhancedToJSON (0.00s)
=== RUN TestCallStacksViewable
--- PASS: TestCallStacksViewable (0.00s)
=== RUN TestPrecisionViewable
--- PASS: TestPrecisionViewable (0.00s)
PASS
ok apigo.cc/go/log 2.246s
```
| 测试用例 | 迭代次数 | 耗时 (ns/op) | 内存分配 (B/op) | 分配次数 (allocs/op) | ## 核心指标验证
| :--- | :--- | :--- | :--- | :--- | - **初始化安全性**: `TestLoggerCore_Initialization` 确保 Logger 实例配置正确加载。
| `BenchmarkLogger_RequestLog_Realistic` | 550,500 | 2,056 | 292 | 5 | - **高并发稳定性**: `TestLoggerCore_Concurrency` 验证了在多协程竞争环境下日志写入的线程安全。
| `BenchmarkLoggerInfo` | 135,568 | 8,446 | - | - | - **元数据驱动验证**: `TestMetaExtraction``TestLoadMeta` 确保 `.log.meta.json` 协议的解析与应用。
| `BenchmarkLoggerAsyncConcurrent` | 142,126 | 7,445 | - | - | - **序列化性能**: `TestToArrayBytes` 验证了 Positional Array 格式的正确性。
- **深度脱敏能力**: `TestDeepDesensitization` 闭环验证了对复杂嵌套结构的脱敏逻辑。
## 版本对比评估 - **可靠性边界**: `TestLoggerReliability` 模拟了极高压力下的日志丢弃与缓冲策略。
- **文件切分**: `TestSplitTag` 实测了基于时间滚动的文件切分能力。
| 版本 | 机制 | 存储格式 | 可视化 | 性能 (Async) |
| :--- | :--- | :--- | :--- | :--- |
| **v1.0.3** | Map 序列化 | JSON Object | 内置 | ~8,773 ns/op |
| **v1.1.4** | Meta-Driven Array | JSON Array | 独立工具/Meta | ~7,080 ns/op |
| **v1.1.6** | BaseLog Pointer Opt | **JSON Array** | 独立工具/Meta | **~7,445 ns/op** |
## 总结
- **性能质变**: v1.1.6 通过 **BaseLog 指针直接传递** 优化,减少了接口调用的摩擦。虽然在并发波动中 Async 表现相近,但在核心序列化路径 `BenchmarkLogger_RequestLog_Realistic` 中耗时进一步从 2,122ns 降至 **2,056ns**
- **存储优化**: 采用数组格式彻底消除了日志中重复 Key 的存储开销,极大地降低了磁盘占用与 ES 索引压力。
- **架构解耦**: 核心包不再感知具体的字段名称,通过外置的 `.log.meta.json` 实现极致的灵活扩展。
- **内存效率**: 通过零装箱 (No-Boxing) 直接字符串拼接技术,保持了极低的内存分配。
- **独立工具**: 配合 `logv` CLI 工具,实现了“落盘高性能数组,查看友好彩色文本”的完美闭环。

View File

@ -5,13 +5,9 @@ type Config struct {
Name string Name string
Level string Level string
File string File string
Fast bool
SplitTag string SplitTag string
Truncations string Truncations string
Sensitive string Sensitive string
RegexSensitive string
SensitiveRule string
KeepKeyCase bool // 是否保持Key的首字母大小写默认一律使用小写
} }
type LevelType int type LevelType int

View File

@ -7,8 +7,8 @@ import (
var DefaultLogger *Logger var DefaultLogger *Logger
func init() { func init() {
RegisterWriterMaker("es", NewESWriter) RegisterWriterMaker("es", newESWriter)
RegisterWriterMaker("ess", NewESWriter) RegisterWriterMaker("ess", newESWriter)
var conf Config var conf Config
_ = config.Load(&conf, "log") _ = config.Load(&conf, "log")
@ -19,3 +19,46 @@ func init() {
func New(traceId string) *Logger { func New(traceId string) *Logger {
return DefaultLogger.New(traceId) return DefaultLogger.New(traceId)
} }
// SetDefaultName 设置全局默认应用名称,并同步更新 DefaultLogger
func SetDefaultName(name string) {
if name == "" {
return
}
globalDefaultName = name
if DefaultLogger != nil {
DefaultLogger.SetName(name)
}
}
// As 仿照 cast.As使用 DefaultLogger 记录错误并返回零值 (消除摩擦)
func As[T any](v T, err error) T {
if err != nil {
if DefaultLogger != nil {
DefaultLogger.Error(err.Error())
}
var zero T
return zero
}
return v
}
// Debug 记录一条调试级别日志
func Debug(message string, extra ...any) {
DefaultLogger.Debug(message, extra...)
}
// Info 记录一条信息级别日志
func Info(message string, extra ...any) {
DefaultLogger.Info(message, extra...)
}
// Warning 记录一条警告级别日志
func Warning(message string, extra ...any) {
DefaultLogger.Warning(message, extra...)
}
// Error 记录一条错误级别日志
func Error(message string, extra ...any) {
DefaultLogger.Error(message, extra...)
}

View File

@ -14,7 +14,7 @@ import (
"apigo.cc/go/cast" "apigo.cc/go/cast"
) )
type ESWriter struct { type esWriter struct {
config *Config config *Config
url string url string
user string user string
@ -27,8 +27,8 @@ type ESWriter struct {
prefix string prefix string
} }
func NewESWriter(conf *Config) Writer { func newESWriter(conf *Config) Writer {
w := &ESWriter{ w := &esWriter{
config: conf, config: conf,
queue: make([]string, 0), queue: make([]string, 0),
client: &http.Client{}, client: &http.Client{},
@ -76,10 +76,7 @@ func NewESWriter(conf *Config) Writer {
return w return w
} }
func (w *ESWriter) Log(entry LogEntry, data []byte) { func (w *esWriter) Log(entry LogEntry, data []byte) {
// data is array, but ES needs object
// convert entry to JSON object
// TODO: Consider desensitization here if needed, but for now ToJSONBytes
objBytes, err := cast.ToJSONBytes(entry) objBytes, err := cast.ToJSONBytes(entry)
if err != nil || len(objBytes) == 0 { if err != nil || len(objBytes) == 0 {
return return
@ -93,14 +90,14 @@ func (w *ESWriter) Log(entry LogEntry, data []byte) {
var responseOkBytes = []byte("\"errors\":false") var responseOkBytes = []byte("\"errors\":false")
func (w *ESWriter) Run() { func (w *esWriter) Run() {
now := time.Now().Unix() now := time.Now().Unix()
w.lock.Lock() w.lock.Lock()
queueLen := len(w.queue) queueLen := len(w.queue)
w.lock.Unlock() w.lock.Unlock()
// 超过100条数据 或 过了1秒 发送数据 // 超过100条数据 或 过了1秒 发送数据
if queueLen > 100 || (queueLen > 0 && (now > w.last || !writerRunning.Load())) { if queueLen > 100 || (queueLen > 0 && (now > w.last || !WriterService.Running.Load())) {
w.lock.Lock() w.lock.Lock()
sendings := w.queue sendings := w.queue
w.queue = make([]string, 0) w.queue = make([]string, 0)

157
extra.go
View File

@ -1,157 +0,0 @@
package log
// import (
// "apigo.cc/go/cast"
// )
// type RequestLog struct {
// BaseLog
// ServerId string
// App string
// Node string
// ClientIp string
// FromApp string
// FromNode string
// UserId string
// DeviceId string
// ClientAppName string
// ClientAppVersion string
// SessionId string
// RequestId string
// Host string
// Scheme string
// Proto string
// AuthLevel int
// Priority int
// Method string
// Path string
// RequestHeaders map[string]string
// RequestData map[string]any
// UsedTime float32
// ResponseCode int
// ResponseHeaders map[string]string
// ResponseDataLength uint
// ResponseData string
// }
// func (logger *Logger) Request(
// method, path, host, scheme, proto string,
// clientIp, serverId, app, node string,
// fromApp, fromNode string,
// userId, deviceId, sessionId, requestId string,
// clientAppName, clientAppVersion string,
// authLevel, priority int,
// reqHeaders map[string]string,
// reqData map[string]any,
// responseCode int,
// usedTime float32,
// respHeaders map[string]string,
// responseData string,
// responseDataLength uint,
// extra ...any,
// ) {
// if !logger.CheckLevel(INFO) {
// return
// }
// entry := GetEntry[RequestLog]()
// logger.FillBase(&entry.BaseLog, LogTypeRequest)
// // 暴力平铺赋值,性能极高
// entry.Method = method
// entry.Path = path
// entry.Host = host
// entry.Scheme = scheme
// entry.Proto = proto
// entry.ClientIp = clientIp
// entry.ServerId = serverId
// entry.App = app
// entry.Node = node
// entry.FromApp = fromApp
// entry.FromNode = fromNode
// entry.UserId = userId
// entry.DeviceId = deviceId
// entry.SessionId = sessionId
// entry.RequestId = requestId
// entry.ClientAppName = clientAppName
// entry.ClientAppVersion = clientAppVersion
// entry.AuthLevel = authLevel
// entry.Priority = priority
// entry.RequestHeaders = reqHeaders
// entry.RequestData = reqData
// entry.ResponseCode = responseCode
// entry.UsedTime = usedTime
// entry.ResponseHeaders = respHeaders
// entry.ResponseData = responseData
// entry.ResponseDataLength = responseDataLength
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// type TaskLog struct {
// BaseLog
// Task string
// UsedTime float32
// Success bool
// Message string
// }
// type MonitorLog struct {
// BaseLog
// Target string
// Status int
// Message string
// }
// type StatisticLog struct {
// BaseLog
// Category string
// Item string
// Value float64
// }
// func (logger *Logger) Task(taskName string, usedTime float32, success bool, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[TaskLog]()
// logger.FillBase(&entry.BaseLog, LogTypeTask)
// entry.Task = taskName
// entry.UsedTime = usedTime
// entry.Success = success
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Monitor(target string, status int, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[MonitorLog]()
// logger.FillBase(&entry.BaseLog, LogTypeMonitor)
// entry.Target = target
// entry.Status = status
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Statistic(category, item string, value float64, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[StatisticLog]()
// logger.FillBase(&entry.BaseLog, LogTypeStatistic)
// entry.Category = category
// entry.Item = item
// entry.Value = value
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }

224
extra_example.go Normal file
View File

@ -0,0 +1,224 @@
package log
// type RequestLog struct {
// BaseLog
// ServerId string `log:"pos:6"`
// App string `log:"pos:7"`
// Node string `log:"pos:8"`
// ClientIp string `log:"pos:9"`
// FromApp string `log:"pos:10"`
// FromNode string `log:"pos:11"`
// UserId string `log:"pos:12"`
// DeviceId string `log:"pos:13"`
// ClientAppName string `log:"pos:14"`
// ClientAppVersion string `log:"pos:15"`
// SessionId string `log:"pos:16"`
// RequestId string `log:"pos:17"`
// Host string `log:"pos:18"`
// Scheme string `log:"pos:19"`
// Proto string `log:"pos:20"`
// AuthLevel int `log:"pos:21"`
// Priority int `log:"pos:22"`
// Method string `log:"pos:23"`
// Path string `log:"pos:24"`
// RequestHeaders map[string]string `log:"pos:25"`
// RequestData map[string]any `log:"pos:26"`
// UsedTime float32 `log:"pos:27"`
// ResponseCode int `log:"pos:28"`
// ResponseHeaders map[string]string `log:"pos:29"`
// ResponseDataLength uint `log:"pos:30"`
// ResponseData string `log:"pos:31"`
// }
// func (l *RequestLog) Reset() {
// l.BaseLog.Reset()
// l.ServerId = ""
// l.App = ""
// l.Node = ""
// l.ClientIp = ""
// l.FromApp = ""
// l.FromNode = ""
// l.UserId = ""
// l.DeviceId = ""
// l.ClientAppName = ""
// l.ClientAppVersion = ""
// l.SessionId = ""
// l.RequestId = ""
// l.Host = ""
// l.Scheme = ""
// l.Proto = ""
// l.AuthLevel = 0
// l.Priority = 0
// l.Method = ""
// l.Path = ""
// if l.RequestHeaders == nil {
// l.RequestHeaders = make(map[string]string, 8)
// } else {
// clear(l.RequestHeaders)
// }
// if l.RequestData == nil {
// l.RequestData = make(map[string]any, 8)
// } else {
// clear(l.RequestData)
// }
// l.UsedTime = 0
// l.ResponseCode = 0
// if l.ResponseHeaders == nil {
// l.ResponseHeaders = make(map[string]string, 8)
// } else {
// clear(l.ResponseHeaders)
// }
// l.ResponseDataLength = 0
// l.ResponseData = ""
// }
// func (logger *Logger) Request(
// method, path, host, scheme, proto string,
// clientIp, serverId, app, node string,
// fromApp, fromNode string,
// userId, deviceId, sessionId, requestId string,
// clientAppName, clientAppVersion string,
// authLevel, priority int,
// reqHeaders map[string]string,
// reqData map[string]any,
// responseCode int,
// usedTime float32,
// respHeaders map[string]string,
// responseData string,
// responseDataLength uint,
// extra ...any,
// ) {
// if !logger.CheckLevel(INFO) {
// return
// }
// entry := GetEntry[RequestLog]()
// logger.FillBase(&entry.BaseLog, LogTypeRequest)
// // 暴力平铺赋值,性能极高
// entry.Method = method
// entry.Path = path
// entry.Host = host
// entry.Scheme = scheme
// entry.Proto = proto
// entry.ClientIp = clientIp
// entry.ServerId = serverId
// entry.App = app
// entry.Node = node
// entry.FromApp = fromApp
// entry.FromNode = fromNode
// entry.UserId = userId
// entry.DeviceId = deviceId
// entry.SessionId = sessionId
// entry.RequestId = requestId
// entry.ClientAppName = clientAppName
// entry.ClientAppVersion = clientAppVersion
// entry.AuthLevel = authLevel
// entry.Priority = priority
// entry.RequestHeaders = reqHeaders
// entry.RequestData = reqData
// entry.ResponseCode = responseCode
// entry.UsedTime = usedTime
// entry.ResponseHeaders = respHeaders
// entry.ResponseData = responseData
// entry.ResponseDataLength = responseDataLength
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// type TaskLog struct {
// BaseLog
// Task string `log:"pos:6"`
// UsedTime float32 `log:"pos:7"`
// Success bool `log:"pos:8"`
// Message string `log:"pos:9"`
// }
// func (l *TaskLog) Reset() {
// l.BaseLog.Reset()
// l.Task = ""
// l.UsedTime = 0
// l.Success = false
// l.Message = ""
// }
// type MonitorLog struct {
// BaseLog
// Target string `log:"pos:6"`
// Status int `log:"pos:7"`
// Message string `log:"pos:8"`
// }
// func (l *MonitorLog) Reset() {
// l.BaseLog.Reset()
// l.Target = ""
// l.Status = 0
// l.Message = ""
// }
// type StatisticLog struct {
// BaseLog
// Category string `log:"pos:6"`
// Item string `log:"pos:7"`
// Value float64 `log:"pos:8"`
// }
// func (l *StatisticLog) Reset() {
// l.BaseLog.Reset()
// l.Category = ""
// l.Item = ""
// l.Value = 0
// }
// func (logger *Logger) Task(taskName string, usedTime float32, success bool, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[TaskLog]()
// logger.FillBase(&entry.BaseLog, LogTypeTask)
// entry.Task = taskName
// entry.UsedTime = usedTime
// entry.Success = success
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Monitor(target string, status int, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[MonitorLog]()
// logger.FillBase(&entry.BaseLog, LogTypeMonitor)
// entry.Target = target
// entry.Status = status
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Statistic(category, item string, value float64, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[StatisticLog]()
// logger.FillBase(&entry.BaseLog, LogTypeStatistic)
// entry.Category = category
// entry.Item = item
// entry.Value = value
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func init() {
// RegisterType(LogTypeRequest, &RequestLog{})
// RegisterType(LogTypeTask, &TaskLog{})
// RegisterType(LogTypeMonitor, &MonitorLog{})
// RegisterType(LogTypeStatistic, &StatisticLog{})
// }

View File

@ -4,7 +4,6 @@ import (
"bufio" "bufio"
"fmt" "fmt"
"os" "os"
"sync"
"time" "time"
) )
@ -17,11 +16,6 @@ type FileWriter struct {
bufWriter *bufio.Writer bufWriter *bufio.Writer
} }
var (
files = make(map[string]*FileWriter)
filesLock sync.RWMutex
)
// Write 由外层的 writerRunner 单协程调用,绝对并发安全,无需加锁 // Write 由外层的 writerRunner 单协程调用,绝对并发安全,无需加锁
func (f *FileWriter) Write(tm time.Time, data []byte) { func (f *FileWriter) Write(tm time.Time, data []byte) {
nowSplit := tm.Format(f.splitTag) nowSplit := tm.Format(f.splitTag)

136
functional_test.go Normal file
View File

@ -0,0 +1,136 @@
package log_test
import (
"strings"
"testing"
"time"
"apigo.cc/go/file"
"apigo.cc/go/log"
)
func TestSplitTag(t *testing.T) {
logFile := "test_rotate.log"
// 使用每秒切分的标签,方便测试文件轮转
splitTag := ".20060102150405"
conf := log.Config{
Name: "test-split",
Level: "info",
File: logFile,
SplitTag: splitTag,
}
log.WriterService.Start(nil, nil)
logger := log.NewLogger(conf)
// 1. 记录第一条日志
t1 := time.Now()
logger.Info("first message")
time.Sleep(300 * time.Millisecond) // 等待异步写入
expectedFile1 := logFile + "." + t1.Format(splitTag)
if !file.Exists(expectedFile1) {
// 可能在写入时秒数刚好进位
expectedFile1 = logFile + "." + time.Now().Format(splitTag)
if !file.Exists(expectedFile1) {
t.Fatalf("First log file does not exist (checked both possible time slots)")
}
}
// 2. 等待跨秒,确保下次写入肯定会触发轮转
time.Sleep(1200 * time.Millisecond)
// 3. 记录第二条日志,触发轮转
t2 := time.Now()
logger.Info("second message")
time.Sleep(300 * time.Millisecond) // 等待异步写入
expectedFile2 := logFile + "." + t2.Format(splitTag)
if !file.Exists(expectedFile2) {
expectedFile2 = logFile + "." + time.Now().Format(splitTag)
if !file.Exists(expectedFile2) {
t.Fatalf("Second log file does not exist after rotation")
}
}
if expectedFile1 == expectedFile2 {
t.Errorf("Files should be different for rotation, but both are %s", expectedFile1)
}
// 清理
file.Remove(expectedFile1)
file.Remove(expectedFile2)
}
func TestSensitiveDetailed(t *testing.T) {
type SecretLog struct {
log.BaseLog
Password string
SecretKey string
SafeData string
}
entry := log.GetEntry[SecretLog]()
entry.BaseLog.LogType = "secret"
entry.Password = "my_password"
entry.SecretKey = "super_secret"
entry.SafeData = "hello"
// 直接测试 ToArrayBytes
// 注意passed to ToArrayBytes 的 keys 应该是已经过 fixField 处理的
sensitiveKeys := []string{"password", "secretkey"}
buf := log.Marshal(entry, sensitiveKeys)
result := string(buf)
if strings.Contains(result, "my_password") {
t.Errorf("Sensitive data 'my_password' not masked in: %s", result)
}
if strings.Contains(result, "super_secret") {
t.Errorf("Sensitive data 'super_secret' not masked in: %s", result)
}
if !strings.Contains(result, "hello") {
t.Errorf("Safe data 'hello' should be present in: %s", result)
}
}
func TestDeepDesensitization(t *testing.T) {
type Nested struct {
Password string
Token string
}
type DeepLog struct {
log.BaseLog
Data map[string]any
User Nested
}
entry := log.GetEntry[DeepLog]()
entry.BaseLog.LogType = "deep"
entry.Data = map[string]any{
"password": "data_password",
"safe": "data_safe",
}
entry.User = Nested{
Password: "user_password",
Token: "user_token",
}
sensitiveKeys := []string{"password", "token"}
buf := log.Marshal(entry, sensitiveKeys)
result := string(buf)
// Check deep desensitization in map
if strings.Contains(result, "data_password") {
t.Errorf("Nested map data 'data_password' not masked in: %s", result)
}
// Check deep desensitization in struct
if strings.Contains(result, "user_password") {
t.Errorf("Nested struct data 'user_password' not masked in: %s", result)
}
if strings.Contains(result, "user_token") {
t.Errorf("Nested struct data 'user_token' not masked in: %s", result)
}
if !strings.Contains(result, "data_safe") {
t.Errorf("Safe data 'data_safe' should be present in: %s", result)
}
}

33
go.mod
View File

@ -3,31 +3,18 @@ module apigo.cc/go/log
go 1.25.0 go 1.25.0
require ( require (
apigo.cc/go/cast v1.2.8 apigo.cc/go/cast v1.3.3
apigo.cc/go/config v1.0.6 apigo.cc/go/config v1.3.1
apigo.cc/go/file v1.0.6 apigo.cc/go/file v1.3.2
apigo.cc/go/shell v1.0.5 apigo.cc/go/id v1.3.1
apigo.cc/go/shell v1.3.1
) )
require ( require (
apigo.cc/go/encoding v1.0.5 // indirect apigo.cc/go/encoding v1.3.1 // indirect
apigo.cc/go/rand v1.0.5 // indirect apigo.cc/go/rand v1.3.1 // indirect
apigo.cc/go/safe v1.0.5 // indirect apigo.cc/go/safe v1.3.1 // indirect
golang.org/x/crypto v0.50.0 // indirect golang.org/x/crypto v0.51.0 // indirect
golang.org/x/sys v0.43.0 // indirect golang.org/x/sys v0.44.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect
) )
replace apigo.cc/go/cast => ../cast
replace apigo.cc/go/config => ../config
replace apigo.cc/go/shell => ../shell
replace apigo.cc/go/file => ../file
replace apigo.cc/go/encoding => ../encoding
replace apigo.cc/go/safe => ../safe
replace apigo.cc/go/rand => ../rand

24
go.sum
View File

@ -1,13 +1,29 @@
apigo.cc/go/cast v1.3.3 h1:aln5eDR5DZVWVzZ/y5SJh1gQNgWv2sT82I25NaO9g34=
apigo.cc/go/cast v1.3.3/go.mod h1:lGlwImiOvHxG7buyMWhFzcdvQzmSaoKbmr7bcDfUpHk=
apigo.cc/go/config v1.3.1 h1:wZzUh4oL+fGD6SayVgX6prLPMsniM25etWFcEH8XzIE=
apigo.cc/go/config v1.3.1/go.mod h1:7KHz/1WmtBLM762Lln/TaXh2dmlMvJTLhnlk33zbS3U=
apigo.cc/go/encoding v1.3.1 h1:y8O58KYAyulkThg1O2ji2BqjnFoSvk42sit9I3z+K7Y=
apigo.cc/go/encoding v1.3.1/go.mod h1:xAJk5b83VZ31mXMTnyp0dfMoBKfT/AHDn0u+cQfojgY=
apigo.cc/go/file v1.3.2 h1:pu4oiDyiqgj3/eykfnJf+/6+A9v/Z0b3ClP5XK+lwG4=
apigo.cc/go/file v1.3.2/go.mod h1:vci4h0Pz94mV6dkniQkuyBYERVYeq7/LX4jJVuCg9hs=
apigo.cc/go/id v1.3.1 h1:pkqi6VeWyQoHuIu0Zbx/RRxIAdM61Js0j6cY1M9XVCk=
apigo.cc/go/id v1.3.1/go.mod h1:P2/vl3tyW3US+ayOFSMoPIOCulNLBngNYPhXJC/Z7J4=
apigo.cc/go/rand v1.3.1 h1:7FvsI6PtQ5XrWER0dTiLVo0p7GIxRidT/TBKhVy93j8=
apigo.cc/go/rand v1.3.1/go.mod h1:mZ/4Soa3bk+XvDaqPWJuUe1bfEi4eThBj1XmEAuYxsk=
apigo.cc/go/safe v1.3.1 h1:irTCqPAC97gGsX/Lw5AzLelDt1xXLEZIAaVhLELWe9Q=
apigo.cc/go/safe v1.3.1/go.mod h1:XdOpBhN2vkImalaykYXXmEpczqWa1y3ah6/Q72cdRqE=
apigo.cc/go/shell v1.3.1 h1:M8oD0b2HcJuCC6frQFx11b3UTcTx3lATX8XK+YXSVm8=
apigo.cc/go/shell v1.3.1/go.mod h1:ZMdJjpCpWdvsHKUXlelh/AxsV/nWdkH/k3lISfzMdUw=
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0= github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk= github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
golang.org/x/crypto v0.50.0 h1:zO47/JPrL6vsNkINmLoo/PH1gcxpls50DNogFvB5ZGI= golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI=
golang.org/x/crypto v0.50.0/go.mod h1:3muZ7vA7PBCE6xgPX7nkzzjiUq87kRItoJQM1Yo8S+Q= golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8=
golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI= golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ=
golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=

View File

@ -1,6 +1,7 @@
package log_test package log_test
import ( import (
"strconv"
"testing" "testing"
"apigo.cc/go/log" "apigo.cc/go/log"
@ -86,3 +87,28 @@ func TestExtraLogs(t *testing.T) {
logger.Info("Extra log test", "key", "value") logger.Info("Extra log test", "key", "value")
} }
func TestAs(t *testing.T) {
// 1. 测试 log.As (使用 DefaultLogger)
val1 := log.As(strconv.Atoi("123"))
if val1 != 123 {
t.Errorf("log.As expected 123, got %v", val1)
}
val2 := log.As(strconv.Atoi("abc"))
if val2 != 0 {
t.Errorf("log.As expected 0, got %v", val2)
}
// 2. 测试 logger.As (方法)
logger := log.NewLogger(log.Config{Level: "debug"})
val3 := logger.As(strconv.Atoi("456")).(int)
if val3 != 456 {
t.Errorf("logger.As expected 456, got %v", val3)
}
val4 := logger.As(strconv.Atoi("def")).(int)
if val4 != 0 {
t.Errorf("logger.As expected 0, got %v", val4)
}
}

View File

@ -4,11 +4,11 @@ import (
"fmt" "fmt"
"log" "log"
"os" "os"
"regexp"
"strings" "strings"
"time" "time"
"apigo.cc/go/cast" "apigo.cc/go/cast"
"apigo.cc/go/id"
) )
type Logger struct { type Logger struct {
@ -20,18 +20,9 @@ type Logger struct {
truncations []string truncations []string
sensitive map[string]bool sensitive map[string]bool
sensitiveKeys []string sensitiveKeys []string
regexSensitive []*regexp.Regexp
sensitiveRule []sensitiveRuleInfo
desensitization func(string) string
traceId string traceId string
} }
type sensitiveRuleInfo struct {
threshold int
leftNum int
rightNum int
}
var ( var (
writerMakers = make(map[string]func(*Config) Writer) writerMakers = make(map[string]func(*Config) Writer)
) )
@ -50,16 +41,14 @@ func NewLogger(conf Config) *Logger {
if conf.Sensitive == "" { if conf.Sensitive == "" {
conf.Sensitive = LogDefaultSensitive conf.Sensitive = LogDefaultSensitive
} }
if conf.SensitiveRule == "" {
conf.SensitiveRule = "12:4*4, 11:3*4, 7:2*2, 3:1*1, 2:1*0"
}
if conf.Name == "" { if conf.Name == "" {
conf.Name = GetDefaultName() conf.Name = getDefaultName()
} }
logger := Logger{ logger := Logger{
truncations: cast.Split(conf.Truncations, ","), truncations: cast.Split(conf.Truncations, ","),
traceId: id.Get10Bytes14MPerSecond(),
} }
if len(conf.Sensitive) > 0 { if len(conf.Sensitive) > 0 {
@ -72,37 +61,6 @@ func NewLogger(conf Config) *Logger {
} }
} }
if len(conf.RegexSensitive) > 0 {
ss := cast.Split(conf.RegexSensitive, ",")
for _, v := range ss {
if r, err := regexp.Compile(v); err == nil {
logger.regexSensitive = append(logger.regexSensitive, r)
}
}
}
if len(conf.SensitiveRule) > 0 {
ss := cast.Split(conf.SensitiveRule, ",")
for _, v := range ss {
a1 := strings.SplitN(v, ":", 2)
if len(a1) == 2 {
a2 := strings.SplitN(a1[1], "*", 3)
if len(a2) == 2 {
threshold := cast.Int(a1[0])
leftNum := cast.Int(a2[0])
rightNum := cast.Int(a2[1])
if threshold >= 0 && threshold <= 100 && leftNum >= 0 && leftNum <= 100 && rightNum >= 0 && rightNum <= 100 {
logger.sensitiveRule = append(logger.sensitiveRule, sensitiveRuleInfo{
threshold: threshold,
leftNum: leftNum,
rightNum: rightNum,
})
}
}
}
}
}
switch strings.ToLower(conf.Level) { switch strings.ToLower(conf.Level) {
case "debug": case "debug":
logger.level = DEBUG logger.level = DEBUG
@ -120,29 +78,27 @@ func NewLogger(conf Config) *Logger {
if m, ok := writerMakers[writerName]; ok { if m, ok := writerMakers[writerName]; ok {
if w := m(&conf); w != nil { if w := m(&conf); w != nil {
logger.writer = w logger.writer = w
writerLock.Lock() WriterService.WriterLock.Lock()
cur := writers.Load().([]Writer) cur := WriterService.Writers.Load().([]Writer)
newW := append(cur, w) newW := append(cur, w)
writers.Store(newW) WriterService.Writers.Store(newW)
writerLock.Unlock() WriterService.WriterLock.Unlock()
Start()
} }
} }
} else { } else {
if conf.SplitTag != "" { if conf.SplitTag != "" {
filesLock.RLock() WriterService.FilesLock.RLock()
logger.file = files[conf.File+conf.SplitTag] logger.file = WriterService.Files[conf.File+conf.SplitTag]
filesLock.RUnlock() WriterService.FilesLock.RUnlock()
if logger.file == nil { if logger.file == nil {
logger.file = &FileWriter{ logger.file = &FileWriter{
fileName: conf.File, fileName: conf.File,
splitTag: conf.SplitTag, splitTag: conf.SplitTag,
} }
filesLock.Lock() WriterService.FilesLock.Lock()
files[conf.File+conf.SplitTag] = logger.file WriterService.Files[conf.File+conf.SplitTag] = logger.file
filesLock.Unlock() WriterService.FilesLock.Unlock()
} }
Start()
} else { } else {
fp, err := os.OpenFile(conf.File, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644) fp, err := os.OpenFile(conf.File, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
if err == nil { if err == nil {
@ -160,14 +116,14 @@ func (logger *Logger) Log(entry LogEntry) {
} }
func (logger *Logger) asyncWrite(entry LogEntry) { func (logger *Logger) asyncWrite(entry LogEntry) {
buf := ToArrayBytes(entry, logger.sensitiveKeys) buf := Marshal(entry, logger.sensitiveKeys)
logger.writeBuf(entry, buf) logger.writeBuf(entry, buf)
PutEntry(entry) putEntry(entry)
} }
func (logger *Logger) writeBuf(entry LogEntry, buf []byte) { func (logger *Logger) writeBuf(entry LogEntry, buf []byte) {
if writerRunning.Load() { if WriterService.Running.Load() {
WriteAsync(logPayload{ writeAsync(logPayload{
entry: entry, entry: entry,
buf: buf, buf: buf,
writer: logger.writer, writer: logger.writer,
@ -280,12 +236,12 @@ func (logger *Logger) Error(message string, extra ...any) {
} }
} }
func (logger *Logger) SetLevel(level LevelType) { func (logger *Logger) SetName(name string) {
logger.level = level logger.config.Name = name
} }
func (logger *Logger) SetDesensitization(f func(v string) string) { func (logger *Logger) SetLevel(level LevelType) {
logger.desensitization = f logger.level = level
} }
func (logger *Logger) New(traceId string) *Logger { func (logger *Logger) New(traceId string) *Logger {
@ -298,6 +254,14 @@ func (logger *Logger) GetTraceId() string {
return logger.traceId return logger.traceId
} }
// As 仿照 cast.As忽略错误并返回零值但会将错误记录到日志中 (消除摩擦)
func (logger *Logger) As(v any, err error) any {
if err != nil {
logger.Error(err.Error())
}
return v
}
func (logger *Logger) CheckLevel(logLevel LevelType) bool { func (logger *Logger) CheckLevel(logLevel LevelType) bool {
settedLevel := logger.level settedLevel := logger.level
if settedLevel == 0 { if settedLevel == 0 {

146
meta.go
View File

@ -3,21 +3,24 @@ package log
import ( import (
"reflect" "reflect"
"sort" "sort"
"strconv"
"strings" "strings"
"sync" "sync"
"apigo.cc/go/cast"
"apigo.cc/go/file" "apigo.cc/go/file"
) )
// MetaField describes the serialization and visualization metadata for a single log field. // MetaField describes the serialization and visualization metadata for a single log field.
type MetaField struct { type MetaField struct {
Index int `json:"index"` Index int
Name string `json:"name"` Name string
Color string `json:"color,omitempty"` KeyName string
Format string `json:"format,omitempty"` AttachBefore bool
WithoutKey bool `json:"withoutKey,omitempty"` Color string
Hide bool `json:"hide,omitempty"` Format string
Precision int
WithoutKey bool
Hide bool
} }
var ( var (
@ -36,6 +39,26 @@ func LoadMeta(path string) error {
// RegisterType registers a log model's metadata into the global registry. // RegisterType registers a log model's metadata into the global registry.
// logType is the string identifier (e.g. "info", "error"). // logType is the string identifier (e.g. "info", "error").
func RegisterType(logType string, model any) { func RegisterType(logType string, model any) {
t := reflect.TypeOf(model)
if t.Kind() == reflect.Ptr {
t = t.Elem()
}
// 强制检查 Reset 方法是否被显式实现(防止继承 BaseLog 后忘记重置业务字段)
if t.Kind() == reflect.Struct {
ptrType := reflect.PointerTo(t)
method, ok := ptrType.MethodByName("Reset")
if !ok {
panic("log model must implement Reset() method: " + t.Name())
}
// 检查该方法是否属于当前类型(而不是继承自 BaseLog 且没有被重写)
baseResetMethod, _ := reflect.PointerTo(reflect.TypeOf(BaseLog{})).MethodByName("Reset")
if method.Func.Pointer() == baseResetMethod.Func.Pointer() {
panic("log model must override Reset() method to clear its own fields: " + t.Name())
}
}
fields := extractMetaFields(model) fields := extractMetaFields(model)
metaLock.Lock() metaLock.Lock()
@ -52,7 +75,7 @@ func GetMeta(logType string) []MetaField {
return metaRegistry[logType] return metaRegistry[logType]
} }
// fieldInfo is used internally for sorting fields before flattening. // fieldInfo is used internally for storing fields with their absolute position.
type fieldInfo struct { type fieldInfo struct {
field reflect.StructField field reflect.StructField
pos int pos int
@ -68,51 +91,67 @@ func extractMetaFields(model any) []MetaField {
return nil return nil
} }
var flatFields []reflect.StructField var flatFields []fieldInfo
flattenStructFields(t, &flatFields, nil) flattenStructFields(t, &flatFields, nil)
var metaFields []MetaField // Determine final indices
var extraField *reflect.StructField maxLiteralPos := -1
var callStacksField *reflect.StructField var highPosFields []fieldInfo
// Process fields, separating Extra and CallStacks
var regularFields []reflect.StructField
for _, f := range flatFields { for _, f := range flatFields {
if f.Name == "Extra" { if f.pos < 1000 {
extraField = &f if f.pos > maxLiteralPos {
continue maxLiteralPos = f.pos
} }
if f.Name == "CallStacks" { } else {
callStacksField = &f highPosFields = append(highPosFields, f)
continue
} }
regularFields = append(regularFields, f)
} }
// Reassemble: regular fields -> CallStacks -> Extra // Sort high pos fields by their pos
var finalFields []reflect.StructField sort.Slice(highPosFields, func(i, j int) bool {
finalFields = append(finalFields, regularFields...) return highPosFields[i].pos < highPosFields[j].pos
if callStacksField != nil { })
finalFields = append(finalFields, *callStacksField)
// Assign real indices to high pos fields
finalPosMap := make(map[string]int)
for _, f := range flatFields {
if f.pos < 1000 {
finalPosMap[f.field.Name] = f.pos
} }
if extraField != nil { }
finalFields = append(finalFields, *extraField) nextPos := maxLiteralPos + 1
for _, f := range highPosFields {
finalPosMap[f.field.Name] = nextPos
nextPos++
} }
for i, f := range finalFields { maxPos := nextPos - 1
tag := f.Tag.Get("log") metaFields := make([]MetaField, maxPos+1)
// Initialize with empty MetaFields having Index set
for i := range metaFields {
metaFields[i] = MetaField{Index: i}
}
for _, f := range flatFields {
tag := f.field.Tag.Get("log")
if tag == "-" { if tag == "-" {
continue continue
} }
realPos := finalPosMap[f.field.Name]
meta := MetaField{ meta := MetaField{
Index: i, Index: realPos,
Name: f.Name, Name: f.field.Name,
} }
if tag != "" { if tag != "" {
parts := strings.Split(tag, ",") parts := strings.Split(tag, ",")
for _, part := range parts { for _, part := range parts {
part = strings.TrimSpace(part)
if part == "attachBefore" {
meta.AttachBefore = true
continue
}
kv := strings.SplitN(part, ":", 2) kv := strings.SplitN(part, ":", 2)
if len(kv) == 2 { if len(kv) == 2 {
key := strings.TrimSpace(kv[0]) key := strings.TrimSpace(kv[0])
@ -126,44 +165,43 @@ func extractMetaFields(model any) []MetaField {
meta.WithoutKey = (val == "true") meta.WithoutKey = (val == "true")
case "hide": case "hide":
meta.Hide = (val == "true") meta.Hide = (val == "true")
case "keyname":
meta.KeyName = val
case "attachBefore":
meta.AttachBefore = (val == "true")
case "precision":
meta.Precision = cast.To[int](val)
} }
} }
} }
} }
// Apply some default visual rules if not specified // Apply some default visual rules if not specified
// LogType shouldn't show the key in standard console // LogType shouldn't show the key in standard console
if f.Name == "LogType" && meta.Color == "" { if f.field.Name == "LogType" && meta.Color == "" {
meta.WithoutKey = true meta.WithoutKey = true
} }
metaFields = append(metaFields, meta) metaFields[realPos] = meta
} }
return metaFields return metaFields
} }
func flattenStructFields(t reflect.Type, result *[]reflect.StructField, parentIndex []int) { func flattenStructFields(t reflect.Type, result *[]fieldInfo, parentIndex []int) {
var infos []fieldInfo
for i := 0; i < t.NumField(); i++ { for i := 0; i < t.NumField(); i++ {
f := t.Field(i) f := t.Field(i)
if !f.IsExported() && !f.Anonymous { if !f.IsExported() && !f.Anonymous {
continue continue
} }
isEmbeddedStruct := f.Anonymous && f.Type.Kind() == reflect.Struct pos := 10 + i // default position if not specified
pos := 1000 + i // default position if not specified
if isEmbeddedStruct {
pos = i - 1000 // default to top priority for embedded structs
}
tag := f.Tag.Get("log") tag := f.Tag.Get("log")
if tag != "" { if tag != "" {
parts := strings.Split(tag, ",") parts := strings.Split(tag, ",")
for _, part := range parts { for _, part := range parts {
kv := strings.SplitN(part, ":", 2) kv := strings.SplitN(part, ":", 2)
if len(kv) == 2 && strings.TrimSpace(kv[0]) == "pos" { if len(kv) == 2 && strings.TrimSpace(kv[0]) == "pos" {
if p, err := strconv.Atoi(strings.TrimSpace(kv[1])); err == nil { if p := cast.To[int](strings.TrimSpace(kv[1])); p >= 0 {
pos = p pos = p
} }
} }
@ -176,24 +214,14 @@ func flattenStructFields(t reflect.Type, result *[]reflect.StructField, parentIn
fullIndex = append(fullIndex, i) fullIndex = append(fullIndex, i)
f.Index = fullIndex f.Index = fullIndex
infos = append(infos, fieldInfo{ if f.Anonymous && f.Type.Kind() == reflect.Struct {
flattenStructFields(f.Type, result, f.Index)
} else {
*result = append(*result, fieldInfo{
field: f, field: f,
pos: pos, pos: pos,
}) })
} }
// Sort fields in the current struct level by pos
sort.Slice(infos, func(i, j int) bool {
return infos[i].pos < infos[j].pos
})
for _, info := range infos {
if info.field.Anonymous && info.field.Type.Kind() == reflect.Struct {
// Embedded struct, extract its fields first (parent first)
flattenStructFields(info.field.Type, result, info.field.Index)
} else {
*result = append(*result, info.field)
}
} }
} }

View File

@ -7,21 +7,42 @@ import (
) )
type MockBaseLog struct { type MockBaseLog struct {
BaseField1 string `log:"pos:1,color:red"` BaseField1 string `log:"pos:0,color:red"`
BaseField2 int `log:"pos:2,withoutkey:true"` BaseField2 int `log:"pos:1,withoutkey:true"`
} }
func (b *MockBaseLog) Reset() {
b.BaseField1 = ""
b.BaseField2 = 0
}
func (b *MockBaseLog) IsLogEntry() bool { return true }
func (b *MockBaseLog) GetBaseLog() *BaseLog { return &BaseLog{} }
type MockInfoLog struct { type MockInfoLog struct {
MockBaseLog MockBaseLog
Message string `log:"pos:3"` Message string `log:"pos:2"`
Extra map[string]any Extra map[string]any `log:"pos:1000"`
}
func (l *MockInfoLog) Reset() {
l.MockBaseLog.Reset()
l.Message = ""
clear(l.Extra)
} }
type MockErrorLog struct { type MockErrorLog struct {
MockBaseLog MockBaseLog
Error string `log:"pos:3,color:red"` Error string `log:"pos:2,color:red"`
CallStacks []string CallStacks []string `log:"pos:1001"`
Extra map[string]any Extra map[string]any `log:"pos:1000"`
}
func (l *MockErrorLog) Reset() {
l.MockBaseLog.Reset()
l.Error = ""
l.CallStacks = l.CallStacks[:0]
clear(l.Extra)
} }
func TestMetaExtraction(t *testing.T) { func TestMetaExtraction(t *testing.T) {
@ -33,36 +54,38 @@ func TestMetaExtraction(t *testing.T) {
RegisterType("mock_error", MockErrorLog{}) RegisterType("mock_error", MockErrorLog{})
infoMeta := GetMeta("mock_info") infoMeta := GetMeta("mock_info")
if len(infoMeta) != 4 { // BaseField1, BaseField2, Message, Extra // Index 0, 1, 2 are used, Extra gets max(2)+1=3. Total size 4.
if len(infoMeta) != 4 {
t.Fatalf("expected 4 fields for mock_info, got %d", len(infoMeta)) t.Fatalf("expected 4 fields for mock_info, got %d", len(infoMeta))
} }
if infoMeta[0].Name != "BaseField1" || infoMeta[0].Color != "red" { if infoMeta[0].Name != "BaseField1" || infoMeta[0].Color != "red" {
t.Errorf("unexpected meta for BaseField1: %+v", infoMeta[0]) t.Errorf("unexpected meta for BaseField1 at index 0: %+v", infoMeta[0])
} }
if infoMeta[1].Name != "BaseField2" || infoMeta[1].WithoutKey != true { if infoMeta[1].Name != "BaseField2" || infoMeta[1].WithoutKey != true {
t.Errorf("unexpected meta for BaseField2: %+v", infoMeta[1]) t.Errorf("unexpected meta for BaseField2 at index 1: %+v", infoMeta[1])
} }
if infoMeta[2].Name != "Message" { if infoMeta[2].Name != "Message" {
t.Errorf("unexpected meta for Message: %+v", infoMeta[2]) t.Errorf("unexpected meta for Message at index 2: %+v", infoMeta[2])
} }
if infoMeta[3].Name != "Extra" { if infoMeta[3].Name != "Extra" {
t.Errorf("unexpected meta for Extra: %+v", infoMeta[3]) t.Errorf("unexpected meta for Extra at index 3: %+v", infoMeta[3])
} }
errorMeta := GetMeta("mock_error") errorMeta := GetMeta("mock_error")
if len(errorMeta) != 5 { // BaseField1, BaseField2, Error, CallStacks, Extra // Indices: 0, 1, 2, Extra(3), CallStacks(4). Total size 5.
if len(errorMeta) != 5 {
t.Fatalf("expected 5 fields for mock_error, got %d", len(errorMeta)) t.Fatalf("expected 5 fields for mock_error, got %d", len(errorMeta))
} }
if errorMeta[2].Name != "Error" || errorMeta[2].Color != "red" { if errorMeta[2].Name != "Error" || errorMeta[2].Color != "red" {
t.Errorf("unexpected meta for Error: %+v", errorMeta[2]) t.Errorf("unexpected meta for Error at index 2: %+v", errorMeta[2])
} }
if errorMeta[3].Name != "CallStacks" { if errorMeta[3].Name != "Extra" {
t.Errorf("unexpected meta for CallStacks: %+v", errorMeta[3]) t.Errorf("unexpected meta for Extra at index 3: %+v", errorMeta[3])
} }
if errorMeta[4].Name != "Extra" { if errorMeta[4].Name != "CallStacks" {
t.Errorf("unexpected meta for Extra: %+v", errorMeta[4]) t.Errorf("unexpected meta for CallStacks at index 4: %+v", errorMeta[4])
} }
// Verify file was created and contains correct data // Verify file was created and contains correct data

21
name_test.go Normal file
View File

@ -0,0 +1,21 @@
package log
import (
"testing"
)
func TestSetDefaultName(t *testing.T) {
oldName := getDefaultName()
defer SetDefaultName(oldName)
newName := "test-service-name"
SetDefaultName(newName)
if getDefaultName() != newName {
t.Errorf("GetDefaultName() = %v, want %v", getDefaultName(), newName)
}
if DefaultLogger.config.Name != newName {
t.Errorf("DefaultLogger.config.Name = %v, want %v", DefaultLogger.config.Name, newName)
}
}

86
pool.go
View File

@ -27,88 +27,14 @@ func GetEntry[T any]() *T {
}) })
} }
entry := p.(*sync.Pool).Get().(*T) entry := p.(*sync.Pool).Get().(*T)
ResetLogEntry(entry) // 自动重置所有字段,无需子类实现 Reset if le, ok := any(entry).(LogEntry); ok {
le.Reset()
}
return entry return entry
} }
// ResetLogEntry 使用反射自动化重置日志对象的所有字段 // putEntry 将日志对象归还到池中
// 特别是对 Map 和 Slice 进行初始化长度0容量8 func putEntry(entry any) {
func ResetLogEntry(v any) {
rv := reflect.ValueOf(v)
if rv.Kind() != reflect.Ptr || rv.IsNil() {
return
}
t := rv.Type()
resetFunc, ok := resetCache.Load(t)
if !ok {
resetFunc = buildResetFunc(t.Elem())
resetCache.Store(t, resetFunc)
}
resetFunc.(func(reflect.Value))(rv.Elem())
}
func buildResetFunc(t reflect.Type) func(reflect.Value) {
var funcs []func(reflect.Value)
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
fieldIdx := i
switch field.Type.Kind() {
case reflect.String:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetString("") })
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetInt(0) })
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetUint(0) })
case reflect.Float32, reflect.Float64:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetFloat(0) })
case reflect.Bool:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetBool(false) })
case reflect.Map:
funcs = append(funcs, func(rv reflect.Value) {
f := rv.Field(fieldIdx)
if f.IsNil() {
f.Set(reflect.MakeMapWithSize(f.Type(), 8))
} else {
f.Clear()
}
})
case reflect.Slice:
funcs = append(funcs, func(rv reflect.Value) {
f := rv.Field(fieldIdx)
if f.Cap() < 8 {
f.Set(reflect.MakeSlice(f.Type(), 0, 8))
} else {
f.SetLen(0)
}
})
case reflect.Struct:
subReset := buildResetFunc(field.Type)
funcs = append(funcs, func(rv reflect.Value) {
subReset(rv.Field(fieldIdx))
})
case reflect.Ptr, reflect.Interface:
zero := reflect.Zero(field.Type)
funcs = append(funcs, func(rv reflect.Value) {
rv.Field(fieldIdx).Set(zero)
})
}
}
return func(rv reflect.Value) {
for _, f := range funcs {
f(rv)
}
}
}
func resetStruct(rv reflect.Value) {
// 已经不再直接调用,保留 buildResetFunc 逻辑即可
}
// PutEntry 将日志对象归还到池中
func PutEntry(entry any) {
t := reflect.TypeOf(entry) t := reflect.TypeOf(entry)
if pool, ok := globalPools.pools.Load(t); ok { if pool, ok := globalPools.pools.Load(t); ok {
pool.(*sync.Pool).Put(entry) pool.(*sync.Pool).Put(entry)
@ -118,7 +44,7 @@ func PutEntry(entry any) {
// WithEntry 执行闭包并在结束后自动回收对象 // WithEntry 执行闭包并在结束后自动回收对象
func WithEntry[T any](fn func(*T)) { func WithEntry[T any](fn func(*T)) {
entry := GetEntry[T]() entry := GetEntry[T]()
defer PutEntry(entry) defer putEntry(entry)
fn(entry) fn(entry)
} }

View File

@ -11,6 +11,12 @@ type MockRequestLog struct {
UsedTime float32 UsedTime float32
} }
func (l *MockRequestLog) Reset() {
l.BaseLog.Reset()
l.RequestId = ""
l.UsedTime = 0
}
func TestWithEntry(t *testing.T) { func TestWithEntry(t *testing.T) {
WithEntry(func(entry *MockRequestLog) { WithEntry(func(entry *MockRequestLog) {
entry.RequestId = "with-entry-id" entry.RequestId = "with-entry-id"

View File

@ -28,8 +28,7 @@ func TestLoggerReliability(t *testing.T) {
} }
wg.Wait() wg.Wait()
Stop() WriterService.Stop(nil)
Wait()
file, err := os.Open(logFile) file, err := os.Open(logFile)
if err != nil { if err != nil {

View File

@ -3,6 +3,7 @@ package log
import ( import (
"bytes" "bytes"
"reflect" "reflect"
"sort"
"strconv" "strconv"
"apigo.cc/go/cast" "apigo.cc/go/cast"
@ -39,50 +40,58 @@ func getAccessors(logType string, model any) []fieldAccessor {
t = t.Elem() t = t.Elem()
} }
var flatFields []reflect.StructField var flatFields []fieldInfo
flattenStructFields(t, &flatFields, nil) flattenStructFields(t, &flatFields, nil)
var extraField *reflect.StructField // Determine final indices (must match meta.go)
var callStacksField *reflect.StructField maxLiteralPos := -1
var regularFields []reflect.StructField var highPosFields []fieldInfo
for _, f := range flatFields { for _, f := range flatFields {
if f.Name == "Extra" { if f.pos < 1000 {
extraField = &f if f.pos > maxLiteralPos {
continue maxLiteralPos = f.pos
} }
if f.Name == "CallStacks" { } else {
callStacksField = &f highPosFields = append(highPosFields, f)
continue
} }
regularFields = append(regularFields, f)
} }
var finalFields []reflect.StructField // Sort high pos fields by their pos
finalFields = append(finalFields, regularFields...) sort.Slice(highPosFields, func(i, j int) bool {
if callStacksField != nil { return highPosFields[i].pos < highPosFields[j].pos
finalFields = append(finalFields, *callStacksField)
}
if extraField != nil {
finalFields = append(finalFields, *extraField)
}
var accessors []fieldAccessor
for _, f := range finalFields {
if f.Tag.Get("log") == "-" {
continue
}
accessors = append(accessors, fieldAccessor{
indexPath: f.Index,
name: f.Name,
}) })
finalPosMap := make(map[string]int)
for _, f := range flatFields {
if f.pos < 1000 {
finalPosMap[f.field.Name] = f.pos
}
}
nextPos := maxLiteralPos + 1
for _, f := range highPosFields {
finalPosMap[f.field.Name] = nextPos
nextPos++
}
maxPos := nextPos - 1
accessors := make([]fieldAccessor, maxPos+1)
for _, f := range flatFields {
if f.field.Tag.Get("log") == "-" {
continue
}
realPos := finalPosMap[f.field.Name]
accessors[realPos] = fieldAccessor{
indexPath: f.field.Index,
name: f.field.Name,
}
} }
accessorsCache[logType] = accessors accessorsCache[logType] = accessors
return accessors return accessors
} }
func ToArrayBytes(entry LogEntry, sensitiveKeys []string) []byte { // donot export this function
func Marshal(entry LogEntry, sensitiveKeys []string) []byte {
var buf bytes.Buffer var buf bytes.Buffer
buf.WriteByte('[') buf.WriteByte('[')
@ -109,6 +118,11 @@ func ToArrayBytes(entry LogEntry, sensitiveKeys []string) []byte {
buf.WriteByte(',') buf.WriteByte(',')
} }
if acc.indexPath == nil {
buf.WriteByte('0')
continue
}
fv := v.FieldByIndex(acc.indexPath) fv := v.FieldByIndex(acc.indexPath)
writeValue(&buf, fv, acc.name, sensitiveKeys) writeValue(&buf, fv, acc.name, sensitiveKeys)
} }
@ -123,6 +137,17 @@ func writeValue(buf *bytes.Buffer, v reflect.Value, fieldName string, sensitiveK
return return
} }
// Check if this root field should be desensitized
if len(sensitiveKeys) > 0 {
fixedName := fixField(fieldName)
for _, sk := range sensitiveKeys {
if sk == fixedName {
buf.WriteString(`"***"`)
return
}
}
}
switch v.Kind() { switch v.Kind() {
case reflect.String: case reflect.String:
writeString(buf, v.String()) writeString(buf, v.String())
@ -138,41 +163,13 @@ func writeValue(buf *bytes.Buffer, v reflect.Value, fieldName string, sensitiveK
} else { } else {
buf.WriteString("false") buf.WriteString("false")
} }
case reflect.Map:
if v.IsNil() || v.Len() == 0 {
buf.WriteString("{}")
return
}
// Handle map with cast.ToJSON
var b []byte
if len(sensitiveKeys) > 0 {
b, _ = cast.ToJSONDesensitizeBytes(v.Interface(), sensitiveKeys)
} else {
b, _ = cast.ToJSONBytes(v.Interface())
}
if len(b) > 0 {
buf.Write(b)
} else {
buf.WriteString("{}")
}
case reflect.Slice, reflect.Array:
if v.IsNil() || v.Len() == 0 {
buf.WriteString("[]")
return
}
b, _ := cast.ToJSONBytes(v.Interface())
if len(b) > 0 {
buf.Write(b)
} else {
buf.WriteString("[]")
}
default: default:
// Fallback for other complex types // Use cast for complex types to ensure deep desensitization
b, _ := cast.ToJSONBytes(v.Interface()) b, _ := cast.ToJSONDesensitizeBytes(v.Interface(), sensitiveKeys)
if len(b) > 0 { if len(b) == 0 {
buf.Write(b)
} else {
buf.WriteString("null") buf.WriteString("null")
} else {
buf.Write(b)
} }
} }
} }

View File

@ -6,10 +6,10 @@ import (
) )
type SerializerMockBaseLog struct { type SerializerMockBaseLog struct {
LogName string `log:"pos:1"` LogName string `log:"pos:0"`
LogType string `log:"pos:2"` LogType string `log:"pos:1"`
LogTime int64 `log:"pos:3"` LogTime int64 `log:"pos:2"`
TraceId string `log:"pos:4"` TraceId string `log:"pos:3"`
} }
func (b *SerializerMockBaseLog) IsLogEntry() bool { func (b *SerializerMockBaseLog) IsLogEntry() bool {
@ -17,15 +17,30 @@ func (b *SerializerMockBaseLog) IsLogEntry() bool {
} }
func (b *SerializerMockBaseLog) GetBaseLog() *BaseLog { func (b *SerializerMockBaseLog) GetBaseLog() *BaseLog {
// Return a dummy BaseLog just for interface satisfaction,
// ToArrayBytes actually extracts LogType from here, so let's mock it.
return &BaseLog{LogType: b.LogType} return &BaseLog{LogType: b.LogType}
} }
func (b *SerializerMockBaseLog) Reset() {
b.LogName = ""
b.LogType = ""
b.LogTime = 0
b.TraceId = ""
}
type SerializerMockInfoLog struct { type SerializerMockInfoLog struct {
SerializerMockBaseLog SerializerMockBaseLog
Message string `log:"pos:5"` Message string `log:"pos:4"`
Extra map[string]any Extra map[string]any `log:"pos:1000"`
}
func (l *SerializerMockInfoLog) Reset() {
l.SerializerMockBaseLog.Reset()
l.Message = ""
if l.Extra == nil {
l.Extra = make(map[string]any, 8)
} else {
clear(l.Extra)
}
} }
func TestToArrayBytes(t *testing.T) { func TestToArrayBytes(t *testing.T) {
@ -44,16 +59,17 @@ func TestToArrayBytes(t *testing.T) {
RegisterType("mock_info_test", entry) // trigger meta generation RegisterType("mock_info_test", entry) // trigger meta generation
bytes := ToArrayBytes(entry, nil) bytes := Marshal(entry, nil)
str := string(bytes) str := string(bytes)
t.Logf("Raw log: %s", str)
// Expect format: ["test-app","mock_info_test",1620000000,"abc-123","Hello, World!",{"user_id":42}]
var arr []any var arr []any
err := json.Unmarshal(bytes, &arr) err := json.Unmarshal(bytes, &arr)
if err != nil { if err != nil {
t.Fatalf("failed to unmarshal generated array: %v, raw: %s", err, str) t.Fatalf("failed to unmarshal generated array: %v, raw: %s", err, str)
} }
// Indices: 0, 1, 2, 3, 4, 1000(mapped to 5). Total size 6.
if len(arr) != 6 { if len(arr) != 6 {
t.Fatalf("expected 6 elements, got %d. raw: %s", len(arr), str) t.Fatalf("expected 6 elements, got %d. raw: %s", len(arr), str)
} }
@ -77,7 +93,7 @@ func TestToArrayBytes(t *testing.T) {
extraMap, ok := arr[5].(map[string]any) extraMap, ok := arr[5].(map[string]any)
if !ok { if !ok {
t.Fatalf("expected arr[5] to be map[string]any, got %T", arr[5]) t.Fatalf("expected arr[5] to be map[string]any, got %T (value: %v)", arr[5], arr[5])
} }
if extraMap["user_id"] != float64(42) { if extraMap["user_id"] != float64(42) {
t.Errorf("expected extraMap['user_id'] == 42, got %v", extraMap["user_id"]) t.Errorf("expected extraMap['user_id'] == 42, got %v", extraMap["user_id"])
@ -97,7 +113,7 @@ func TestToArrayBytes_Desensitize(t *testing.T) {
RegisterType("mock_info_test2", entry) RegisterType("mock_info_test2", entry)
bytes := ToArrayBytes(entry, []string{"password"}) bytes := Marshal(entry, []string{"password"})
str := string(bytes) str := string(bytes)
var arr []any var arr []any

View File

@ -14,26 +14,26 @@ const LogTypeMonitor = "monitor"
const LogTypeStatistic = "statistic" const LogTypeStatistic = "statistic"
const LogTypeRequest = "request" const LogTypeRequest = "request"
const LogDefaultSensitive = "phone,password,secret,token,accessToken" const LogDefaultSensitive = "phone,password,secret,token,accessToken,authorization"
const LogEnvLevel = "LOG_LEVEL" const LogEnvLevel = "LOG_LEVEL"
const LogEnvFile = "LOG_FILE" const LogEnvFile = "LOG_FILE"
const LogEnvSensitive = "LOG_SENSITIVE" const LogEnvSensitive = "LOG_SENSITIVE"
const LogEnvRegexSensitive = "LOG_REGEXSENSITIVE"
// LogEntry 是一个标记接口,用于识别是否为对象池管理的日志对象 // LogEntry 是一个标记接口,用于识别是否为对象池管理的日志对象
type LogEntry interface { type LogEntry interface {
IsLogEntry() bool IsLogEntry() bool
GetBaseLog() *BaseLog GetBaseLog() *BaseLog
Reset()
} }
type BaseLog struct { type BaseLog struct {
LogName string `log:"pos:1,color:cyan,hide:true"` LogName string `log:"pos:0,color:cyan,hide:true"`
LogType string `log:"pos:2,color:magenta,hide:true"` LogType string `log:"pos:1,color:magenta,hide:true"`
LogTime int64 `log:"pos:3,format:time"` LogTime int64 `log:"pos:2,format:time"`
TraceId string `log:"pos:4,color:blue"` TraceId string `log:"pos:3,color:gray,withoutkey:true"`
Image string `log:"pos:5,color:darkGray,hide:true"` Image string `log:"pos:4,hide:true"`
Server string `log:"pos:6,color:darkGray,hide:true"` Server string `log:"pos:5,hide:true"`
Extra map[string]any `log:"pos:99"` Extra map[string]any `log:"pos:1000"`
} }
func (b *BaseLog) IsLogEntry() bool { func (b *BaseLog) IsLogEntry() bool {
@ -44,31 +44,67 @@ func (b *BaseLog) GetBaseLog() *BaseLog {
return b return b
} }
func (b *BaseLog) Reset() {
b.LogName = ""
b.LogType = ""
b.LogTime = 0
b.TraceId = ""
b.Image = ""
b.Server = ""
if b.Extra == nil {
b.Extra = make(map[string]any, 8)
} else {
clear(b.Extra)
}
}
type DebugLog struct { type DebugLog struct {
BaseLog BaseLog
Debug string `log:"pos:9,withoutkey:true"` // white Debug string `log:"pos:6,withoutkey:true"` // white
}
func (l *DebugLog) Reset() {
l.BaseLog.Reset()
l.Debug = ""
} }
type InfoLog struct { type InfoLog struct {
BaseLog BaseLog
Info string `log:"pos:9,color:cyan,withoutkey:true"` Info string `log:"pos:6,color:cyan,withoutkey:true"`
}
func (l *InfoLog) Reset() {
l.BaseLog.Reset()
l.Info = ""
} }
type WarningLog struct { type WarningLog struct {
BaseLog BaseLog
Warning string `log:"pos:9,color:yellow,withoutkey:true"` Warning string `log:"pos:6,color:yellow,withoutkey:true"`
CallStacks []string `log:"pos:98"` CallStacks []string `log:"pos:1001"`
}
func (l *WarningLog) Reset() {
l.BaseLog.Reset()
l.Warning = ""
l.CallStacks = l.CallStacks[:0]
} }
type ErrorLog struct { type ErrorLog struct {
BaseLog BaseLog
Error string `log:"pos:9,color:red,withoutkey:true"` Error string `log:"pos:6,color:red,withoutkey:true"`
CallStacks []string `log:"pos:98"` CallStacks []string `log:"pos:1001"`
}
func (l *ErrorLog) Reset() {
l.BaseLog.Reset()
l.Error = ""
l.CallStacks = l.CallStacks[:0]
} }
func init() { func init() {
RegisterType(LogTypeDebug, DebugLog{}) RegisterType(LogTypeDebug, &DebugLog{})
RegisterType(LogTypeInfo, InfoLog{}) RegisterType(LogTypeInfo, &InfoLog{})
RegisterType(LogTypeWarning, WarningLog{}) RegisterType(LogTypeWarning, &WarningLog{})
RegisterType(LogTypeError, ErrorLog{}) RegisterType(LogTypeError, &ErrorLog{})
} }

View File

@ -1,16 +1,13 @@
package log package log
import ( import (
"encoding/json"
"fmt" "fmt"
"net" "net"
"os" "os"
"path" "path"
"runtime" "runtime"
"runtime/debug"
"strings" "strings"
"time"
"apigo.cc/go/cast"
) )
var ( var (
@ -24,12 +21,21 @@ func init() {
dockerImageName = os.Getenv("DOCKER_IMAGE_NAME") dockerImageName = os.Getenv("DOCKER_IMAGE_NAME")
dockerImageTag = os.Getenv("DOCKER_IMAGE_TAG") dockerImageTag = os.Getenv("DOCKER_IMAGE_TAG")
serverName, _ = os.Hostname() serverName, _ = os.Hostname()
// 获取真实局域网 IP (UDP 8.8.8.8 伪拨号法)
conn, err := net.Dial("udp", "8.8.8.8:80")
if err == nil {
localAddr := conn.LocalAddr().(*net.UDPAddr)
serverIp = localAddr.IP.String()
_ = conn.Close()
}
if serverIp == "" {
addrs, err := net.InterfaceAddrs() addrs, err := net.InterfaceAddrs()
if err == nil { if err == nil {
for _, a := range addrs { for _, a := range addrs {
if an, ok := a.(*net.IPNet); ok { if an, ok := a.(*net.IPNet); ok {
// 忽略 Docker 私有网段 if an.IP.IsGlobalUnicast() {
if an.IP.IsGlobalUnicast() && !strings.HasPrefix(an.IP.To4().String(), "172.17.") {
serverIp = an.IP.To4().String() serverIp = an.IP.To4().String()
break break
} }
@ -37,107 +43,13 @@ func init() {
} }
} }
} }
// MakeTime 解析纳秒时间戳或 RFC3339 字符串
func MakeTime(v any) time.Time {
if ts, ok := cast.ToInt64E(v); ok == nil {
return time.Unix(0, ts)
}
tm, _ := time.Parse(time.RFC3339Nano, cast.String(v))
return tm
} }
// MakeUsedTime 计算消耗时间(毫秒) // fixField 格式化字段名(去横线、下划线,小写)
func MakeUsedTime(startTime, endTime time.Time) float32 {
return float32(endTime.UnixNano()-startTime.UnixNano()) / 1e6
}
// ParseBaseLog 解析基础日志行
func ParseBaseLog(line string) *BaseLog {
pos := strings.IndexByte(line, '{')
if pos == -1 {
return ParseBadLog(line)
}
l := make(map[string]any)
err := json.Unmarshal([]byte(line[pos:]), &l)
if err != nil {
return ParseBadLog(line)
}
baseLog := BaseLog{Extra: make(map[string]any)}
for k, v := range l {
lk := strings.ToLower(k)
switch lk {
case "logname":
baseLog.LogName = cast.String(v)
case "logtype":
baseLog.LogType = cast.String(v)
case "logtime":
baseLog.LogTime = cast.Int64(v)
case "traceid":
baseLog.TraceId = cast.String(v)
case "imagename":
if baseLog.Image != "" {
baseLog.Image = cast.String(v) + ":" + baseLog.Image
} else {
baseLog.Image = cast.String(v)
}
case "imagetag":
if baseLog.Image != "" {
baseLog.Image = baseLog.Image + ":" + cast.String(v)
} else {
baseLog.Image = cast.String(v)
}
case "servername":
if baseLog.Server != "" {
baseLog.Server = cast.String(v) + ":" + baseLog.Server
} else {
baseLog.Server = cast.String(v)
}
case "serverip":
if baseLog.Server != "" {
baseLog.Server = baseLog.Server + ":" + cast.String(v)
} else {
baseLog.Server = cast.String(v)
}
default:
baseLog.Extra[lk] = v
}
}
return &baseLog
}
// ParseBadLog 解析非 JSON 格式的日志
func ParseBadLog(line string) *BaseLog {
baseLog := BaseLog{Extra: make(map[string]any)}
baseLog.LogType = LogTypeUndefined
if len(line) > 19 && line[19] == ' ' {
tm, err := time.Parse("2006/01/02 15:04:05", line[0:19])
if err == nil {
baseLog.LogTime = tm.UnixNano()
line = line[20:]
} else {
return nil
}
} else if len(line) > 26 && line[26] == ' ' {
tm, err := time.Parse("2006/01/02 15:04:05.000000", line[0:26])
if err == nil {
baseLog.LogTime = tm.UnixNano()
line = line[27:]
} else {
return nil
}
} else {
return nil
}
baseLog.Extra["info"] = line
return &baseLog
}
// fixField 格式化字段名(去横线,小写)
func fixField(s string) string { func fixField(s string) string {
return strings.ToLower(strings.ReplaceAll(s, "-", "")) s = strings.ReplaceAll(s, "-", "")
s = strings.ReplaceAll(s, "_", "")
return strings.ToLower(s)
} }
// getCallStacks 获取调用栈 // getCallStacks 获取调用栈
@ -157,6 +69,7 @@ func getCallStacks(truncations []string) []string {
isLogInternal := (strings.Contains(file, "/log/logger.go") || isLogInternal := (strings.Contains(file, "/log/logger.go") ||
strings.Contains(file, "/log/utility.go") || strings.Contains(file, "/log/utility.go") ||
strings.Contains(file, "/log/standard.go") || strings.Contains(file, "/log/standard.go") ||
strings.Contains(file, "/log/default_logger.go") ||
strings.Contains(file, "/log/extra.go")) strings.Contains(file, "/log/extra.go"))
if isLogInternal { if isLogInternal {
@ -179,24 +92,21 @@ func getCallStacks(truncations []string) []string {
return callStacks return callStacks
} }
// GetDefaultName 获取默认应用名称 var globalDefaultName string
func GetDefaultName() string {
name := os.Getenv("DISCOVER_APP") // getDefaultName 获取默认应用名称
if name == "" { func getDefaultName() string {
name = os.Getenv("discover_app") if globalDefaultName != "" {
} return globalDefaultName
if name == "" {
imageName := os.Getenv("DOCKER_IMAGE_NAME")
if imageName != "" {
parts := strings.Split(imageName, "/")
imageName = parts[len(parts)-1]
imageName = strings.SplitN(imageName, ":", 2)[0]
imageName = strings.SplitN(imageName, "#", 2)[0]
name = imageName
} }
name := ""
if info, ok := debug.ReadBuildInfo(); ok && info.Path != "" && info.Path != "command-line-arguments" {
name = path.Base(info.Path)
} }
if name == "" { if name == "" {
name = path.Base(os.Args[0]) name = path.Base(os.Args[0])
} }
// 处理 Windows 下的 .exe 后缀
name = strings.TrimSuffix(name, ".exe")
return name return name
} }

168
viewer.go
View File

@ -2,7 +2,9 @@ package log
import ( import (
"fmt" "fmt"
"os"
"regexp" "regexp"
"sort"
"strings" "strings"
"time" "time"
@ -12,6 +14,7 @@ import (
var errorLineMatcher = regexp.MustCompile(`(\w+\.go:\d+)`) var errorLineMatcher = regexp.MustCompile(`(\w+\.go:\d+)`)
var codeFileMatcher = regexp.MustCompile(`(\w+?\.)(go|js)`) var codeFileMatcher = regexp.MustCompile(`(\w+?\.)(go|js)`)
var workspaceRoot, _ = os.Getwd()
func Viewable(line string) string { func Viewable(line string) string {
line = strings.TrimSpace(line) line = strings.TrimSpace(line)
@ -34,16 +37,20 @@ func Viewable(line string) string {
return line return line
} }
if len(arr) < 3 { if len(arr) < 2 {
return line // At least Name, Type, Time return line
} }
logType := cast.String(arr[1]) logType := ""
if logType == "" { if len(arr) > 2 {
logType = "undefined" logType = cast.String(arr[2])
} }
meta := GetMeta(logType) meta := GetMeta(logType)
if len(meta) == 0 {
logType = cast.String(arr[1])
meta = GetMeta(logType)
}
if len(meta) == 0 { if len(meta) == 0 {
// Fallback rendering // Fallback rendering
return fallbackRenderArray(arr) return fallbackRenderArray(arr)
@ -52,7 +59,7 @@ func Viewable(line string) string {
var builder strings.Builder var builder strings.Builder
for i, v := range arr { for i, v := range arr {
if v == nil { if v == nil || cast.String(v) == "0" { // 0 is gap
continue continue
} }
if i >= len(meta) { if i >= len(meta) {
@ -65,24 +72,23 @@ func Viewable(line string) string {
m := meta[i] m := meta[i]
if m.Hide { if m.Hide || m.Name == "" {
continue continue
} }
if m.Name == "Extra" { if m.Name == "Extra" {
extraMap, ok := v.(map[string]any) extraMap, ok := v.(map[string]any)
if ok && len(extraMap) > 0 { if ok && len(extraMap) > 0 {
for k, ev := range extraMap { keys := make([]string, 0, len(extraMap))
for k := range extraMap {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
ev := extraMap[k]
builder.WriteString(" ") builder.WriteString(" ")
builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Italic, k+":")) builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Italic, k+":"))
vStr := "" builder.WriteString(renderValue(ev, 0, ""))
switch ev.(type) {
case map[string]any, []any:
vStr, _ = cast.ToJSON(ev)
default:
vStr = cast.String(ev)
}
builder.WriteString(vStr)
} }
} }
continue continue
@ -91,9 +97,18 @@ func Viewable(line string) string {
if m.Name == "CallStacks" { if m.Name == "CallStacks" {
callStacksList, ok := v.([]any) callStacksList, ok := v.([]any)
if ok && len(callStacksList) > 0 { if ok && len(callStacksList) > 0 {
stackColor := shell.TextRed
if strings.Contains(strings.ToLower(logType), "warn") {
stackColor = shell.TextYellow
}
builder.WriteString("\n") builder.WriteString("\n")
for _, vi := range callStacksList { for _, vi := range callStacksList {
vStr := cast.String(vi) vStr := cast.String(vi)
if workspaceRoot != "" {
vStr = strings.TrimPrefix(vStr, workspaceRoot)
vStr = strings.TrimPrefix(vStr, "/")
}
postfix := "" postfix := ""
if pos := strings.LastIndexByte(vStr, '/'); pos != -1 { if pos := strings.LastIndexByte(vStr, '/'); pos != -1 {
postfix = vStr[pos+1:] postfix = vStr[pos+1:]
@ -104,7 +119,7 @@ func Viewable(line string) string {
} }
builder.WriteString(" ") builder.WriteString(" ")
builder.WriteString(shell.Style(shell.Dim, vStr)) builder.WriteString(shell.Style(shell.Dim, vStr))
builder.WriteString(shell.Style(shell.TextWhite, postfix)) builder.WriteString(shell.Style(stackColor, postfix))
builder.WriteString("\n") builder.WriteString("\n")
} }
} }
@ -116,28 +131,42 @@ func Viewable(line string) string {
if m.Format == "time" { if m.Format == "time" {
// Convert int64 ns to time string // Convert int64 ns to time string
logTime := time.Unix(0, cast.Int64(v)) logTime := time.Unix(0, cast.Int64(v))
vStr = logTime.Format("01-02 15:04:05.000") dateStr := logTime.Format("01-02")
if m.Color == "" { timeStr := logTime.Format("15:04:05")
builder.WriteString(shell.White(shell.Bold, vStr)) milliStr := logTime.Format(".000")
if builder.Len() > 0 {
builder.WriteString(" ") builder.WriteString(" ")
continue
} }
builder.WriteString(shell.Style(shell.Dim, dateStr))
builder.WriteString(" ")
builder.WriteString(shell.White(shell.Bold, timeStr))
builder.WriteString(shell.Style(shell.Dim, milliStr))
continue
} else { } else {
vStr = cast.String(v) vStr = renderValue(v, m.Precision, m.Color)
if vStr == "" { if vStr == "" {
continue continue
} }
} }
if builder.Len() > 0 { if builder.Len() > 0 {
if m.AttachBefore {
builder.WriteString(":")
} else {
builder.WriteString(" ") builder.WriteString(" ")
} }
if !m.WithoutKey {
builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Italic, m.Name+":"))
} }
builder.WriteString(applyColor(vStr, m.Color)) if !m.WithoutKey && !m.AttachBefore {
name := m.KeyName
if name == "" {
name = m.Name
}
builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Italic, name+":"))
}
builder.WriteString(vStr)
} }
return builder.String() return builder.String()
@ -155,12 +184,20 @@ func ToJSON(line string) string {
return line return line
} }
if len(arr) < 3 { if len(arr) < 2 {
return line return line
} }
logType := cast.String(arr[1]) logType := ""
if len(arr) > 2 {
logType = cast.String(arr[2])
}
meta := GetMeta(logType) meta := GetMeta(logType)
if len(meta) == 0 {
logType = cast.String(arr[1])
meta = GetMeta(logType)
}
if len(meta) == 0 { if len(meta) == 0 {
return line return line
} }
@ -169,6 +206,15 @@ func ToJSON(line string) string {
for i, v := range arr { for i, v := range arr {
if i < len(meta) { if i < len(meta) {
m := meta[i] m := meta[i]
if m.Name == "" {
continue
}
name := m.KeyName
if name == "" {
name = m.Name
}
if m.Name == "Extra" { if m.Name == "Extra" {
if extraMap, ok := v.(map[string]any); ok { if extraMap, ok := v.(map[string]any); ok {
for k, ev := range extraMap { for k, ev := range extraMap {
@ -176,9 +222,9 @@ func ToJSON(line string) string {
} }
} }
} else { } else {
result[m.Name] = v result[name] = v
} }
} else { } else if cast.String(v) != "0" {
result[fmt.Sprintf("Extra%d", i)] = v result[fmt.Sprintf("Extra%d", i)] = v
} }
} }
@ -201,7 +247,7 @@ func applyColor(text string, color string) string {
return shell.Yellow(text) return shell.Yellow(text)
case "green": case "green":
return shell.Green(text) return shell.Green(text)
case "gray", "darkGray": case "gray":
return shell.Style(shell.Dim, text) return shell.Style(shell.Dim, text)
default: default:
return text return text
@ -218,3 +264,61 @@ func fallbackRenderArray(arr []any) string {
} }
return builder.String() return builder.String()
} }
func renderValue(v any, precision int, color string) string {
if v == nil {
return ""
}
switch val := v.(type) {
case float32, float64:
vStr := ""
if precision > 0 {
vStr = fmt.Sprintf("%.*f", precision, cast.To[float64](v))
} else {
vStr = cast.String(v)
}
return applyColor(vStr, color)
case map[string]any:
if len(val) == 0 {
return ""
}
var parts []string
keys := make([]string, 0, len(val))
for k := range val {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
// Key is always dim, value is colored
vStr := renderValue(val[k], precision, color)
if vStr != "" {
parts = append(parts, fmt.Sprintf("%s:%s", shell.Style(shell.Dim, k), vStr))
}
}
if len(parts) == 0 {
return ""
}
return "[ " + strings.Join(parts, " ") + " ]"
case []any:
if len(val) == 0 {
return ""
}
var parts []string
for _, iv := range val {
vStr := renderValue(iv, precision, color)
if vStr != "" {
parts = append(parts, vStr)
}
}
if len(parts) == 0 {
return ""
}
return "[ " + strings.Join(parts, " ") + " ]"
default:
s := cast.String(v)
if s == "" {
return ""
}
return applyColor(s, color)
}
}

View File

@ -20,7 +20,7 @@ func TestViewable(t *testing.T) {
} }
log.RegisterType("info", entry) log.RegisterType("info", entry)
line := string(log.ToArrayBytes(entry, nil)) line := string(log.Marshal(entry, nil))
out := log.Viewable(line) out := log.Viewable(line)
if !strings.Contains(out, "hello world") { if !strings.Contains(out, "hello world") {
@ -44,7 +44,7 @@ func TestToJSON(t *testing.T) {
entry.Extra = map[string]any{"key": "value"} entry.Extra = map[string]any{"key": "value"}
log.RegisterType("info", entry) log.RegisterType("info", entry)
line := string(log.ToArrayBytes(entry, nil)) line := string(log.Marshal(entry, nil))
jsonStr := log.ToJSON(line) jsonStr := log.ToJSON(line)
if !strings.Contains(jsonStr, `"Info":"hello world"`) { if !strings.Contains(jsonStr, `"Info":"hello world"`) {
@ -77,3 +77,154 @@ func TestLoadMeta(t *testing.T) {
t.Errorf("expected Field1, got %s", meta[0].Name) t.Errorf("expected Field1, got %s", meta[0].Name)
} }
} }
type EnhancedLog struct {
log.BaseLog
App string `log:"pos:10,withoutkey:true"`
Node string `log:"pos:11,attachBefore,withoutkey:true"`
RequestHeaders map[string]string `log:"pos:13,keyname:reqH"`
ClientIP string `log:"pos:12,keyname:ip"`
Tags []string `log:"pos:14"`
}
func (l *EnhancedLog) Reset() {
l.BaseLog.Reset()
l.App = ""
l.Node = ""
l.ClientIP = ""
l.RequestHeaders = nil
l.Tags = nil
}
func TestEnhancedViewable(t *testing.T) {
entry := &EnhancedLog{
BaseLog: log.BaseLog{
LogType: "enhanced",
LogTime: 1714896000000000000,
},
App: "MyApp",
Node: "Node1",
ClientIP: "127.0.0.1",
RequestHeaders: map[string]string{
"User-Agent": "Go-http-cli",
},
Tags: []string{"tag1", "tag2"},
}
log.RegisterType("enhanced", entry)
line := string(log.Marshal(entry, nil))
out := log.Viewable(line)
// Check attachBefore: MyApp:Node1 (since both are withoutkey)
if !strings.Contains(out, "MyApp:Node1") {
t.Errorf("expected MyApp:Node1, got: %s", out)
}
// Check pos ordering and keyname: ip:127.0.0.1 should come before reqH
if !strings.Contains(out, "ip:") || !strings.Contains(out, "127.0.0.1") {
t.Errorf("expected ip:127.0.0.1, got: %s", out)
}
if !strings.Contains(out, "reqH:") || !strings.Contains(out, "User-Agent") || !strings.Contains(out, "Go-http-cli") {
t.Errorf("expected reqH:[ User-Agent:Go-http-cli ], got: %s", out)
}
ipIdx := strings.Index(out, "ip:")
reqHIdx := strings.Index(out, "reqH:")
if ipIdx > reqHIdx {
t.Errorf("expected ip to come before reqH, but ipIdx=%d, reqHIdx=%d", ipIdx, reqHIdx)
}
// Check array rendering: Tags:[ tag1 tag2 ]
if !strings.Contains(out, "Tags:") || !strings.Contains(out, "[ tag1 tag2 ]") {
t.Errorf("expected Tags:[ tag1 tag2 ], got: %s", out)
}
}
func TestEnhancedToJSON(t *testing.T) {
entry := &EnhancedLog{
BaseLog: log.BaseLog{
LogType: "enhanced",
LogTime: 1714896000000000000,
},
App: "MyApp",
Node: "Node1",
ClientIP: "127.0.0.1",
RequestHeaders: map[string]string{
"User-Agent": "Go-http-cli",
},
}
log.RegisterType("enhanced", entry)
line := string(log.Marshal(entry, nil))
jsonStr := log.ToJSON(line)
// Check keyname in JSON
if !strings.Contains(jsonStr, `"ip":"127.0.0.1"`) {
t.Errorf("expected ip field in JSON, got: %s", jsonStr)
}
if !strings.Contains(jsonStr, `"reqH":{"User-Agent":"Go-http-cli"}`) {
t.Errorf("expected reqH field in JSON, got: %s", jsonStr)
}
}
type CallStackLog struct {
log.BaseLog
CallStacks []string `log:"pos:6"`
}
func (l *CallStackLog) Reset() {
l.BaseLog.Reset()
l.CallStacks = nil
}
func TestCallStacksViewable(t *testing.T) {
wd, _ := os.Getwd()
entry := &CallStackLog{
BaseLog: log.BaseLog{
LogType: "test_error",
LogTime: 1714896000000000000,
},
CallStacks: []string{wd + "/main.go:10", "/usr/local/go/src/runtime/panic.go:100"},
}
log.RegisterType("test_error", entry)
line := string(log.Marshal(entry, nil))
out := log.Viewable(line)
// Check path truncation (should contain relative "main.go:10")
if !strings.Contains(out, "main.go:10") {
t.Errorf("expected relative path main.go:10, got: %s", out)
}
// Absolute path should be removed if it matches wd
if strings.Contains(out, wd) {
t.Errorf("absolute path should be truncated, but still found: %s", out)
}
}
type PrecisionLog struct {
log.BaseLog
Value float64 `log:"pos:6,precision:2"`
}
func (l *PrecisionLog) Reset() {
l.BaseLog.Reset()
l.Value = 0
}
func TestPrecisionViewable(t *testing.T) {
entry := &PrecisionLog{
BaseLog: log.BaseLog{
LogType: "precision",
LogTime: 1714896000000000000,
},
Value: 3.14159,
}
log.RegisterType("precision", entry)
line := string(log.Marshal(entry, nil))
out := log.Viewable(line)
if !strings.Contains(out, "3.14") || strings.Contains(out, "3.141") {
t.Errorf("expected 3.14 (precision 2), got: %s", out)
}
}

118
writer.go
View File

@ -1,6 +1,7 @@
package log package log
import ( import (
"context"
"fmt" "fmt"
"sync" "sync"
"sync/atomic" "sync/atomic"
@ -21,12 +22,22 @@ type logPayload struct {
file *FileWriter // 目标文件 Writer file *FileWriter // 目标文件 Writer
} }
// writerService manages the background writing of log entries.
type writerService struct {
Running atomic.Bool
StopChan chan bool
LogChannel chan logPayload
Dropped atomic.Uint64
Writers atomic.Value // []Writer
WriterLock sync.Mutex
Files map[string]*FileWriter
FilesLock sync.RWMutex
}
var ( var (
writerRunning atomic.Bool // WriterService is the global instance of defaultService.
writerLock sync.Mutex // 仅用于注册时锁定 WriterService = &writerService{}
writerStopChan chan bool
writers atomic.Value // 存储 []Writer
logChannel chan logPayload
) )
// ConsoleWriter 控制台写入器 // ConsoleWriter 控制台写入器
@ -41,93 +52,112 @@ func (w *ConsoleWriter) Run() {
} }
func init() { func init() {
logChannel = make(chan logPayload, 10000) WriterService.LogChannel = make(chan logPayload, 10000)
writers.Store([]Writer{}) WriterService.Writers.Store([]Writer{})
WriterService.Files = make(map[string]*FileWriter)
RegisterWriterMaker("console", func(conf *Config) Writer { RegisterWriterMaker("console", func(conf *Config) Writer {
return &ConsoleWriter{} return &ConsoleWriter{}
}) })
} }
// WriteAsync 异步写入日志 // writeAsync 异步写入日志
func WriteAsync(payload logPayload) { func writeAsync(payload logPayload) {
defer func() { defer func() {
recover() recover()
}() }()
if !writerRunning.Load() { if !WriterService.Running.Load() {
return return
} }
select { select {
case logChannel <- payload: case WriterService.LogChannel <- payload:
default: default:
// 丢弃或处理过载,此处简单丢弃 // 丢弃或处理过载
dropped := WriterService.Dropped.Add(1)
if dropped%1000 == 1 {
if DefaultLogger != nil {
// 注意:这里可能会产生递归调用,但 select default 保证了不会死锁
DefaultLogger.Error(fmt.Sprintf("log channel full, dropped %d logs", dropped))
}
}
} }
} }
// Start 启动写入器 // GetDroppedLogs 获取被丢弃的日志数量
func Start() { func GetDroppedLogs() uint64 {
if !writerRunning.CompareAndSwap(false, true) { return WriterService.Dropped.Load()
return
}
writerStopChan = make(chan bool)
go writerRunner()
} }
// Stop 停止写入器 // Start implements starter.Service interface.
func Stop() { func (s *writerService) Start(_ context.Context, _ *Logger) error {
if writerRunning.CompareAndSwap(true, false) { if !s.Running.CompareAndSwap(false, true) {
close(logChannel) return nil
} }
s.StopChan = make(chan bool)
go s.writerRunner()
return nil
} }
// Wait 等待写入器停止 // Stop implements starter.Service interface.
func Wait() { func (s *writerService) Stop(_ context.Context) error {
if writerStopChan != nil { if s.Running.CompareAndSwap(true, false) {
<-writerStopChan close(s.LogChannel)
writerStopChan = nil if s.StopChan != nil {
<-s.StopChan
s.StopChan = nil
} }
} }
return nil
}
func writerRunner() { // Status implements starter.Service interface.
func (s *writerService) Status() (string, error) {
if s.Running.Load() {
return fmt.Sprintf("queue: %d, dropped: %d", len(s.LogChannel), s.Dropped.Load()), nil
}
return "stopped", nil
}
func (s *writerService) writerRunner() {
ticker := time.NewTicker(200 * time.Millisecond) ticker := time.NewTicker(200 * time.Millisecond)
defer ticker.Stop() defer ticker.Stop()
defer func() { defer func() {
if writerStopChan != nil { if s.StopChan != nil {
close(writerStopChan) close(s.StopChan)
} }
}() }()
for { for {
select { select {
case payload, ok := <-logChannel: case payload, ok := <-s.LogChannel:
if !ok { if !ok {
flushWriters() s.flushWriters()
return return
} }
processLog(payload) s.processLog(payload)
// 尝试批量处理更多日志 // 尝试批量处理更多日志
batchCount := 0 batchCount := 0
for batchCount < 100 { for batchCount < 100 {
select { select {
case nextPayload, nextOk := <-logChannel: case nextPayload, nextOk := <-s.LogChannel:
if !nextOk { if !nextOk {
flushWriters() s.flushWriters()
return return
} }
processLog(nextPayload) s.processLog(nextPayload)
batchCount++ batchCount++
default: default:
batchCount = 100 // break outer loop batchCount = 100 // break outer loop
} }
} }
case <-ticker.C: case <-ticker.C:
flushWriters() s.flushWriters()
} }
} }
} }
func processLog(payload logPayload) { func (s *writerService) processLog(payload logPayload) {
// 精准路由:根据包裹信息决定写入目标 // 精准路由:根据包裹信息决定写入目标
if payload.writer != nil { if payload.writer != nil {
payload.writer.Log(payload.entry, payload.buf) payload.writer.Log(payload.entry, payload.buf)
@ -136,15 +166,15 @@ func processLog(payload logPayload) {
} }
} }
func flushWriters() { func (s *writerService) flushWriters() {
curWriters, _ := writers.Load().([]Writer) curWriters, _ := s.Writers.Load().([]Writer)
for _, w := range curWriters { for _, w := range curWriters {
w.Run() w.Run()
} }
filesLock.RLock() s.FilesLock.RLock()
for _, f := range files { for _, f := range s.Files {
f.Run() f.Run()
} }
filesLock.RUnlock() s.FilesLock.RUnlock()
} }