Compare commits

...

25 Commits
v1.1.1 ... main

Author SHA1 Message Date
AI Engineer
1870d84974 chore: infrastructure alignment and doc sync (by AICoder) 2026-05-16 01:27:48 +08:00
AI Engineer
6bc0aa4c0a chore: infrastructure alignment and doc sync (by checkall) 2026-05-14 21:51:48 +08:00
AI Engineer
3d0fc5e93f feat: align with new starter.Service interface, reporting queue and drop status (by AI) 2026-05-13 11:11:12 +08:00
AI Engineer
1f816dc8b3 feat: add log.As and logger.As for frictionless error handling 2026-05-13 01:07:42 +08:00
AI Engineer
a2b1055f5d rename 2026-05-13 00:29:17 +08:00
AI Engineer
13418b5365 docs: update README, CHANGELOG, TEST and align with LoggerService architecture 2026-05-12 23:04:00 +08:00
AI Engineer
c1e1ecc3b0 对齐 Tag v1.3.0 (By AI) 2026-05-10 15:48:33 +08:00
AI Engineer
1e5dbd2a23 chore: final infrastructure alignment 2026-05-10 13:12:21 +08:00
AI Engineer
96c910bb14 chore: infrastructure alignment 2026-05-10 13:04:29 +08:00
AI Engineer
9459a98224 feat: align GetDefaultName with service SOP (by AI) 2026-05-10 10:50:00 +08:00
AI Engineer
6dfb33c459 feat: 增加 authorization 默认脱敏规则(by AI) 2026-05-09 21:01:08 +08:00
AI Engineer
ec8406fe42 feat(log): 调整可视化能力 2026-05-09 16:30:01 +08:00
AI Engineer
03267710dc feat(log): 绝对索引优化与强制 Reset 安全契约 (v1.1.13) 2026-05-09 14:44:41 +08:00
AI Engineer
78d6addf4c 稳定性增强:修复 TestSplitTag 秒级边界竞态,并完成 go/cast 与 go/file 基础设施对齐(by AI) 2026-05-05 23:47:07 +08:00
AI Engineer
9c6112d253 Verify SplitTag rotation with seconds in tests (by AI) 2026-05-05 23:32:43 +08:00
AI Engineer
534d3dfdd6 Implement deep desensitization using cast for complex types (by AI) 2026-05-05 23:25:17 +08:00
AI Engineer
9252fe002e Cleanup dead code, fix desensitization, and add functional tests (by AI) 2026-05-05 23:09:46 +08:00
AI Engineer
c988b8d88b Optimize FillBase to take *BaseLog and update documentation (by AI) 2026-05-05 22:52:55 +08:00
AI Engineer
00a677492d chore: align dependencies 2026-05-05 22:03:57 +08:00
AI Engineer
0fe4a51407 chore: update dependencies 2026-05-05 21:58:02 +08:00
AI Engineer
be893e1b99 feat: 实现高性能 Meta 驱动的 JSON 数组日志架构 (by AI) 2026-05-05 21:45:43 +08:00
AI Engineer
8a44c1ace6 temp version for new plan 2026-05-05 20:35:27 +08:00
AI Engineer
b2f91d37be temp version for new plan 2026-05-05 20:35:15 +08:00
AI Engineer
a720cfb63b 自动化增强:引入 StackTraceable 接口,Log 方法支持自动补全调用栈 (by AI) 2026-05-05 18:10:17 +08:00
AI Engineer
f9a10486e3 架构解耦:移除内置 DB 日志支持,回归私有填充逻辑,导出 GetCallStacks (by AI) 2026-05-05 17:59:43 +08:00
30 changed files with 2308 additions and 847 deletions

9
.gitignore vendored Normal file
View File

@ -0,0 +1,9 @@
.log.meta.json
.test.meta.json
.ai/
.geminiignore
.gemini
env.json
env.yml
env.yaml
/CODE-FULL.md

View File

@ -1,6 +1,107 @@
# Changelog
## [1.1.1] - 2026-05-05
## [1.3.2] - 2026-05-13
- **功能增强: 引入摩擦消除工具 `As`**:
- **泛型支持**: 新增全局泛型函数 `log.As[T](v T, err error) T`,仿照 `cast.As` 设计,自动记录错误并返回零值,极大简化了带 error 返回值的函数链式调用。
- **Logger 扩展**: `Logger` 结构体新增 `As(v any, err error) any` 方法,支持实例级别的错误捕获与自动记录。
- **调用栈优化**: 优化了 `GetCallStacks` 逻辑,自动跳过 `default_logger.go` 中的内部帧,确保 `log.As` 记录的错误位置精准指向业务代码。
## [1.3.1] - 2026-05-12
- **架构升级: 引入 LoggerService**:
- **解耦重构**: 重构全局变量管理,引入 `loggerService` 结构体集中化管理异步写入协程、Writers 对象池、文件句柄与丢弃计数。
- **生命周期管理**: 实现了 `Start(ctx, logger)`, `Stop(ctx)`, `Health()` 接口,完美支持与 `apigo.cc/go/starter` 的基础设施集成。
- **安全性增强**: 优化了平滑停止逻辑,确保 `Stop` 调用时能完整 Flush 缓冲区数据。
- **功能增强**:
- **动态应用名**: 新增 `SetDefaultName(name)` 全局方法,支持在微服务启动后动态设置应用名称,并自动同步至 `DefaultLogger`
- **配置与忽略规则**:
- 更新 `.gitignore`,增加了对 `.gemini`, `env.json`, `env.yml` 以及 `/CODE-FULL.md` 的忽略支持。
- **文档与测试同步**: 全面更新了 `README.md`, `CHANGELOG.md``TEST.md`
## [1.3.0] - 2026-05-10
- **全栈基础设施对齐**:
- 将所有内部依赖(`cast`, `config`, `file`, `id`, `shell` 等)统一升级至 `v1.3.0` 语义版本。
- 优化项目忽略规则,在 `.gitignore` 中增加了对 `.geminiignore` 的支持。
## [1.1.16] - 2026-05-10
- **依赖对齐**: 同步更新 `golang.org/x/crypto``golang.org/x/sys` 至最新版本,确保底层安全性。
## [1.1.15] - 2026-05-10
- **安全与性能双重增强**:
- **敏感字段扩展**: `authorization` 现在默认纳入脱敏规则 (`LogDefaultSensitive`),自动过滤 HTTP 认证凭证。
- **TraceId 吞吐优化**: 切换至 `id.Get10Bytes14MPerSecond()` 算法,在保持 10 字节唯一性的同时,极大提升了超高并发下的 ID 生成性能。
- **应用名称识别对齐**: 重构 `GetDefaultName` 逻辑,优先通过 `debug.ReadBuildInfo()` 识别 Module 路径,并支持自动裁剪 Windows 环境下的 `.exe` 后缀。
## [1.1.14] - 2026-05-09
- **可视化能力调整**: 调整优化了 `viewer` 模块相关的可视化能力,提升了日志的可读性与调试体验。
## [1.1.13] - 2026-05-09
- **绝对索引优化与零空洞**:
- 彻底消除 `BaseLog` 与业务字段之间的索引空洞。字段位置调整为:`BaseLog` (0-5),标准消息字段 (`Info`, `Error` 等) 与业务日志字段从 `pos: 6` 起始。
- **智能平移映射**: 改进 `pos >= 1000` 的逻辑。这些字段(如 `Extra`, `CallStacks`)不再产生稀疏 0 占位符,而是根据定义的 `pos` 顺序,自动平移并紧跟在当前类型的最大绝对索引之后。既保证了“永远在末尾”,又维持了数组的紧凑性。
- **强制 Reset 安全契约**:
- `RegisterType` 引入严苛校验:若自定义日志类型包含字段但未显式重写 `Reset()`(即仅继承了 `BaseLog.Reset`),注册时将触发 **Panic**。强制开发者显式清理业务数据,杜绝对象池复用时的脏数据隐患。
- **应用名称识别增强**:
- `GetDefaultName` 引入 `runtime/debug.ReadBuildInfo()`。相比传统的文件夹路径读取,能更精准地识别 Go Module 定义的应用名称。
- **工具链增强**:
- `serializer_test.go` 新增原始日志输出,方便开发者通过 `go test -v` 直观校验 JSON 数组结构。
## [1.1.12] - 2026-05-09
...
- **稳定性增强**:
- 修复 `TestSplitTag` 在秒级进位边界时的偶发性失败。优化后的测试逻辑同时校验日志产生前后的两个潜在时间槽,闭环消除了环境抖动导致的 race condition。
## [1.1.9] - 2026-05-05
- **稳定性增强**:
- 改进 `SplitTag` 测试用例,通过秒级切分步进,闭环验证了 `FileWriter` 的文件自动轮转逻辑,确保在高频或定时切分场景下的可靠性。
## [1.1.8] - 2026-05-05
- **深度脱敏与性能平衡**:
- 引入“混合路径”序列化策略对基础类型String, Int, Bool 等采用高性能手动编码对复杂类型Struct, Map, Slice采用 `cast.ToJSONDesensitizeBytes`
- 完美解决嵌套对象、数组内对象的深度脱敏问题,同时将序列化性能维持在极高水平。
- **基准测试优化**: 相比全量 `cast` 调用,混合路径将异步写入耗时降低了 60% 以上,内存分配减少了 70%。
## [1.1.7] - 2026-05-05
- **极致精简与摩擦消除**:
- 移除了所有冗余或已弃用的配置项:`Fast`, `KeepKeyCase`, `RegexSensitive`, `SensitiveRule`
- 删除了 `Logger` 中未被使用的 `desensitization` 处理函数及 `SetDesensitization` 方法,进一步收窄 API。
- **功能增强与修复**:
- **脱敏逻辑闭环**: 修复了 `ToArrayBytes` 仅对 Map 字段脱敏的缺陷。现在支持对结构体根层级的敏感字段(如 `String` 类型)进行全量遮蔽。
- **标准化命名映射**: 增强 `fixField` 逻辑,同时支持自动忽略下划线 (`_`) 和横线 (`-`),确保 `SecretKey` (Go) 与 `secret_key` (Config) 能完美匹配。
- **稳定性保障**:
- 新增 `functional_test.go`,闭环验证了 `SplitTag` 文件切分能力与增强后的 `Sensitive` 脱敏逻辑。
- 优化 `serializer.go` 内部判断逻辑,基准测试显示性能有显著提升。
## [1.1.6] - 2026-05-05
- **性能优化**:
- 重构 `FillBase` 方法签名,由接收接口 `LogEntry` 改为接收指针 `*BaseLog`
- 此项改动消除了在填充元数据时通过接口调用 `GetBaseLog()` 的开销,直接操作结构体指针,进一步提升日志预处理性能。
- **文档与示例对齐**:
- 完善 `README.md` 中的 `Config` 配置项说明,涵盖 `Fast`, `KeepKeyCase`, `Truncations`, `SensitiveRule` 等所有字段。
- 修正 `BusinessLog` 扩展示例,确保与最新的 `FillBase` 签名及元数据填充逻辑保持一致。
- **扩展与迁移指引**: 同步更新 `extra.go` 中的注释示例,为从旧版本或其他项目迁移提供准确的参考实现。
## [1.1.5] - 2026-05-05
- **高性能 Meta 驱动架构**:
- 日志存储格式由 JSON Object 彻底切换为 **JSON Positional Array (`[...]`)**,通过位置索引消除重复 Key 的存储与传输开销。
- 实现基于反射的 **零装箱 (No-Boxing) 序列化**,直接拼接 JSON 字符串,大幅降低内存分配与 CPU 占用。
- **元数据外置与可视化**:
- 引入 `.log.meta.json` 机制,将字段顺序、颜色、格式化等可视化逻辑从核心包中剥离。
- 新增独立 CLI 工具 `logv`,支持基于元数据文件的 stdin 流式日志渲染。
- 改造 `Viewable` 接口,支持基于 MetaRegistry 的动态终端彩色输出渲染。
- **字段优化与压缩**:
- 合并 `ImageName/Tag``Image`,合并 `ServerName/Ip``Server`,精简日志槽位。
- 引入 `hide:true` 标签,支持在控制台隐藏结构性元数据(如 `LogName`, `LogType`),保持输出极致纯净。
- **架构兼容性**:
- 调整 `Writer` 接口,支持自定义 Writer`ESWriter`)自行决定序列化格式(如转回 Object 以适配 ES 索引)。
- `utility.go` 新增对旧版 JSON 对象日志的解析兼容。
- **安全加固**: 为 `http`, `db`, `redis`, `discover` 等子包同步添加 `.gitignore` 以排除自动生成的 `.log.meta.json`
## [1.1.2] - 2026-05-05
- **架构解耦**:
- 正式移除 `log` 包对数据库日志(`DB` 方法及 `DBLog` 结构)的内置支持,推动“日志格式随业务走”的架构对齐。
- 导出 `GetCallStacks` 方法,支持外部包捕获符合截断配置的调用栈。
- **内部优化**:
- `fillBase` 回归私有,强化 `Log(entry)` 的自动填充契约,降低外部摩擦。
- **体验对齐**:
- 修复 `viewer.go` 中空错误字符串导致日志类型被跳过显示的 Bug。
- `Log` 方法集成 `fillBase` 自动补全逻辑,对于直接调用 `Log(entry)` 的自定义日志类型,若未设置时间戳则自动填充元数据,无需手动干预。

167
README.md
View File

@ -6,9 +6,10 @@
`@go/log` 旨在提供高性能、零摩擦的异步日志系统。其核心目标是:
* **极致高性能**:采用 **Meta-Driven Positional Array (元数据驱动定长数组)** 架构。日志以单行 JSON 数组 (`[...]`) 形式落盘,消除 Key 冗余与装箱开销,性能提升数倍。
* **架构解耦**:元数据外置于 `.log.meta.json`。日志包仅负责高速序列化,可视化由外部工具或 `Viewable` 接口根据元数据动态渲染。
* **零摩擦入口**自动识别环境上下文应用名、IP等无需手动构建。
* **极致高性能**:异步写入架构,支持对象池复用,大幅降低内存分配。
* **语义脱敏**:内置敏感信息(如手机号、密钥)的自动脱敏与正则过滤。
* **语义脱敏**:内置敏感信息(如手机号、密钥)的自动脱敏。
* **高度可扩展**支持多种写入渠道文件切分、Elasticsearch批量传输
## 📦 安装
@ -22,14 +23,65 @@ go get apigo.cc/go/log
```go
import "apigo.cc/go/log"
// 使用默认配置初始化 (或在配置中指定)
logger := log.NewLogger(log.Config{Name: "my-app", Level: "info"})
// 默认 logger (通过 log.json 或环境变量配置)
func main() {
// 在微服务场景下动态设置应用名称
log.SetDefaultName("my-microservice")
// 记录业务日志 (自动通过 cast.ToMap 处理变长参数)
logger.Info("用户登录", "userId", 10086, "ip", "1.2.3.4")
logger.Error("数据库连接失败", "db", "mysql", "err", err)
log.Info("服务启动", "port", 8080)
log.Error("数据库连接失败", "db", "mysql")
// 创建带 traceId 的新 logger 实例
logger := log.New("trace-xyz-123")
}
```
## ⚙️ 配置 (Configuration)
本包深度集成 `@go/config`,支持多种灵活的配置方式,优先级从高到低:
1. **环境变量** (最高优先级)
2. **环境特定文件** (`env.json` / `env.yml`,需增加层级 `log:`)
3. **基础配置文件** (`log.json` / `log.yml`)
### 1. 配置文件 (`log.json`)
在项目根目录创建 `log.json``log.yml`
```json
{
"name": "my-cool-app",
"level": "info",
"file": "logs/app.log",
"splitTag": ".2006-01-02",
"sensitive": "phone,password,secret,token,accessToken,authorization"
}
```
### 2. 环境变量 (最高优先级)
任何配置都可以通过环境变量覆盖,变量名规则为 `LOG_` + `字段名`
```bash
# 覆盖日志级别和输出文件
export LOG_LEVEL=debug
export LOG_FILE=console
```
### 配置项说明
* `name`: 应用名称 (默认通过 `debug.ReadBuildInfo()``os.Args[0]` 自动识别)。
* `level`: 日志级别 (`debug`, `info`, `warning`, `error`)。
* `file`: 输出目标。
* `console`: 直接输出到控制台(默认)。
* `path/to/file.log`: 输出到指定文件。
* `es://...``ess://...`: 输出到 Elasticsearch。
* `splitTag`: 文件切分格式,仅当 `file` 为文件路径时有效。
* 语法遵循 Go 标准的 `time.Format` 布局,如 `".2006-01-02"` (按天切分)`".2006-01-02-15"` (按小时切分)。
* `truncations`: 堆栈信息截断前缀(多个以逗号分隔,默认截断 `github.com/`, `golang.org/`, `/apigo.cc/`)。
* `sensitive`: 需要自动脱敏的字段名(多个以逗号分隔,不区分大小写),默认处理 `phone,password,secret,token,accessToken,authorization`
## 🛠 API 指南
### 核心功能
@ -37,45 +89,90 @@ logger.Error("数据库连接失败", "db", "mysql", "err", err)
1. **分级记录**
* `Debug`, `Info`, `Warning`, `Error` —— 标准日志方法,支持 `message` + 变长 `extra` 参数。
2. **通用记录 (`Log`)**
* `Log(LogEntry)` —— 记录自定义结构的日志。注意:仅支持实现 `LogEntry` 接口的类型(即嵌入了 `BaseLog` 的结构体)。
2. **摩擦消除 (`As`)**
* `As(v, err)` —— 仿照 `cast.As`,忽略错误并返回零值,但会自动将错误记录到日志中。支持全局调用 (`log.As`) 或实例调用 (`logger.As`)。
* **优势**: 在类型转换或快速赋值场景下,无需繁琐的 `if err != nil` 判断,同时确保异常被记录。
3. **专业日志扩展**
* **请求日志 (`Request`)**: 记录 HTTP 请求,包含方法、路径、状态码、耗时等。
* **数据库日志 (`DB`)**: 自动计算耗时、捕获调用栈并支持脱敏。
* **监控与统计 (`Monitor`, `Statistic`)**: 用于应用指标监控。
* **任务执行 (`Task`)**: 用于任务耗时与状态记录。
3. **通用记录 (`Log`)**
* `Log(LogEntry)` —— 记录自定义结构的日志。
### 自定义日志扩展
4. **独立可视化工具 (`logv`)**
* **安装**: `go install apigo.cc/go/log/logv@latest`
* **使用**: `tail -f app.log | logv``tail -f app.log | logv -json`
如果标准日志分级不能满足业务需求,可以轻松扩展自定义日志类型:
### 自定义日志扩展 (规范)
1. **定义结构体**:必须嵌入 `log.BaseLog` 以自动获得基础字段和池化能力。
2. **获取对象**:使用 `log.GetEntry[MyLog]()` 从对象池获取,避免频繁分配内存。
3. **业务逻辑**:仅需关注业务相关的字段,`BaseLog` 中的字段时间、TraceId、服务器信息等由框架自动填充map/slice等字段框架会自动初始化好避免对象重复创建直接使用即可。
4. **发送日志**:调用 `logger.Log(entry)`
为保证高性能与内存安全,扩展自定义日志类型必须遵循以下规范:
1. **定义结构体**
* 必须嵌入 `log.BaseLog` (或其子类,如 `log.ErrorLog`)。
* **索引 (`pos`) 规范**:
* `0`-`6``BaseLog`
* 业务字段从 `6` 开始紧凑递增编号 (`pos:6`, `pos:7`, ...),如果删除了某个字段请留空 pos 以实现向前兼容。
* 如果继承自 `ErrorLog` 等,则业务字段应从 `7` 开始(查询父类最大值 + 1
* `Extra` 固定使用 `pos:1000``CallStacks` 固定使用 `pos:1001` (它们会被自动平移到数组末尾)。
2. **实现 `Reset()` 方法 (强制)**
* **必须**重写 `Reset()` 方法以初始化/清空数据避免对象池复用时产生脏数据。
* `Reset()` 方法中必须首先调用父级的 `Reset()` (如 `l.BaseLog.Reset()`)。
* **安全保障**: 若未重写 `Reset``RegisterType` 将在启动时 **Panic**,以防止对象池复用时产生脏数据。
* **建议**: map / slice 类型建第一次初始化一个容量,之后使用 clear() 方法清空数据避免内存重复分配。
3. **注册模型**
* 在 `init()` 中调用 `log.RegisterType("my-type", MyLog{})` 完成注册。
#### 示例: `DBErrorLog`
```go
type BusinessLog struct {
log.BaseLog // 必须嵌入
Action string
UserId string
package main
import "apigo.cc/go/log"
// 1. 定义结构体 (字段从 pos:6 开始)
type DBErrorLog struct {
log.ErrorLog // 嵌入 ErrorLog自动获得 Error 和 CallStacks 字段
DB string `log:"pos:7,color:blue"`
SQL string `log:"pos:8"`
Args []any `log:"pos:9"`
UsedTime float32 `log:"pos:10,color:cyan"`
}
func LogBusiness(logger *log.Logger, action, userId string) {
entry := log.GetEntry[BusinessLog]()
entry.Action = action
entry.UserId = userId
logger.Log(entry) // 框架会自动填充 BaseLog 并异步写入后回收对象
// 2. 实现 Reset() 方法 (强制)
func (l *DBErrorLog) Reset() {
l.ErrorLog.Reset() // 必须先调用父级 Reset
l.DB = ""
l.SQL = ""
if l.Args == nil {
l.Args = make([]any, 0, 10)
} else {
clear(l.Args) // 清空内容
l.Args = l.Args[:0] // 清空长度
}
l.UsedTime = 0
}
// 3. 注册
func init() {
log.RegisterType("dbError", &DBErrorLog{})
}
// 4. 使用示例
func LogDBError(logger *log.Logger, db, sql string, args []any, err error, usedTime float32) {
entry := log.GetEntry[DBErrorLog]()
// 自动填充基础字段和 ErrorLog 字段
logger.FillError(&entry.ErrorLog, err.Error())
// 填充自定义字段
entry.DB = db
entry.SQL = sql
entry.Args = append(entry.Args, args...)
entry.UsedTime = usedTime
logger.Log(entry)
}
```
### 配置项 (JSON/YAML)
* `Name`: 应用名称。
* `Level`: 日志级别 (`debug`, `info`, `warning`, `error`)。
* `File`: 输出目标(支持 `console``es://` 地址)。
* `Sensitive`, `RegexSensitive`: 脱敏配置。
## 🧪 验证状态
测试全部通过,异步写入与性能达标。

84
TEST.md
View File

@ -1,28 +1,62 @@
# 日志性能测试报告
# Test Results
## 测试环境
- 操作系统: darwin
- 架构: amd64
- CPU: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
## 单元测试报告
## 基准测试结果 (v1.0.3)
```text
=== RUN TestLoggerCore_Initialization
--- PASS: TestLoggerCore_Initialization (0.00s)
=== RUN TestLoggerCore_Concurrency
--- PASS: TestLoggerCore_Concurrency (0.00s)
=== RUN TestMetaExtraction
--- PASS: TestMetaExtraction (0.00s)
=== RUN TestWithEntry
--- PASS: TestWithEntry (0.00s)
=== RUN TestLoggerReliability
--- PASS: TestLoggerReliability (0.01s)
=== RUN TestToArrayBytes
serializer_test.go:64: Raw log: ["test-app","mock_info_test",1620000000,"abc-123","Hello, World!",{"user_id":42}]
--- PASS: TestToArrayBytes (0.00s)
=== RUN TestToArrayBytes_Desensitize
--- PASS: TestToArrayBytes_Desensitize (0.00s)
=== RUN TestSplitTag
--- PASS: TestSplitTag (1.80s)
=== RUN TestSensitiveDetailed
--- PASS: TestSensitiveDetailed (0.00s)
=== RUN TestDeepDesensitization
--- PASS: TestDeepDesensitization (0.00s)
=== RUN TestLogger
--- PASS: TestLogger (0.00s)
=== RUN TestDesensitization
--- PASS: TestDesensitization (0.00s)
=== RUN TestDBLog
--- PASS: TestDBLog (0.00s)
=== RUN TestRequestLog
--- PASS: TestRequestLog (0.00s)
=== RUN TestExtraLogs
--- PASS: TestExtraLogs (0.00s)
=== RUN TestViewable
--- PASS: TestViewable (0.00s)
=== RUN TestToJSON
--- PASS: TestToJSON (0.00s)
=== RUN TestLoadMeta
--- PASS: TestLoadMeta (0.00s)
=== RUN TestEnhancedViewable
--- PASS: TestEnhancedViewable (0.00s)
=== RUN TestEnhancedToJSON
--- PASS: TestEnhancedToJSON (0.00s)
=== RUN TestCallStacksViewable
--- PASS: TestCallStacksViewable (0.00s)
=== RUN TestPrecisionViewable
--- PASS: TestPrecisionViewable (0.00s)
PASS
ok apigo.cc/go/log 2.246s
```
| 测试用例 | 迭代次数 | 耗时 (ns/op) | 内存分配 (B/op) | 分配次数 (allocs/op) |
| :--- | :--- | :--- | :--- | :--- |
| `BenchmarkLogger_RequestLog_Realistic` | 2,324,065 | 544.1 | 72 | 2 |
| `BenchmarkLoggerInfo` | 122,059 | 9,706 | - | - |
| `BenchmarkLoggerAsyncConcurrent` | 127,830 | 8,773 | - | - |
## 版本对比评估
| 版本 | 机制 | 耗时 (ns/op) | 易用性 |
| :--- | :--- | :--- | :--- |
| **v1.0.1** | 手动 Reset | ~270 | 较低 (需编写大量样板代码) |
| **v1.0.2** | 自动化 Reset | ~475 | 极高 (嵌入 BaseLog 即可) |
| **v1.0.3** | 参数封装与解耦架构 | ~544 | 极高 (核心框架与业务结构完全分离) |
## 总结
- **性能评估**: v1.0.3 在核心日志记录上保持高性能。应用端自定义结构与框架对象池的结合被证明是高效的。
- **解耦架构**: `extra.go` 中的示例代码已被注释,成功将业务日志结构的定义权移交给应用层。框架仅保留最核心的异步写入和对象池管理能力。
- **内存效率**: 持续保持极低分配。
- **最佳实践**: 引导应用通过定义局部结构体并封装 `Logger` 扩展方法来记录日志,这不仅符合 Go 的工程规范,也极大地提升了系统的可维护性。
## 核心指标验证
- **初始化安全性**: `TestLoggerCore_Initialization` 确保 Logger 实例配置正确加载。
- **高并发稳定性**: `TestLoggerCore_Concurrency` 验证了在多协程竞争环境下日志写入的线程安全。
- **元数据驱动验证**: `TestMetaExtraction``TestLoadMeta` 确保 `.log.meta.json` 协议的解析与应用。
- **序列化性能**: `TestToArrayBytes` 验证了 Positional Array 格式的正确性。
- **深度脱敏能力**: `TestDeepDesensitization` 闭环验证了对复杂嵌套结构的脱敏逻辑。
- **可靠性边界**: `TestLoggerReliability` 模拟了极高压力下的日志丢弃与缓冲策略。
- **文件切分**: `TestSplitTag` 实测了基于时间滚动的文件切分能力。

View File

@ -5,14 +5,9 @@ type Config struct {
Name string
Level string
File string
Fast bool
SplitTag string
Truncations string
Sensitive string
RegexSensitive string
SensitiveRule string
KeepKeyCase bool // 是否保持Key的首字母大小写默认一律使用小写
Formatter Formatter
}
type LevelType int

View File

@ -7,8 +7,8 @@ import (
var DefaultLogger *Logger
func init() {
RegisterWriterMaker("es", NewESWriter)
RegisterWriterMaker("ess", NewESWriter)
RegisterWriterMaker("es", newESWriter)
RegisterWriterMaker("ess", newESWriter)
var conf Config
_ = config.Load(&conf, "log")
@ -19,3 +19,46 @@ func init() {
func New(traceId string) *Logger {
return DefaultLogger.New(traceId)
}
// SetDefaultName 设置全局默认应用名称,并同步更新 DefaultLogger
func SetDefaultName(name string) {
if name == "" {
return
}
globalDefaultName = name
if DefaultLogger != nil {
DefaultLogger.SetName(name)
}
}
// As 仿照 cast.As使用 DefaultLogger 记录错误并返回零值 (消除摩擦)
func As[T any](v T, err error) T {
if err != nil {
if DefaultLogger != nil {
DefaultLogger.Error(err.Error())
}
var zero T
return zero
}
return v
}
// Debug 记录一条调试级别日志
func Debug(message string, extra ...any) {
DefaultLogger.Debug(message, extra...)
}
// Info 记录一条信息级别日志
func Info(message string, extra ...any) {
DefaultLogger.Info(message, extra...)
}
// Warning 记录一条警告级别日志
func Warning(message string, extra ...any) {
DefaultLogger.Warning(message, extra...)
}
// Error 记录一条错误级别日志
func Error(message string, extra ...any) {
DefaultLogger.Error(message, extra...)
}

View File

@ -14,7 +14,7 @@ import (
"apigo.cc/go/cast"
)
type ESWriter struct {
type esWriter struct {
config *Config
url string
user string
@ -27,8 +27,8 @@ type ESWriter struct {
prefix string
}
func NewESWriter(conf *Config) Writer {
w := &ESWriter{
func newESWriter(conf *Config) Writer {
w := &esWriter{
config: conf,
queue: make([]string, 0),
client: &http.Client{},
@ -76,11 +76,12 @@ func NewESWriter(conf *Config) Writer {
return w
}
func (w *ESWriter) Log(data []byte) {
if len(data) == 0 {
func (w *esWriter) Log(entry LogEntry, data []byte) {
objBytes, err := cast.ToJSONBytes(entry)
if err != nil || len(objBytes) == 0 {
return
}
dataString := string(data)
dataString := string(objBytes)
w.lock.Lock()
w.queue = append(w.queue, w.prefix, dataString)
@ -89,14 +90,14 @@ func (w *ESWriter) Log(data []byte) {
var responseOkBytes = []byte("\"errors\":false")
func (w *ESWriter) Run() {
func (w *esWriter) Run() {
now := time.Now().Unix()
w.lock.Lock()
queueLen := len(w.queue)
w.lock.Unlock()
// 超过100条数据 或 过了1秒 发送数据
if queueLen > 100 || (queueLen > 0 && (now > w.last || !writerRunning.Load())) {
if queueLen > 100 || (queueLen > 0 && (now > w.last || !WriterService.Running.Load())) {
w.lock.Lock()
sendings := w.queue
w.queue = make([]string, 0)

197
extra.go
View File

@ -1,197 +0,0 @@
package log
// import (
// "apigo.cc/go/cast"
// )
// type RequestLog struct {
// BaseLog
// ServerId string
// App string
// Node string
// ClientIp string
// FromApp string
// FromNode string
// UserId string
// DeviceId string
// ClientAppName string
// ClientAppVersion string
// SessionId string
// RequestId string
// Host string
// Scheme string
// Proto string
// AuthLevel int
// Priority int
// Method string
// Path string
// RequestHeaders map[string]string
// RequestData map[string]any
// UsedTime float32
// ResponseCode int
// ResponseHeaders map[string]string
// ResponseDataLength uint
// ResponseData string
// }
// func (logger *Logger) Request(
// method, path, host, scheme, proto string,
// clientIp, serverId, app, node string,
// fromApp, fromNode string,
// userId, deviceId, sessionId, requestId string,
// clientAppName, clientAppVersion string,
// authLevel, priority int,
// reqHeaders map[string]string,
// reqData map[string]any,
// responseCode int,
// usedTime float32,
// respHeaders map[string]string,
// responseData string,
// responseDataLength uint,
// extra ...any,
// ) {
// if !logger.CheckLevel(INFO) {
// return
// }
// entry := GetEntry[RequestLog]()
// logger.fillBase(entry, LogTypeRequest)
// // 暴力平铺赋值,性能极高
// entry.Method = method
// entry.Path = path
// entry.Host = host
// entry.Scheme = scheme
// entry.Proto = proto
// entry.ClientIp = clientIp
// entry.ServerId = serverId
// entry.App = app
// entry.Node = node
// entry.FromApp = fromApp
// entry.FromNode = fromNode
// entry.UserId = userId
// entry.DeviceId = deviceId
// entry.SessionId = sessionId
// entry.RequestId = requestId
// entry.ClientAppName = clientAppName
// entry.ClientAppVersion = clientAppVersion
// entry.AuthLevel = authLevel
// entry.Priority = priority
// entry.RequestHeaders = reqHeaders
// entry.RequestData = reqData
// entry.ResponseCode = responseCode
// entry.UsedTime = usedTime
// entry.ResponseHeaders = respHeaders
// entry.ResponseData = responseData
// entry.ResponseDataLength = responseDataLength
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// type TaskLog struct {
// BaseLog
// Task string
// UsedTime float32
// Success bool
// Message string
// }
// type MonitorLog struct {
// BaseLog
// Target string
// Status int
// Message string
// }
// type StatisticLog struct {
// BaseLog
// Category string
// Item string
// Value float64
// }
// func (logger *Logger) Task(taskName string, usedTime float32, success bool, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[TaskLog]()
// logger.fillBase(entry, LogTypeTask)
// entry.Task = taskName
// entry.UsedTime = usedTime
// entry.Success = success
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Monitor(target string, status int, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[MonitorLog]()
// logger.fillBase(entry, LogTypeMonitor)
// entry.Target = target
// entry.Status = status
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Statistic(category, item string, value float64, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[StatisticLog]()
// logger.fillBase(entry, LogTypeStatistic)
// entry.Category = category
// entry.Item = item
// entry.Value = value
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// type DBLog struct {
// BaseLog
// DbType string
// Dsn string
// Query string
// QueryArgs string
// UsedTime float32
// Error string
// CallStacks []string
// }
// func (logger *Logger) DB(dbType, dsn, query string, args []any, usedTime float32, err error, extra ...any) {
// logType := LogTypeDb
// level := INFO
// var e string
// if err != nil {
// logType = LogTypeDbError
// level = ERROR
// e = err.Error()
// }
// if logger.CheckLevel(level) {
// entry := GetEntry[DBLog]()
// logger.fillBase(entry, logType)
// entry.DbType = dbType
// entry.Dsn = dsn
// entry.Query = query
// entry.QueryArgs = cast.To[string](args)
// entry.UsedTime = usedTime
// if e != "" {
// entry.Error = e
// entry.CallStacks = getCallStacks(logger.truncations)
// }
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }

224
extra_example.go Normal file
View File

@ -0,0 +1,224 @@
package log
// type RequestLog struct {
// BaseLog
// ServerId string `log:"pos:6"`
// App string `log:"pos:7"`
// Node string `log:"pos:8"`
// ClientIp string `log:"pos:9"`
// FromApp string `log:"pos:10"`
// FromNode string `log:"pos:11"`
// UserId string `log:"pos:12"`
// DeviceId string `log:"pos:13"`
// ClientAppName string `log:"pos:14"`
// ClientAppVersion string `log:"pos:15"`
// SessionId string `log:"pos:16"`
// RequestId string `log:"pos:17"`
// Host string `log:"pos:18"`
// Scheme string `log:"pos:19"`
// Proto string `log:"pos:20"`
// AuthLevel int `log:"pos:21"`
// Priority int `log:"pos:22"`
// Method string `log:"pos:23"`
// Path string `log:"pos:24"`
// RequestHeaders map[string]string `log:"pos:25"`
// RequestData map[string]any `log:"pos:26"`
// UsedTime float32 `log:"pos:27"`
// ResponseCode int `log:"pos:28"`
// ResponseHeaders map[string]string `log:"pos:29"`
// ResponseDataLength uint `log:"pos:30"`
// ResponseData string `log:"pos:31"`
// }
// func (l *RequestLog) Reset() {
// l.BaseLog.Reset()
// l.ServerId = ""
// l.App = ""
// l.Node = ""
// l.ClientIp = ""
// l.FromApp = ""
// l.FromNode = ""
// l.UserId = ""
// l.DeviceId = ""
// l.ClientAppName = ""
// l.ClientAppVersion = ""
// l.SessionId = ""
// l.RequestId = ""
// l.Host = ""
// l.Scheme = ""
// l.Proto = ""
// l.AuthLevel = 0
// l.Priority = 0
// l.Method = ""
// l.Path = ""
// if l.RequestHeaders == nil {
// l.RequestHeaders = make(map[string]string, 8)
// } else {
// clear(l.RequestHeaders)
// }
// if l.RequestData == nil {
// l.RequestData = make(map[string]any, 8)
// } else {
// clear(l.RequestData)
// }
// l.UsedTime = 0
// l.ResponseCode = 0
// if l.ResponseHeaders == nil {
// l.ResponseHeaders = make(map[string]string, 8)
// } else {
// clear(l.ResponseHeaders)
// }
// l.ResponseDataLength = 0
// l.ResponseData = ""
// }
// func (logger *Logger) Request(
// method, path, host, scheme, proto string,
// clientIp, serverId, app, node string,
// fromApp, fromNode string,
// userId, deviceId, sessionId, requestId string,
// clientAppName, clientAppVersion string,
// authLevel, priority int,
// reqHeaders map[string]string,
// reqData map[string]any,
// responseCode int,
// usedTime float32,
// respHeaders map[string]string,
// responseData string,
// responseDataLength uint,
// extra ...any,
// ) {
// if !logger.CheckLevel(INFO) {
// return
// }
// entry := GetEntry[RequestLog]()
// logger.FillBase(&entry.BaseLog, LogTypeRequest)
// // 暴力平铺赋值,性能极高
// entry.Method = method
// entry.Path = path
// entry.Host = host
// entry.Scheme = scheme
// entry.Proto = proto
// entry.ClientIp = clientIp
// entry.ServerId = serverId
// entry.App = app
// entry.Node = node
// entry.FromApp = fromApp
// entry.FromNode = fromNode
// entry.UserId = userId
// entry.DeviceId = deviceId
// entry.SessionId = sessionId
// entry.RequestId = requestId
// entry.ClientAppName = clientAppName
// entry.ClientAppVersion = clientAppVersion
// entry.AuthLevel = authLevel
// entry.Priority = priority
// entry.RequestHeaders = reqHeaders
// entry.RequestData = reqData
// entry.ResponseCode = responseCode
// entry.UsedTime = usedTime
// entry.ResponseHeaders = respHeaders
// entry.ResponseData = responseData
// entry.ResponseDataLength = responseDataLength
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// type TaskLog struct {
// BaseLog
// Task string `log:"pos:6"`
// UsedTime float32 `log:"pos:7"`
// Success bool `log:"pos:8"`
// Message string `log:"pos:9"`
// }
// func (l *TaskLog) Reset() {
// l.BaseLog.Reset()
// l.Task = ""
// l.UsedTime = 0
// l.Success = false
// l.Message = ""
// }
// type MonitorLog struct {
// BaseLog
// Target string `log:"pos:6"`
// Status int `log:"pos:7"`
// Message string `log:"pos:8"`
// }
// func (l *MonitorLog) Reset() {
// l.BaseLog.Reset()
// l.Target = ""
// l.Status = 0
// l.Message = ""
// }
// type StatisticLog struct {
// BaseLog
// Category string `log:"pos:6"`
// Item string `log:"pos:7"`
// Value float64 `log:"pos:8"`
// }
// func (l *StatisticLog) Reset() {
// l.BaseLog.Reset()
// l.Category = ""
// l.Item = ""
// l.Value = 0
// }
// func (logger *Logger) Task(taskName string, usedTime float32, success bool, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[TaskLog]()
// logger.FillBase(&entry.BaseLog, LogTypeTask)
// entry.Task = taskName
// entry.UsedTime = usedTime
// entry.Success = success
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Monitor(target string, status int, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[MonitorLog]()
// logger.FillBase(&entry.BaseLog, LogTypeMonitor)
// entry.Target = target
// entry.Status = status
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Statistic(category, item string, value float64, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[StatisticLog]()
// logger.FillBase(&entry.BaseLog, LogTypeStatistic)
// entry.Category = category
// entry.Item = item
// entry.Value = value
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func init() {
// RegisterType(LogTypeRequest, &RequestLog{})
// RegisterType(LogTypeTask, &TaskLog{})
// RegisterType(LogTypeMonitor, &MonitorLog{})
// RegisterType(LogTypeStatistic, &StatisticLog{})
// }

View File

@ -4,7 +4,6 @@ import (
"bufio"
"fmt"
"os"
"sync"
"time"
)
@ -17,11 +16,6 @@ type FileWriter struct {
bufWriter *bufio.Writer
}
var (
files = make(map[string]*FileWriter)
filesLock sync.RWMutex
)
// Write 由外层的 writerRunner 单协程调用,绝对并发安全,无需加锁
func (f *FileWriter) Write(tm time.Time, data []byte) {
nowSplit := tm.Format(f.splitTag)

View File

@ -1,28 +0,0 @@
package log
import (
"apigo.cc/go/cast"
)
// Formatter 日志格式化接口
type Formatter interface {
Format(data any, sensitiveKeys []string) ([]byte, error)
}
// JSONFormatter 默认的 JSON 格式化器
type JSONFormatter struct{}
func (f *JSONFormatter) Format(data any, sensitiveKeys []string) ([]byte, error) {
if len(sensitiveKeys) > 0 {
return cast.ToJSONDesensitizeBytes(data, sensitiveKeys)
}
return cast.ToJSONBytes(data)
}
// TextFormatter 文本格式化器 (示例)
type TextFormatter struct{}
func (f *TextFormatter) Format(data any, sensitiveKeys []string) ([]byte, error) {
// 简单的文本格式化实现
return []byte(cast.String(data)), nil
}

136
functional_test.go Normal file
View File

@ -0,0 +1,136 @@
package log_test
import (
"strings"
"testing"
"time"
"apigo.cc/go/file"
"apigo.cc/go/log"
)
func TestSplitTag(t *testing.T) {
logFile := "test_rotate.log"
// 使用每秒切分的标签,方便测试文件轮转
splitTag := ".20060102150405"
conf := log.Config{
Name: "test-split",
Level: "info",
File: logFile,
SplitTag: splitTag,
}
log.WriterService.Start(nil, nil)
logger := log.NewLogger(conf)
// 1. 记录第一条日志
t1 := time.Now()
logger.Info("first message")
time.Sleep(300 * time.Millisecond) // 等待异步写入
expectedFile1 := logFile + "." + t1.Format(splitTag)
if !file.Exists(expectedFile1) {
// 可能在写入时秒数刚好进位
expectedFile1 = logFile + "." + time.Now().Format(splitTag)
if !file.Exists(expectedFile1) {
t.Fatalf("First log file does not exist (checked both possible time slots)")
}
}
// 2. 等待跨秒,确保下次写入肯定会触发轮转
time.Sleep(1200 * time.Millisecond)
// 3. 记录第二条日志,触发轮转
t2 := time.Now()
logger.Info("second message")
time.Sleep(300 * time.Millisecond) // 等待异步写入
expectedFile2 := logFile + "." + t2.Format(splitTag)
if !file.Exists(expectedFile2) {
expectedFile2 = logFile + "." + time.Now().Format(splitTag)
if !file.Exists(expectedFile2) {
t.Fatalf("Second log file does not exist after rotation")
}
}
if expectedFile1 == expectedFile2 {
t.Errorf("Files should be different for rotation, but both are %s", expectedFile1)
}
// 清理
file.Remove(expectedFile1)
file.Remove(expectedFile2)
}
func TestSensitiveDetailed(t *testing.T) {
type SecretLog struct {
log.BaseLog
Password string
SecretKey string
SafeData string
}
entry := log.GetEntry[SecretLog]()
entry.BaseLog.LogType = "secret"
entry.Password = "my_password"
entry.SecretKey = "super_secret"
entry.SafeData = "hello"
// 直接测试 ToArrayBytes
// 注意passed to ToArrayBytes 的 keys 应该是已经过 fixField 处理的
sensitiveKeys := []string{"password", "secretkey"}
buf := log.Marshal(entry, sensitiveKeys)
result := string(buf)
if strings.Contains(result, "my_password") {
t.Errorf("Sensitive data 'my_password' not masked in: %s", result)
}
if strings.Contains(result, "super_secret") {
t.Errorf("Sensitive data 'super_secret' not masked in: %s", result)
}
if !strings.Contains(result, "hello") {
t.Errorf("Safe data 'hello' should be present in: %s", result)
}
}
func TestDeepDesensitization(t *testing.T) {
type Nested struct {
Password string
Token string
}
type DeepLog struct {
log.BaseLog
Data map[string]any
User Nested
}
entry := log.GetEntry[DeepLog]()
entry.BaseLog.LogType = "deep"
entry.Data = map[string]any{
"password": "data_password",
"safe": "data_safe",
}
entry.User = Nested{
Password: "user_password",
Token: "user_token",
}
sensitiveKeys := []string{"password", "token"}
buf := log.Marshal(entry, sensitiveKeys)
result := string(buf)
// Check deep desensitization in map
if strings.Contains(result, "data_password") {
t.Errorf("Nested map data 'data_password' not masked in: %s", result)
}
// Check deep desensitization in struct
if strings.Contains(result, "user_password") {
t.Errorf("Nested struct data 'user_password' not masked in: %s", result)
}
if strings.Contains(result, "user_token") {
t.Errorf("Nested struct data 'user_token' not masked in: %s", result)
}
if !strings.Contains(result, "data_safe") {
t.Errorf("Safe data 'data_safe' should be present in: %s", result)
}
}

23
go.mod
View File

@ -3,21 +3,18 @@ module apigo.cc/go/log
go 1.25.0
require (
apigo.cc/go/cast v1.2.6
apigo.cc/go/config v1.0.5
apigo.cc/go/shell v1.0.4
apigo.cc/go/cast v1.3.3
apigo.cc/go/config v1.3.1
apigo.cc/go/file v1.3.2
apigo.cc/go/id v1.3.1
apigo.cc/go/shell v1.3.1
)
require (
apigo.cc/go/convert v1.0.4 // indirect
apigo.cc/go/encoding v1.0.4 // indirect
apigo.cc/go/file v1.0.4 // indirect
apigo.cc/go/rand v1.0.4 // indirect
apigo.cc/go/safe v1.0.4 // indirect
github.com/kr/pretty v0.3.0 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
golang.org/x/crypto v0.50.0 // indirect
golang.org/x/sys v0.43.0 // indirect
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
apigo.cc/go/encoding v1.3.1 // indirect
apigo.cc/go/rand v1.3.1 // indirect
apigo.cc/go/safe v1.3.1 // indirect
golang.org/x/crypto v0.51.0 // indirect
golang.org/x/sys v0.44.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

48
go.sum
View File

@ -1,39 +1,31 @@
apigo.cc/go/cast v1.2.6 h1:xnWiaQAGsRCrnu1p8fIFQfg5HFSc7CxR+3ItiDIDMaY=
apigo.cc/go/cast v1.2.6/go.mod h1:lGlwImiOvHxG7buyMWhFzcdvQzmSaoKbmr7bcDfUpHk=
apigo.cc/go/config v1.0.4 h1:WG9zrQkqfFPkrKIL7RNvvAbbkuUBt1Av11ZP/aIfldM=
apigo.cc/go/config v1.0.4/go.mod h1:obryzJiK6j7lQex/58d5eWYOGx5O5IABguqNWxyyXJo=
apigo.cc/go/convert v1.0.4 h1:5+qPjC3dlPB59GnWZRlmthxcaXQtKvN+iOuiLdJ1GvQ=
apigo.cc/go/convert v1.0.4/go.mod h1:Hp+geeSyhqg/zwIKPOrDoceIREzcwM14t1I5q/dtbfU=
apigo.cc/go/encoding v1.0.4 h1:aezB0J/qFuHs6iXkbtuJP5JIHUtmjsr5SFb0NNvbObY=
apigo.cc/go/encoding v1.0.4/go.mod h1:V5CgT7rBbCxy+uCU20q0ptcNNRSgMtpA8cNOs6r8IeI=
apigo.cc/go/file v1.0.4 h1:qCKegV7OYh7r0qc3jZjGA/aKh0vIHgmr1OEbhfEmGX8=
apigo.cc/go/file v1.0.4/go.mod h1:C9gNo7386iA21OiBmuWh6CznKWlVBDFkhE4f0H0Susg=
apigo.cc/go/rand v1.0.4 h1:we070eWSL0dB8NEMaWjXj43+EekXQTm/h0kKpZ/frqw=
apigo.cc/go/rand v1.0.4/go.mod h1:mZ/4Soa3bk+XvDaqPWJuUe1bfEi4eThBj1XmEAuYxsk=
apigo.cc/go/safe v1.0.4 h1:07pRSdEHprF/2v6SsqAjICYFoeLcqjjvHGEdh6Dzrzg=
apigo.cc/go/safe v1.0.4/go.mod h1:o568sHS5rTRSVPmhxWod0tGdc+8l1KjidsNY1/OVZr0=
apigo.cc/go/shell v1.0.4 h1:EL9zjI39YBe1h+kRYQeAi/8zVGHe5W198DYYN7cENiY=
apigo.cc/go/shell v1.0.4/go.mod h1:N2gDkgK4tJ9TadD60/+gAGuWxyVAWHs5YPBmytw6ELA=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
apigo.cc/go/cast v1.3.3 h1:aln5eDR5DZVWVzZ/y5SJh1gQNgWv2sT82I25NaO9g34=
apigo.cc/go/cast v1.3.3/go.mod h1:lGlwImiOvHxG7buyMWhFzcdvQzmSaoKbmr7bcDfUpHk=
apigo.cc/go/config v1.3.1 h1:wZzUh4oL+fGD6SayVgX6prLPMsniM25etWFcEH8XzIE=
apigo.cc/go/config v1.3.1/go.mod h1:7KHz/1WmtBLM762Lln/TaXh2dmlMvJTLhnlk33zbS3U=
apigo.cc/go/encoding v1.3.1 h1:y8O58KYAyulkThg1O2ji2BqjnFoSvk42sit9I3z+K7Y=
apigo.cc/go/encoding v1.3.1/go.mod h1:xAJk5b83VZ31mXMTnyp0dfMoBKfT/AHDn0u+cQfojgY=
apigo.cc/go/file v1.3.2 h1:pu4oiDyiqgj3/eykfnJf+/6+A9v/Z0b3ClP5XK+lwG4=
apigo.cc/go/file v1.3.2/go.mod h1:vci4h0Pz94mV6dkniQkuyBYERVYeq7/LX4jJVuCg9hs=
apigo.cc/go/id v1.3.1 h1:pkqi6VeWyQoHuIu0Zbx/RRxIAdM61Js0j6cY1M9XVCk=
apigo.cc/go/id v1.3.1/go.mod h1:P2/vl3tyW3US+ayOFSMoPIOCulNLBngNYPhXJC/Z7J4=
apigo.cc/go/rand v1.3.1 h1:7FvsI6PtQ5XrWER0dTiLVo0p7GIxRidT/TBKhVy93j8=
apigo.cc/go/rand v1.3.1/go.mod h1:mZ/4Soa3bk+XvDaqPWJuUe1bfEi4eThBj1XmEAuYxsk=
apigo.cc/go/safe v1.3.1 h1:irTCqPAC97gGsX/Lw5AzLelDt1xXLEZIAaVhLELWe9Q=
apigo.cc/go/safe v1.3.1/go.mod h1:XdOpBhN2vkImalaykYXXmEpczqWa1y3ah6/Q72cdRqE=
apigo.cc/go/shell v1.3.1 h1:M8oD0b2HcJuCC6frQFx11b3UTcTx3lATX8XK+YXSVm8=
apigo.cc/go/shell v1.3.1/go.mod h1:ZMdJjpCpWdvsHKUXlelh/AxsV/nWdkH/k3lISfzMdUw=
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
golang.org/x/crypto v0.50.0 h1:zO47/JPrL6vsNkINmLoo/PH1gcxpls50DNogFvB5ZGI=
golang.org/x/crypto v0.50.0/go.mod h1:3muZ7vA7PBCE6xgPX7nkzzjiUq87kRItoJQM1Yo8S+Q=
golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI=
golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8=
golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ=
golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

View File

@ -1,6 +1,7 @@
package log_test
import (
"strconv"
"testing"
"apigo.cc/go/log"
@ -47,6 +48,7 @@ func TestDesensitization(t *testing.T) {
}
entry := log.GetEntry[MyLog]()
logger.FillBase(&entry.BaseLog, "test")
entry.Phone = "13812345678"
logger.Log(entry) // 应该在输出中脱敏
}
@ -57,7 +59,7 @@ func TestDBLog(t *testing.T) {
})
entry := log.GetEntry[DBEntry]()
entry.LogType = "db"
logger.FillBase(&entry.BaseLog, "db")
entry.DbType = "mysql"
entry.Query = "SELECT * FROM users"
entry.UsedTime = 10.5
@ -70,7 +72,7 @@ func TestRequestLog(t *testing.T) {
})
entry := log.GetEntry[RequestEntry]()
entry.LogType = "request"
logger.FillBase(&entry.BaseLog, "request")
entry.Method = "GET"
entry.Path = "/api/user"
entry.ResponseCode = 200
@ -85,3 +87,28 @@ func TestExtraLogs(t *testing.T) {
logger.Info("Extra log test", "key", "value")
}
func TestAs(t *testing.T) {
// 1. 测试 log.As (使用 DefaultLogger)
val1 := log.As(strconv.Atoi("123"))
if val1 != 123 {
t.Errorf("log.As expected 123, got %v", val1)
}
val2 := log.As(strconv.Atoi("abc"))
if val2 != 0 {
t.Errorf("log.As expected 0, got %v", val2)
}
// 2. 测试 logger.As (方法)
logger := log.NewLogger(log.Config{Level: "debug"})
val3 := logger.As(strconv.Atoi("456")).(int)
if val3 != 456 {
t.Errorf("logger.As expected 456, got %v", val3)
}
val4 := logger.As(strconv.Atoi("def")).(int)
if val4 != 0 {
t.Errorf("logger.As expected 0, got %v", val4)
}
}

171
logger.go
View File

@ -4,11 +4,11 @@ import (
"fmt"
"log"
"os"
"regexp"
"strings"
"time"
"apigo.cc/go/cast"
"apigo.cc/go/id"
)
type Logger struct {
@ -17,22 +17,12 @@ type Logger struct {
goLogger *log.Logger
file *FileWriter
writer Writer
formatter Formatter
truncations []string
sensitive map[string]bool
sensitiveKeys []string
regexSensitive []*regexp.Regexp
sensitiveRule []sensitiveRuleInfo
desensitization func(string) string
traceId string
}
type sensitiveRuleInfo struct {
threshold int
leftNum int
rightNum int
}
var (
writerMakers = make(map[string]func(*Config) Writer)
)
@ -51,20 +41,14 @@ func NewLogger(conf Config) *Logger {
if conf.Sensitive == "" {
conf.Sensitive = LogDefaultSensitive
}
if conf.SensitiveRule == "" {
conf.SensitiveRule = "12:4*4, 11:3*4, 7:2*2, 3:1*1, 2:1*0"
}
if conf.Name == "" {
conf.Name = GetDefaultName()
conf.Name = getDefaultName()
}
logger := Logger{
truncations: cast.Split(conf.Truncations, ","),
formatter: conf.Formatter,
}
if logger.formatter == nil {
logger.formatter = &JSONFormatter{}
traceId: id.Get10Bytes14MPerSecond(),
}
if len(conf.Sensitive) > 0 {
@ -77,37 +61,6 @@ func NewLogger(conf Config) *Logger {
}
}
if len(conf.RegexSensitive) > 0 {
ss := cast.Split(conf.RegexSensitive, ",")
for _, v := range ss {
if r, err := regexp.Compile(v); err == nil {
logger.regexSensitive = append(logger.regexSensitive, r)
}
}
}
if len(conf.SensitiveRule) > 0 {
ss := cast.Split(conf.SensitiveRule, ",")
for _, v := range ss {
a1 := strings.SplitN(v, ":", 2)
if len(a1) == 2 {
a2 := strings.SplitN(a1[1], "*", 3)
if len(a2) == 2 {
threshold := cast.Int(a1[0])
leftNum := cast.Int(a2[0])
rightNum := cast.Int(a2[1])
if threshold >= 0 && threshold <= 100 && leftNum >= 0 && leftNum <= 100 && rightNum >= 0 && rightNum <= 100 {
logger.sensitiveRule = append(logger.sensitiveRule, sensitiveRuleInfo{
threshold: threshold,
leftNum: leftNum,
rightNum: rightNum,
})
}
}
}
}
}
switch strings.ToLower(conf.Level) {
case "debug":
logger.level = DEBUG
@ -125,29 +78,27 @@ func NewLogger(conf Config) *Logger {
if m, ok := writerMakers[writerName]; ok {
if w := m(&conf); w != nil {
logger.writer = w
writerLock.Lock()
cur := writers.Load().([]Writer)
WriterService.WriterLock.Lock()
cur := WriterService.Writers.Load().([]Writer)
newW := append(cur, w)
writers.Store(newW)
writerLock.Unlock()
Start()
WriterService.Writers.Store(newW)
WriterService.WriterLock.Unlock()
}
}
} else {
if conf.SplitTag != "" {
filesLock.RLock()
logger.file = files[conf.File+conf.SplitTag]
filesLock.RUnlock()
WriterService.FilesLock.RLock()
logger.file = WriterService.Files[conf.File+conf.SplitTag]
WriterService.FilesLock.RUnlock()
if logger.file == nil {
logger.file = &FileWriter{
fileName: conf.File,
splitTag: conf.SplitTag,
}
filesLock.Lock()
files[conf.File+conf.SplitTag] = logger.file
filesLock.Unlock()
WriterService.FilesLock.Lock()
WriterService.Files[conf.File+conf.SplitTag] = logger.file
WriterService.FilesLock.Unlock()
}
Start()
} else {
fp, err := os.OpenFile(conf.File, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
if err == nil {
@ -161,26 +112,19 @@ func NewLogger(conf Config) *Logger {
}
func (logger *Logger) Log(entry LogEntry) {
// 如果是自动生成的 LogType如 Debug/Info/Warning/Error 对应的类型),这里通常已经在方法内 fillBase 了
// 但对于像 discover.Log 这样直接调用 Log(entry) 的,需要在这里补全
if entry.GetBaseLog().LogTime == 0 {
logger.fillBase(entry, "")
}
logger.asyncWrite(entry)
}
func (logger *Logger) asyncWrite(entry LogEntry) {
buf, err := logger.formatter.Format(entry, logger.sensitiveKeys)
if err == nil {
logger.writeBuf(buf)
}
PutEntry(entry)
buf := Marshal(entry, logger.sensitiveKeys)
logger.writeBuf(entry, buf)
putEntry(entry)
}
func (logger *Logger) writeBuf(buf []byte) {
if writerRunning.Load() {
WriteAsync(logPayload{
func (logger *Logger) writeBuf(entry LogEntry, buf []byte) {
if WriterService.Running.Load() {
writeAsync(logPayload{
entry: entry,
buf: buf,
writer: logger.writer,
file: logger.file,
@ -189,7 +133,7 @@ func (logger *Logger) writeBuf(buf []byte) {
}
if logger.writer != nil {
logger.writer.Log(buf)
logger.writer.Log(entry, buf)
} else if logger.file != nil {
fmt.Println(Viewable(string(buf)))
} else if logger.goLogger == nil {
@ -199,8 +143,7 @@ func (logger *Logger) writeBuf(buf []byte) {
}
}
func (logger *Logger) fillBase(entry LogEntry, logType string) {
base := entry.GetBaseLog()
func (logger *Logger) FillBase(base *BaseLog, logType string) {
if base == nil {
return
}
@ -211,17 +154,48 @@ func (logger *Logger) fillBase(entry LogEntry, logType string) {
}
base.LogTime = time.Now().UnixNano()
base.TraceId = logger.traceId
base.ImageName = dockerImageName
base.ImageTag = dockerImageTag
base.ServerName = serverName
base.ServerIp = serverIp
if dockerImageTag != "" {
base.Image = dockerImageName + ":" + dockerImageTag
} else {
base.Image = dockerImageName
}
if serverIp != "" {
base.Server = serverName + ":" + serverIp
} else {
base.Server = serverName
}
}
func (logger *Logger) FillDebug(entry *DebugLog, message string) {
logger.FillBase(&entry.BaseLog, LogTypeDebug)
entry.Debug = message
}
func (logger *Logger) FillInfo(entry *InfoLog, message string) {
logger.FillBase(&entry.BaseLog, LogTypeInfo)
entry.Info = message
}
func (logger *Logger) FillWarning(entry *WarningLog, message string) {
logger.FillBase(&entry.BaseLog, LogTypeWarning)
entry.Warning = message
entry.CallStacks = getCallStacks(logger.truncations)
}
func (logger *Logger) FillError(entry *ErrorLog, message string) {
logger.FillBase(&entry.BaseLog, LogTypeError)
entry.Error = message
entry.CallStacks = getCallStacks(logger.truncations)
}
func (logger *Logger) GetCallStacks() []string {
return getCallStacks(logger.truncations)
}
func (logger *Logger) Debug(message string, extra ...any) {
if logger.CheckLevel(DEBUG) {
entry := GetEntry[DebugLog]()
logger.fillBase(entry, LogTypeDebug)
entry.Debug = message
logger.FillDebug(entry, message)
if len(extra) > 0 {
cast.FillMap(&entry.Extra, extra)
}
@ -232,8 +206,7 @@ func (logger *Logger) Debug(message string, extra ...any) {
func (logger *Logger) Info(message string, extra ...any) {
if logger.CheckLevel(INFO) {
entry := GetEntry[InfoLog]()
logger.fillBase(entry, LogTypeInfo)
entry.Info = message
logger.FillInfo(entry, message)
if len(extra) > 0 {
cast.FillMap(&entry.Extra, extra)
}
@ -244,9 +217,7 @@ func (logger *Logger) Info(message string, extra ...any) {
func (logger *Logger) Warning(message string, extra ...any) {
if logger.CheckLevel(WARNING) {
entry := GetEntry[WarningLog]()
logger.fillBase(entry, LogTypeWarning)
entry.Warning = message
entry.CallStacks = getCallStacks(logger.truncations)
logger.FillWarning(entry, message)
if len(extra) > 0 {
cast.FillMap(&entry.Extra, extra)
}
@ -257,9 +228,7 @@ func (logger *Logger) Warning(message string, extra ...any) {
func (logger *Logger) Error(message string, extra ...any) {
if logger.CheckLevel(ERROR) {
entry := GetEntry[ErrorLog]()
logger.fillBase(entry, LogTypeError)
entry.Error = message
entry.CallStacks = getCallStacks(logger.truncations)
logger.FillError(entry, message)
if len(extra) > 0 {
cast.FillMap(&entry.Extra, extra)
}
@ -267,12 +236,12 @@ func (logger *Logger) Error(message string, extra ...any) {
}
}
func (logger *Logger) SetLevel(level LevelType) {
logger.level = level
func (logger *Logger) SetName(name string) {
logger.config.Name = name
}
func (logger *Logger) SetDesensitization(f func(v string) string) {
logger.desensitization = f
func (logger *Logger) SetLevel(level LevelType) {
logger.level = level
}
func (logger *Logger) New(traceId string) *Logger {
@ -285,6 +254,14 @@ func (logger *Logger) GetTraceId() string {
return logger.traceId
}
// As 仿照 cast.As忽略错误并返回零值但会将错误记录到日志中 (消除摩擦)
func (logger *Logger) As(v any, err error) any {
if err != nil {
logger.Error(err.Error())
}
return v
}
func (logger *Logger) CheckLevel(logLevel LevelType) bool {
settedLevel := logger.level
if settedLevel == 0 {

84
logv/main.go Normal file
View File

@ -0,0 +1,84 @@
package main
import (
"bufio"
"flag"
"fmt"
"io"
"os"
"apigo.cc/go/log"
)
func main() {
jsonMode := flag.Bool("json", false, "output in JSON format")
helpMode := flag.Bool("h", false, "show help")
flag.BoolVar(helpMode, "help", false, "show help")
flag.Usage = func() {
fmt.Fprintf(os.Stderr, "Usage: logv [options] [file1 file2 ...]\n")
fmt.Fprintf(os.Stderr, "Options:\n")
flag.PrintDefaults()
fmt.Fprintf(os.Stderr, "\nExamples:\n")
fmt.Fprintf(os.Stderr, " 1. 彩色可视化实时日志:\n")
fmt.Fprintf(os.Stderr, " tail -f app.log | logv\n")
fmt.Fprintf(os.Stderr, " 2. 格式化查看历史日志文件:\n")
fmt.Fprintf(os.Stderr, " logv error.log\n")
fmt.Fprintf(os.Stderr, " 3. 还原为标准 JSON 格式 (供 Filebeat / Logstash 收集):\n")
fmt.Fprintf(os.Stderr, " tail -f app.log | logv -json\n")
fmt.Fprintf(os.Stderr, " 4. 批量转换日志文件为 JSON:\n")
fmt.Fprintf(os.Stderr, " logv -json app.log.2026* > all_logs.json\n")
}
flag.Parse()
if *helpMode {
flag.Usage()
return
}
// Try to load meta file from current directory
_ = log.LoadMeta(".log.meta.json")
args := flag.Args()
if len(args) == 0 {
// Check if stdin is a terminal
stat, _ := os.Stdin.Stat()
if (stat.Mode() & os.ModeCharDevice) != 0 {
// Stdin is a terminal and no files provided, show usage
flag.Usage()
return
}
process(os.Stdin, *jsonMode)
} else {
for _, arg := range args {
f, err := os.Open(arg)
if err != nil {
fmt.Fprintf(os.Stderr, "logv: %v\n", err)
continue
}
process(f, *jsonMode)
f.Close()
}
}
}
func process(r io.Reader, jsonMode bool) {
scanner := bufio.NewScanner(r)
for scanner.Scan() {
line := scanner.Text()
if len(line) == 0 {
continue
}
if jsonMode {
fmt.Println(log.ToJSON(line))
} else {
fmt.Println(log.Viewable(line))
}
}
if err := scanner.Err(); err != nil {
fmt.Fprintf(os.Stderr, "logv: error reading input: %v\n", err)
}
}

237
meta.go Normal file
View File

@ -0,0 +1,237 @@
package log
import (
"reflect"
"sort"
"strings"
"sync"
"apigo.cc/go/cast"
"apigo.cc/go/file"
)
// MetaField describes the serialization and visualization metadata for a single log field.
type MetaField struct {
Index int
Name string
KeyName string
AttachBefore bool
Color string
Format string
Precision int
WithoutKey bool
Hide bool
}
var (
metaRegistry = make(map[string][]MetaField)
metaLock sync.RWMutex
metaFilePath = ".log.meta.json"
)
// LoadMeta loads metadata from the specified file into the global registry.
func LoadMeta(path string) error {
metaLock.Lock()
defer metaLock.Unlock()
return file.UnmarshalFile(path, &metaRegistry)
}
// RegisterType registers a log model's metadata into the global registry.
// logType is the string identifier (e.g. "info", "error").
func RegisterType(logType string, model any) {
t := reflect.TypeOf(model)
if t.Kind() == reflect.Ptr {
t = t.Elem()
}
// 强制检查 Reset 方法是否被显式实现(防止继承 BaseLog 后忘记重置业务字段)
if t.Kind() == reflect.Struct {
ptrType := reflect.PointerTo(t)
method, ok := ptrType.MethodByName("Reset")
if !ok {
panic("log model must implement Reset() method: " + t.Name())
}
// 检查该方法是否属于当前类型(而不是继承自 BaseLog 且没有被重写)
baseResetMethod, _ := reflect.PointerTo(reflect.TypeOf(BaseLog{})).MethodByName("Reset")
if method.Func.Pointer() == baseResetMethod.Func.Pointer() {
panic("log model must override Reset() method to clear its own fields: " + t.Name())
}
}
fields := extractMetaFields(model)
metaLock.Lock()
metaRegistry[logType] = fields
metaLock.Unlock()
syncMetaFile()
}
// GetMeta returns the metadata fields for a given logType.
func GetMeta(logType string) []MetaField {
metaLock.RLock()
defer metaLock.RUnlock()
return metaRegistry[logType]
}
// fieldInfo is used internally for storing fields with their absolute position.
type fieldInfo struct {
field reflect.StructField
pos int
}
func extractMetaFields(model any) []MetaField {
t := reflect.TypeOf(model)
if t.Kind() == reflect.Ptr {
t = t.Elem()
}
if t.Kind() != reflect.Struct {
return nil
}
var flatFields []fieldInfo
flattenStructFields(t, &flatFields, nil)
// Determine final indices
maxLiteralPos := -1
var highPosFields []fieldInfo
for _, f := range flatFields {
if f.pos < 1000 {
if f.pos > maxLiteralPos {
maxLiteralPos = f.pos
}
} else {
highPosFields = append(highPosFields, f)
}
}
// Sort high pos fields by their pos
sort.Slice(highPosFields, func(i, j int) bool {
return highPosFields[i].pos < highPosFields[j].pos
})
// Assign real indices to high pos fields
finalPosMap := make(map[string]int)
for _, f := range flatFields {
if f.pos < 1000 {
finalPosMap[f.field.Name] = f.pos
}
}
nextPos := maxLiteralPos + 1
for _, f := range highPosFields {
finalPosMap[f.field.Name] = nextPos
nextPos++
}
maxPos := nextPos - 1
metaFields := make([]MetaField, maxPos+1)
// Initialize with empty MetaFields having Index set
for i := range metaFields {
metaFields[i] = MetaField{Index: i}
}
for _, f := range flatFields {
tag := f.field.Tag.Get("log")
if tag == "-" {
continue
}
realPos := finalPosMap[f.field.Name]
meta := MetaField{
Index: realPos,
Name: f.field.Name,
}
if tag != "" {
parts := strings.Split(tag, ",")
for _, part := range parts {
part = strings.TrimSpace(part)
if part == "attachBefore" {
meta.AttachBefore = true
continue
}
kv := strings.SplitN(part, ":", 2)
if len(kv) == 2 {
key := strings.TrimSpace(kv[0])
val := strings.TrimSpace(kv[1])
switch key {
case "color":
meta.Color = val
case "format":
meta.Format = val
case "withoutkey":
meta.WithoutKey = (val == "true")
case "hide":
meta.Hide = (val == "true")
case "keyname":
meta.KeyName = val
case "attachBefore":
meta.AttachBefore = (val == "true")
case "precision":
meta.Precision = cast.To[int](val)
}
}
}
}
// Apply some default visual rules if not specified
// LogType shouldn't show the key in standard console
if f.field.Name == "LogType" && meta.Color == "" {
meta.WithoutKey = true
}
metaFields[realPos] = meta
}
return metaFields
}
func flattenStructFields(t reflect.Type, result *[]fieldInfo, parentIndex []int) {
for i := 0; i < t.NumField(); i++ {
f := t.Field(i)
if !f.IsExported() && !f.Anonymous {
continue
}
pos := 10 + i // default position if not specified
tag := f.Tag.Get("log")
if tag != "" {
parts := strings.Split(tag, ",")
for _, part := range parts {
kv := strings.SplitN(part, ":", 2)
if len(kv) == 2 && strings.TrimSpace(kv[0]) == "pos" {
if p := cast.To[int](strings.TrimSpace(kv[1])); p >= 0 {
pos = p
}
}
}
}
// Compute the full index path from the root
fullIndex := make([]int, len(parentIndex), len(parentIndex)+1)
copy(fullIndex, parentIndex)
fullIndex = append(fullIndex, i)
f.Index = fullIndex
if f.Anonymous && f.Type.Kind() == reflect.Struct {
flattenStructFields(f.Type, result, f.Index)
} else {
*result = append(*result, fieldInfo{
field: f,
pos: pos,
})
}
}
}
func syncMetaFile() {
metaLock.RLock()
defer metaLock.RUnlock()
_ = file.MarshalFilePretty(metaFilePath, metaRegistry)
}
// SetMetaFilePath allows changing the path for testing or configuration purposes
func SetMetaFilePath(path string) {
metaFilePath = path
}

105
meta_test.go Normal file
View File

@ -0,0 +1,105 @@
package log
import (
"encoding/json"
"os"
"testing"
)
type MockBaseLog struct {
BaseField1 string `log:"pos:0,color:red"`
BaseField2 int `log:"pos:1,withoutkey:true"`
}
func (b *MockBaseLog) Reset() {
b.BaseField1 = ""
b.BaseField2 = 0
}
func (b *MockBaseLog) IsLogEntry() bool { return true }
func (b *MockBaseLog) GetBaseLog() *BaseLog { return &BaseLog{} }
type MockInfoLog struct {
MockBaseLog
Message string `log:"pos:2"`
Extra map[string]any `log:"pos:1000"`
}
func (l *MockInfoLog) Reset() {
l.MockBaseLog.Reset()
l.Message = ""
clear(l.Extra)
}
type MockErrorLog struct {
MockBaseLog
Error string `log:"pos:2,color:red"`
CallStacks []string `log:"pos:1001"`
Extra map[string]any `log:"pos:1000"`
}
func (l *MockErrorLog) Reset() {
l.MockBaseLog.Reset()
l.Error = ""
l.CallStacks = l.CallStacks[:0]
clear(l.Extra)
}
func TestMetaExtraction(t *testing.T) {
// Setup custom meta file path for testing
SetMetaFilePath(".test.meta.json")
defer os.Remove(".test.meta.json")
RegisterType("mock_info", MockInfoLog{})
RegisterType("mock_error", MockErrorLog{})
infoMeta := GetMeta("mock_info")
// Index 0, 1, 2 are used, Extra gets max(2)+1=3. Total size 4.
if len(infoMeta) != 4 {
t.Fatalf("expected 4 fields for mock_info, got %d", len(infoMeta))
}
if infoMeta[0].Name != "BaseField1" || infoMeta[0].Color != "red" {
t.Errorf("unexpected meta for BaseField1 at index 0: %+v", infoMeta[0])
}
if infoMeta[1].Name != "BaseField2" || infoMeta[1].WithoutKey != true {
t.Errorf("unexpected meta for BaseField2 at index 1: %+v", infoMeta[1])
}
if infoMeta[2].Name != "Message" {
t.Errorf("unexpected meta for Message at index 2: %+v", infoMeta[2])
}
if infoMeta[3].Name != "Extra" {
t.Errorf("unexpected meta for Extra at index 3: %+v", infoMeta[3])
}
errorMeta := GetMeta("mock_error")
// Indices: 0, 1, 2, Extra(3), CallStacks(4). Total size 5.
if len(errorMeta) != 5 {
t.Fatalf("expected 5 fields for mock_error, got %d", len(errorMeta))
}
if errorMeta[2].Name != "Error" || errorMeta[2].Color != "red" {
t.Errorf("unexpected meta for Error at index 2: %+v", errorMeta[2])
}
if errorMeta[3].Name != "Extra" {
t.Errorf("unexpected meta for Extra at index 3: %+v", errorMeta[3])
}
if errorMeta[4].Name != "CallStacks" {
t.Errorf("unexpected meta for CallStacks at index 4: %+v", errorMeta[4])
}
// Verify file was created and contains correct data
data, err := os.ReadFile(".test.meta.json")
if err != nil {
t.Fatalf("failed to read test meta file: %v", err)
}
var registry map[string][]MetaField
if err := json.Unmarshal(data, &registry); err != nil {
t.Fatalf("failed to unmarshal test meta file: %v", err)
}
if len(registry) < 2 {
t.Errorf("expected at least 2 types in registry, got %d", len(registry))
}
}

21
name_test.go Normal file
View File

@ -0,0 +1,21 @@
package log
import (
"testing"
)
func TestSetDefaultName(t *testing.T) {
oldName := getDefaultName()
defer SetDefaultName(oldName)
newName := "test-service-name"
SetDefaultName(newName)
if getDefaultName() != newName {
t.Errorf("GetDefaultName() = %v, want %v", getDefaultName(), newName)
}
if DefaultLogger.config.Name != newName {
t.Errorf("DefaultLogger.config.Name = %v, want %v", DefaultLogger.config.Name, newName)
}
}

86
pool.go
View File

@ -27,88 +27,14 @@ func GetEntry[T any]() *T {
})
}
entry := p.(*sync.Pool).Get().(*T)
ResetLogEntry(entry) // 自动重置所有字段,无需子类实现 Reset
if le, ok := any(entry).(LogEntry); ok {
le.Reset()
}
return entry
}
// ResetLogEntry 使用反射自动化重置日志对象的所有字段
// 特别是对 Map 和 Slice 进行初始化长度0容量8
func ResetLogEntry(v any) {
rv := reflect.ValueOf(v)
if rv.Kind() != reflect.Ptr || rv.IsNil() {
return
}
t := rv.Type()
resetFunc, ok := resetCache.Load(t)
if !ok {
resetFunc = buildResetFunc(t.Elem())
resetCache.Store(t, resetFunc)
}
resetFunc.(func(reflect.Value))(rv.Elem())
}
func buildResetFunc(t reflect.Type) func(reflect.Value) {
var funcs []func(reflect.Value)
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
fieldIdx := i
switch field.Type.Kind() {
case reflect.String:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetString("") })
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetInt(0) })
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetUint(0) })
case reflect.Float32, reflect.Float64:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetFloat(0) })
case reflect.Bool:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetBool(false) })
case reflect.Map:
funcs = append(funcs, func(rv reflect.Value) {
f := rv.Field(fieldIdx)
if f.IsNil() {
f.Set(reflect.MakeMapWithSize(f.Type(), 8))
} else {
f.Clear()
}
})
case reflect.Slice:
funcs = append(funcs, func(rv reflect.Value) {
f := rv.Field(fieldIdx)
if f.Cap() < 8 {
f.Set(reflect.MakeSlice(f.Type(), 0, 8))
} else {
f.SetLen(0)
}
})
case reflect.Struct:
subReset := buildResetFunc(field.Type)
funcs = append(funcs, func(rv reflect.Value) {
subReset(rv.Field(fieldIdx))
})
case reflect.Ptr, reflect.Interface:
zero := reflect.Zero(field.Type)
funcs = append(funcs, func(rv reflect.Value) {
rv.Field(fieldIdx).Set(zero)
})
}
}
return func(rv reflect.Value) {
for _, f := range funcs {
f(rv)
}
}
}
func resetStruct(rv reflect.Value) {
// 已经不再直接调用,保留 buildResetFunc 逻辑即可
}
// PutEntry 将日志对象归还到池中
func PutEntry(entry any) {
// putEntry 将日志对象归还到池中
func putEntry(entry any) {
t := reflect.TypeOf(entry)
if pool, ok := globalPools.pools.Load(t); ok {
pool.(*sync.Pool).Put(entry)
@ -118,7 +44,7 @@ func PutEntry(entry any) {
// WithEntry 执行闭包并在结束后自动回收对象
func WithEntry[T any](fn func(*T)) {
entry := GetEntry[T]()
defer PutEntry(entry)
defer putEntry(entry)
fn(entry)
}

View File

@ -11,6 +11,12 @@ type MockRequestLog struct {
UsedTime float32
}
func (l *MockRequestLog) Reset() {
l.BaseLog.Reset()
l.RequestId = ""
l.UsedTime = 0
}
func TestWithEntry(t *testing.T) {
WithEntry(func(entry *MockRequestLog) {
entry.RequestId = "with-entry-id"

View File

@ -28,8 +28,7 @@ func TestLoggerReliability(t *testing.T) {
}
wg.Wait()
Stop()
Wait()
WriterService.Stop(nil)
file, err := os.Open(logFile)
if err != nil {

197
serializer.go Normal file
View File

@ -0,0 +1,197 @@
package log
import (
"bytes"
"reflect"
"sort"
"strconv"
"apigo.cc/go/cast"
)
type fieldAccessor struct {
indexPath []int
name string
}
var (
accessorsCache = make(map[string][]fieldAccessor)
)
// getAccessors caches the reflection index paths for the flattened fields.
func getAccessors(logType string, model any) []fieldAccessor {
metaLock.RLock()
if acc, ok := accessorsCache[logType]; ok {
metaLock.RUnlock()
return acc
}
metaLock.RUnlock()
metaLock.Lock()
defer metaLock.Unlock()
// Double check
if acc, ok := accessorsCache[logType]; ok {
return acc
}
t := reflect.TypeOf(model)
if t.Kind() == reflect.Ptr {
t = t.Elem()
}
var flatFields []fieldInfo
flattenStructFields(t, &flatFields, nil)
// Determine final indices (must match meta.go)
maxLiteralPos := -1
var highPosFields []fieldInfo
for _, f := range flatFields {
if f.pos < 1000 {
if f.pos > maxLiteralPos {
maxLiteralPos = f.pos
}
} else {
highPosFields = append(highPosFields, f)
}
}
// Sort high pos fields by their pos
sort.Slice(highPosFields, func(i, j int) bool {
return highPosFields[i].pos < highPosFields[j].pos
})
finalPosMap := make(map[string]int)
for _, f := range flatFields {
if f.pos < 1000 {
finalPosMap[f.field.Name] = f.pos
}
}
nextPos := maxLiteralPos + 1
for _, f := range highPosFields {
finalPosMap[f.field.Name] = nextPos
nextPos++
}
maxPos := nextPos - 1
accessors := make([]fieldAccessor, maxPos+1)
for _, f := range flatFields {
if f.field.Tag.Get("log") == "-" {
continue
}
realPos := finalPosMap[f.field.Name]
accessors[realPos] = fieldAccessor{
indexPath: f.field.Index,
name: f.field.Name,
}
}
accessorsCache[logType] = accessors
return accessors
}
// donot export this function
func Marshal(entry LogEntry, sensitiveKeys []string) []byte {
var buf bytes.Buffer
buf.WriteByte('[')
base := entry.GetBaseLog()
if base == nil {
buf.WriteByte(']')
return buf.Bytes()
}
logType := base.LogType
if logType == "" {
// Fallback for undefined types
logType = "undefined"
}
accessors := getAccessors(logType, entry)
v := reflect.ValueOf(entry)
if v.Kind() == reflect.Ptr {
v = v.Elem()
}
for i, acc := range accessors {
if i > 0 {
buf.WriteByte(',')
}
if acc.indexPath == nil {
buf.WriteByte('0')
continue
}
fv := v.FieldByIndex(acc.indexPath)
writeValue(&buf, fv, acc.name, sensitiveKeys)
}
buf.WriteByte(']')
return buf.Bytes()
}
func writeValue(buf *bytes.Buffer, v reflect.Value, fieldName string, sensitiveKeys []string) {
if !v.IsValid() {
buf.WriteString("null")
return
}
// Check if this root field should be desensitized
if len(sensitiveKeys) > 0 {
fixedName := fixField(fieldName)
for _, sk := range sensitiveKeys {
if sk == fixedName {
buf.WriteString(`"***"`)
return
}
}
}
switch v.Kind() {
case reflect.String:
writeString(buf, v.String())
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
buf.WriteString(strconv.FormatInt(v.Int(), 10))
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
buf.WriteString(strconv.FormatUint(v.Uint(), 10))
case reflect.Float32, reflect.Float64:
buf.WriteString(strconv.FormatFloat(v.Float(), 'g', -1, 64))
case reflect.Bool:
if v.Bool() {
buf.WriteString("true")
} else {
buf.WriteString("false")
}
default:
// Use cast for complex types to ensure deep desensitization
b, _ := cast.ToJSONDesensitizeBytes(v.Interface(), sensitiveKeys)
if len(b) == 0 {
buf.WriteString("null")
} else {
buf.Write(b)
}
}
}
func writeString(buf *bytes.Buffer, s string) {
buf.WriteByte('"')
for i := 0; i < len(s); i++ {
c := s[i]
switch c {
case '\\':
buf.WriteString(`\\`)
case '"':
buf.WriteString(`\"`)
case '\n':
buf.WriteString(`\n`)
case '\r':
buf.WriteString(`\r`)
case '\t':
buf.WriteString(`\t`)
default:
buf.WriteByte(c)
}
}
buf.WriteByte('"')
}

129
serializer_test.go Normal file
View File

@ -0,0 +1,129 @@
package log
import (
"encoding/json"
"testing"
)
type SerializerMockBaseLog struct {
LogName string `log:"pos:0"`
LogType string `log:"pos:1"`
LogTime int64 `log:"pos:2"`
TraceId string `log:"pos:3"`
}
func (b *SerializerMockBaseLog) IsLogEntry() bool {
return true
}
func (b *SerializerMockBaseLog) GetBaseLog() *BaseLog {
return &BaseLog{LogType: b.LogType}
}
func (b *SerializerMockBaseLog) Reset() {
b.LogName = ""
b.LogType = ""
b.LogTime = 0
b.TraceId = ""
}
type SerializerMockInfoLog struct {
SerializerMockBaseLog
Message string `log:"pos:4"`
Extra map[string]any `log:"pos:1000"`
}
func (l *SerializerMockInfoLog) Reset() {
l.SerializerMockBaseLog.Reset()
l.Message = ""
if l.Extra == nil {
l.Extra = make(map[string]any, 8)
} else {
clear(l.Extra)
}
}
func TestToArrayBytes(t *testing.T) {
entry := &SerializerMockInfoLog{
SerializerMockBaseLog: SerializerMockBaseLog{
LogName: "test-app",
LogType: "mock_info_test",
LogTime: 1620000000,
TraceId: "abc-123",
},
Message: "Hello, World!",
Extra: map[string]any{
"user_id": 42,
},
}
RegisterType("mock_info_test", entry) // trigger meta generation
bytes := Marshal(entry, nil)
str := string(bytes)
t.Logf("Raw log: %s", str)
var arr []any
err := json.Unmarshal(bytes, &arr)
if err != nil {
t.Fatalf("failed to unmarshal generated array: %v, raw: %s", err, str)
}
// Indices: 0, 1, 2, 3, 4, 1000(mapped to 5). Total size 6.
if len(arr) != 6 {
t.Fatalf("expected 6 elements, got %d. raw: %s", len(arr), str)
}
if arr[0] != "test-app" {
t.Errorf("expected arr[0] == 'test-app', got %v", arr[0])
}
if arr[1] != "mock_info_test" {
t.Errorf("expected arr[1] == 'mock_info_test', got %v", arr[1])
}
// JSON numbers are parsed as float64
if arr[2] != float64(1620000000) {
t.Errorf("expected arr[2] == 1620000000, got %v", arr[2])
}
if arr[3] != "abc-123" {
t.Errorf("expected arr[3] == 'abc-123', got %v", arr[3])
}
if arr[4] != "Hello, World!" {
t.Errorf("expected arr[4] == 'Hello, World!', got %v", arr[4])
}
extraMap, ok := arr[5].(map[string]any)
if !ok {
t.Fatalf("expected arr[5] to be map[string]any, got %T (value: %v)", arr[5], arr[5])
}
if extraMap["user_id"] != float64(42) {
t.Errorf("expected extraMap['user_id'] == 42, got %v", extraMap["user_id"])
}
}
func TestToArrayBytes_Desensitize(t *testing.T) {
entry := &SerializerMockInfoLog{
SerializerMockBaseLog: SerializerMockBaseLog{
LogType: "mock_info_test2",
},
Message: "Sensitive Info",
Extra: map[string]any{
"password": "my-secret-password",
},
}
RegisterType("mock_info_test2", entry)
bytes := Marshal(entry, []string{"password"})
str := string(bytes)
var arr []any
err := json.Unmarshal(bytes, &arr)
if err != nil {
t.Fatalf("failed to unmarshal generated array: %v, raw: %s", err, str)
}
extraMap := arr[5].(map[string]any)
if extraMap["password"] != "***" {
t.Errorf("expected password to be desensitized, got %v", extraMap["password"])
}
}

View File

@ -14,28 +14,26 @@ const LogTypeMonitor = "monitor"
const LogTypeStatistic = "statistic"
const LogTypeRequest = "request"
const LogDefaultSensitive = "phone,password,secret,token,accessToken"
const LogDefaultSensitive = "phone,password,secret,token,accessToken,authorization"
const LogEnvLevel = "LOG_LEVEL"
const LogEnvFile = "LOG_FILE"
const LogEnvSensitive = "LOG_SENSITIVE"
const LogEnvRegexSensitive = "LOG_REGEXSENSITIVE"
// LogEntry 是一个标记接口,用于识别是否为对象池管理的日志对象
type LogEntry interface {
IsLogEntry() bool
GetBaseLog() *BaseLog
Reset()
}
type BaseLog struct {
LogName string
LogType string
LogTime int64
TraceId string
ImageName string
ImageTag string
ServerName string
ServerIp string
Extra map[string]any
LogName string `log:"pos:0,color:cyan,hide:true"`
LogType string `log:"pos:1,color:magenta,hide:true"`
LogTime int64 `log:"pos:2,format:time"`
TraceId string `log:"pos:3,color:gray,withoutkey:true"`
Image string `log:"pos:4,hide:true"`
Server string `log:"pos:5,hide:true"`
Extra map[string]any `log:"pos:1000"`
}
func (b *BaseLog) IsLogEntry() bool {
@ -46,24 +44,67 @@ func (b *BaseLog) GetBaseLog() *BaseLog {
return b
}
func (b *BaseLog) Reset() {
b.LogName = ""
b.LogType = ""
b.LogTime = 0
b.TraceId = ""
b.Image = ""
b.Server = ""
if b.Extra == nil {
b.Extra = make(map[string]any, 8)
} else {
clear(b.Extra)
}
}
type DebugLog struct {
BaseLog
Debug string
Debug string `log:"pos:6,withoutkey:true"` // white
}
func (l *DebugLog) Reset() {
l.BaseLog.Reset()
l.Debug = ""
}
type InfoLog struct {
BaseLog
Info string
Info string `log:"pos:6,color:cyan,withoutkey:true"`
}
func (l *InfoLog) Reset() {
l.BaseLog.Reset()
l.Info = ""
}
type WarningLog struct {
BaseLog
Warning string
CallStacks []string
Warning string `log:"pos:6,color:yellow,withoutkey:true"`
CallStacks []string `log:"pos:1001"`
}
func (l *WarningLog) Reset() {
l.BaseLog.Reset()
l.Warning = ""
l.CallStacks = l.CallStacks[:0]
}
type ErrorLog struct {
BaseLog
Error string
CallStacks []string
Error string `log:"pos:6,color:red,withoutkey:true"`
CallStacks []string `log:"pos:1001"`
}
func (l *ErrorLog) Reset() {
l.BaseLog.Reset()
l.Error = ""
l.CallStacks = l.CallStacks[:0]
}
func init() {
RegisterType(LogTypeDebug, &DebugLog{})
RegisterType(LogTypeInfo, &InfoLog{})
RegisterType(LogTypeWarning, &WarningLog{})
RegisterType(LogTypeError, &ErrorLog{})
}

View File

@ -1,16 +1,13 @@
package log
import (
"encoding/json"
"fmt"
"net"
"os"
"path"
"runtime"
"runtime/debug"
"strings"
"time"
"apigo.cc/go/cast"
)
var (
@ -24,104 +21,35 @@ func init() {
dockerImageName = os.Getenv("DOCKER_IMAGE_NAME")
dockerImageTag = os.Getenv("DOCKER_IMAGE_TAG")
serverName, _ = os.Hostname()
// 获取真实局域网 IP (UDP 8.8.8.8 伪拨号法)
conn, err := net.Dial("udp", "8.8.8.8:80")
if err == nil {
localAddr := conn.LocalAddr().(*net.UDPAddr)
serverIp = localAddr.IP.String()
_ = conn.Close()
}
if serverIp == "" {
addrs, err := net.InterfaceAddrs()
if err == nil {
for _, a := range addrs {
if an, ok := a.(*net.IPNet); ok {
// 忽略 Docker 私有网段
if an.IP.IsGlobalUnicast() && !strings.HasPrefix(an.IP.To4().String(), "172.17.") {
if an.IP.IsGlobalUnicast() {
serverIp = an.IP.To4().String()
break
}
}
}
}
}
}
// MakeTime 解析纳秒时间戳或 RFC3339 字符串
func MakeTime(v any) time.Time {
if ts, ok := cast.ToInt64E(v); ok == nil {
return time.Unix(0, ts)
}
tm, _ := time.Parse(time.RFC3339Nano, cast.String(v))
return tm
}
// MakeUsedTime 计算消耗时间(毫秒)
func MakeUsedTime(startTime, endTime time.Time) float32 {
return float32(endTime.UnixNano()-startTime.UnixNano()) / 1e6
}
// ParseBaseLog 解析基础日志行
func ParseBaseLog(line string) *BaseLog {
pos := strings.IndexByte(line, '{')
if pos == -1 {
return ParseBadLog(line)
}
l := make(map[string]any)
err := json.Unmarshal([]byte(line[pos:]), &l)
if err != nil {
return ParseBadLog(line)
}
baseLog := BaseLog{Extra: make(map[string]any)}
for k, v := range l {
lk := strings.ToLower(k)
switch lk {
case "logname":
baseLog.LogName = cast.String(v)
case "logtype":
baseLog.LogType = cast.String(v)
case "logtime":
baseLog.LogTime = cast.Int64(v)
case "traceid":
baseLog.TraceId = cast.String(v)
case "imagename":
baseLog.ImageName = cast.String(v)
case "imagetag":
baseLog.ImageTag = cast.String(v)
case "servername":
baseLog.ServerName = cast.String(v)
case "serverip":
baseLog.ServerIp = cast.String(v)
default:
baseLog.Extra[lk] = v
}
}
return &baseLog
}
// ParseBadLog 解析非 JSON 格式的日志
func ParseBadLog(line string) *BaseLog {
baseLog := BaseLog{Extra: make(map[string]any)}
baseLog.LogType = LogTypeUndefined
if len(line) > 19 && line[19] == ' ' {
tm, err := time.Parse("2006/01/02 15:04:05", line[0:19])
if err == nil {
baseLog.LogTime = tm.UnixNano()
line = line[20:]
} else {
return nil
}
} else if len(line) > 26 && line[26] == ' ' {
tm, err := time.Parse("2006/01/02 15:04:05.000000", line[0:26])
if err == nil {
baseLog.LogTime = tm.UnixNano()
line = line[27:]
} else {
return nil
}
} else {
return nil
}
baseLog.Extra["info"] = line
return &baseLog
}
// fixField 格式化字段名(去横线,小写)
// fixField 格式化字段名(去横线、下划线,小写)
func fixField(s string) string {
return strings.ToLower(strings.ReplaceAll(s, "-", ""))
s = strings.ReplaceAll(s, "-", "")
s = strings.ReplaceAll(s, "_", "")
return strings.ToLower(s)
}
// getCallStacks 获取调用栈
@ -141,6 +69,7 @@ func getCallStacks(truncations []string) []string {
isLogInternal := (strings.Contains(file, "/log/logger.go") ||
strings.Contains(file, "/log/utility.go") ||
strings.Contains(file, "/log/standard.go") ||
strings.Contains(file, "/log/default_logger.go") ||
strings.Contains(file, "/log/extra.go"))
if isLogInternal {
@ -163,24 +92,21 @@ func getCallStacks(truncations []string) []string {
return callStacks
}
// GetDefaultName 获取默认应用名称
func GetDefaultName() string {
name := os.Getenv("DISCOVER_APP")
if name == "" {
name = os.Getenv("discover_app")
}
if name == "" {
imageName := os.Getenv("DOCKER_IMAGE_NAME")
if imageName != "" {
parts := strings.Split(imageName, "/")
imageName = parts[len(parts)-1]
imageName = strings.SplitN(imageName, ":", 2)[0]
imageName = strings.SplitN(imageName, "#", 2)[0]
name = imageName
var globalDefaultName string
// getDefaultName 获取默认应用名称
func getDefaultName() string {
if globalDefaultName != "" {
return globalDefaultName
}
name := ""
if info, ok := debug.ReadBuildInfo(); ok && info.Path != "" && info.Path != "command-line-arguments" {
name = path.Base(info.Path)
}
if name == "" {
name = path.Base(os.Args[0])
}
// 处理 Windows 下的 .exe 后缀
name = strings.TrimSuffix(name, ".exe")
return name
}

378
viewer.go
View File

@ -1,9 +1,10 @@
package log
import (
"encoding/json"
"fmt"
"os"
"regexp"
"sort"
"strings"
"time"
@ -13,11 +14,12 @@ import (
var errorLineMatcher = regexp.MustCompile(`(\w+\.go:\d+)`)
var codeFileMatcher = regexp.MustCompile(`(\w+?\.)(go|js)`)
var workspaceRoot, _ = os.Getwd()
func Viewable(line string) string {
b := ParseBaseLog(line)
if b == nil {
// 高亮错误代码
line = strings.TrimSpace(line)
if !strings.HasPrefix(line, "[") {
// Fallback highlight for non-array strings
if strings.Contains(line, ".go:") {
if strings.Contains(line, "/ssgo/") || strings.Contains(line, "/ssdo/") || strings.Contains(line, "/gojs/") {
line = errorLineMatcher.ReplaceAllString(line, shell.BYellow("$1"))
@ -30,139 +32,293 @@ func Viewable(line string) string {
return line
}
logTime := time.Unix(0, b.LogTime)
var arr []any
if err := cast.UnmarshalJSON([]byte(line), &arr); err != nil {
return line
}
if len(arr) < 2 {
return line
}
logType := ""
if len(arr) > 2 {
logType = cast.String(arr[2])
}
meta := GetMeta(logType)
if len(meta) == 0 {
logType = cast.String(arr[1])
meta = GetMeta(logType)
}
if len(meta) == 0 {
// Fallback rendering
return fallbackRenderArray(arr)
}
var builder strings.Builder
builder.WriteString(shell.White(shell.Bold, logTime.Format("01-02 15:04:05.000")))
builder.WriteString(" ")
builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Underline, b.TraceId))
level := ""
for _, k := range []string{"info", "warning", "error", "debug"} {
if v := b.Extra[k]; v != nil && cast.String(v) != "" {
level = k
break
}
}
if b.LogType == LogTypeRequest {
method := cast.String(b.Extra["method"])
path := cast.String(b.Extra["path"])
code := cast.Int(b.Extra["responsecode"])
used := float32(cast.Float64(b.Extra["usedtime"]))
builder.WriteString(" ")
builder.WriteString(shell.Cyan(shell.Bold, "REQUEST"))
builder.WriteString(" ")
builder.WriteString(shell.Cyan(method))
builder.WriteString(" ")
builder.WriteString(path)
builder.WriteString(" ")
if code >= 500 {
builder.WriteString(shell.BRed(cast.String(code)))
} else if code >= 400 {
builder.WriteString(shell.BYellow(cast.String(code)))
} else {
builder.WriteString(shell.BGreen(cast.String(code)))
}
builder.WriteString(" ")
builder.WriteString(shell.Style(shell.Dim, fmt.Sprintf("%.2fms", used)))
for _, k := range []string{"method", "path", "responsecode", "usedtime", "host", "scheme", "proto", "clientip", "serverid", "app", "node", "fromapp", "fromnode", "userid", "deviceid", "clientappname", "clientappversion", "sessionid", "requestid", "authlevel", "priority", "requestheaders", "requestdata", "responseheaders", "responsedatalength", "responsedata", "logname", "logtype", "logtime", "traceid", "imagename", "imagetag", "servername", "serverip"} {
delete(b.Extra, k)
}
} else if b.LogType == LogTypeStatistic {
builder.WriteString(" ")
builder.WriteString(shell.Cyan(shell.Bold, "STATISTIC"))
} else if b.LogType == LogTypeTask {
builder.WriteString(" ")
builder.WriteString(shell.Cyan(shell.Bold, "TASK"))
} else {
if level != "" {
msg := cast.String(b.Extra[level])
delete(b.Extra, level)
builder.WriteString(" ")
switch level {
case "info":
builder.WriteString(shell.Cyan(msg))
case "warning":
builder.WriteString(shell.Yellow(msg))
case "error":
builder.WriteString(shell.Red(msg))
case "debug":
builder.WriteString(msg)
}
} else if b.LogType == "undefined" {
builder.WriteString(" ")
builder.WriteString(shell.Style(shell.Dim, "-"))
} else {
builder.WriteString(" ")
builder.WriteString(shell.Cyan(shell.Bold, b.LogType))
}
}
callStacks := b.Extra["callstacks"]
delete(b.Extra, "callstacks")
if b.Extra != nil {
for k, v := range b.Extra {
vStr := ""
if v == nil {
for i, v := range arr {
if v == nil || cast.String(v) == "0" { // 0 is gap
continue
}
switch v.(type) {
case map[string]any, []any:
vStr, _ = cast.ToJSON(v)
default:
vStr = cast.String(v)
if i >= len(meta) {
// Unmapped trailing values, just print them
builder.WriteString(" ")
builder.WriteString(shell.Style(shell.Dim, fmt.Sprintf("Index%d:", i)))
builder.WriteString(cast.String(v))
continue
}
if k == "extra" && len(vStr) > 0 && vStr[0] == '{' {
extra, err := cast.ToMap[string, any](vStr)
if err == nil {
for k2, v2 := range extra {
builder.WriteString(" ")
builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Italic, k2+":"))
builder.WriteString(cast.String(v2))
m := meta[i]
if m.Hide || m.Name == "" {
continue
}
if m.Name == "Extra" {
extraMap, ok := v.(map[string]any)
if ok && len(extraMap) > 0 {
keys := make([]string, 0, len(extraMap))
for k := range extraMap {
keys = append(keys, k)
}
} else {
sort.Strings(keys)
for _, k := range keys {
ev := extraMap[k]
builder.WriteString(" ")
builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Italic, k+":"))
builder.WriteString(vStr)
builder.WriteString(renderValue(ev, 0, ""))
}
}
continue
}
if callStacks != nil {
var callStacksList []any
switch cs := callStacks.(type) {
case string:
if len(cs) > 2 && cs[0] == '[' {
_ = json.Unmarshal([]byte(cs), &callStacksList)
}
case []any:
callStacksList = cs
if m.Name == "CallStacks" {
callStacksList, ok := v.([]any)
if ok && len(callStacksList) > 0 {
stackColor := shell.TextRed
if strings.Contains(strings.ToLower(logType), "warn") {
stackColor = shell.TextYellow
}
if len(callStacksList) > 0 {
builder.WriteString("\n")
for _, vi := range callStacksList {
v := cast.String(vi)
vStr := cast.String(vi)
if workspaceRoot != "" {
vStr = strings.TrimPrefix(vStr, workspaceRoot)
vStr = strings.TrimPrefix(vStr, "/")
}
postfix := ""
if pos := strings.LastIndexByte(v, '/'); pos != -1 {
postfix = v[pos+1:]
v = v[:pos+1]
if pos := strings.LastIndexByte(vStr, '/'); pos != -1 {
postfix = vStr[pos+1:]
vStr = vStr[:pos+1]
} else {
postfix = v
v = ""
postfix = vStr
vStr = ""
}
builder.WriteString(" ")
builder.WriteString(shell.Style(shell.Dim, v))
builder.WriteString(shell.Style(shell.TextWhite, postfix))
builder.WriteString(shell.Style(shell.Dim, vStr))
builder.WriteString(shell.Style(stackColor, postfix))
builder.WriteString("\n")
}
}
continue
}
// Handle normal fields
vStr := ""
if m.Format == "time" {
// Convert int64 ns to time string
logTime := time.Unix(0, cast.Int64(v))
dateStr := logTime.Format("01-02")
timeStr := logTime.Format("15:04:05")
milliStr := logTime.Format(".000")
if builder.Len() > 0 {
builder.WriteString(" ")
}
builder.WriteString(shell.Style(shell.Dim, dateStr))
builder.WriteString(" ")
builder.WriteString(shell.White(shell.Bold, timeStr))
builder.WriteString(shell.Style(shell.Dim, milliStr))
continue
} else {
vStr = renderValue(v, m.Precision, m.Color)
if vStr == "" {
continue
}
}
if builder.Len() > 0 {
if m.AttachBefore {
builder.WriteString(":")
} else {
builder.WriteString(" ")
}
}
if !m.WithoutKey && !m.AttachBefore {
name := m.KeyName
if name == "" {
name = m.Name
}
builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Italic, name+":"))
}
builder.WriteString(vStr)
}
return builder.String()
}
// ToJSON converts a JSON array log line to a standard JSON object string based on metadata.
func ToJSON(line string) string {
line = strings.TrimSpace(line)
if !strings.HasPrefix(line, "[") {
return line
}
var arr []any
if err := cast.UnmarshalJSON([]byte(line), &arr); err != nil {
return line
}
if len(arr) < 2 {
return line
}
logType := ""
if len(arr) > 2 {
logType = cast.String(arr[2])
}
meta := GetMeta(logType)
if len(meta) == 0 {
logType = cast.String(arr[1])
meta = GetMeta(logType)
}
if len(meta) == 0 {
return line
}
result := make(map[string]any)
for i, v := range arr {
if i < len(meta) {
m := meta[i]
if m.Name == "" {
continue
}
name := m.KeyName
if name == "" {
name = m.Name
}
if m.Name == "Extra" {
if extraMap, ok := v.(map[string]any); ok {
for k, ev := range extraMap {
result[k] = ev
}
}
} else {
result[name] = v
}
} else if cast.String(v) != "0" {
result[fmt.Sprintf("Extra%d", i)] = v
}
}
jsonStr, _ := cast.ToJSON(result)
return jsonStr
}
func applyColor(text string, color string) string {
switch color {
case "red":
return shell.Red(text)
case "cyan":
return shell.Cyan(text)
case "blue":
return shell.Blue(text)
case "magenta":
return shell.Magenta(text)
case "yellow":
return shell.Yellow(text)
case "green":
return shell.Green(text)
case "gray":
return shell.Style(shell.Dim, text)
default:
return text
}
}
func fallbackRenderArray(arr []any) string {
var builder strings.Builder
for i, v := range arr {
if i > 0 {
builder.WriteString(" ")
}
builder.WriteString(cast.String(v))
}
return builder.String()
}
func renderValue(v any, precision int, color string) string {
if v == nil {
return ""
}
switch val := v.(type) {
case float32, float64:
vStr := ""
if precision > 0 {
vStr = fmt.Sprintf("%.*f", precision, cast.To[float64](v))
} else {
vStr = cast.String(v)
}
return applyColor(vStr, color)
case map[string]any:
if len(val) == 0 {
return ""
}
var parts []string
keys := make([]string, 0, len(val))
for k := range val {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
// Key is always dim, value is colored
vStr := renderValue(val[k], precision, color)
if vStr != "" {
parts = append(parts, fmt.Sprintf("%s:%s", shell.Style(shell.Dim, k), vStr))
}
}
if len(parts) == 0 {
return ""
}
return "[ " + strings.Join(parts, " ") + " ]"
case []any:
if len(val) == 0 {
return ""
}
var parts []string
for _, iv := range val {
vStr := renderValue(iv, precision, color)
if vStr != "" {
parts = append(parts, vStr)
}
}
if len(parts) == 0 {
return ""
}
return "[ " + strings.Join(parts, " ") + " ]"
default:
s := cast.String(v)
if s == "" {
return ""
}
return applyColor(s, color)
}
}

View File

@ -1,29 +1,230 @@
package log_test
import (
"os"
"strings"
"testing"
"apigo.cc/go/log"
)
func BenchmarkViewable(b *testing.B) {
// 准备一个典型的 JSON 日志行,注意 Info, Warning 等在顶层
line := `{"LogName":"test-app","LogType":"info","LogTime":1714896000000000000,"TraceId":"trace-123","info":"hello world","Extra":{"key":"value"}}`
func TestViewable(t *testing.T) {
entry := &log.InfoLog{
BaseLog: log.BaseLog{
LogName: "test-app",
LogType: "info",
LogTime: 1714896000000000000,
TraceId: "trace-123",
},
Info: "hello world",
}
log.RegisterType("info", entry)
b.ResetTimer()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
_ = log.Viewable(line)
line := string(log.Marshal(entry, nil))
out := log.Viewable(line)
if !strings.Contains(out, "hello world") {
t.Errorf("expected 'hello world' in output, got: %s", out)
}
if !strings.Contains(out, "trace-123") {
t.Errorf("expected 'trace-123' in output, got: %s", out)
}
}
func BenchmarkViewable_Request(b *testing.B) {
// RequestLog 的字段也在顶层
line := `{"LogName":"test-app","LogType":"request","LogTime":1714896000000000000,"TraceId":"trace-123","method":"GET","path":"/api/user","responsecode":200,"usedtime":10.5}`
func TestToJSON(t *testing.T) {
entry := &log.InfoLog{
BaseLog: log.BaseLog{
LogName: "test-app",
LogType: "info",
LogTime: 1714896000000000000,
TraceId: "trace-123",
},
Info: "hello world",
}
entry.Extra = map[string]any{"key": "value"}
log.RegisterType("info", entry)
b.ResetTimer()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
_ = log.Viewable(line)
line := string(log.Marshal(entry, nil))
jsonStr := log.ToJSON(line)
if !strings.Contains(jsonStr, `"Info":"hello world"`) {
t.Errorf("expected Info field in JSON, got: %s", jsonStr)
}
if !strings.Contains(jsonStr, `"TraceId":"trace-123"`) {
t.Errorf("expected TraceId field in JSON, got: %s", jsonStr)
}
if !strings.Contains(jsonStr, `"key":"value"`) {
t.Errorf("expected Extra fields merged in JSON, got: %s", jsonStr)
}
}
func TestLoadMeta(t *testing.T) {
// Create a temporary meta file
metaData := `{"test-type":[{"index":0,"name":"Field1"},{"index":1,"name":"Field2"}]}`
_ = os.WriteFile(".test_meta.json", []byte(metaData), 0644)
defer os.Remove(".test_meta.json")
err := log.LoadMeta(".test_meta.json")
if err != nil {
t.Fatalf("failed to load meta: %v", err)
}
meta := log.GetMeta("test-type")
if len(meta) != 2 {
t.Errorf("expected 2 meta fields, got %d", len(meta))
}
if meta[0].Name != "Field1" {
t.Errorf("expected Field1, got %s", meta[0].Name)
}
}
type EnhancedLog struct {
log.BaseLog
App string `log:"pos:10,withoutkey:true"`
Node string `log:"pos:11,attachBefore,withoutkey:true"`
RequestHeaders map[string]string `log:"pos:13,keyname:reqH"`
ClientIP string `log:"pos:12,keyname:ip"`
Tags []string `log:"pos:14"`
}
func (l *EnhancedLog) Reset() {
l.BaseLog.Reset()
l.App = ""
l.Node = ""
l.ClientIP = ""
l.RequestHeaders = nil
l.Tags = nil
}
func TestEnhancedViewable(t *testing.T) {
entry := &EnhancedLog{
BaseLog: log.BaseLog{
LogType: "enhanced",
LogTime: 1714896000000000000,
},
App: "MyApp",
Node: "Node1",
ClientIP: "127.0.0.1",
RequestHeaders: map[string]string{
"User-Agent": "Go-http-cli",
},
Tags: []string{"tag1", "tag2"},
}
log.RegisterType("enhanced", entry)
line := string(log.Marshal(entry, nil))
out := log.Viewable(line)
// Check attachBefore: MyApp:Node1 (since both are withoutkey)
if !strings.Contains(out, "MyApp:Node1") {
t.Errorf("expected MyApp:Node1, got: %s", out)
}
// Check pos ordering and keyname: ip:127.0.0.1 should come before reqH
if !strings.Contains(out, "ip:") || !strings.Contains(out, "127.0.0.1") {
t.Errorf("expected ip:127.0.0.1, got: %s", out)
}
if !strings.Contains(out, "reqH:") || !strings.Contains(out, "User-Agent") || !strings.Contains(out, "Go-http-cli") {
t.Errorf("expected reqH:[ User-Agent:Go-http-cli ], got: %s", out)
}
ipIdx := strings.Index(out, "ip:")
reqHIdx := strings.Index(out, "reqH:")
if ipIdx > reqHIdx {
t.Errorf("expected ip to come before reqH, but ipIdx=%d, reqHIdx=%d", ipIdx, reqHIdx)
}
// Check array rendering: Tags:[ tag1 tag2 ]
if !strings.Contains(out, "Tags:") || !strings.Contains(out, "[ tag1 tag2 ]") {
t.Errorf("expected Tags:[ tag1 tag2 ], got: %s", out)
}
}
func TestEnhancedToJSON(t *testing.T) {
entry := &EnhancedLog{
BaseLog: log.BaseLog{
LogType: "enhanced",
LogTime: 1714896000000000000,
},
App: "MyApp",
Node: "Node1",
ClientIP: "127.0.0.1",
RequestHeaders: map[string]string{
"User-Agent": "Go-http-cli",
},
}
log.RegisterType("enhanced", entry)
line := string(log.Marshal(entry, nil))
jsonStr := log.ToJSON(line)
// Check keyname in JSON
if !strings.Contains(jsonStr, `"ip":"127.0.0.1"`) {
t.Errorf("expected ip field in JSON, got: %s", jsonStr)
}
if !strings.Contains(jsonStr, `"reqH":{"User-Agent":"Go-http-cli"}`) {
t.Errorf("expected reqH field in JSON, got: %s", jsonStr)
}
}
type CallStackLog struct {
log.BaseLog
CallStacks []string `log:"pos:6"`
}
func (l *CallStackLog) Reset() {
l.BaseLog.Reset()
l.CallStacks = nil
}
func TestCallStacksViewable(t *testing.T) {
wd, _ := os.Getwd()
entry := &CallStackLog{
BaseLog: log.BaseLog{
LogType: "test_error",
LogTime: 1714896000000000000,
},
CallStacks: []string{wd + "/main.go:10", "/usr/local/go/src/runtime/panic.go:100"},
}
log.RegisterType("test_error", entry)
line := string(log.Marshal(entry, nil))
out := log.Viewable(line)
// Check path truncation (should contain relative "main.go:10")
if !strings.Contains(out, "main.go:10") {
t.Errorf("expected relative path main.go:10, got: %s", out)
}
// Absolute path should be removed if it matches wd
if strings.Contains(out, wd) {
t.Errorf("absolute path should be truncated, but still found: %s", out)
}
}
type PrecisionLog struct {
log.BaseLog
Value float64 `log:"pos:6,precision:2"`
}
func (l *PrecisionLog) Reset() {
l.BaseLog.Reset()
l.Value = 0
}
func TestPrecisionViewable(t *testing.T) {
entry := &PrecisionLog{
BaseLog: log.BaseLog{
LogType: "precision",
LogTime: 1714896000000000000,
},
Value: 3.14159,
}
log.RegisterType("precision", entry)
line := string(log.Marshal(entry, nil))
out := log.Viewable(line)
if !strings.Contains(out, "3.14") || strings.Contains(out, "3.141") {
t.Errorf("expected 3.14 (precision 2), got: %s", out)
}
}

125
writer.go
View File

@ -1,6 +1,7 @@
package log
import (
"context"
"fmt"
"sync"
"sync/atomic"
@ -9,30 +10,41 @@ import (
// Writer 日志写入接口
type Writer interface {
Log([]byte)
Log(LogEntry, []byte)
Run()
}
// logPayload 包含路由信息的包裹
type logPayload struct {
entry LogEntry
buf []byte
writer Writer // 目标自定义 Writer
file *FileWriter // 目标文件 Writer
}
// writerService manages the background writing of log entries.
type writerService struct {
Running atomic.Bool
StopChan chan bool
LogChannel chan logPayload
Dropped atomic.Uint64
Writers atomic.Value // []Writer
WriterLock sync.Mutex
Files map[string]*FileWriter
FilesLock sync.RWMutex
}
var (
writerRunning atomic.Bool
writerLock sync.Mutex // 仅用于注册时锁定
writerStopChan chan bool
writers atomic.Value // 存储 []Writer
logChannel chan logPayload
// WriterService is the global instance of defaultService.
WriterService = &writerService{}
)
// ConsoleWriter 控制台写入器
type ConsoleWriter struct {
}
func (w *ConsoleWriter) Log(data []byte) {
func (w *ConsoleWriter) Log(entry LogEntry, data []byte) {
fmt.Println(Viewable(string(data)))
}
@ -40,110 +52,129 @@ func (w *ConsoleWriter) Run() {
}
func init() {
logChannel = make(chan logPayload, 10000)
writers.Store([]Writer{})
WriterService.LogChannel = make(chan logPayload, 10000)
WriterService.Writers.Store([]Writer{})
WriterService.Files = make(map[string]*FileWriter)
RegisterWriterMaker("console", func(conf *Config) Writer {
return &ConsoleWriter{}
})
}
// WriteAsync 异步写入日志
func WriteAsync(payload logPayload) {
// writeAsync 异步写入日志
func writeAsync(payload logPayload) {
defer func() {
recover()
}()
if !writerRunning.Load() {
if !WriterService.Running.Load() {
return
}
select {
case logChannel <- payload:
case WriterService.LogChannel <- payload:
default:
// 丢弃或处理过载,此处简单丢弃
// 丢弃或处理过载
dropped := WriterService.Dropped.Add(1)
if dropped%1000 == 1 {
if DefaultLogger != nil {
// 注意:这里可能会产生递归调用,但 select default 保证了不会死锁
DefaultLogger.Error(fmt.Sprintf("log channel full, dropped %d logs", dropped))
}
}
}
}
// Start 启动写入器
func Start() {
if !writerRunning.CompareAndSwap(false, true) {
return
}
writerStopChan = make(chan bool)
go writerRunner()
// GetDroppedLogs 获取被丢弃的日志数量
func GetDroppedLogs() uint64 {
return WriterService.Dropped.Load()
}
// Stop 停止写入器
func Stop() {
if writerRunning.CompareAndSwap(true, false) {
close(logChannel)
// Start implements starter.Service interface.
func (s *writerService) Start(_ context.Context, _ *Logger) error {
if !s.Running.CompareAndSwap(false, true) {
return nil
}
s.StopChan = make(chan bool)
go s.writerRunner()
return nil
}
// Wait 等待写入器停止
func Wait() {
if writerStopChan != nil {
<-writerStopChan
writerStopChan = nil
// Stop implements starter.Service interface.
func (s *writerService) Stop(_ context.Context) error {
if s.Running.CompareAndSwap(true, false) {
close(s.LogChannel)
if s.StopChan != nil {
<-s.StopChan
s.StopChan = nil
}
}
return nil
}
func writerRunner() {
// Status implements starter.Service interface.
func (s *writerService) Status() (string, error) {
if s.Running.Load() {
return fmt.Sprintf("queue: %d, dropped: %d", len(s.LogChannel), s.Dropped.Load()), nil
}
return "stopped", nil
}
func (s *writerService) writerRunner() {
ticker := time.NewTicker(200 * time.Millisecond)
defer ticker.Stop()
defer func() {
if writerStopChan != nil {
close(writerStopChan)
if s.StopChan != nil {
close(s.StopChan)
}
}()
for {
select {
case payload, ok := <-logChannel:
case payload, ok := <-s.LogChannel:
if !ok {
flushWriters()
s.flushWriters()
return
}
processLog(payload)
s.processLog(payload)
// 尝试批量处理更多日志
batchCount := 0
for batchCount < 100 {
select {
case nextPayload, nextOk := <-logChannel:
case nextPayload, nextOk := <-s.LogChannel:
if !nextOk {
flushWriters()
s.flushWriters()
return
}
processLog(nextPayload)
s.processLog(nextPayload)
batchCount++
default:
batchCount = 100 // break outer loop
}
}
case <-ticker.C:
flushWriters()
s.flushWriters()
}
}
}
func processLog(payload logPayload) {
func (s *writerService) processLog(payload logPayload) {
// 精准路由:根据包裹信息决定写入目标
if payload.writer != nil {
payload.writer.Log(payload.buf)
payload.writer.Log(payload.entry, payload.buf)
} else if payload.file != nil {
payload.file.Write(time.Now(), payload.buf)
}
}
func flushWriters() {
curWriters, _ := writers.Load().([]Writer)
func (s *writerService) flushWriters() {
curWriters, _ := s.Writers.Load().([]Writer)
for _, w := range curWriters {
w.Run()
}
filesLock.RLock()
for _, f := range files {
s.FilesLock.RLock()
for _, f := range s.Files {
f.Run()
}
filesLock.RUnlock()
s.FilesLock.RUnlock()
}