Compare commits

..

No commits in common. "main" and "v1.1.5" have entirely different histories.
main ... v1.1.5

29 changed files with 855 additions and 1663 deletions

7
.gitignore vendored
View File

@ -1,9 +1,2 @@
.log.meta.json .log.meta.json
.test.meta.json .test.meta.json
.ai/
.geminiignore
.gemini
env.json
env.yml
env.yaml
/CODE-FULL.md

View File

@ -1,86 +1,6 @@
# Changelog # Changelog
## [1.3.2] - 2026-05-13 ## [1.1.4] - 2026-05-05
- **功能增强: 引入摩擦消除工具 `As`**:
- **泛型支持**: 新增全局泛型函数 `log.As[T](v T, err error) T`,仿照 `cast.As` 设计,自动记录错误并返回零值,极大简化了带 error 返回值的函数链式调用。
- **Logger 扩展**: `Logger` 结构体新增 `As(v any, err error) any` 方法,支持实例级别的错误捕获与自动记录。
- **调用栈优化**: 优化了 `GetCallStacks` 逻辑,自动跳过 `default_logger.go` 中的内部帧,确保 `log.As` 记录的错误位置精准指向业务代码。
## [1.3.1] - 2026-05-12
- **架构升级: 引入 LoggerService**:
- **解耦重构**: 重构全局变量管理,引入 `loggerService` 结构体集中化管理异步写入协程、Writers 对象池、文件句柄与丢弃计数。
- **生命周期管理**: 实现了 `Start(ctx, logger)`, `Stop(ctx)`, `Health()` 接口,完美支持与 `apigo.cc/go/starter` 的基础设施集成。
- **安全性增强**: 优化了平滑停止逻辑,确保 `Stop` 调用时能完整 Flush 缓冲区数据。
- **功能增强**:
- **动态应用名**: 新增 `SetDefaultName(name)` 全局方法,支持在微服务启动后动态设置应用名称,并自动同步至 `DefaultLogger`
- **配置与忽略规则**:
- 更新 `.gitignore`,增加了对 `.gemini`, `env.json`, `env.yml` 以及 `/CODE-FULL.md` 的忽略支持。
- **文档与测试同步**: 全面更新了 `README.md`, `CHANGELOG.md``TEST.md`
## [1.3.0] - 2026-05-10
- **全栈基础设施对齐**:
- 将所有内部依赖(`cast`, `config`, `file`, `id`, `shell` 等)统一升级至 `v1.3.0` 语义版本。
- 优化项目忽略规则,在 `.gitignore` 中增加了对 `.geminiignore` 的支持。
## [1.1.16] - 2026-05-10
- **依赖对齐**: 同步更新 `golang.org/x/crypto``golang.org/x/sys` 至最新版本,确保底层安全性。
## [1.1.15] - 2026-05-10
- **安全与性能双重增强**:
- **敏感字段扩展**: `authorization` 现在默认纳入脱敏规则 (`LogDefaultSensitive`),自动过滤 HTTP 认证凭证。
- **TraceId 吞吐优化**: 切换至 `id.Get10Bytes14MPerSecond()` 算法,在保持 10 字节唯一性的同时,极大提升了超高并发下的 ID 生成性能。
- **应用名称识别对齐**: 重构 `GetDefaultName` 逻辑,优先通过 `debug.ReadBuildInfo()` 识别 Module 路径,并支持自动裁剪 Windows 环境下的 `.exe` 后缀。
## [1.1.14] - 2026-05-09
- **可视化能力调整**: 调整优化了 `viewer` 模块相关的可视化能力,提升了日志的可读性与调试体验。
## [1.1.13] - 2026-05-09
- **绝对索引优化与零空洞**:
- 彻底消除 `BaseLog` 与业务字段之间的索引空洞。字段位置调整为:`BaseLog` (0-5),标准消息字段 (`Info`, `Error` 等) 与业务日志字段从 `pos: 6` 起始。
- **智能平移映射**: 改进 `pos >= 1000` 的逻辑。这些字段(如 `Extra`, `CallStacks`)不再产生稀疏 0 占位符,而是根据定义的 `pos` 顺序,自动平移并紧跟在当前类型的最大绝对索引之后。既保证了“永远在末尾”,又维持了数组的紧凑性。
- **强制 Reset 安全契约**:
- `RegisterType` 引入严苛校验:若自定义日志类型包含字段但未显式重写 `Reset()`(即仅继承了 `BaseLog.Reset`),注册时将触发 **Panic**。强制开发者显式清理业务数据,杜绝对象池复用时的脏数据隐患。
- **应用名称识别增强**:
- `GetDefaultName` 引入 `runtime/debug.ReadBuildInfo()`。相比传统的文件夹路径读取,能更精准地识别 Go Module 定义的应用名称。
- **工具链增强**:
- `serializer_test.go` 新增原始日志输出,方便开发者通过 `go test -v` 直观校验 JSON 数组结构。
## [1.1.12] - 2026-05-09
...
- **稳定性增强**:
- 修复 `TestSplitTag` 在秒级进位边界时的偶发性失败。优化后的测试逻辑同时校验日志产生前后的两个潜在时间槽,闭环消除了环境抖动导致的 race condition。
## [1.1.9] - 2026-05-05
- **稳定性增强**:
- 改进 `SplitTag` 测试用例,通过秒级切分步进,闭环验证了 `FileWriter` 的文件自动轮转逻辑,确保在高频或定时切分场景下的可靠性。
## [1.1.8] - 2026-05-05
- **深度脱敏与性能平衡**:
- 引入“混合路径”序列化策略对基础类型String, Int, Bool 等采用高性能手动编码对复杂类型Struct, Map, Slice采用 `cast.ToJSONDesensitizeBytes`
- 完美解决嵌套对象、数组内对象的深度脱敏问题,同时将序列化性能维持在极高水平。
- **基准测试优化**: 相比全量 `cast` 调用,混合路径将异步写入耗时降低了 60% 以上,内存分配减少了 70%。
## [1.1.7] - 2026-05-05
- **极致精简与摩擦消除**:
- 移除了所有冗余或已弃用的配置项:`Fast`, `KeepKeyCase`, `RegexSensitive`, `SensitiveRule`
- 删除了 `Logger` 中未被使用的 `desensitization` 处理函数及 `SetDesensitization` 方法,进一步收窄 API。
- **功能增强与修复**:
- **脱敏逻辑闭环**: 修复了 `ToArrayBytes` 仅对 Map 字段脱敏的缺陷。现在支持对结构体根层级的敏感字段(如 `String` 类型)进行全量遮蔽。
- **标准化命名映射**: 增强 `fixField` 逻辑,同时支持自动忽略下划线 (`_`) 和横线 (`-`),确保 `SecretKey` (Go) 与 `secret_key` (Config) 能完美匹配。
- **稳定性保障**:
- 新增 `functional_test.go`,闭环验证了 `SplitTag` 文件切分能力与增强后的 `Sensitive` 脱敏逻辑。
- 优化 `serializer.go` 内部判断逻辑,基准测试显示性能有显著提升。
## [1.1.6] - 2026-05-05
- **性能优化**:
- 重构 `FillBase` 方法签名,由接收接口 `LogEntry` 改为接收指针 `*BaseLog`
- 此项改动消除了在填充元数据时通过接口调用 `GetBaseLog()` 的开销,直接操作结构体指针,进一步提升日志预处理性能。
- **文档与示例对齐**:
- 完善 `README.md` 中的 `Config` 配置项说明,涵盖 `Fast`, `KeepKeyCase`, `Truncations`, `SensitiveRule` 等所有字段。
- 修正 `BusinessLog` 扩展示例,确保与最新的 `FillBase` 签名及元数据填充逻辑保持一致。
- **扩展与迁移指引**: 同步更新 `extra.go` 中的注释示例,为从旧版本或其他项目迁移提供准确的参考实现。
## [1.1.5] - 2026-05-05
- **高性能 Meta 驱动架构**: - **高性能 Meta 驱动架构**:
- 日志存储格式由 JSON Object 彻底切换为 **JSON Positional Array (`[...]`)**,通过位置索引消除重复 Key 的存储与传输开销。 - 日志存储格式由 JSON Object 彻底切换为 **JSON Positional Array (`[...]`)**,通过位置索引消除重复 Key 的存储与传输开销。
- 实现基于反射的 **零装箱 (No-Boxing) 序列化**,直接拼接 JSON 字符串,大幅降低内存分配与 CPU 占用。 - 实现基于反射的 **零装箱 (No-Boxing) 序列化**,直接拼接 JSON 字符串,大幅降低内存分配与 CPU 占用。

160
README.md
View File

@ -9,7 +9,7 @@
* **极致高性能**:采用 **Meta-Driven Positional Array (元数据驱动定长数组)** 架构。日志以单行 JSON 数组 (`[...]`) 形式落盘,消除 Key 冗余与装箱开销,性能提升数倍。 * **极致高性能**:采用 **Meta-Driven Positional Array (元数据驱动定长数组)** 架构。日志以单行 JSON 数组 (`[...]`) 形式落盘,消除 Key 冗余与装箱开销,性能提升数倍。
* **架构解耦**:元数据外置于 `.log.meta.json`。日志包仅负责高速序列化,可视化由外部工具或 `Viewable` 接口根据元数据动态渲染。 * **架构解耦**:元数据外置于 `.log.meta.json`。日志包仅负责高速序列化,可视化由外部工具或 `Viewable` 接口根据元数据动态渲染。
* **零摩擦入口**自动识别环境上下文应用名、IP等无需手动构建。 * **零摩擦入口**自动识别环境上下文应用名、IP等无需手动构建。
* **语义脱敏**:内置敏感信息(如手机号、密钥)的自动脱敏。 * **语义脱敏**:内置敏感信息(如手机号、密钥)的自动脱敏与正则过滤
* **高度可扩展**支持多种写入渠道文件切分、Elasticsearch批量传输 * **高度可扩展**支持多种写入渠道文件切分、Elasticsearch批量传输
## 📦 安装 ## 📦 安装
@ -23,65 +23,14 @@ go get apigo.cc/go/log
```go ```go
import "apigo.cc/go/log" import "apigo.cc/go/log"
// 默认 logger (通过 log.json 或环境变量配置) // 使用默认配置初始化 (或在配置中指定)
func main() { logger := log.NewLogger(log.Config{Name: "my-app", Level: "info"})
// 在微服务场景下动态设置应用名称
log.SetDefaultName("my-microservice")
log.Info("服务启动", "port", 8080) // 记录业务日志 (自动通过 cast.ToMap 处理变长参数)
log.Error("数据库连接失败", "db", "mysql") logger.Info("用户登录", "userId", 10086, "ip", "1.2.3.4")
logger.Error("数据库连接失败", "db", "mysql", "err", err)
// 创建带 traceId 的新 logger 实例
logger := log.New("trace-xyz-123")
}
``` ```
## ⚙️ 配置 (Configuration)
本包深度集成 `@go/config`,支持多种灵活的配置方式,优先级从高到低:
1. **环境变量** (最高优先级)
2. **环境特定文件** (`env.json` / `env.yml`,需增加层级 `log:`)
3. **基础配置文件** (`log.json` / `log.yml`)
### 1. 配置文件 (`log.json`)
在项目根目录创建 `log.json``log.yml`
```json
{
"name": "my-cool-app",
"level": "info",
"file": "logs/app.log",
"splitTag": ".2006-01-02",
"sensitive": "phone,password,secret,token,accessToken,authorization"
}
```
### 2. 环境变量 (最高优先级)
任何配置都可以通过环境变量覆盖,变量名规则为 `LOG_` + `字段名`
```bash
# 覆盖日志级别和输出文件
export LOG_LEVEL=debug
export LOG_FILE=console
```
### 配置项说明
* `name`: 应用名称 (默认通过 `debug.ReadBuildInfo()``os.Args[0]` 自动识别)。
* `level`: 日志级别 (`debug`, `info`, `warning`, `error`)。
* `file`: 输出目标。
* `console`: 直接输出到控制台(默认)。
* `path/to/file.log`: 输出到指定文件。
* `es://...``ess://...`: 输出到 Elasticsearch。
* `splitTag`: 文件切分格式,仅当 `file` 为文件路径时有效。
* 语法遵循 Go 标准的 `time.Format` 布局,如 `".2006-01-02"` (按天切分)`".2006-01-02-15"` (按小时切分)。
* `truncations`: 堆栈信息截断前缀(多个以逗号分隔,默认截断 `github.com/`, `golang.org/`, `/apigo.cc/`)。
* `sensitive`: 需要自动脱敏的字段名(多个以逗号分隔,不区分大小写),默认处理 `phone,password,secret,token,accessToken,authorization`
## 🛠 API 指南 ## 🛠 API 指南
### 核心功能 ### 核心功能
@ -89,90 +38,47 @@ export LOG_FILE=console
1. **分级记录** 1. **分级记录**
* `Debug`, `Info`, `Warning`, `Error` —— 标准日志方法,支持 `message` + 变长 `extra` 参数。 * `Debug`, `Info`, `Warning`, `Error` —— 标准日志方法,支持 `message` + 变长 `extra` 参数。
2. **摩擦消除 (`As`)** 2. **通用记录 (`Log`)**
* `As(v, err)` —— 仿照 `cast.As`,忽略错误并返回零值,但会自动将错误记录到日志中。支持全局调用 (`log.As`) 或实例调用 (`logger.As`)。 * `Log(LogEntry)` —— 记录自定义结构的日志。注意:仅支持实现 `LogEntry` 接口的类型(即嵌入了 `BaseLog` 的结构体)。
* **优势**: 在类型转换或快速赋值场景下,无需繁琐的 `if err != nil` 判断,同时确保异常被记录。
3. **通用记录 (`Log`)** 3. **独立可视化工具 (`logv`)**
* `Log(LogEntry)` —— 记录自定义结构的日志 * 在项目根目录下运行 `go run apigo.cc/go/log/logv` 或将其编译为二进制。该工具从 `stdin` 读取 JSON 数组日志,并根据当前目录的 `.log.meta.json` 自动渲染为带颜色和格式化的彩色文本
4. **独立可视化工具 (`logv`)** ### 自定义日志扩展
* **安装**: `go install apigo.cc/go/log/logv@latest`
* **使用**: `tail -f app.log | logv``tail -f app.log | logv -json`
### 自定义日志扩展 (规范) 如果标准日志分级不能满足业务需求,可以轻松扩展自定义日志类型:
为保证高性能与内存安全,扩展自定义日志类型必须遵循以下规范: 1. **定义结构体**:必须嵌入 `log.BaseLog`
2. **标注位置与样式**:使用 `log:"pos:N,color:xxx,hide:true"` 标签定义字段在数组中的位置及在 `logv` 中的显示样式。
1. **定义结构体** 3. **注册模型**:在 `init()` 中调用 `log.RegisterType("my-type", MyLog{})`
* 必须嵌入 `log.BaseLog` (或其子类,如 `log.ErrorLog`)。 4. **获取与发送**:使用 `log.GetEntry[MyLog]()` 并调用 `logger.Log(entry)`
* **索引 (`pos`) 规范**:
* `0`-`6``BaseLog`
* 业务字段从 `6` 开始紧凑递增编号 (`pos:6`, `pos:7`, ...),如果删除了某个字段请留空 pos 以实现向前兼容。
* 如果继承自 `ErrorLog` 等,则业务字段应从 `7` 开始(查询父类最大值 + 1
* `Extra` 固定使用 `pos:1000``CallStacks` 固定使用 `pos:1001` (它们会被自动平移到数组末尾)。
2. **实现 `Reset()` 方法 (强制)**
* **必须**重写 `Reset()` 方法以初始化/清空数据避免对象池复用时产生脏数据。
* `Reset()` 方法中必须首先调用父级的 `Reset()` (如 `l.BaseLog.Reset()`)。
* **安全保障**: 若未重写 `Reset``RegisterType` 将在启动时 **Panic**,以防止对象池复用时产生脏数据。
* **建议**: map / slice 类型建第一次初始化一个容量,之后使用 clear() 方法清空数据避免内存重复分配。
3. **注册模型**
* 在 `init()` 中调用 `log.RegisterType("my-type", MyLog{})` 完成注册。
#### 示例: `DBErrorLog`
```go ```go
package main type BusinessLog struct {
log.BaseLog // 必须嵌入
import "apigo.cc/go/log" Action string `log:"pos:10,color:cyan"`
UserId string `log:"pos:11"`
// 1. 定义结构体 (字段从 pos:6 开始)
type DBErrorLog struct {
log.ErrorLog // 嵌入 ErrorLog自动获得 Error 和 CallStacks 字段
DB string `log:"pos:7,color:blue"`
SQL string `log:"pos:8"`
Args []any `log:"pos:9"`
UsedTime float32 `log:"pos:10,color:cyan"`
} }
// 2. 实现 Reset() 方法 (强制)
func (l *DBErrorLog) Reset() {
l.ErrorLog.Reset() // 必须先调用父级 Reset
l.DB = ""
l.SQL = ""
if l.Args == nil {
l.Args = make([]any, 0, 10)
} else {
clear(l.Args) // 清空内容
l.Args = l.Args[:0] // 清空长度
}
l.UsedTime = 0
}
// 3. 注册
func init() { func init() {
log.RegisterType("dbError", &DBErrorLog{}) log.RegisterType("business", BusinessLog{})
} }
// 4. 使用示例 func LogBusiness(logger *log.Logger, action, userId string) {
func LogDBError(logger *log.Logger, db, sql string, args []any, err error, usedTime float32) { entry := log.GetEntry[BusinessLog]()
entry := log.GetEntry[DBErrorLog]() entry.LogType = "business"
entry.Action = action
// 自动填充基础字段和 ErrorLog 字段 entry.UserId = userId
logger.FillError(&entry.ErrorLog, err.Error()) logger.Log(entry)
// 填充自定义字段
entry.DB = db
entry.SQL = sql
entry.Args = append(entry.Args, args...)
entry.UsedTime = usedTime
logger.Log(entry)
} }
``` ```
### 配置项 (JSON/YAML)
* `Name`: 应用名称。
* `Level`: 日志级别 (`debug`, `info`, `warning`, `error`)。
* `File`: 输出目标(支持 `console``es://` 地址)。
* `Sensitive`, `RegexSensitive`: 脱敏配置。
## 🧪 验证状态 ## 🧪 验证状态
测试全部通过,异步写入与性能达标。 测试全部通过,异步写入与性能达标。

84
TEST.md
View File

@ -1,62 +1,28 @@
# Test Results # 日志性能测试报告
## 单元测试报告 ## 测试环境
- 操作系统: darwin
- 架构: amd64
- CPU: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
```text ## 基准测试结果 (v1.1.4)
=== RUN TestLoggerCore_Initialization
--- PASS: TestLoggerCore_Initialization (0.00s)
=== RUN TestLoggerCore_Concurrency
--- PASS: TestLoggerCore_Concurrency (0.00s)
=== RUN TestMetaExtraction
--- PASS: TestMetaExtraction (0.00s)
=== RUN TestWithEntry
--- PASS: TestWithEntry (0.00s)
=== RUN TestLoggerReliability
--- PASS: TestLoggerReliability (0.01s)
=== RUN TestToArrayBytes
serializer_test.go:64: Raw log: ["test-app","mock_info_test",1620000000,"abc-123","Hello, World!",{"user_id":42}]
--- PASS: TestToArrayBytes (0.00s)
=== RUN TestToArrayBytes_Desensitize
--- PASS: TestToArrayBytes_Desensitize (0.00s)
=== RUN TestSplitTag
--- PASS: TestSplitTag (1.80s)
=== RUN TestSensitiveDetailed
--- PASS: TestSensitiveDetailed (0.00s)
=== RUN TestDeepDesensitization
--- PASS: TestDeepDesensitization (0.00s)
=== RUN TestLogger
--- PASS: TestLogger (0.00s)
=== RUN TestDesensitization
--- PASS: TestDesensitization (0.00s)
=== RUN TestDBLog
--- PASS: TestDBLog (0.00s)
=== RUN TestRequestLog
--- PASS: TestRequestLog (0.00s)
=== RUN TestExtraLogs
--- PASS: TestExtraLogs (0.00s)
=== RUN TestViewable
--- PASS: TestViewable (0.00s)
=== RUN TestToJSON
--- PASS: TestToJSON (0.00s)
=== RUN TestLoadMeta
--- PASS: TestLoadMeta (0.00s)
=== RUN TestEnhancedViewable
--- PASS: TestEnhancedViewable (0.00s)
=== RUN TestEnhancedToJSON
--- PASS: TestEnhancedToJSON (0.00s)
=== RUN TestCallStacksViewable
--- PASS: TestCallStacksViewable (0.00s)
=== RUN TestPrecisionViewable
--- PASS: TestPrecisionViewable (0.00s)
PASS
ok apigo.cc/go/log 2.246s
```
## 核心指标验证 | 测试用例 | 迭代次数 | 耗时 (ns/op) | 内存分配 (B/op) | 分配次数 (allocs/op) |
- **初始化安全性**: `TestLoggerCore_Initialization` 确保 Logger 实例配置正确加载。 | :--- | :--- | :--- | :--- | :--- |
- **高并发稳定性**: `TestLoggerCore_Concurrency` 验证了在多协程竞争环境下日志写入的线程安全。 | `BenchmarkLogger_RequestLog_Realistic` | 510,711 | 2,122 | 292 | 5 |
- **元数据驱动验证**: `TestMetaExtraction``TestLoadMeta` 确保 `.log.meta.json` 协议的解析与应用。 | `BenchmarkLoggerInfo` | 144,194 | 9,547 | - | - |
- **序列化性能**: `TestToArrayBytes` 验证了 Positional Array 格式的正确性。 | `BenchmarkLoggerAsyncConcurrent` | 159,004 | 7,080 | - | - |
- **深度脱敏能力**: `TestDeepDesensitization` 闭环验证了对复杂嵌套结构的脱敏逻辑。
- **可靠性边界**: `TestLoggerReliability` 模拟了极高压力下的日志丢弃与缓冲策略。 ## 版本对比评估
- **文件切分**: `TestSplitTag` 实测了基于时间滚动的文件切分能力。
| 版本 | 机制 | 存储格式 | 可视化 | 性能 (Async) |
| :--- | :--- | :--- | :--- | :--- |
| **v1.0.3** | Map 序列化 | JSON Object | 内置 | ~8,773 ns/op |
| **v1.1.4** | Meta-Driven Array | **JSON Array** | 独立工具/Meta | **~7,080 ns/op** |
## 总结
- **性能质变**: v1.1.4 通过 **Meta-Driven Positional Array** 架构,在异步并发场景下性能提升了约 20%。
- **存储优化**: 采用数组格式彻底消除了日志中重复 Key 的存储开销,极大地降低了磁盘占用与 ES 索引压力。
- **架构解耦**: 核心包不再感知具体的字段名称,通过外置的 `.log.meta.json` 实现极致的灵活扩展。
- **内存效率**: 通过零装箱 (No-Boxing) 直接字符串拼接技术,保持了极低的内存分配。
- **独立工具**: 配合 `logv` CLI 工具,实现了“落盘高性能数组,查看友好彩色文本”的完美闭环。

View File

@ -2,12 +2,16 @@ package log
// Config 日志配置 // Config 日志配置
type Config struct { type Config struct {
Name string Name string
Level string Level string
File string File string
SplitTag string Fast bool
Truncations string SplitTag string
Sensitive string Truncations string
Sensitive string
RegexSensitive string
SensitiveRule string
KeepKeyCase bool // 是否保持Key的首字母大小写默认一律使用小写
} }
type LevelType int type LevelType int

View File

@ -7,8 +7,8 @@ import (
var DefaultLogger *Logger var DefaultLogger *Logger
func init() { func init() {
RegisterWriterMaker("es", newESWriter) RegisterWriterMaker("es", NewESWriter)
RegisterWriterMaker("ess", newESWriter) RegisterWriterMaker("ess", NewESWriter)
var conf Config var conf Config
_ = config.Load(&conf, "log") _ = config.Load(&conf, "log")
@ -19,46 +19,3 @@ func init() {
func New(traceId string) *Logger { func New(traceId string) *Logger {
return DefaultLogger.New(traceId) return DefaultLogger.New(traceId)
} }
// SetDefaultName 设置全局默认应用名称,并同步更新 DefaultLogger
func SetDefaultName(name string) {
if name == "" {
return
}
globalDefaultName = name
if DefaultLogger != nil {
DefaultLogger.SetName(name)
}
}
// As 仿照 cast.As使用 DefaultLogger 记录错误并返回零值 (消除摩擦)
func As[T any](v T, err error) T {
if err != nil {
if DefaultLogger != nil {
DefaultLogger.Error(err.Error())
}
var zero T
return zero
}
return v
}
// Debug 记录一条调试级别日志
func Debug(message string, extra ...any) {
DefaultLogger.Debug(message, extra...)
}
// Info 记录一条信息级别日志
func Info(message string, extra ...any) {
DefaultLogger.Info(message, extra...)
}
// Warning 记录一条警告级别日志
func Warning(message string, extra ...any) {
DefaultLogger.Warning(message, extra...)
}
// Error 记录一条错误级别日志
func Error(message string, extra ...any) {
DefaultLogger.Error(message, extra...)
}

View File

@ -14,7 +14,7 @@ import (
"apigo.cc/go/cast" "apigo.cc/go/cast"
) )
type esWriter struct { type ESWriter struct {
config *Config config *Config
url string url string
user string user string
@ -27,8 +27,8 @@ type esWriter struct {
prefix string prefix string
} }
func newESWriter(conf *Config) Writer { func NewESWriter(conf *Config) Writer {
w := &esWriter{ w := &ESWriter{
config: conf, config: conf,
queue: make([]string, 0), queue: make([]string, 0),
client: &http.Client{}, client: &http.Client{},
@ -76,7 +76,10 @@ func newESWriter(conf *Config) Writer {
return w return w
} }
func (w *esWriter) Log(entry LogEntry, data []byte) { func (w *ESWriter) Log(entry LogEntry, data []byte) {
// data is array, but ES needs object
// convert entry to JSON object
// TODO: Consider desensitization here if needed, but for now ToJSONBytes
objBytes, err := cast.ToJSONBytes(entry) objBytes, err := cast.ToJSONBytes(entry)
if err != nil || len(objBytes) == 0 { if err != nil || len(objBytes) == 0 {
return return
@ -90,14 +93,14 @@ func (w *esWriter) Log(entry LogEntry, data []byte) {
var responseOkBytes = []byte("\"errors\":false") var responseOkBytes = []byte("\"errors\":false")
func (w *esWriter) Run() { func (w *ESWriter) Run() {
now := time.Now().Unix() now := time.Now().Unix()
w.lock.Lock() w.lock.Lock()
queueLen := len(w.queue) queueLen := len(w.queue)
w.lock.Unlock() w.lock.Unlock()
// 超过100条数据 或 过了1秒 发送数据 // 超过100条数据 或 过了1秒 发送数据
if queueLen > 100 || (queueLen > 0 && (now > w.last || !WriterService.Running.Load())) { if queueLen > 100 || (queueLen > 0 && (now > w.last || !writerRunning.Load())) {
w.lock.Lock() w.lock.Lock()
sendings := w.queue sendings := w.queue
w.queue = make([]string, 0) w.queue = make([]string, 0)

157
extra.go Normal file
View File

@ -0,0 +1,157 @@
package log
// import (
// "apigo.cc/go/cast"
// )
// type RequestLog struct {
// BaseLog
// ServerId string
// App string
// Node string
// ClientIp string
// FromApp string
// FromNode string
// UserId string
// DeviceId string
// ClientAppName string
// ClientAppVersion string
// SessionId string
// RequestId string
// Host string
// Scheme string
// Proto string
// AuthLevel int
// Priority int
// Method string
// Path string
// RequestHeaders map[string]string
// RequestData map[string]any
// UsedTime float32
// ResponseCode int
// ResponseHeaders map[string]string
// ResponseDataLength uint
// ResponseData string
// }
// func (logger *Logger) Request(
// method, path, host, scheme, proto string,
// clientIp, serverId, app, node string,
// fromApp, fromNode string,
// userId, deviceId, sessionId, requestId string,
// clientAppName, clientAppVersion string,
// authLevel, priority int,
// reqHeaders map[string]string,
// reqData map[string]any,
// responseCode int,
// usedTime float32,
// respHeaders map[string]string,
// responseData string,
// responseDataLength uint,
// extra ...any,
// ) {
// if !logger.CheckLevel(INFO) {
// return
// }
// entry := GetEntry[RequestLog]()
// logger.fillBase(entry, LogTypeRequest)
// // 暴力平铺赋值,性能极高
// entry.Method = method
// entry.Path = path
// entry.Host = host
// entry.Scheme = scheme
// entry.Proto = proto
// entry.ClientIp = clientIp
// entry.ServerId = serverId
// entry.App = app
// entry.Node = node
// entry.FromApp = fromApp
// entry.FromNode = fromNode
// entry.UserId = userId
// entry.DeviceId = deviceId
// entry.SessionId = sessionId
// entry.RequestId = requestId
// entry.ClientAppName = clientAppName
// entry.ClientAppVersion = clientAppVersion
// entry.AuthLevel = authLevel
// entry.Priority = priority
// entry.RequestHeaders = reqHeaders
// entry.RequestData = reqData
// entry.ResponseCode = responseCode
// entry.UsedTime = usedTime
// entry.ResponseHeaders = respHeaders
// entry.ResponseData = responseData
// entry.ResponseDataLength = responseDataLength
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// type TaskLog struct {
// BaseLog
// Task string
// UsedTime float32
// Success bool
// Message string
// }
// type MonitorLog struct {
// BaseLog
// Target string
// Status int
// Message string
// }
// type StatisticLog struct {
// BaseLog
// Category string
// Item string
// Value float64
// }
// func (logger *Logger) Task(taskName string, usedTime float32, success bool, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[TaskLog]()
// logger.fillBase(entry, LogTypeTask)
// entry.Task = taskName
// entry.UsedTime = usedTime
// entry.Success = success
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Monitor(target string, status int, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[MonitorLog]()
// logger.fillBase(entry, LogTypeMonitor)
// entry.Target = target
// entry.Status = status
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Statistic(category, item string, value float64, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[StatisticLog]()
// logger.fillBase(entry, LogTypeStatistic)
// entry.Category = category
// entry.Item = item
// entry.Value = value
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }

View File

@ -1,224 +0,0 @@
package log
// type RequestLog struct {
// BaseLog
// ServerId string `log:"pos:6"`
// App string `log:"pos:7"`
// Node string `log:"pos:8"`
// ClientIp string `log:"pos:9"`
// FromApp string `log:"pos:10"`
// FromNode string `log:"pos:11"`
// UserId string `log:"pos:12"`
// DeviceId string `log:"pos:13"`
// ClientAppName string `log:"pos:14"`
// ClientAppVersion string `log:"pos:15"`
// SessionId string `log:"pos:16"`
// RequestId string `log:"pos:17"`
// Host string `log:"pos:18"`
// Scheme string `log:"pos:19"`
// Proto string `log:"pos:20"`
// AuthLevel int `log:"pos:21"`
// Priority int `log:"pos:22"`
// Method string `log:"pos:23"`
// Path string `log:"pos:24"`
// RequestHeaders map[string]string `log:"pos:25"`
// RequestData map[string]any `log:"pos:26"`
// UsedTime float32 `log:"pos:27"`
// ResponseCode int `log:"pos:28"`
// ResponseHeaders map[string]string `log:"pos:29"`
// ResponseDataLength uint `log:"pos:30"`
// ResponseData string `log:"pos:31"`
// }
// func (l *RequestLog) Reset() {
// l.BaseLog.Reset()
// l.ServerId = ""
// l.App = ""
// l.Node = ""
// l.ClientIp = ""
// l.FromApp = ""
// l.FromNode = ""
// l.UserId = ""
// l.DeviceId = ""
// l.ClientAppName = ""
// l.ClientAppVersion = ""
// l.SessionId = ""
// l.RequestId = ""
// l.Host = ""
// l.Scheme = ""
// l.Proto = ""
// l.AuthLevel = 0
// l.Priority = 0
// l.Method = ""
// l.Path = ""
// if l.RequestHeaders == nil {
// l.RequestHeaders = make(map[string]string, 8)
// } else {
// clear(l.RequestHeaders)
// }
// if l.RequestData == nil {
// l.RequestData = make(map[string]any, 8)
// } else {
// clear(l.RequestData)
// }
// l.UsedTime = 0
// l.ResponseCode = 0
// if l.ResponseHeaders == nil {
// l.ResponseHeaders = make(map[string]string, 8)
// } else {
// clear(l.ResponseHeaders)
// }
// l.ResponseDataLength = 0
// l.ResponseData = ""
// }
// func (logger *Logger) Request(
// method, path, host, scheme, proto string,
// clientIp, serverId, app, node string,
// fromApp, fromNode string,
// userId, deviceId, sessionId, requestId string,
// clientAppName, clientAppVersion string,
// authLevel, priority int,
// reqHeaders map[string]string,
// reqData map[string]any,
// responseCode int,
// usedTime float32,
// respHeaders map[string]string,
// responseData string,
// responseDataLength uint,
// extra ...any,
// ) {
// if !logger.CheckLevel(INFO) {
// return
// }
// entry := GetEntry[RequestLog]()
// logger.FillBase(&entry.BaseLog, LogTypeRequest)
// // 暴力平铺赋值,性能极高
// entry.Method = method
// entry.Path = path
// entry.Host = host
// entry.Scheme = scheme
// entry.Proto = proto
// entry.ClientIp = clientIp
// entry.ServerId = serverId
// entry.App = app
// entry.Node = node
// entry.FromApp = fromApp
// entry.FromNode = fromNode
// entry.UserId = userId
// entry.DeviceId = deviceId
// entry.SessionId = sessionId
// entry.RequestId = requestId
// entry.ClientAppName = clientAppName
// entry.ClientAppVersion = clientAppVersion
// entry.AuthLevel = authLevel
// entry.Priority = priority
// entry.RequestHeaders = reqHeaders
// entry.RequestData = reqData
// entry.ResponseCode = responseCode
// entry.UsedTime = usedTime
// entry.ResponseHeaders = respHeaders
// entry.ResponseData = responseData
// entry.ResponseDataLength = responseDataLength
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// type TaskLog struct {
// BaseLog
// Task string `log:"pos:6"`
// UsedTime float32 `log:"pos:7"`
// Success bool `log:"pos:8"`
// Message string `log:"pos:9"`
// }
// func (l *TaskLog) Reset() {
// l.BaseLog.Reset()
// l.Task = ""
// l.UsedTime = 0
// l.Success = false
// l.Message = ""
// }
// type MonitorLog struct {
// BaseLog
// Target string `log:"pos:6"`
// Status int `log:"pos:7"`
// Message string `log:"pos:8"`
// }
// func (l *MonitorLog) Reset() {
// l.BaseLog.Reset()
// l.Target = ""
// l.Status = 0
// l.Message = ""
// }
// type StatisticLog struct {
// BaseLog
// Category string `log:"pos:6"`
// Item string `log:"pos:7"`
// Value float64 `log:"pos:8"`
// }
// func (l *StatisticLog) Reset() {
// l.BaseLog.Reset()
// l.Category = ""
// l.Item = ""
// l.Value = 0
// }
// func (logger *Logger) Task(taskName string, usedTime float32, success bool, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[TaskLog]()
// logger.FillBase(&entry.BaseLog, LogTypeTask)
// entry.Task = taskName
// entry.UsedTime = usedTime
// entry.Success = success
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Monitor(target string, status int, message string, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[MonitorLog]()
// logger.FillBase(&entry.BaseLog, LogTypeMonitor)
// entry.Target = target
// entry.Status = status
// entry.Message = message
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func (logger *Logger) Statistic(category, item string, value float64, extra ...any) {
// if logger.CheckLevel(INFO) {
// entry := GetEntry[StatisticLog]()
// logger.FillBase(&entry.BaseLog, LogTypeStatistic)
// entry.Category = category
// entry.Item = item
// entry.Value = value
// if len(extra) > 0 {
// cast.FillMap(&entry.Extra, extra)
// }
// logger.Log(entry)
// }
// }
// func init() {
// RegisterType(LogTypeRequest, &RequestLog{})
// RegisterType(LogTypeTask, &TaskLog{})
// RegisterType(LogTypeMonitor, &MonitorLog{})
// RegisterType(LogTypeStatistic, &StatisticLog{})
// }

View File

@ -4,6 +4,7 @@ import (
"bufio" "bufio"
"fmt" "fmt"
"os" "os"
"sync"
"time" "time"
) )
@ -16,6 +17,11 @@ type FileWriter struct {
bufWriter *bufio.Writer bufWriter *bufio.Writer
} }
var (
files = make(map[string]*FileWriter)
filesLock sync.RWMutex
)
// Write 由外层的 writerRunner 单协程调用,绝对并发安全,无需加锁 // Write 由外层的 writerRunner 单协程调用,绝对并发安全,无需加锁
func (f *FileWriter) Write(tm time.Time, data []byte) { func (f *FileWriter) Write(tm time.Time, data []byte) {
nowSplit := tm.Format(f.splitTag) nowSplit := tm.Format(f.splitTag)

View File

@ -1,136 +0,0 @@
package log_test
import (
"strings"
"testing"
"time"
"apigo.cc/go/file"
"apigo.cc/go/log"
)
func TestSplitTag(t *testing.T) {
logFile := "test_rotate.log"
// 使用每秒切分的标签,方便测试文件轮转
splitTag := ".20060102150405"
conf := log.Config{
Name: "test-split",
Level: "info",
File: logFile,
SplitTag: splitTag,
}
log.WriterService.Start(nil, nil)
logger := log.NewLogger(conf)
// 1. 记录第一条日志
t1 := time.Now()
logger.Info("first message")
time.Sleep(300 * time.Millisecond) // 等待异步写入
expectedFile1 := logFile + "." + t1.Format(splitTag)
if !file.Exists(expectedFile1) {
// 可能在写入时秒数刚好进位
expectedFile1 = logFile + "." + time.Now().Format(splitTag)
if !file.Exists(expectedFile1) {
t.Fatalf("First log file does not exist (checked both possible time slots)")
}
}
// 2. 等待跨秒,确保下次写入肯定会触发轮转
time.Sleep(1200 * time.Millisecond)
// 3. 记录第二条日志,触发轮转
t2 := time.Now()
logger.Info("second message")
time.Sleep(300 * time.Millisecond) // 等待异步写入
expectedFile2 := logFile + "." + t2.Format(splitTag)
if !file.Exists(expectedFile2) {
expectedFile2 = logFile + "." + time.Now().Format(splitTag)
if !file.Exists(expectedFile2) {
t.Fatalf("Second log file does not exist after rotation")
}
}
if expectedFile1 == expectedFile2 {
t.Errorf("Files should be different for rotation, but both are %s", expectedFile1)
}
// 清理
file.Remove(expectedFile1)
file.Remove(expectedFile2)
}
func TestSensitiveDetailed(t *testing.T) {
type SecretLog struct {
log.BaseLog
Password string
SecretKey string
SafeData string
}
entry := log.GetEntry[SecretLog]()
entry.BaseLog.LogType = "secret"
entry.Password = "my_password"
entry.SecretKey = "super_secret"
entry.SafeData = "hello"
// 直接测试 ToArrayBytes
// 注意passed to ToArrayBytes 的 keys 应该是已经过 fixField 处理的
sensitiveKeys := []string{"password", "secretkey"}
buf := log.Marshal(entry, sensitiveKeys)
result := string(buf)
if strings.Contains(result, "my_password") {
t.Errorf("Sensitive data 'my_password' not masked in: %s", result)
}
if strings.Contains(result, "super_secret") {
t.Errorf("Sensitive data 'super_secret' not masked in: %s", result)
}
if !strings.Contains(result, "hello") {
t.Errorf("Safe data 'hello' should be present in: %s", result)
}
}
func TestDeepDesensitization(t *testing.T) {
type Nested struct {
Password string
Token string
}
type DeepLog struct {
log.BaseLog
Data map[string]any
User Nested
}
entry := log.GetEntry[DeepLog]()
entry.BaseLog.LogType = "deep"
entry.Data = map[string]any{
"password": "data_password",
"safe": "data_safe",
}
entry.User = Nested{
Password: "user_password",
Token: "user_token",
}
sensitiveKeys := []string{"password", "token"}
buf := log.Marshal(entry, sensitiveKeys)
result := string(buf)
// Check deep desensitization in map
if strings.Contains(result, "data_password") {
t.Errorf("Nested map data 'data_password' not masked in: %s", result)
}
// Check deep desensitization in struct
if strings.Contains(result, "user_password") {
t.Errorf("Nested struct data 'user_password' not masked in: %s", result)
}
if strings.Contains(result, "user_token") {
t.Errorf("Nested struct data 'user_token' not masked in: %s", result)
}
if !strings.Contains(result, "data_safe") {
t.Errorf("Safe data 'data_safe' should be present in: %s", result)
}
}

33
go.mod
View File

@ -3,18 +3,31 @@ module apigo.cc/go/log
go 1.25.0 go 1.25.0
require ( require (
apigo.cc/go/cast v1.3.3 apigo.cc/go/cast v1.2.7
apigo.cc/go/config v1.3.1 apigo.cc/go/config v1.0.6
apigo.cc/go/file v1.3.2 apigo.cc/go/shell v1.0.5
apigo.cc/go/id v1.3.1
apigo.cc/go/shell v1.3.1
) )
require ( require (
apigo.cc/go/encoding v1.3.1 // indirect apigo.cc/go/encoding v1.0.5 // indirect
apigo.cc/go/rand v1.3.1 // indirect apigo.cc/go/file v1.0.6 // indirect
apigo.cc/go/safe v1.3.1 // indirect apigo.cc/go/rand v1.0.5 // indirect
golang.org/x/crypto v0.51.0 // indirect apigo.cc/go/safe v1.0.5 // indirect
golang.org/x/sys v0.44.0 // indirect golang.org/x/crypto v0.50.0 // indirect
golang.org/x/sys v0.43.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect
) )
replace apigo.cc/go/cast => ../cast
replace apigo.cc/go/config => ../config
replace apigo.cc/go/shell => ../shell
replace apigo.cc/go/file => ../file
replace apigo.cc/go/encoding => ../encoding
replace apigo.cc/go/safe => ../safe
replace apigo.cc/go/rand => ../rand

24
go.sum
View File

@ -1,29 +1,13 @@
apigo.cc/go/cast v1.3.3 h1:aln5eDR5DZVWVzZ/y5SJh1gQNgWv2sT82I25NaO9g34=
apigo.cc/go/cast v1.3.3/go.mod h1:lGlwImiOvHxG7buyMWhFzcdvQzmSaoKbmr7bcDfUpHk=
apigo.cc/go/config v1.3.1 h1:wZzUh4oL+fGD6SayVgX6prLPMsniM25etWFcEH8XzIE=
apigo.cc/go/config v1.3.1/go.mod h1:7KHz/1WmtBLM762Lln/TaXh2dmlMvJTLhnlk33zbS3U=
apigo.cc/go/encoding v1.3.1 h1:y8O58KYAyulkThg1O2ji2BqjnFoSvk42sit9I3z+K7Y=
apigo.cc/go/encoding v1.3.1/go.mod h1:xAJk5b83VZ31mXMTnyp0dfMoBKfT/AHDn0u+cQfojgY=
apigo.cc/go/file v1.3.2 h1:pu4oiDyiqgj3/eykfnJf+/6+A9v/Z0b3ClP5XK+lwG4=
apigo.cc/go/file v1.3.2/go.mod h1:vci4h0Pz94mV6dkniQkuyBYERVYeq7/LX4jJVuCg9hs=
apigo.cc/go/id v1.3.1 h1:pkqi6VeWyQoHuIu0Zbx/RRxIAdM61Js0j6cY1M9XVCk=
apigo.cc/go/id v1.3.1/go.mod h1:P2/vl3tyW3US+ayOFSMoPIOCulNLBngNYPhXJC/Z7J4=
apigo.cc/go/rand v1.3.1 h1:7FvsI6PtQ5XrWER0dTiLVo0p7GIxRidT/TBKhVy93j8=
apigo.cc/go/rand v1.3.1/go.mod h1:mZ/4Soa3bk+XvDaqPWJuUe1bfEi4eThBj1XmEAuYxsk=
apigo.cc/go/safe v1.3.1 h1:irTCqPAC97gGsX/Lw5AzLelDt1xXLEZIAaVhLELWe9Q=
apigo.cc/go/safe v1.3.1/go.mod h1:XdOpBhN2vkImalaykYXXmEpczqWa1y3ah6/Q72cdRqE=
apigo.cc/go/shell v1.3.1 h1:M8oD0b2HcJuCC6frQFx11b3UTcTx3lATX8XK+YXSVm8=
apigo.cc/go/shell v1.3.1/go.mod h1:ZMdJjpCpWdvsHKUXlelh/AxsV/nWdkH/k3lISfzMdUw=
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0= github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk= github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI= golang.org/x/crypto v0.50.0 h1:zO47/JPrL6vsNkINmLoo/PH1gcxpls50DNogFvB5ZGI=
golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8= golang.org/x/crypto v0.50.0/go.mod h1:3muZ7vA7PBCE6xgPX7nkzzjiUq87kRItoJQM1Yo8S+Q=
golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ= golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=

View File

@ -1,7 +1,6 @@
package log_test package log_test
import ( import (
"strconv"
"testing" "testing"
"apigo.cc/go/log" "apigo.cc/go/log"
@ -48,7 +47,7 @@ func TestDesensitization(t *testing.T) {
} }
entry := log.GetEntry[MyLog]() entry := log.GetEntry[MyLog]()
logger.FillBase(&entry.BaseLog, "test") logger.FillBase(entry, "test")
entry.Phone = "13812345678" entry.Phone = "13812345678"
logger.Log(entry) // 应该在输出中脱敏 logger.Log(entry) // 应该在输出中脱敏
} }
@ -59,7 +58,7 @@ func TestDBLog(t *testing.T) {
}) })
entry := log.GetEntry[DBEntry]() entry := log.GetEntry[DBEntry]()
logger.FillBase(&entry.BaseLog, "db") logger.FillBase(entry, "db")
entry.DbType = "mysql" entry.DbType = "mysql"
entry.Query = "SELECT * FROM users" entry.Query = "SELECT * FROM users"
entry.UsedTime = 10.5 entry.UsedTime = 10.5
@ -72,7 +71,7 @@ func TestRequestLog(t *testing.T) {
}) })
entry := log.GetEntry[RequestEntry]() entry := log.GetEntry[RequestEntry]()
logger.FillBase(&entry.BaseLog, "request") logger.FillBase(entry, "request")
entry.Method = "GET" entry.Method = "GET"
entry.Path = "/api/user" entry.Path = "/api/user"
entry.ResponseCode = 200 entry.ResponseCode = 200
@ -87,28 +86,3 @@ func TestExtraLogs(t *testing.T) {
logger.Info("Extra log test", "key", "value") logger.Info("Extra log test", "key", "value")
} }
func TestAs(t *testing.T) {
// 1. 测试 log.As (使用 DefaultLogger)
val1 := log.As(strconv.Atoi("123"))
if val1 != 123 {
t.Errorf("log.As expected 123, got %v", val1)
}
val2 := log.As(strconv.Atoi("abc"))
if val2 != 0 {
t.Errorf("log.As expected 0, got %v", val2)
}
// 2. 测试 logger.As (方法)
logger := log.NewLogger(log.Config{Level: "debug"})
val3 := logger.As(strconv.Atoi("456")).(int)
if val3 != 456 {
t.Errorf("logger.As expected 456, got %v", val3)
}
val4 := logger.As(strconv.Atoi("def")).(int)
if val4 != 0 {
t.Errorf("logger.As expected 0, got %v", val4)
}
}

123
logger.go
View File

@ -4,23 +4,32 @@ import (
"fmt" "fmt"
"log" "log"
"os" "os"
"regexp"
"strings" "strings"
"time" "time"
"apigo.cc/go/cast" "apigo.cc/go/cast"
"apigo.cc/go/id"
) )
type Logger struct { type Logger struct {
config Config config Config
level LevelType level LevelType
goLogger *log.Logger goLogger *log.Logger
file *FileWriter file *FileWriter
writer Writer writer Writer
truncations []string truncations []string
sensitive map[string]bool sensitive map[string]bool
sensitiveKeys []string sensitiveKeys []string
traceId string regexSensitive []*regexp.Regexp
sensitiveRule []sensitiveRuleInfo
desensitization func(string) string
traceId string
}
type sensitiveRuleInfo struct {
threshold int
leftNum int
rightNum int
} }
var ( var (
@ -41,14 +50,16 @@ func NewLogger(conf Config) *Logger {
if conf.Sensitive == "" { if conf.Sensitive == "" {
conf.Sensitive = LogDefaultSensitive conf.Sensitive = LogDefaultSensitive
} }
if conf.SensitiveRule == "" {
conf.SensitiveRule = "12:4*4, 11:3*4, 7:2*2, 3:1*1, 2:1*0"
}
if conf.Name == "" { if conf.Name == "" {
conf.Name = getDefaultName() conf.Name = GetDefaultName()
} }
logger := Logger{ logger := Logger{
truncations: cast.Split(conf.Truncations, ","), truncations: cast.Split(conf.Truncations, ","),
traceId: id.Get10Bytes14MPerSecond(),
} }
if len(conf.Sensitive) > 0 { if len(conf.Sensitive) > 0 {
@ -61,6 +72,37 @@ func NewLogger(conf Config) *Logger {
} }
} }
if len(conf.RegexSensitive) > 0 {
ss := cast.Split(conf.RegexSensitive, ",")
for _, v := range ss {
if r, err := regexp.Compile(v); err == nil {
logger.regexSensitive = append(logger.regexSensitive, r)
}
}
}
if len(conf.SensitiveRule) > 0 {
ss := cast.Split(conf.SensitiveRule, ",")
for _, v := range ss {
a1 := strings.SplitN(v, ":", 2)
if len(a1) == 2 {
a2 := strings.SplitN(a1[1], "*", 3)
if len(a2) == 2 {
threshold := cast.Int(a1[0])
leftNum := cast.Int(a2[0])
rightNum := cast.Int(a2[1])
if threshold >= 0 && threshold <= 100 && leftNum >= 0 && leftNum <= 100 && rightNum >= 0 && rightNum <= 100 {
logger.sensitiveRule = append(logger.sensitiveRule, sensitiveRuleInfo{
threshold: threshold,
leftNum: leftNum,
rightNum: rightNum,
})
}
}
}
}
}
switch strings.ToLower(conf.Level) { switch strings.ToLower(conf.Level) {
case "debug": case "debug":
logger.level = DEBUG logger.level = DEBUG
@ -78,27 +120,29 @@ func NewLogger(conf Config) *Logger {
if m, ok := writerMakers[writerName]; ok { if m, ok := writerMakers[writerName]; ok {
if w := m(&conf); w != nil { if w := m(&conf); w != nil {
logger.writer = w logger.writer = w
WriterService.WriterLock.Lock() writerLock.Lock()
cur := WriterService.Writers.Load().([]Writer) cur := writers.Load().([]Writer)
newW := append(cur, w) newW := append(cur, w)
WriterService.Writers.Store(newW) writers.Store(newW)
WriterService.WriterLock.Unlock() writerLock.Unlock()
Start()
} }
} }
} else { } else {
if conf.SplitTag != "" { if conf.SplitTag != "" {
WriterService.FilesLock.RLock() filesLock.RLock()
logger.file = WriterService.Files[conf.File+conf.SplitTag] logger.file = files[conf.File+conf.SplitTag]
WriterService.FilesLock.RUnlock() filesLock.RUnlock()
if logger.file == nil { if logger.file == nil {
logger.file = &FileWriter{ logger.file = &FileWriter{
fileName: conf.File, fileName: conf.File,
splitTag: conf.SplitTag, splitTag: conf.SplitTag,
} }
WriterService.FilesLock.Lock() filesLock.Lock()
WriterService.Files[conf.File+conf.SplitTag] = logger.file files[conf.File+conf.SplitTag] = logger.file
WriterService.FilesLock.Unlock() filesLock.Unlock()
} }
Start()
} else { } else {
fp, err := os.OpenFile(conf.File, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644) fp, err := os.OpenFile(conf.File, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
if err == nil { if err == nil {
@ -116,14 +160,14 @@ func (logger *Logger) Log(entry LogEntry) {
} }
func (logger *Logger) asyncWrite(entry LogEntry) { func (logger *Logger) asyncWrite(entry LogEntry) {
buf := Marshal(entry, logger.sensitiveKeys) buf := ToArrayBytes(entry, logger.sensitiveKeys)
logger.writeBuf(entry, buf) logger.writeBuf(entry, buf)
putEntry(entry) PutEntry(entry)
} }
func (logger *Logger) writeBuf(entry LogEntry, buf []byte) { func (logger *Logger) writeBuf(entry LogEntry, buf []byte) {
if WriterService.Running.Load() { if writerRunning.Load() {
writeAsync(logPayload{ WriteAsync(logPayload{
entry: entry, entry: entry,
buf: buf, buf: buf,
writer: logger.writer, writer: logger.writer,
@ -143,7 +187,8 @@ func (logger *Logger) writeBuf(entry LogEntry, buf []byte) {
} }
} }
func (logger *Logger) FillBase(base *BaseLog, logType string) { func (logger *Logger) FillBase(entry LogEntry, logType string) {
base := entry.GetBaseLog()
if base == nil { if base == nil {
return return
} }
@ -167,23 +212,23 @@ func (logger *Logger) FillBase(base *BaseLog, logType string) {
} }
func (logger *Logger) FillDebug(entry *DebugLog, message string) { func (logger *Logger) FillDebug(entry *DebugLog, message string) {
logger.FillBase(&entry.BaseLog, LogTypeDebug) logger.FillBase(entry, LogTypeDebug)
entry.Debug = message entry.Debug = message
} }
func (logger *Logger) FillInfo(entry *InfoLog, message string) { func (logger *Logger) FillInfo(entry *InfoLog, message string) {
logger.FillBase(&entry.BaseLog, LogTypeInfo) logger.FillBase(entry, LogTypeInfo)
entry.Info = message entry.Info = message
} }
func (logger *Logger) FillWarning(entry *WarningLog, message string) { func (logger *Logger) FillWarning(entry *WarningLog, message string) {
logger.FillBase(&entry.BaseLog, LogTypeWarning) logger.FillBase(entry, LogTypeWarning)
entry.Warning = message entry.Warning = message
entry.CallStacks = getCallStacks(logger.truncations) entry.CallStacks = getCallStacks(logger.truncations)
} }
func (logger *Logger) FillError(entry *ErrorLog, message string) { func (logger *Logger) FillError(entry *ErrorLog, message string) {
logger.FillBase(&entry.BaseLog, LogTypeError) logger.FillBase(entry, LogTypeError)
entry.Error = message entry.Error = message
entry.CallStacks = getCallStacks(logger.truncations) entry.CallStacks = getCallStacks(logger.truncations)
} }
@ -236,14 +281,14 @@ func (logger *Logger) Error(message string, extra ...any) {
} }
} }
func (logger *Logger) SetName(name string) {
logger.config.Name = name
}
func (logger *Logger) SetLevel(level LevelType) { func (logger *Logger) SetLevel(level LevelType) {
logger.level = level logger.level = level
} }
func (logger *Logger) SetDesensitization(f func(v string) string) {
logger.desensitization = f
}
func (logger *Logger) New(traceId string) *Logger { func (logger *Logger) New(traceId string) *Logger {
newLogger := *logger newLogger := *logger
newLogger.traceId = traceId newLogger.traceId = traceId
@ -254,14 +299,6 @@ func (logger *Logger) GetTraceId() string {
return logger.traceId return logger.traceId
} }
// As 仿照 cast.As忽略错误并返回零值但会将错误记录到日志中 (消除摩擦)
func (logger *Logger) As(v any, err error) any {
if err != nil {
logger.Error(err.Error())
}
return v
}
func (logger *Logger) CheckLevel(logLevel LevelType) bool { func (logger *Logger) CheckLevel(logLevel LevelType) bool {
settedLevel := logger.level settedLevel := logger.level
if settedLevel == 0 { if settedLevel == 0 {

View File

@ -2,83 +2,36 @@ package main
import ( import (
"bufio" "bufio"
"flag"
"fmt" "fmt"
"io"
"os" "os"
"apigo.cc/go/log" "apigo.cc/go/log"
) )
func main() { func main() {
jsonMode := flag.Bool("json", false, "output in JSON format") // Ensure built-in types are registered to get basic meta if .log.meta.json is missing
helpMode := flag.Bool("h", false, "show help") // log package init() handles most of it, but we can also just run it.
flag.BoolVar(helpMode, "help", false, "show help")
flag.Usage = func() { // Reading from stdin
fmt.Fprintf(os.Stderr, "Usage: logv [options] [file1 file2 ...]\n") scanner := bufio.NewScanner(os.Stdin)
fmt.Fprintf(os.Stderr, "Options:\n")
flag.PrintDefaults()
fmt.Fprintf(os.Stderr, "\nExamples:\n")
fmt.Fprintf(os.Stderr, " 1. 彩色可视化实时日志:\n")
fmt.Fprintf(os.Stderr, " tail -f app.log | logv\n")
fmt.Fprintf(os.Stderr, " 2. 格式化查看历史日志文件:\n")
fmt.Fprintf(os.Stderr, " logv error.log\n")
fmt.Fprintf(os.Stderr, " 3. 还原为标准 JSON 格式 (供 Filebeat / Logstash 收集):\n")
fmt.Fprintf(os.Stderr, " tail -f app.log | logv -json\n")
fmt.Fprintf(os.Stderr, " 4. 批量转换日志文件为 JSON:\n")
fmt.Fprintf(os.Stderr, " logv -json app.log.2026* > all_logs.json\n")
}
flag.Parse() // Optional: Adjust max token size if log lines are extremely long
// buf := make([]byte, 0, 64*1024)
// scanner.Buffer(buf, 1024*1024)
if *helpMode {
flag.Usage()
return
}
// Try to load meta file from current directory
_ = log.LoadMeta(".log.meta.json")
args := flag.Args()
if len(args) == 0 {
// Check if stdin is a terminal
stat, _ := os.Stdin.Stat()
if (stat.Mode() & os.ModeCharDevice) != 0 {
// Stdin is a terminal and no files provided, show usage
flag.Usage()
return
}
process(os.Stdin, *jsonMode)
} else {
for _, arg := range args {
f, err := os.Open(arg)
if err != nil {
fmt.Fprintf(os.Stderr, "logv: %v\n", err)
continue
}
process(f, *jsonMode)
f.Close()
}
}
}
func process(r io.Reader, jsonMode bool) {
scanner := bufio.NewScanner(r)
for scanner.Scan() { for scanner.Scan() {
line := scanner.Text() line := scanner.Text()
if len(line) == 0 { if len(line) == 0 {
continue continue
} }
if jsonMode { // Render and print the log line
fmt.Println(log.ToJSON(line)) rendered := log.Viewable(line)
} else { fmt.Println(rendered)
fmt.Println(log.Viewable(line))
}
} }
if err := scanner.Err(); err != nil { if err := scanner.Err(); err != nil {
fmt.Fprintf(os.Stderr, "logv: error reading input: %v\n", err) fmt.Fprintf(os.Stderr, "logv: error reading standard input: %v\n", err)
os.Exit(1)
} }
} }

176
meta.go
View File

@ -1,26 +1,23 @@
package log package log
import ( import (
"encoding/json"
"os"
"reflect" "reflect"
"sort" "sort"
"strconv"
"strings" "strings"
"sync" "sync"
"apigo.cc/go/cast"
"apigo.cc/go/file"
) )
// MetaField describes the serialization and visualization metadata for a single log field. // MetaField describes the serialization and visualization metadata for a single log field.
type MetaField struct { type MetaField struct {
Index int Index int `json:"index"`
Name string Name string `json:"name"`
KeyName string Color string `json:"color,omitempty"`
AttachBefore bool Format string `json:"format,omitempty"`
Color string WithoutKey bool `json:"withoutKey,omitempty"`
Format string Hide bool `json:"hide,omitempty"`
Precision int
WithoutKey bool
Hide bool
} }
var ( var (
@ -29,36 +26,9 @@ var (
metaFilePath = ".log.meta.json" metaFilePath = ".log.meta.json"
) )
// LoadMeta loads metadata from the specified file into the global registry.
func LoadMeta(path string) error {
metaLock.Lock()
defer metaLock.Unlock()
return file.UnmarshalFile(path, &metaRegistry)
}
// RegisterType registers a log model's metadata into the global registry. // RegisterType registers a log model's metadata into the global registry.
// logType is the string identifier (e.g. "info", "error"). // logType is the string identifier (e.g. "info", "error").
func RegisterType(logType string, model any) { func RegisterType(logType string, model any) {
t := reflect.TypeOf(model)
if t.Kind() == reflect.Ptr {
t = t.Elem()
}
// 强制检查 Reset 方法是否被显式实现(防止继承 BaseLog 后忘记重置业务字段)
if t.Kind() == reflect.Struct {
ptrType := reflect.PointerTo(t)
method, ok := ptrType.MethodByName("Reset")
if !ok {
panic("log model must implement Reset() method: " + t.Name())
}
// 检查该方法是否属于当前类型(而不是继承自 BaseLog 且没有被重写)
baseResetMethod, _ := reflect.PointerTo(reflect.TypeOf(BaseLog{})).MethodByName("Reset")
if method.Func.Pointer() == baseResetMethod.Func.Pointer() {
panic("log model must override Reset() method to clear its own fields: " + t.Name())
}
}
fields := extractMetaFields(model) fields := extractMetaFields(model)
metaLock.Lock() metaLock.Lock()
@ -75,7 +45,7 @@ func GetMeta(logType string) []MetaField {
return metaRegistry[logType] return metaRegistry[logType]
} }
// fieldInfo is used internally for storing fields with their absolute position. // fieldInfo is used internally for sorting fields before flattening.
type fieldInfo struct { type fieldInfo struct {
field reflect.StructField field reflect.StructField
pos int pos int
@ -91,67 +61,51 @@ func extractMetaFields(model any) []MetaField {
return nil return nil
} }
var flatFields []fieldInfo var flatFields []reflect.StructField
flattenStructFields(t, &flatFields, nil) flattenStructFields(t, &flatFields, nil)
// Determine final indices var metaFields []MetaField
maxLiteralPos := -1 var extraField *reflect.StructField
var highPosFields []fieldInfo var callStacksField *reflect.StructField
// Process fields, separating Extra and CallStacks
var regularFields []reflect.StructField
for _, f := range flatFields { for _, f := range flatFields {
if f.pos < 1000 { if f.Name == "Extra" {
if f.pos > maxLiteralPos { extraField = &f
maxLiteralPos = f.pos continue
}
} else {
highPosFields = append(highPosFields, f)
} }
} if f.Name == "CallStacks" {
callStacksField = &f
// Sort high pos fields by their pos continue
sort.Slice(highPosFields, func(i, j int) bool {
return highPosFields[i].pos < highPosFields[j].pos
})
// Assign real indices to high pos fields
finalPosMap := make(map[string]int)
for _, f := range flatFields {
if f.pos < 1000 {
finalPosMap[f.field.Name] = f.pos
} }
} regularFields = append(regularFields, f)
nextPos := maxLiteralPos + 1
for _, f := range highPosFields {
finalPosMap[f.field.Name] = nextPos
nextPos++
} }
maxPos := nextPos - 1 // Reassemble: regular fields -> CallStacks -> Extra
metaFields := make([]MetaField, maxPos+1) var finalFields []reflect.StructField
// Initialize with empty MetaFields having Index set finalFields = append(finalFields, regularFields...)
for i := range metaFields { if callStacksField != nil {
metaFields[i] = MetaField{Index: i} finalFields = append(finalFields, *callStacksField)
}
if extraField != nil {
finalFields = append(finalFields, *extraField)
} }
for _, f := range flatFields { for i, f := range finalFields {
tag := f.field.Tag.Get("log") tag := f.Tag.Get("log")
if tag == "-" { if tag == "-" {
continue continue
} }
realPos := finalPosMap[f.field.Name]
meta := MetaField{ meta := MetaField{
Index: realPos, Index: i,
Name: f.field.Name, Name: f.Name,
} }
if tag != "" { if tag != "" {
parts := strings.Split(tag, ",") parts := strings.Split(tag, ",")
for _, part := range parts { for _, part := range parts {
part = strings.TrimSpace(part)
if part == "attachBefore" {
meta.AttachBefore = true
continue
}
kv := strings.SplitN(part, ":", 2) kv := strings.SplitN(part, ":", 2)
if len(kv) == 2 { if len(kv) == 2 {
key := strings.TrimSpace(kv[0]) key := strings.TrimSpace(kv[0])
@ -165,43 +119,44 @@ func extractMetaFields(model any) []MetaField {
meta.WithoutKey = (val == "true") meta.WithoutKey = (val == "true")
case "hide": case "hide":
meta.Hide = (val == "true") meta.Hide = (val == "true")
case "keyname":
meta.KeyName = val
case "attachBefore":
meta.AttachBefore = (val == "true")
case "precision":
meta.Precision = cast.To[int](val)
} }
} }
} }
} }
// Apply some default visual rules if not specified // Apply some default visual rules if not specified
// LogType shouldn't show the key in standard console // LogType shouldn't show the key in standard console
if f.field.Name == "LogType" && meta.Color == "" { if f.Name == "LogType" && meta.Color == "" {
meta.WithoutKey = true meta.WithoutKey = true
} }
metaFields[realPos] = meta metaFields = append(metaFields, meta)
} }
return metaFields return metaFields
} }
func flattenStructFields(t reflect.Type, result *[]fieldInfo, parentIndex []int) { func flattenStructFields(t reflect.Type, result *[]reflect.StructField, parentIndex []int) {
var infos []fieldInfo
for i := 0; i < t.NumField(); i++ { for i := 0; i < t.NumField(); i++ {
f := t.Field(i) f := t.Field(i)
if !f.IsExported() && !f.Anonymous { if !f.IsExported() && !f.Anonymous {
continue continue
} }
pos := 10 + i // default position if not specified isEmbeddedStruct := f.Anonymous && f.Type.Kind() == reflect.Struct
pos := 1000 + i // default position if not specified
if isEmbeddedStruct {
pos = i - 1000 // default to top priority for embedded structs
}
tag := f.Tag.Get("log") tag := f.Tag.Get("log")
if tag != "" { if tag != "" {
parts := strings.Split(tag, ",") parts := strings.Split(tag, ",")
for _, part := range parts { for _, part := range parts {
kv := strings.SplitN(part, ":", 2) kv := strings.SplitN(part, ":", 2)
if len(kv) == 2 && strings.TrimSpace(kv[0]) == "pos" { if len(kv) == 2 && strings.TrimSpace(kv[0]) == "pos" {
if p := cast.To[int](strings.TrimSpace(kv[1])); p >= 0 { if p, err := strconv.Atoi(strings.TrimSpace(kv[1])); err == nil {
pos = p pos = p
} }
} }
@ -214,21 +169,40 @@ func flattenStructFields(t reflect.Type, result *[]fieldInfo, parentIndex []int)
fullIndex = append(fullIndex, i) fullIndex = append(fullIndex, i)
f.Index = fullIndex f.Index = fullIndex
if f.Anonymous && f.Type.Kind() == reflect.Struct { infos = append(infos, fieldInfo{
flattenStructFields(f.Type, result, f.Index) field: f,
pos: pos,
})
}
// Sort fields in the current struct level by pos
sort.Slice(infos, func(i, j int) bool {
return infos[i].pos < infos[j].pos
})
for _, info := range infos {
if info.field.Anonymous && info.field.Type.Kind() == reflect.Struct {
// Embedded struct, extract its fields first (parent first)
flattenStructFields(info.field.Type, result, info.field.Index)
} else { } else {
*result = append(*result, fieldInfo{ *result = append(*result, info.field)
field: f,
pos: pos,
})
} }
} }
} }
func syncMetaFile() { func syncMetaFile() {
metaLock.RLock() metaLock.RLock()
defer metaLock.RUnlock() data, err := json.MarshalIndent(metaRegistry, "", " ")
_ = file.MarshalFilePretty(metaFilePath, metaRegistry) metaLock.RUnlock()
if err != nil {
return
}
// Determine the path. If running in tests or from another dir, it might be better
// to allow setting the meta file path, but for now we write to current working dir.
// You could also write to executable dir.
_ = os.WriteFile(metaFilePath, append(data, '\n'), 0644)
} }
// SetMetaFilePath allows changing the path for testing or configuration purposes // SetMetaFilePath allows changing the path for testing or configuration purposes

View File

@ -7,42 +7,21 @@ import (
) )
type MockBaseLog struct { type MockBaseLog struct {
BaseField1 string `log:"pos:0,color:red"` BaseField1 string `log:"pos:1,color:red"`
BaseField2 int `log:"pos:1,withoutkey:true"` BaseField2 int `log:"pos:2,withoutkey:true"`
} }
func (b *MockBaseLog) Reset() {
b.BaseField1 = ""
b.BaseField2 = 0
}
func (b *MockBaseLog) IsLogEntry() bool { return true }
func (b *MockBaseLog) GetBaseLog() *BaseLog { return &BaseLog{} }
type MockInfoLog struct { type MockInfoLog struct {
MockBaseLog MockBaseLog
Message string `log:"pos:2"` Message string `log:"pos:3"`
Extra map[string]any `log:"pos:1000"` Extra map[string]any
}
func (l *MockInfoLog) Reset() {
l.MockBaseLog.Reset()
l.Message = ""
clear(l.Extra)
} }
type MockErrorLog struct { type MockErrorLog struct {
MockBaseLog MockBaseLog
Error string `log:"pos:2,color:red"` Error string `log:"pos:3,color:red"`
CallStacks []string `log:"pos:1001"` CallStacks []string
Extra map[string]any `log:"pos:1000"` Extra map[string]any
}
func (l *MockErrorLog) Reset() {
l.MockBaseLog.Reset()
l.Error = ""
l.CallStacks = l.CallStacks[:0]
clear(l.Extra)
} }
func TestMetaExtraction(t *testing.T) { func TestMetaExtraction(t *testing.T) {
@ -54,38 +33,36 @@ func TestMetaExtraction(t *testing.T) {
RegisterType("mock_error", MockErrorLog{}) RegisterType("mock_error", MockErrorLog{})
infoMeta := GetMeta("mock_info") infoMeta := GetMeta("mock_info")
// Index 0, 1, 2 are used, Extra gets max(2)+1=3. Total size 4. if len(infoMeta) != 4 { // BaseField1, BaseField2, Message, Extra
if len(infoMeta) != 4 {
t.Fatalf("expected 4 fields for mock_info, got %d", len(infoMeta)) t.Fatalf("expected 4 fields for mock_info, got %d", len(infoMeta))
} }
if infoMeta[0].Name != "BaseField1" || infoMeta[0].Color != "red" { if infoMeta[0].Name != "BaseField1" || infoMeta[0].Color != "red" {
t.Errorf("unexpected meta for BaseField1 at index 0: %+v", infoMeta[0]) t.Errorf("unexpected meta for BaseField1: %+v", infoMeta[0])
} }
if infoMeta[1].Name != "BaseField2" || infoMeta[1].WithoutKey != true { if infoMeta[1].Name != "BaseField2" || infoMeta[1].WithoutKey != true {
t.Errorf("unexpected meta for BaseField2 at index 1: %+v", infoMeta[1]) t.Errorf("unexpected meta for BaseField2: %+v", infoMeta[1])
} }
if infoMeta[2].Name != "Message" { if infoMeta[2].Name != "Message" {
t.Errorf("unexpected meta for Message at index 2: %+v", infoMeta[2]) t.Errorf("unexpected meta for Message: %+v", infoMeta[2])
} }
if infoMeta[3].Name != "Extra" { if infoMeta[3].Name != "Extra" {
t.Errorf("unexpected meta for Extra at index 3: %+v", infoMeta[3]) t.Errorf("unexpected meta for Extra: %+v", infoMeta[3])
} }
errorMeta := GetMeta("mock_error") errorMeta := GetMeta("mock_error")
// Indices: 0, 1, 2, Extra(3), CallStacks(4). Total size 5. if len(errorMeta) != 5 { // BaseField1, BaseField2, Error, CallStacks, Extra
if len(errorMeta) != 5 {
t.Fatalf("expected 5 fields for mock_error, got %d", len(errorMeta)) t.Fatalf("expected 5 fields for mock_error, got %d", len(errorMeta))
} }
if errorMeta[2].Name != "Error" || errorMeta[2].Color != "red" { if errorMeta[2].Name != "Error" || errorMeta[2].Color != "red" {
t.Errorf("unexpected meta for Error at index 2: %+v", errorMeta[2]) t.Errorf("unexpected meta for Error: %+v", errorMeta[2])
} }
if errorMeta[3].Name != "Extra" { if errorMeta[3].Name != "CallStacks" {
t.Errorf("unexpected meta for Extra at index 3: %+v", errorMeta[3]) t.Errorf("unexpected meta for CallStacks: %+v", errorMeta[3])
} }
if errorMeta[4].Name != "CallStacks" { if errorMeta[4].Name != "Extra" {
t.Errorf("unexpected meta for CallStacks at index 4: %+v", errorMeta[4]) t.Errorf("unexpected meta for Extra: %+v", errorMeta[4])
} }
// Verify file was created and contains correct data // Verify file was created and contains correct data

View File

@ -1,21 +0,0 @@
package log
import (
"testing"
)
func TestSetDefaultName(t *testing.T) {
oldName := getDefaultName()
defer SetDefaultName(oldName)
newName := "test-service-name"
SetDefaultName(newName)
if getDefaultName() != newName {
t.Errorf("GetDefaultName() = %v, want %v", getDefaultName(), newName)
}
if DefaultLogger.config.Name != newName {
t.Errorf("DefaultLogger.config.Name = %v, want %v", DefaultLogger.config.Name, newName)
}
}

86
pool.go
View File

@ -27,14 +27,88 @@ func GetEntry[T any]() *T {
}) })
} }
entry := p.(*sync.Pool).Get().(*T) entry := p.(*sync.Pool).Get().(*T)
if le, ok := any(entry).(LogEntry); ok { ResetLogEntry(entry) // 自动重置所有字段,无需子类实现 Reset
le.Reset()
}
return entry return entry
} }
// putEntry 将日志对象归还到池中 // ResetLogEntry 使用反射自动化重置日志对象的所有字段
func putEntry(entry any) { // 特别是对 Map 和 Slice 进行初始化长度0容量8
func ResetLogEntry(v any) {
rv := reflect.ValueOf(v)
if rv.Kind() != reflect.Ptr || rv.IsNil() {
return
}
t := rv.Type()
resetFunc, ok := resetCache.Load(t)
if !ok {
resetFunc = buildResetFunc(t.Elem())
resetCache.Store(t, resetFunc)
}
resetFunc.(func(reflect.Value))(rv.Elem())
}
func buildResetFunc(t reflect.Type) func(reflect.Value) {
var funcs []func(reflect.Value)
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
fieldIdx := i
switch field.Type.Kind() {
case reflect.String:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetString("") })
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetInt(0) })
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetUint(0) })
case reflect.Float32, reflect.Float64:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetFloat(0) })
case reflect.Bool:
funcs = append(funcs, func(rv reflect.Value) { rv.Field(fieldIdx).SetBool(false) })
case reflect.Map:
funcs = append(funcs, func(rv reflect.Value) {
f := rv.Field(fieldIdx)
if f.IsNil() {
f.Set(reflect.MakeMapWithSize(f.Type(), 8))
} else {
f.Clear()
}
})
case reflect.Slice:
funcs = append(funcs, func(rv reflect.Value) {
f := rv.Field(fieldIdx)
if f.Cap() < 8 {
f.Set(reflect.MakeSlice(f.Type(), 0, 8))
} else {
f.SetLen(0)
}
})
case reflect.Struct:
subReset := buildResetFunc(field.Type)
funcs = append(funcs, func(rv reflect.Value) {
subReset(rv.Field(fieldIdx))
})
case reflect.Ptr, reflect.Interface:
zero := reflect.Zero(field.Type)
funcs = append(funcs, func(rv reflect.Value) {
rv.Field(fieldIdx).Set(zero)
})
}
}
return func(rv reflect.Value) {
for _, f := range funcs {
f(rv)
}
}
}
func resetStruct(rv reflect.Value) {
// 已经不再直接调用,保留 buildResetFunc 逻辑即可
}
// PutEntry 将日志对象归还到池中
func PutEntry(entry any) {
t := reflect.TypeOf(entry) t := reflect.TypeOf(entry)
if pool, ok := globalPools.pools.Load(t); ok { if pool, ok := globalPools.pools.Load(t); ok {
pool.(*sync.Pool).Put(entry) pool.(*sync.Pool).Put(entry)
@ -44,7 +118,7 @@ func putEntry(entry any) {
// WithEntry 执行闭包并在结束后自动回收对象 // WithEntry 执行闭包并在结束后自动回收对象
func WithEntry[T any](fn func(*T)) { func WithEntry[T any](fn func(*T)) {
entry := GetEntry[T]() entry := GetEntry[T]()
defer putEntry(entry) defer PutEntry(entry)
fn(entry) fn(entry)
} }

View File

@ -11,12 +11,6 @@ type MockRequestLog struct {
UsedTime float32 UsedTime float32
} }
func (l *MockRequestLog) Reset() {
l.BaseLog.Reset()
l.RequestId = ""
l.UsedTime = 0
}
func TestWithEntry(t *testing.T) { func TestWithEntry(t *testing.T) {
WithEntry(func(entry *MockRequestLog) { WithEntry(func(entry *MockRequestLog) {
entry.RequestId = "with-entry-id" entry.RequestId = "with-entry-id"

View File

@ -28,7 +28,8 @@ func TestLoggerReliability(t *testing.T) {
} }
wg.Wait() wg.Wait()
WriterService.Stop(nil) Stop()
Wait()
file, err := os.Open(logFile) file, err := os.Open(logFile)
if err != nil { if err != nil {

View File

@ -3,7 +3,6 @@ package log
import ( import (
"bytes" "bytes"
"reflect" "reflect"
"sort"
"strconv" "strconv"
"apigo.cc/go/cast" "apigo.cc/go/cast"
@ -40,58 +39,50 @@ func getAccessors(logType string, model any) []fieldAccessor {
t = t.Elem() t = t.Elem()
} }
var flatFields []fieldInfo var flatFields []reflect.StructField
flattenStructFields(t, &flatFields, nil) flattenStructFields(t, &flatFields, nil)
// Determine final indices (must match meta.go) var extraField *reflect.StructField
maxLiteralPos := -1 var callStacksField *reflect.StructField
var highPosFields []fieldInfo var regularFields []reflect.StructField
for _, f := range flatFields {
if f.pos < 1000 {
if f.pos > maxLiteralPos {
maxLiteralPos = f.pos
}
} else {
highPosFields = append(highPosFields, f)
}
}
// Sort high pos fields by their pos
sort.Slice(highPosFields, func(i, j int) bool {
return highPosFields[i].pos < highPosFields[j].pos
})
finalPosMap := make(map[string]int)
for _, f := range flatFields { for _, f := range flatFields {
if f.pos < 1000 { if f.Name == "Extra" {
finalPosMap[f.field.Name] = f.pos extraField = &f
}
}
nextPos := maxLiteralPos + 1
for _, f := range highPosFields {
finalPosMap[f.field.Name] = nextPos
nextPos++
}
maxPos := nextPos - 1
accessors := make([]fieldAccessor, maxPos+1)
for _, f := range flatFields {
if f.field.Tag.Get("log") == "-" {
continue continue
} }
realPos := finalPosMap[f.field.Name] if f.Name == "CallStacks" {
accessors[realPos] = fieldAccessor{ callStacksField = &f
indexPath: f.field.Index, continue
name: f.field.Name,
} }
regularFields = append(regularFields, f)
}
var finalFields []reflect.StructField
finalFields = append(finalFields, regularFields...)
if callStacksField != nil {
finalFields = append(finalFields, *callStacksField)
}
if extraField != nil {
finalFields = append(finalFields, *extraField)
}
var accessors []fieldAccessor
for _, f := range finalFields {
if f.Tag.Get("log") == "-" {
continue
}
accessors = append(accessors, fieldAccessor{
indexPath: f.Index,
name: f.Name,
})
} }
accessorsCache[logType] = accessors accessorsCache[logType] = accessors
return accessors return accessors
} }
// donot export this function func ToArrayBytes(entry LogEntry, sensitiveKeys []string) []byte {
func Marshal(entry LogEntry, sensitiveKeys []string) []byte {
var buf bytes.Buffer var buf bytes.Buffer
buf.WriteByte('[') buf.WriteByte('[')
@ -118,11 +109,6 @@ func Marshal(entry LogEntry, sensitiveKeys []string) []byte {
buf.WriteByte(',') buf.WriteByte(',')
} }
if acc.indexPath == nil {
buf.WriteByte('0')
continue
}
fv := v.FieldByIndex(acc.indexPath) fv := v.FieldByIndex(acc.indexPath)
writeValue(&buf, fv, acc.name, sensitiveKeys) writeValue(&buf, fv, acc.name, sensitiveKeys)
} }
@ -137,17 +123,6 @@ func writeValue(buf *bytes.Buffer, v reflect.Value, fieldName string, sensitiveK
return return
} }
// Check if this root field should be desensitized
if len(sensitiveKeys) > 0 {
fixedName := fixField(fieldName)
for _, sk := range sensitiveKeys {
if sk == fixedName {
buf.WriteString(`"***"`)
return
}
}
}
switch v.Kind() { switch v.Kind() {
case reflect.String: case reflect.String:
writeString(buf, v.String()) writeString(buf, v.String())
@ -163,13 +138,41 @@ func writeValue(buf *bytes.Buffer, v reflect.Value, fieldName string, sensitiveK
} else { } else {
buf.WriteString("false") buf.WriteString("false")
} }
default: case reflect.Map:
// Use cast for complex types to ensure deep desensitization if v.IsNil() || v.Len() == 0 {
b, _ := cast.ToJSONDesensitizeBytes(v.Interface(), sensitiveKeys) buf.WriteString("{}")
if len(b) == 0 { return
buf.WriteString("null") }
// Handle map with cast.ToJSON
var b []byte
if len(sensitiveKeys) > 0 {
b, _ = cast.ToJSONDesensitizeBytes(v.Interface(), sensitiveKeys)
} else { } else {
b, _ = cast.ToJSONBytes(v.Interface())
}
if len(b) > 0 {
buf.Write(b) buf.Write(b)
} else {
buf.WriteString("{}")
}
case reflect.Slice, reflect.Array:
if v.IsNil() || v.Len() == 0 {
buf.WriteString("[]")
return
}
b, _ := cast.ToJSONBytes(v.Interface())
if len(b) > 0 {
buf.Write(b)
} else {
buf.WriteString("[]")
}
default:
// Fallback for other complex types
b, _ := cast.ToJSONBytes(v.Interface())
if len(b) > 0 {
buf.Write(b)
} else {
buf.WriteString("null")
} }
} }
} }

View File

@ -6,10 +6,10 @@ import (
) )
type SerializerMockBaseLog struct { type SerializerMockBaseLog struct {
LogName string `log:"pos:0"` LogName string `log:"pos:1"`
LogType string `log:"pos:1"` LogType string `log:"pos:2"`
LogTime int64 `log:"pos:2"` LogTime int64 `log:"pos:3"`
TraceId string `log:"pos:3"` TraceId string `log:"pos:4"`
} }
func (b *SerializerMockBaseLog) IsLogEntry() bool { func (b *SerializerMockBaseLog) IsLogEntry() bool {
@ -17,30 +17,15 @@ func (b *SerializerMockBaseLog) IsLogEntry() bool {
} }
func (b *SerializerMockBaseLog) GetBaseLog() *BaseLog { func (b *SerializerMockBaseLog) GetBaseLog() *BaseLog {
// Return a dummy BaseLog just for interface satisfaction,
// ToArrayBytes actually extracts LogType from here, so let's mock it.
return &BaseLog{LogType: b.LogType} return &BaseLog{LogType: b.LogType}
} }
func (b *SerializerMockBaseLog) Reset() {
b.LogName = ""
b.LogType = ""
b.LogTime = 0
b.TraceId = ""
}
type SerializerMockInfoLog struct { type SerializerMockInfoLog struct {
SerializerMockBaseLog SerializerMockBaseLog
Message string `log:"pos:4"` Message string `log:"pos:5"`
Extra map[string]any `log:"pos:1000"` Extra map[string]any
}
func (l *SerializerMockInfoLog) Reset() {
l.SerializerMockBaseLog.Reset()
l.Message = ""
if l.Extra == nil {
l.Extra = make(map[string]any, 8)
} else {
clear(l.Extra)
}
} }
func TestToArrayBytes(t *testing.T) { func TestToArrayBytes(t *testing.T) {
@ -59,17 +44,16 @@ func TestToArrayBytes(t *testing.T) {
RegisterType("mock_info_test", entry) // trigger meta generation RegisterType("mock_info_test", entry) // trigger meta generation
bytes := Marshal(entry, nil) bytes := ToArrayBytes(entry, nil)
str := string(bytes) str := string(bytes)
t.Logf("Raw log: %s", str)
// Expect format: ["test-app","mock_info_test",1620000000,"abc-123","Hello, World!",{"user_id":42}]
var arr []any var arr []any
err := json.Unmarshal(bytes, &arr) err := json.Unmarshal(bytes, &arr)
if err != nil { if err != nil {
t.Fatalf("failed to unmarshal generated array: %v, raw: %s", err, str) t.Fatalf("failed to unmarshal generated array: %v, raw: %s", err, str)
} }
// Indices: 0, 1, 2, 3, 4, 1000(mapped to 5). Total size 6.
if len(arr) != 6 { if len(arr) != 6 {
t.Fatalf("expected 6 elements, got %d. raw: %s", len(arr), str) t.Fatalf("expected 6 elements, got %d. raw: %s", len(arr), str)
} }
@ -93,7 +77,7 @@ func TestToArrayBytes(t *testing.T) {
extraMap, ok := arr[5].(map[string]any) extraMap, ok := arr[5].(map[string]any)
if !ok { if !ok {
t.Fatalf("expected arr[5] to be map[string]any, got %T (value: %v)", arr[5], arr[5]) t.Fatalf("expected arr[5] to be map[string]any, got %T", arr[5])
} }
if extraMap["user_id"] != float64(42) { if extraMap["user_id"] != float64(42) {
t.Errorf("expected extraMap['user_id'] == 42, got %v", extraMap["user_id"]) t.Errorf("expected extraMap['user_id'] == 42, got %v", extraMap["user_id"])
@ -113,7 +97,7 @@ func TestToArrayBytes_Desensitize(t *testing.T) {
RegisterType("mock_info_test2", entry) RegisterType("mock_info_test2", entry)
bytes := Marshal(entry, []string{"password"}) bytes := ToArrayBytes(entry, []string{"password"})
str := string(bytes) str := string(bytes)
var arr []any var arr []any

View File

@ -14,26 +14,26 @@ const LogTypeMonitor = "monitor"
const LogTypeStatistic = "statistic" const LogTypeStatistic = "statistic"
const LogTypeRequest = "request" const LogTypeRequest = "request"
const LogDefaultSensitive = "phone,password,secret,token,accessToken,authorization" const LogDefaultSensitive = "phone,password,secret,token,accessToken"
const LogEnvLevel = "LOG_LEVEL" const LogEnvLevel = "LOG_LEVEL"
const LogEnvFile = "LOG_FILE" const LogEnvFile = "LOG_FILE"
const LogEnvSensitive = "LOG_SENSITIVE" const LogEnvSensitive = "LOG_SENSITIVE"
const LogEnvRegexSensitive = "LOG_REGEXSENSITIVE"
// LogEntry 是一个标记接口,用于识别是否为对象池管理的日志对象 // LogEntry 是一个标记接口,用于识别是否为对象池管理的日志对象
type LogEntry interface { type LogEntry interface {
IsLogEntry() bool IsLogEntry() bool
GetBaseLog() *BaseLog GetBaseLog() *BaseLog
Reset()
} }
type BaseLog struct { type BaseLog struct {
LogName string `log:"pos:0,color:cyan,hide:true"` LogName string `log:"pos:1,color:cyan,hide:true"`
LogType string `log:"pos:1,color:magenta,hide:true"` LogType string `log:"pos:2,color:magenta,hide:true"`
LogTime int64 `log:"pos:2,format:time"` LogTime int64 `log:"pos:3,format:time"`
TraceId string `log:"pos:3,color:gray,withoutkey:true"` TraceId string `log:"pos:4,color:blue"`
Image string `log:"pos:4,hide:true"` Image string `log:"pos:5,color:darkGray,hide:true"`
Server string `log:"pos:5,hide:true"` Server string `log:"pos:6,color:darkGray,hide:true"`
Extra map[string]any `log:"pos:1000"` Extra map[string]any `log:"pos:99"`
} }
func (b *BaseLog) IsLogEntry() bool { func (b *BaseLog) IsLogEntry() bool {
@ -44,67 +44,31 @@ func (b *BaseLog) GetBaseLog() *BaseLog {
return b return b
} }
func (b *BaseLog) Reset() {
b.LogName = ""
b.LogType = ""
b.LogTime = 0
b.TraceId = ""
b.Image = ""
b.Server = ""
if b.Extra == nil {
b.Extra = make(map[string]any, 8)
} else {
clear(b.Extra)
}
}
type DebugLog struct { type DebugLog struct {
BaseLog BaseLog
Debug string `log:"pos:6,withoutkey:true"` // white Debug string `log:"pos:9,withoutkey:true"` // white
}
func (l *DebugLog) Reset() {
l.BaseLog.Reset()
l.Debug = ""
} }
type InfoLog struct { type InfoLog struct {
BaseLog BaseLog
Info string `log:"pos:6,color:cyan,withoutkey:true"` Info string `log:"pos:9,color:cyan,withoutkey:true"`
}
func (l *InfoLog) Reset() {
l.BaseLog.Reset()
l.Info = ""
} }
type WarningLog struct { type WarningLog struct {
BaseLog BaseLog
Warning string `log:"pos:6,color:yellow,withoutkey:true"` Warning string `log:"pos:9,color:yellow,withoutkey:true"`
CallStacks []string `log:"pos:1001"` CallStacks []string `log:"pos:98"`
}
func (l *WarningLog) Reset() {
l.BaseLog.Reset()
l.Warning = ""
l.CallStacks = l.CallStacks[:0]
} }
type ErrorLog struct { type ErrorLog struct {
BaseLog BaseLog
Error string `log:"pos:6,color:red,withoutkey:true"` Error string `log:"pos:9,color:red,withoutkey:true"`
CallStacks []string `log:"pos:1001"` CallStacks []string `log:"pos:98"`
}
func (l *ErrorLog) Reset() {
l.BaseLog.Reset()
l.Error = ""
l.CallStacks = l.CallStacks[:0]
} }
func init() { func init() {
RegisterType(LogTypeDebug, &DebugLog{}) RegisterType(LogTypeDebug, DebugLog{})
RegisterType(LogTypeInfo, &InfoLog{}) RegisterType(LogTypeInfo, InfoLog{})
RegisterType(LogTypeWarning, &WarningLog{}) RegisterType(LogTypeWarning, WarningLog{})
RegisterType(LogTypeError, &ErrorLog{}) RegisterType(LogTypeError, ErrorLog{})
} }

View File

@ -1,13 +1,16 @@
package log package log
import ( import (
"encoding/json"
"fmt" "fmt"
"net" "net"
"os" "os"
"path" "path"
"runtime" "runtime"
"runtime/debug"
"strings" "strings"
"time"
"apigo.cc/go/cast"
) )
var ( var (
@ -21,35 +24,120 @@ func init() {
dockerImageName = os.Getenv("DOCKER_IMAGE_NAME") dockerImageName = os.Getenv("DOCKER_IMAGE_NAME")
dockerImageTag = os.Getenv("DOCKER_IMAGE_TAG") dockerImageTag = os.Getenv("DOCKER_IMAGE_TAG")
serverName, _ = os.Hostname() serverName, _ = os.Hostname()
addrs, err := net.InterfaceAddrs()
// 获取真实局域网 IP (UDP 8.8.8.8 伪拨号法)
conn, err := net.Dial("udp", "8.8.8.8:80")
if err == nil { if err == nil {
localAddr := conn.LocalAddr().(*net.UDPAddr) for _, a := range addrs {
serverIp = localAddr.IP.String() if an, ok := a.(*net.IPNet); ok {
_ = conn.Close() // 忽略 Docker 私有网段
} if an.IP.IsGlobalUnicast() && !strings.HasPrefix(an.IP.To4().String(), "172.17.") {
serverIp = an.IP.To4().String()
if serverIp == "" { break
addrs, err := net.InterfaceAddrs()
if err == nil {
for _, a := range addrs {
if an, ok := a.(*net.IPNet); ok {
if an.IP.IsGlobalUnicast() {
serverIp = an.IP.To4().String()
break
}
} }
} }
} }
} }
} }
// fixField 格式化字段名(去横线、下划线,小写) // MakeTime 解析纳秒时间戳或 RFC3339 字符串
func MakeTime(v any) time.Time {
if ts, ok := cast.ToInt64E(v); ok == nil {
return time.Unix(0, ts)
}
tm, _ := time.Parse(time.RFC3339Nano, cast.String(v))
return tm
}
// MakeUsedTime 计算消耗时间(毫秒)
func MakeUsedTime(startTime, endTime time.Time) float32 {
return float32(endTime.UnixNano()-startTime.UnixNano()) / 1e6
}
// ParseBaseLog 解析基础日志行
func ParseBaseLog(line string) *BaseLog {
pos := strings.IndexByte(line, '{')
if pos == -1 {
return ParseBadLog(line)
}
l := make(map[string]any)
err := json.Unmarshal([]byte(line[pos:]), &l)
if err != nil {
return ParseBadLog(line)
}
baseLog := BaseLog{Extra: make(map[string]any)}
for k, v := range l {
lk := strings.ToLower(k)
switch lk {
case "logname":
baseLog.LogName = cast.String(v)
case "logtype":
baseLog.LogType = cast.String(v)
case "logtime":
baseLog.LogTime = cast.Int64(v)
case "traceid":
baseLog.TraceId = cast.String(v)
case "imagename":
if baseLog.Image != "" {
baseLog.Image = cast.String(v) + ":" + baseLog.Image
} else {
baseLog.Image = cast.String(v)
}
case "imagetag":
if baseLog.Image != "" {
baseLog.Image = baseLog.Image + ":" + cast.String(v)
} else {
baseLog.Image = cast.String(v)
}
case "servername":
if baseLog.Server != "" {
baseLog.Server = cast.String(v) + ":" + baseLog.Server
} else {
baseLog.Server = cast.String(v)
}
case "serverip":
if baseLog.Server != "" {
baseLog.Server = baseLog.Server + ":" + cast.String(v)
} else {
baseLog.Server = cast.String(v)
}
default:
baseLog.Extra[lk] = v
}
}
return &baseLog
}
// ParseBadLog 解析非 JSON 格式的日志
func ParseBadLog(line string) *BaseLog {
baseLog := BaseLog{Extra: make(map[string]any)}
baseLog.LogType = LogTypeUndefined
if len(line) > 19 && line[19] == ' ' {
tm, err := time.Parse("2006/01/02 15:04:05", line[0:19])
if err == nil {
baseLog.LogTime = tm.UnixNano()
line = line[20:]
} else {
return nil
}
} else if len(line) > 26 && line[26] == ' ' {
tm, err := time.Parse("2006/01/02 15:04:05.000000", line[0:26])
if err == nil {
baseLog.LogTime = tm.UnixNano()
line = line[27:]
} else {
return nil
}
} else {
return nil
}
baseLog.Extra["info"] = line
return &baseLog
}
// fixField 格式化字段名(去横线,小写)
func fixField(s string) string { func fixField(s string) string {
s = strings.ReplaceAll(s, "-", "") return strings.ToLower(strings.ReplaceAll(s, "-", ""))
s = strings.ReplaceAll(s, "_", "")
return strings.ToLower(s)
} }
// getCallStacks 获取调用栈 // getCallStacks 获取调用栈
@ -69,7 +157,6 @@ func getCallStacks(truncations []string) []string {
isLogInternal := (strings.Contains(file, "/log/logger.go") || isLogInternal := (strings.Contains(file, "/log/logger.go") ||
strings.Contains(file, "/log/utility.go") || strings.Contains(file, "/log/utility.go") ||
strings.Contains(file, "/log/standard.go") || strings.Contains(file, "/log/standard.go") ||
strings.Contains(file, "/log/default_logger.go") ||
strings.Contains(file, "/log/extra.go")) strings.Contains(file, "/log/extra.go"))
if isLogInternal { if isLogInternal {
@ -92,21 +179,24 @@ func getCallStacks(truncations []string) []string {
return callStacks return callStacks
} }
var globalDefaultName string // GetDefaultName 获取默认应用名称
func GetDefaultName() string {
// getDefaultName 获取默认应用名称 name := os.Getenv("DISCOVER_APP")
func getDefaultName() string { if name == "" {
if globalDefaultName != "" { name = os.Getenv("discover_app")
return globalDefaultName
} }
name := "" if name == "" {
if info, ok := debug.ReadBuildInfo(); ok && info.Path != "" && info.Path != "command-line-arguments" { imageName := os.Getenv("DOCKER_IMAGE_NAME")
name = path.Base(info.Path) if imageName != "" {
parts := strings.Split(imageName, "/")
imageName = parts[len(parts)-1]
imageName = strings.SplitN(imageName, ":", 2)[0]
imageName = strings.SplitN(imageName, "#", 2)[0]
name = imageName
}
} }
if name == "" { if name == "" {
name = path.Base(os.Args[0]) name = path.Base(os.Args[0])
} }
// 处理 Windows 下的 .exe 后缀
name = strings.TrimSuffix(name, ".exe")
return name return name
} }

207
viewer.go
View File

@ -1,10 +1,9 @@
package log package log
import ( import (
"encoding/json"
"fmt" "fmt"
"os"
"regexp" "regexp"
"sort"
"strings" "strings"
"time" "time"
@ -14,7 +13,6 @@ import (
var errorLineMatcher = regexp.MustCompile(`(\w+\.go:\d+)`) var errorLineMatcher = regexp.MustCompile(`(\w+\.go:\d+)`)
var codeFileMatcher = regexp.MustCompile(`(\w+?\.)(go|js)`) var codeFileMatcher = regexp.MustCompile(`(\w+?\.)(go|js)`)
var workspaceRoot, _ = os.Getwd()
func Viewable(line string) string { func Viewable(line string) string {
line = strings.TrimSpace(line) line = strings.TrimSpace(line)
@ -33,24 +31,20 @@ func Viewable(line string) string {
} }
var arr []any var arr []any
if err := cast.UnmarshalJSON([]byte(line), &arr); err != nil { if err := json.Unmarshal([]byte(line), &arr); err != nil {
return line return line
} }
if len(arr) < 2 { if len(arr) < 3 {
return line return line // At least Name, Type, Time
} }
logType := "" logType := cast.String(arr[1])
if len(arr) > 2 { if logType == "" {
logType = cast.String(arr[2]) logType = "undefined"
} }
meta := GetMeta(logType) meta := GetMeta(logType)
if len(meta) == 0 {
logType = cast.String(arr[1])
meta = GetMeta(logType)
}
if len(meta) == 0 { if len(meta) == 0 {
// Fallback rendering // Fallback rendering
return fallbackRenderArray(arr) return fallbackRenderArray(arr)
@ -59,7 +53,7 @@ func Viewable(line string) string {
var builder strings.Builder var builder strings.Builder
for i, v := range arr { for i, v := range arr {
if v == nil || cast.String(v) == "0" { // 0 is gap if v == nil {
continue continue
} }
if i >= len(meta) { if i >= len(meta) {
@ -72,23 +66,24 @@ func Viewable(line string) string {
m := meta[i] m := meta[i]
if m.Hide || m.Name == "" { if m.Hide {
continue continue
} }
if m.Name == "Extra" { if m.Name == "Extra" {
extraMap, ok := v.(map[string]any) extraMap, ok := v.(map[string]any)
if ok && len(extraMap) > 0 { if ok && len(extraMap) > 0 {
keys := make([]string, 0, len(extraMap)) for k, ev := range extraMap {
for k := range extraMap {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
ev := extraMap[k]
builder.WriteString(" ") builder.WriteString(" ")
builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Italic, k+":")) builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Italic, k+":"))
builder.WriteString(renderValue(ev, 0, "")) vStr := ""
switch ev.(type) {
case map[string]any, []any:
vStr, _ = cast.ToJSON(ev)
default:
vStr = cast.String(ev)
}
builder.WriteString(vStr)
} }
} }
continue continue
@ -97,18 +92,9 @@ func Viewable(line string) string {
if m.Name == "CallStacks" { if m.Name == "CallStacks" {
callStacksList, ok := v.([]any) callStacksList, ok := v.([]any)
if ok && len(callStacksList) > 0 { if ok && len(callStacksList) > 0 {
stackColor := shell.TextRed
if strings.Contains(strings.ToLower(logType), "warn") {
stackColor = shell.TextYellow
}
builder.WriteString("\n") builder.WriteString("\n")
for _, vi := range callStacksList { for _, vi := range callStacksList {
vStr := cast.String(vi) vStr := cast.String(vi)
if workspaceRoot != "" {
vStr = strings.TrimPrefix(vStr, workspaceRoot)
vStr = strings.TrimPrefix(vStr, "/")
}
postfix := "" postfix := ""
if pos := strings.LastIndexByte(vStr, '/'); pos != -1 { if pos := strings.LastIndexByte(vStr, '/'); pos != -1 {
postfix = vStr[pos+1:] postfix = vStr[pos+1:]
@ -119,7 +105,7 @@ func Viewable(line string) string {
} }
builder.WriteString(" ") builder.WriteString(" ")
builder.WriteString(shell.Style(shell.Dim, vStr)) builder.WriteString(shell.Style(shell.Dim, vStr))
builder.WriteString(shell.Style(stackColor, postfix)) builder.WriteString(shell.Style(shell.TextWhite, postfix))
builder.WriteString("\n") builder.WriteString("\n")
} }
} }
@ -131,108 +117,33 @@ func Viewable(line string) string {
if m.Format == "time" { if m.Format == "time" {
// Convert int64 ns to time string // Convert int64 ns to time string
logTime := time.Unix(0, cast.Int64(v)) logTime := time.Unix(0, cast.Int64(v))
dateStr := logTime.Format("01-02") vStr = logTime.Format("01-02 15:04:05.000")
timeStr := logTime.Format("15:04:05") if m.Color == "" {
milliStr := logTime.Format(".000") builder.WriteString(shell.White(shell.Bold, vStr))
if builder.Len() > 0 {
builder.WriteString(" ") builder.WriteString(" ")
continue
} }
builder.WriteString(shell.Style(shell.Dim, dateStr))
builder.WriteString(" ")
builder.WriteString(shell.White(shell.Bold, timeStr))
builder.WriteString(shell.Style(shell.Dim, milliStr))
continue
} else { } else {
vStr = renderValue(v, m.Precision, m.Color) vStr = cast.String(v)
if vStr == "" { if vStr == "" {
continue continue
} }
} }
if builder.Len() > 0 { if builder.Len() > 0 {
if m.AttachBefore { builder.WriteString(" ")
builder.WriteString(":")
} else {
builder.WriteString(" ")
}
} }
if !m.WithoutKey && !m.AttachBefore { if !m.WithoutKey {
name := m.KeyName builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Italic, m.Name+":"))
if name == "" {
name = m.Name
}
builder.WriteString(shell.Style(shell.TextWhite, shell.Dim, shell.Italic, name+":"))
} }
builder.WriteString(vStr) builder.WriteString(applyColor(vStr, m.Color))
} }
return builder.String() return builder.String()
} }
// ToJSON converts a JSON array log line to a standard JSON object string based on metadata.
func ToJSON(line string) string {
line = strings.TrimSpace(line)
if !strings.HasPrefix(line, "[") {
return line
}
var arr []any
if err := cast.UnmarshalJSON([]byte(line), &arr); err != nil {
return line
}
if len(arr) < 2 {
return line
}
logType := ""
if len(arr) > 2 {
logType = cast.String(arr[2])
}
meta := GetMeta(logType)
if len(meta) == 0 {
logType = cast.String(arr[1])
meta = GetMeta(logType)
}
if len(meta) == 0 {
return line
}
result := make(map[string]any)
for i, v := range arr {
if i < len(meta) {
m := meta[i]
if m.Name == "" {
continue
}
name := m.KeyName
if name == "" {
name = m.Name
}
if m.Name == "Extra" {
if extraMap, ok := v.(map[string]any); ok {
for k, ev := range extraMap {
result[k] = ev
}
}
} else {
result[name] = v
}
} else if cast.String(v) != "0" {
result[fmt.Sprintf("Extra%d", i)] = v
}
}
jsonStr, _ := cast.ToJSON(result)
return jsonStr
}
func applyColor(text string, color string) string { func applyColor(text string, color string) string {
switch color { switch color {
case "red": case "red":
@ -247,7 +158,7 @@ func applyColor(text string, color string) string {
return shell.Yellow(text) return shell.Yellow(text)
case "green": case "green":
return shell.Green(text) return shell.Green(text)
case "gray": case "gray", "darkGray":
return shell.Style(shell.Dim, text) return shell.Style(shell.Dim, text)
default: default:
return text return text
@ -264,61 +175,3 @@ func fallbackRenderArray(arr []any) string {
} }
return builder.String() return builder.String()
} }
func renderValue(v any, precision int, color string) string {
if v == nil {
return ""
}
switch val := v.(type) {
case float32, float64:
vStr := ""
if precision > 0 {
vStr = fmt.Sprintf("%.*f", precision, cast.To[float64](v))
} else {
vStr = cast.String(v)
}
return applyColor(vStr, color)
case map[string]any:
if len(val) == 0 {
return ""
}
var parts []string
keys := make([]string, 0, len(val))
for k := range val {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
// Key is always dim, value is colored
vStr := renderValue(val[k], precision, color)
if vStr != "" {
parts = append(parts, fmt.Sprintf("%s:%s", shell.Style(shell.Dim, k), vStr))
}
}
if len(parts) == 0 {
return ""
}
return "[ " + strings.Join(parts, " ") + " ]"
case []any:
if len(val) == 0 {
return ""
}
var parts []string
for _, iv := range val {
vStr := renderValue(iv, precision, color)
if vStr != "" {
parts = append(parts, vStr)
}
}
if len(parts) == 0 {
return ""
}
return "[ " + strings.Join(parts, " ") + " ]"
default:
s := cast.String(v)
if s == "" {
return ""
}
return applyColor(s, color)
}
}

View File

@ -1,7 +1,6 @@
package log_test package log_test
import ( import (
"os"
"strings" "strings"
"testing" "testing"
@ -9,18 +8,17 @@ import (
) )
func TestViewable(t *testing.T) { func TestViewable(t *testing.T) {
// First ensure mock_info type is registered so we have meta
entry := &log.InfoLog{ entry := &log.InfoLog{
BaseLog: log.BaseLog{ BaseLog: log.BaseLog{
LogName: "test-app", LogName: "test-app",
LogType: "info", LogType: "info",
LogTime: 1714896000000000000,
TraceId: "trace-123",
}, },
Info: "hello world", Info: "hello world",
} }
log.RegisterType("info", entry) log.RegisterType("info", entry)
line := string(log.Marshal(entry, nil)) line := `["test-app","info",1714896000000000000,"trace-123","","","","","hello world",{"key":"value"}]`
out := log.Viewable(line) out := log.Viewable(line)
if !strings.Contains(out, "hello world") { if !strings.Contains(out, "hello world") {
@ -29,202 +27,20 @@ func TestViewable(t *testing.T) {
if !strings.Contains(out, "trace-123") { if !strings.Contains(out, "trace-123") {
t.Errorf("expected 'trace-123' in output, got: %s", out) t.Errorf("expected 'trace-123' in output, got: %s", out)
} }
} if !strings.Contains(out, "key:") {
t.Errorf("expected 'key:' in output, got: %s", out)
func TestToJSON(t *testing.T) {
entry := &log.InfoLog{
BaseLog: log.BaseLog{
LogName: "test-app",
LogType: "info",
LogTime: 1714896000000000000,
TraceId: "trace-123",
},
Info: "hello world",
} }
entry.Extra = map[string]any{"key": "value"} if !strings.Contains(out, "value") {
log.RegisterType("info", entry) t.Errorf("expected 'value' in output, got: %s", out)
line := string(log.Marshal(entry, nil))
jsonStr := log.ToJSON(line)
if !strings.Contains(jsonStr, `"Info":"hello world"`) {
t.Errorf("expected Info field in JSON, got: %s", jsonStr)
}
if !strings.Contains(jsonStr, `"TraceId":"trace-123"`) {
t.Errorf("expected TraceId field in JSON, got: %s", jsonStr)
}
if !strings.Contains(jsonStr, `"key":"value"`) {
t.Errorf("expected Extra fields merged in JSON, got: %s", jsonStr)
} }
} }
func TestLoadMeta(t *testing.T) { func BenchmarkViewable(b *testing.B) {
// Create a temporary meta file line := `["test-app","info",1714896000000000000,"trace-123","","","","","hello world",{"key":"value"}]`
metaData := `{"test-type":[{"index":0,"name":"Field1"},{"index":1,"name":"Field2"}]}`
_ = os.WriteFile(".test_meta.json", []byte(metaData), 0644)
defer os.Remove(".test_meta.json")
err := log.LoadMeta(".test_meta.json") b.ResetTimer()
if err != nil { b.ReportAllocs()
t.Fatalf("failed to load meta: %v", err) for i := 0; i < b.N; i++ {
} _ = log.Viewable(line)
meta := log.GetMeta("test-type")
if len(meta) != 2 {
t.Errorf("expected 2 meta fields, got %d", len(meta))
}
if meta[0].Name != "Field1" {
t.Errorf("expected Field1, got %s", meta[0].Name)
}
}
type EnhancedLog struct {
log.BaseLog
App string `log:"pos:10,withoutkey:true"`
Node string `log:"pos:11,attachBefore,withoutkey:true"`
RequestHeaders map[string]string `log:"pos:13,keyname:reqH"`
ClientIP string `log:"pos:12,keyname:ip"`
Tags []string `log:"pos:14"`
}
func (l *EnhancedLog) Reset() {
l.BaseLog.Reset()
l.App = ""
l.Node = ""
l.ClientIP = ""
l.RequestHeaders = nil
l.Tags = nil
}
func TestEnhancedViewable(t *testing.T) {
entry := &EnhancedLog{
BaseLog: log.BaseLog{
LogType: "enhanced",
LogTime: 1714896000000000000,
},
App: "MyApp",
Node: "Node1",
ClientIP: "127.0.0.1",
RequestHeaders: map[string]string{
"User-Agent": "Go-http-cli",
},
Tags: []string{"tag1", "tag2"},
}
log.RegisterType("enhanced", entry)
line := string(log.Marshal(entry, nil))
out := log.Viewable(line)
// Check attachBefore: MyApp:Node1 (since both are withoutkey)
if !strings.Contains(out, "MyApp:Node1") {
t.Errorf("expected MyApp:Node1, got: %s", out)
}
// Check pos ordering and keyname: ip:127.0.0.1 should come before reqH
if !strings.Contains(out, "ip:") || !strings.Contains(out, "127.0.0.1") {
t.Errorf("expected ip:127.0.0.1, got: %s", out)
}
if !strings.Contains(out, "reqH:") || !strings.Contains(out, "User-Agent") || !strings.Contains(out, "Go-http-cli") {
t.Errorf("expected reqH:[ User-Agent:Go-http-cli ], got: %s", out)
}
ipIdx := strings.Index(out, "ip:")
reqHIdx := strings.Index(out, "reqH:")
if ipIdx > reqHIdx {
t.Errorf("expected ip to come before reqH, but ipIdx=%d, reqHIdx=%d", ipIdx, reqHIdx)
}
// Check array rendering: Tags:[ tag1 tag2 ]
if !strings.Contains(out, "Tags:") || !strings.Contains(out, "[ tag1 tag2 ]") {
t.Errorf("expected Tags:[ tag1 tag2 ], got: %s", out)
}
}
func TestEnhancedToJSON(t *testing.T) {
entry := &EnhancedLog{
BaseLog: log.BaseLog{
LogType: "enhanced",
LogTime: 1714896000000000000,
},
App: "MyApp",
Node: "Node1",
ClientIP: "127.0.0.1",
RequestHeaders: map[string]string{
"User-Agent": "Go-http-cli",
},
}
log.RegisterType("enhanced", entry)
line := string(log.Marshal(entry, nil))
jsonStr := log.ToJSON(line)
// Check keyname in JSON
if !strings.Contains(jsonStr, `"ip":"127.0.0.1"`) {
t.Errorf("expected ip field in JSON, got: %s", jsonStr)
}
if !strings.Contains(jsonStr, `"reqH":{"User-Agent":"Go-http-cli"}`) {
t.Errorf("expected reqH field in JSON, got: %s", jsonStr)
}
}
type CallStackLog struct {
log.BaseLog
CallStacks []string `log:"pos:6"`
}
func (l *CallStackLog) Reset() {
l.BaseLog.Reset()
l.CallStacks = nil
}
func TestCallStacksViewable(t *testing.T) {
wd, _ := os.Getwd()
entry := &CallStackLog{
BaseLog: log.BaseLog{
LogType: "test_error",
LogTime: 1714896000000000000,
},
CallStacks: []string{wd + "/main.go:10", "/usr/local/go/src/runtime/panic.go:100"},
}
log.RegisterType("test_error", entry)
line := string(log.Marshal(entry, nil))
out := log.Viewable(line)
// Check path truncation (should contain relative "main.go:10")
if !strings.Contains(out, "main.go:10") {
t.Errorf("expected relative path main.go:10, got: %s", out)
}
// Absolute path should be removed if it matches wd
if strings.Contains(out, wd) {
t.Errorf("absolute path should be truncated, but still found: %s", out)
}
}
type PrecisionLog struct {
log.BaseLog
Value float64 `log:"pos:6,precision:2"`
}
func (l *PrecisionLog) Reset() {
l.BaseLog.Reset()
l.Value = 0
}
func TestPrecisionViewable(t *testing.T) {
entry := &PrecisionLog{
BaseLog: log.BaseLog{
LogType: "precision",
LogTime: 1714896000000000000,
},
Value: 3.14159,
}
log.RegisterType("precision", entry)
line := string(log.Marshal(entry, nil))
out := log.Viewable(line)
if !strings.Contains(out, "3.14") || strings.Contains(out, "3.141") {
t.Errorf("expected 3.14 (precision 2), got: %s", out)
} }
} }

118
writer.go
View File

@ -1,7 +1,6 @@
package log package log
import ( import (
"context"
"fmt" "fmt"
"sync" "sync"
"sync/atomic" "sync/atomic"
@ -22,22 +21,12 @@ type logPayload struct {
file *FileWriter // 目标文件 Writer file *FileWriter // 目标文件 Writer
} }
// writerService manages the background writing of log entries.
type writerService struct {
Running atomic.Bool
StopChan chan bool
LogChannel chan logPayload
Dropped atomic.Uint64
Writers atomic.Value // []Writer
WriterLock sync.Mutex
Files map[string]*FileWriter
FilesLock sync.RWMutex
}
var ( var (
// WriterService is the global instance of defaultService. writerRunning atomic.Bool
WriterService = &writerService{} writerLock sync.Mutex // 仅用于注册时锁定
writerStopChan chan bool
writers atomic.Value // 存储 []Writer
logChannel chan logPayload
) )
// ConsoleWriter 控制台写入器 // ConsoleWriter 控制台写入器
@ -52,112 +41,93 @@ func (w *ConsoleWriter) Run() {
} }
func init() { func init() {
WriterService.LogChannel = make(chan logPayload, 10000) logChannel = make(chan logPayload, 10000)
WriterService.Writers.Store([]Writer{}) writers.Store([]Writer{})
WriterService.Files = make(map[string]*FileWriter)
RegisterWriterMaker("console", func(conf *Config) Writer { RegisterWriterMaker("console", func(conf *Config) Writer {
return &ConsoleWriter{} return &ConsoleWriter{}
}) })
} }
// writeAsync 异步写入日志 // WriteAsync 异步写入日志
func writeAsync(payload logPayload) { func WriteAsync(payload logPayload) {
defer func() { defer func() {
recover() recover()
}() }()
if !WriterService.Running.Load() { if !writerRunning.Load() {
return return
} }
select { select {
case WriterService.LogChannel <- payload: case logChannel <- payload:
default: default:
// 丢弃或处理过载 // 丢弃或处理过载,此处简单丢弃
dropped := WriterService.Dropped.Add(1)
if dropped%1000 == 1 {
if DefaultLogger != nil {
// 注意:这里可能会产生递归调用,但 select default 保证了不会死锁
DefaultLogger.Error(fmt.Sprintf("log channel full, dropped %d logs", dropped))
}
}
} }
} }
// GetDroppedLogs 获取被丢弃的日志数量 // Start 启动写入器
func GetDroppedLogs() uint64 { func Start() {
return WriterService.Dropped.Load() if !writerRunning.CompareAndSwap(false, true) {
return
}
writerStopChan = make(chan bool)
go writerRunner()
} }
// Start implements starter.Service interface. // Stop 停止写入器
func (s *writerService) Start(_ context.Context, _ *Logger) error { func Stop() {
if !s.Running.CompareAndSwap(false, true) { if writerRunning.CompareAndSwap(true, false) {
return nil close(logChannel)
} }
s.StopChan = make(chan bool)
go s.writerRunner()
return nil
} }
// Stop implements starter.Service interface. // Wait 等待写入器停止
func (s *writerService) Stop(_ context.Context) error { func Wait() {
if s.Running.CompareAndSwap(true, false) { if writerStopChan != nil {
close(s.LogChannel) <-writerStopChan
if s.StopChan != nil { writerStopChan = nil
<-s.StopChan
s.StopChan = nil
}
} }
return nil
} }
// Status implements starter.Service interface. func writerRunner() {
func (s *writerService) Status() (string, error) {
if s.Running.Load() {
return fmt.Sprintf("queue: %d, dropped: %d", len(s.LogChannel), s.Dropped.Load()), nil
}
return "stopped", nil
}
func (s *writerService) writerRunner() {
ticker := time.NewTicker(200 * time.Millisecond) ticker := time.NewTicker(200 * time.Millisecond)
defer ticker.Stop() defer ticker.Stop()
defer func() { defer func() {
if s.StopChan != nil { if writerStopChan != nil {
close(s.StopChan) close(writerStopChan)
} }
}() }()
for { for {
select { select {
case payload, ok := <-s.LogChannel: case payload, ok := <-logChannel:
if !ok { if !ok {
s.flushWriters() flushWriters()
return return
} }
s.processLog(payload) processLog(payload)
// 尝试批量处理更多日志 // 尝试批量处理更多日志
batchCount := 0 batchCount := 0
for batchCount < 100 { for batchCount < 100 {
select { select {
case nextPayload, nextOk := <-s.LogChannel: case nextPayload, nextOk := <-logChannel:
if !nextOk { if !nextOk {
s.flushWriters() flushWriters()
return return
} }
s.processLog(nextPayload) processLog(nextPayload)
batchCount++ batchCount++
default: default:
batchCount = 100 // break outer loop batchCount = 100 // break outer loop
} }
} }
case <-ticker.C: case <-ticker.C:
s.flushWriters() flushWriters()
} }
} }
} }
func (s *writerService) processLog(payload logPayload) { func processLog(payload logPayload) {
// 精准路由:根据包裹信息决定写入目标 // 精准路由:根据包裹信息决定写入目标
if payload.writer != nil { if payload.writer != nil {
payload.writer.Log(payload.entry, payload.buf) payload.writer.Log(payload.entry, payload.buf)
@ -166,15 +136,15 @@ func (s *writerService) processLog(payload logPayload) {
} }
} }
func (s *writerService) flushWriters() { func flushWriters() {
curWriters, _ := s.Writers.Load().([]Writer) curWriters, _ := writers.Load().([]Writer)
for _, w := range curWriters { for _, w := range curWriters {
w.Run() w.Run()
} }
s.FilesLock.RLock() filesLock.RLock()
for _, f := range s.Files { for _, f := range files {
f.Run() f.Run()
} }
s.FilesLock.RUnlock() filesLock.RUnlock()
} }