go-ethereum/metrics
Daniel Liu cc4ea7a685 metrics: refactor metrics (#28035)
This change includes a lot of things, listed below.

The interfaces have been split up into one write-interface and one read-interface, with `Snapshot` being the gateway from write to read. This simplifies the semantics _a lot_.

Example of splitting up an interface into one readonly 'snapshot' part, and one updatable writeonly part:

```golang
type MeterSnapshot interface {
	Count() int64
	Rate1() float64
	Rate5() float64
	Rate15() float64
	RateMean() float64
}

// Meters count events to produce exponentially-weighted moving average rates
// at one-, five-, and fifteen-minutes and a mean rate.
type Meter interface {
	Mark(int64)
	Snapshot() MeterSnapshot
	Stop()
}
```

This PR makes the concurrency model clearer. We have actual meters and snapshot of meters. The `meter` is the thing which can be accessed from the registry, and updates can be made to it.

- For all `meters`, (`Gauge`, `Timer` etc), it is assumed that they are accessed by different threads, making updates. Therefore, all `meters` update-methods (`Inc`, `Add`, `Update`, `Clear` etc) need to be concurrency-safe.
- All `meters` have a `Snapshot()` method. This method is _usually_ called from one thread, a backend-exporter. But it's fully possible to have several exporters simultaneously: therefore this method should also be concurrency-safe.

TLDR: `meter`s are accessible via registry, all their methods must be concurrency-safe.

For all `Snapshot`s, it is assumed that an individual exporter-thread has obtained a `meter` from the registry, and called the `Snapshot` method to obtain a readonly snapshot. This snapshot is _not_ guaranteed to be concurrency-safe. There's no need for a snapshot to be concurrency-safe, since exporters should not share snapshots.

Note, though: that by happenstance a lot of the snapshots _are_ concurrency-safe, being unmutable minimal representations of a value. Only the more complex ones are _not_ threadsafe, those that lazily calculate things like `Variance()`, `Mean()`.

Example of how a background exporter typically works, obtaining the snapshot and sequentially accessing the non-threadsafe methods in it:
```golang
		ms := metric.Snapshot()
                ...
		fields := map[string]interface{}{
			"count":    ms.Count(),
			"max":      ms.Max(),
			"mean":     ms.Mean(),
			"min":      ms.Min(),
			"stddev":   ms.StdDev(),
			"variance": ms.Variance(),
```

TLDR: `snapshots` are not guaranteed to be concurrency-safe (but often are).

I also changed the `Sample` type: previously, it iterated the samples fully every time `Mean()`,`Sum()`, `Min()` or `Max()` was invoked. Since we now have readonly base data, we can just iterate it once, in the constructor, and set all four values at once.

The same thing has been done for runtimehistogram.

Back when ResettingTImer was implemented, as part of https://github.com/ethereum/go-ethereum/pull/15910, Anton implemented a `Percentiles` on the new type. However, the method did not conform to the other existing types which also had a `Percentiles`.

1. The existing ones, on input, took `0.5` to mean `50%`. Anton used `50` to mean `50%`.
2. The existing ones returned `float64` outputs, thus interpolating between values. A value-set of `0, 10`, at `50%` would return `5`, whereas Anton's would return either `0` or `10`.

This PR removes the 'new' version, and uses only the 'legacy' percentiles, also for the ResettingTimer type.

The resetting timer snapshot was also defined so that it would expose the internal values. This has been removed, and getters for `Max, Min, Mean` have been added instead.

A lot of types were exported, but do not need to be. This PR unexports quite a lot of them.

metrics: refactor metrics (28035)
2024-12-13 14:00:13 +08:00
..
exp metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
influxdb metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
internal metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
librato metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
prometheus metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
testdata metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
config.go metrics/influxdb: support V2 (#23194) 2024-12-13 14:00:12 +08:00
counter.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
counter_float64.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
counter_float_64_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
counter_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
cpu.go metrics: improve accuracy of CPU gauges (#26793) 2024-12-13 14:00:13 +08:00
cpu_disabled.go metrics: add go:build lines (#23468) 2024-12-13 14:00:12 +08:00
cpu_enabled.go metrics: improve accuracy of CPU gauges (#26793) 2024-12-13 14:00:13 +08:00
cputime_nop.go metrics: improve accuracy of CPU gauges (#26793) 2024-12-13 14:00:13 +08:00
cputime_unix.go metrics: improve accuracy of CPU gauges (#26793) 2024-12-13 14:00:13 +08:00
debug.go metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910) 2018-02-23 11:56:08 +02:00
debug_test.go metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910) 2018-02-23 11:56:08 +02:00
disk.go all: fix license headers one more time 2015-07-23 18:35:11 +02:00
disk_linux.go all: fix ineffectual assignments and remove uses of crypto.Sha3 2017-01-09 16:24:42 +01:00
disk_nop.go all: fix staticcheck warning ST1005: incorrectly formatted error string 2024-10-24 09:48:20 +08:00
ewma.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
ewma_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
FORK.md metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910) 2018-02-23 11:56:08 +02:00
gauge.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
gauge_float64.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
gauge_float64_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
gauge_info.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
gauge_info_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
gauge_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
graphite.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
graphite_test.go metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910) 2018-02-23 11:56:08 +02:00
healthcheck.go metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910) 2018-02-23 11:56:08 +02:00
histogram.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
histogram_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
inactive.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
init_test.go metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910) 2018-02-23 11:56:08 +02:00
json.go metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910) 2018-02-23 11:56:08 +02:00
json_test.go all: fix staticcheck warning SA1006 2024-10-28 15:13:21 +08:00
LICENSE metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910) 2018-02-23 11:56:08 +02:00
log.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
memory.md metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910) 2018-02-23 11:56:08 +02:00
meter.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
meter_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
metrics.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
metrics_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
opentsdb.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
opentsdb_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
README.md metrics: change links in README.md to https (#20182) 2024-12-13 14:00:11 +08:00
registry.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
registry_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
resetting_sample.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
resetting_timer.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
resetting_timer_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
runtimehistogram.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
runtimehistogram_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
sample.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
sample_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
syslog.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
timer.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
timer_test.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
validate.sh metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910) 2018-02-23 11:56:08 +02:00
writer.go metrics: refactor metrics (#28035) 2024-12-13 14:00:13 +08:00
writer_test.go metrics: use slices package for sorting (#27493 #27909) 2024-12-13 14:00:13 +08:00

go-metrics

travis build status

Go port of Coda Hale's Metrics library: https://github.com/dropwizard/metrics.

Documentation: https://godoc.org/github.com/rcrowley/go-metrics.

Usage

Create and update metrics:

c := metrics.NewCounter()
metrics.Register("foo", c)
c.Inc(47)

g := metrics.NewGauge()
metrics.Register("bar", g)
g.Update(47)

r := NewRegistry()
g := metrics.NewRegisteredFunctionalGauge("cache-evictions", r, func() int64 { return cache.getEvictionsCount() })

s := metrics.NewExpDecaySample(1028, 0.015) // or metrics.NewUniformSample(1028)
h := metrics.NewHistogram(s)
metrics.Register("baz", h)
h.Update(47)

m := metrics.NewMeter()
metrics.Register("quux", m)
m.Mark(47)

t := metrics.NewTimer()
metrics.Register("bang", t)
t.Time(func() {})
t.Update(47)

Register() is not threadsafe. For threadsafe metric registration use GetOrRegister:

t := metrics.GetOrRegisterTimer("account.create.latency", nil)
t.Time(func() {})
t.Update(47)

NOTE: Be sure to unregister short-lived meters and timers otherwise they will leak memory:

// Will call Stop() on the Meter to allow for garbage collection
metrics.Unregister("quux")
// Or similarly for a Timer that embeds a Meter
metrics.Unregister("bang")

Periodically log every metric in human-readable form to standard error:

go metrics.Log(metrics.DefaultRegistry, 5 * time.Second, log.New(os.Stderr, "metrics: ", log.Lmicroseconds))

Periodically log every metric in slightly-more-parseable form to syslog:

w, _ := syslog.Dial("unixgram", "/dev/log", syslog.LOG_INFO, "metrics")
go metrics.Syslog(metrics.DefaultRegistry, 60e9, w)

Periodically emit every metric to Graphite using the Graphite client:


import "github.com/cyberdelia/go-metrics-graphite"

addr, _ := net.ResolveTCPAddr("tcp", "127.0.0.1:2003")
go graphite.Graphite(metrics.DefaultRegistry, 10e9, "metrics", addr)

Periodically emit every metric into InfluxDB:

NOTE: this has been pulled out of the library due to constant fluctuations in the InfluxDB API. In fact, all client libraries are on their way out. see issues #121 and #124 for progress and details.

import "github.com/vrischmann/go-metrics-influxdb"

go influxdb.InfluxDB(metrics.DefaultRegistry,
  10e9, 
  "127.0.0.1:8086", 
  "database-name", 
  "username", 
  "password"
)

Periodically upload every metric to Librato using the Librato client:

Note: the client included with this repository under the librato package has been deprecated and moved to the repository linked above.

import "github.com/mihasya/go-metrics-librato"

go librato.Librato(metrics.DefaultRegistry,
    10e9,                  // interval
    "example@example.com", // account owner email address
    "token",               // Librato API token
    "hostname",            // source
    []float64{0.95},       // percentiles to send
    time.Millisecond,      // time unit
)

Periodically emit every metric to StatHat:

import "github.com/rcrowley/go-metrics/stathat"

go stathat.Stathat(metrics.DefaultRegistry, 10e9, "example@example.com")

Maintain all metrics along with expvars at /debug/metrics:

This uses the same mechanism as the official expvar but exposed under /debug/metrics, which shows a json representation of all your usual expvars as well as all your go-metrics.

import "github.com/rcrowley/go-metrics/exp"

exp.Exp(metrics.DefaultRegistry)

Installation

go get github.com/rcrowley/go-metrics

StatHat support additionally requires their Go client:

go get github.com/stathat/go

Publishing Metrics

Clients are available for the following destinations: