Jolokia

tl;dr: no pretty graphs 😦 

I’ve never heard of this before, it’s a DevOps kind of thing. Jolokia will expose the Actuator JMX beans over HTTP at the Jolokia endpoint. This is so that the metrics exposed by Actuator can be sent to other services to more easily make sense of them and to make pretty graphs.

The idea in the tutorial is to send the data from Spring Boot Actuator via Telegraf  to InfluxDB. Then Grafana pulls the data to make pretty Graphs.

• InfluxDB – time series database
• Telegraf – server for collecting and writing metrics
• Grafana – server used to visualize time-series data

This seems a bit nuts to me, how many tools do we need ffs. Anyway, lets go with the flow.

Add jolokia to the pom.xml file so Spring Boot can auto-configure the jolokia end-point.

<dependency>
<groupId>org.jolokia</groupId>
<artifactId>jolokia-core</artifactId>
</dependency>

version: '2.1'

[tags]
dc = "local-1"

[agent]
interval = "5s"

[[inputs.jolokia]]
context = "/jolokia"

[[inputs.jolokia.servers]]
name = "jolokia-server"
host = "section-2-3-service"
port = "8080"

[[inputs.jolokia.metrics]]
name = "metrics"
mbean = "org.springframework.boot:name=metricsEndpoint,type=Endpoint"
attribute = "Data"

[outputs]
[outputs.influxdb]
url = "http://influxdb:8086"
database = "metrics"
precision = "s"

#[[inputs.jolokia]]
# context = "/jolokia/"

#[[inputs.jolokia.servers]]
# name = "jolokia-server"
# host = "section-2-3-service"
# port = "8080"

#[[inputs.jolokia.metrics]]
# name = "metrics"
# mbean = "org.springframework.boot:name=metricsEndpoint,type=Endpoint"
# attribute = "Data"

# https://github.com/influxdata/telegraf/blob/master/etc/telegraf.conf
# # Read JMX metrics from a Jolokia REST agent endpoint
[[inputs.jolokia2_agent]]
# # default_tag_prefix = ""
# # default_field_prefix = ""
# # default_field_separator = "."
#
# # Add agents URLs to query
urls = ["http://localhost:8080/application/jolokia"]
# # username = ""
# # password = ""
# # response_timeout = "5s"
#
# ## Optional TLS config
# # tls_ca = "/var/private/ca.pem"
# # tls_cert = "/var/private/client.pem"
# # tls_key = "/var/private/client-key.pem"
# # insecure_skip_verify = false
#
# ## Add metrics to read
[[inputs.jolokia2_agent.metric]]
name = "metrics"
mbean = "org.springframework.boot:name=metricsEndpoint,type=Endpoint"
paths = ["Data"]

I changed the urls to urls = [“http://localhost:8080/application/jolokia”%5D because that was what was being mapped at start-up. I removed the old target folder, built the application again and re-ran the docker-compose up stuff. Now a different error:

telegraf | 2018-11-07T22:06:50Z E! Error in plugin [inputs.jolokia2_agent]: Unable to gather metrics for http://localhost:8080/application/jolokia: Post http://localhost:8080/application/jolokia/read: dial tcp 127.0.0.1:8080:
connect: connection refused

Urgh! However, looking at http://localhost:8080/application/jolokia in the browser, I got some JSON back!!!! I formatted it below to have a good look at it.

{"request":{"type":"version"},
 "value":{"agent":"1.3.7",
          "protocol":"7.2",
          "config":{"agentId":"172.18.0.3-6-2e77b8cf-servlet",
                    "agentType":"servlet"},
          "info":{"product":"tomcat",
                  "vendor":"Apache",
                  "version":"8.5.16"}
         },
 "timestamp":1541628160,
 "status":200
}

When I navigate to http://localhost:8080/application/jolokia/read I get:

{"stacktrace":"java.lang.IllegalArgumentException: 
Invalid arguments in pathinfo read for command READ\n\t
at org.jolokia.request.JmxRequestFactory.createGetRequest
bla bla bla

Found this from stackoverflow: https://stackoverflow.com/questions/38838526/how-to-use-jolokia-to-retrive-mbean-for-a-class

In general, you can get a list of all your available mbeans like this:

http://yourserver/jolokia/list

You should end up with a large json document that contains everything you might want to fetch. You will see things like

"foo.bar.Log4j": {
  "name=foo,type=MyLogger": {
    "desc": ...
    "attr": {
        ...
}}}

You can now get the attributes using something like this:

http://yourserver/jolokia/read/foo.bar.Log4j:type=name=foo,type=MyLogger

In addition to type and name, you may see other fields as well, for example context or id. This a:b key is the Java ObjectName for your mbean.

Cool, I did it and got back loads of stuff. I searched for metricsEndpoint from the tutorial telegraf.conf:

#[[inputs.jolokia.metrics]]
# name = "metrics"
# mbean = "org.springframework.boot:name=metricsEndpoint,type=Endpoint"
# attribute = "Data"

and found it. Used JSONLint online to prettify it:

"class": "org.springframework.boot.actuate.endpoint.jmx.AuditEventsJmxEndpoint", "desc": ""
}, "name=metricsEndpoint,type=Endpoint": {
	"op": {
		"isSensitive": {
			"args": [],
			"ret": "boolean",
			"desc": "Indicates whether the underlying endpoint exposes sensitive information"
		},
		"getEndpointClass": {
			"args": [],
			"ret": "java.lang.String",
			"desc": "Returns the class of the underlying endpoint"
		},
		"getData": {
			"args": [],
			"ret": "java.lang.Object",
			"desc": "Invoke the underlying endpoint"
		}
	},
	"attr": {
		"EndpointClass": {
			"rw": false,
			"type": "java.lang.String",
			"desc": "Returns the class of the underlying endpoint"
		},
		"Sensitive": {
			"rw": false,
			"type": "boolean",
			"desc": "Indicates whether the underlying endpoint exposes sensitive information"
		},
		"Data": {
			"rw": false,
			"type": "java.lang.Object",
			"desc": "Invoke the underlying endpoint"
		}
	},

Look, there’s an attribute for Data like before.

So, like the stackoverflow advice, I should be able to go

http://localhost:8080/application/jolokia/read/org.springframework.boot.actuate.endpoint.jmx.AuditEventsJmxEndpoint:name=metricsEndpoint,type=Endpoint

Got stuff 😀 but it seems that the name was wrong.

{“request”:{“mbean”:”org.springframework.boot.actuate.endpoint.jmx.AuditEventsJmxEndpoint:name=metricsEndpoint,type=Endpoint”,”type”:”read”},”stacktrace”:”javax.management.InstanceNotFoundException:

I just tried typing in this from the mbean part:

http://localhost:8080/application/jolokia/read/org.springframework.boot:name=metricsEndpoint,type=Endpoint

and I got better stuff 😀

{
	"request": {
		"mbean": "org.springframework.boot:name=metricsEndpoint,type=Endpoint",
		"type": "read"
	},
	"value": {
		"EndpointClass": "org.springframework.boot.actuate.endpoint.MetricsEndpoint",
		"Sensitive": false,
		"Data": {
			"heap.committed": 204800,
			"heap.used": 144214,
			"classes": 7670,
			"httpsessions.max": -1,
			"systemload.average": 0.01,
			"mem.free": 60585,
			"processors": 2,
			"gc.ps_marksweep.count": 2,
			"threads.peak": 23,
			"mem": 263926,
			"counter.status.200.application.root": 1,
			"counter.status.404.application.metrics.name": 1,
			"nonheap.used": 59126,
			"gauge.response.application.jolokia.star-star": 14.0,
			"heap": 451584,
			"gc.ps_marksweep.time": 157,
			"nonheap.init": 2496,
			"counter.status.200.application.metrics": 1,
			"threads": 21,
			"heap.init": 32768,
			"gauge.response.application.metrics": 29.0,
			"threads.totalStarted": 26,
			"gc.ps_scavenge.count": 12,
			"uptime": 1952627,
			"nonheap": 0,
			"gauge.response.application.root": 209.0,
			"instance.uptime": 1939639,
			"httpsessions.active": 0,
			"nonheap.committed": 60800,
			"threads.daemon": 19,
			"classes.unloaded": 0,
			"counter.status.200.application.jolokia.star-star": 8,
			"classes.loaded": 7670,
			"gc.ps_scavenge.time": 159,
			"gauge.response.application.metrics.name": 83.0
		}
	},
	"timestamp": 1541632046,
	"status": 200
}

This is brilliant, but why can’t the application get this data???

I’m next going to focus on why it’s doing a Post and not a Get. Looking at https://jolokia.org/reference/html/protocol.html

Jolokia requests can be sent in two ways: Either as a HTTP GET request, in which case the request parameters are encoded completely in the URL. Or as a POST request where the request is put into a JSON payload in the HTTP request’s body. GET based requests are mostly suitable for simple use cases and for testing the agent via a browser. The focus here is on simplicity. POST based requests uses a JSON representation of the request within the HTTP body. They are more appropriate for complex requests and provide some additional features (e.g. bulk requests are only possible with POST).

The response returned by the agent uses always JSON for its data representation. It has the same format regardless whether GET or POST requests are used.

I used Postman to do the browser GET call again and it worked. Then I tried to do the POST. After playing around a bit (I forgot to use ‘:’ instead of ‘=’) I put the following JSON in the Body:

{
“type” : “read”,
“name” : “metrics”,
“mbean” : “org.springframework.boot:name=metricsEndpoint,type=Endpoint”
}

It couldn’t find it!!!! Yey!!!

So, I re-checked the telegraf.conf file and noticed something… my mbean definition was different:

mbean = “org.springframework.boot.actuate.metrics:name=metricsEndpoint,type=Endpoint”

This was left over from something else I tried. I renamed it to the tutorial example again, and removed the Data bit.

# ## Add metrics to read
[[inputs.jolokia2_agent.metric]]
name = "metrics"
mbean = "org.springframework.boot:name=metricsEndpoint,type=Endpoint"

Re-built project, docker-compose up and… same error.

Ok, changed telegraf.conf again:

# ## Add metrics to read
[[inputs.jolokia2_agent.metric]]
type = "read"
name = "metrics"
mbean = "org.springframework.boot:name=metricsEndpoint,type=Endpoint"

It didn’t like the type stuff in the telegraf.conf file:

telegraf | 2018/11/08 00:05:57 E! Error parsing /etc/telegraf/telegraf.conf, line 81: field corresponding to `type’ is not defined in `*jolokia2.MetricConfig’

Removed type and added paths for the Data bit I forgot about… maybe that will help?

# ## Add metrics to read
[[inputs.jolokia2_agent.metric]]
name = "metrics"
mbean = "org.springframework.boot:name=metricsEndpoint,type=Endpoint"
paths = ["Data"]

Nope. Added paths in Postman and it returned just the same. Hmmmm… it’s not that.

Google: https://github.com/influxdata/telegraf/issues/4371

Since you are running in docker you will need to make sure Telegraf can reach the container you application is in. This depends a bit on how you are starting your containers, in some simple cases you may just need to start the Telegraf container with something like --net=container:yourappcontainer. I also suggest using the docker exec command to run a shell in your Telegraf container for debugging.

Honestly, this makes no sense to me. I was hoping for a tutorial to just click through and learn something. This has been a  minefield and a good look into the DevOps world… urgh! Scratching around on Google for information was impossible. I suppose I have an appreciation of more technologies, just wish i could have had a pretty graph 😦