Bring Elastic Observability to your Java application

How to plan and implement observability of a Java application with Elasticsearch — a step-by-step guide

1 2 Page 2
Page 2 of 2

API test results and response time measurements

In order to measure proper functionality of the API endpoints we need to POST some URL encoded data, read the response, and verify it. This is often done manually by using curl or the Postman API Client. By automating the testing with Heartbeat, the response time and test results are available alongside the logs, APM, and other metrics for the service. Heartbeat monitors the availability of services by testing API endpoints for proper responses, checking websites for content and response codes, verifying ICMP pings, etc. 

Install Heartbeat

Follow the instructions in Kibana Home > Add metric data > Uptime monitors. When you are instructed to edit the heartbeat.monitors setting in the heartbeat.yml file, replace the existing monitor with this API test:

# Configure monitors inline
heartbeat.monitors:
- type: http
  name: SpringToDoApp
  schedule: '@every 5s'
  urls: ["http://roscigno-obs:8080/demo/add"]
  check.request:
    method: POST
    headers:
      'Content-Type': 'application/x-www-form-urlencoded'
    body: "name=first&email=someemail%40someemailprovider.com"
  check.response:
    status: 200
    body:
      - Saved
      - saved
  response.include_body: 'always'

Run the setup command and start Heartbeat as directed in Kibana > Add metric data > Uptime monitors. At the bottom of that page is a link to the Uptime App.

elastic uptime 1 Elastic

And

elastic uptime 2 Elastic

Distributed tracing of the application including the database

Elastic APM instruments your applications to ship performance metrics to Elasticsearch for visualization in Kibana with the APM app. By adding the APM jar file to the command used to launch the application I get distributed tracing so I can see where my app is spending time (whether it is in the Java code or in the calls to MariaDB).

The process is provided in Kibana Home > Add APM > Java and consists of downloading the jar file and using the Java instrumentation API to start the agent. Open the Home > Add data > APM page, choose Java, and download the jar file:

elastic apm 1 Elastic

Load the APM Kibana objects

elastic apm 2 Elastic

I prefer to use environment variables for the javaagent properties, so I take the details provided and set the environment variables:

$ cat environment
export
ELASTIC_APM_SERVER_URL=https://1530f7c8afdf402eb281750f0b127bc4.apm.us-central1.gcp.cloud.es.io:443

export ELASTIC_APM_SECRET_TOKEN=WjyW67R0eSWDhILWDD

export ELASTIC_APM_SERVICE_NAME=winter-mysql

export ELASTIC_APM_APP_PACKAGES=com.example

I am launching the app via ./mvnw spring-boot:run and sourcing the environment variables in the Maven Wrapper:

exec "$JAVACMD" \
  -javaagent:./elastic-apm-agent.jar \
  -Delastic.apm.service_name=${ELASTIC_APM_SERVICE_NAME:-demo-app} \
  -Delastic.apm.server_url=${ELASTIC_APM_SERVER_URL} \
  -Delastic.apm.secret_token=${ELASTIC_APM_SECRET_TOKEN} \
  -Delastic.apm.application_packages=org.example \
  $MAVEN_OPTS \
  -classpath "$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar" \
  "-Dmaven.home=${M2_HOME}" \
  "-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \
  ${WRAPPER_LAUNCHER} "$@"

As soon as the application is started, the API tests set up earlier with Heartbeat will result in traces in Elasticsearch. Launch APM and navigate to your service:

elastic apm 3 Elastic

Out of the box and without any code changes you will see detailed performance information on response time for incoming requests, database queries, calls to caches, external HTTP requests, JVM metrics, errors, and more. This makes it easy to pinpoint and fix performance problems quickly. Looking into the AddNewUser transactions (which is where Heartbeat is performing synthetic API testing) I can see the breakdown of a transaction:

elastic apm 4 Elastic

And digging into the “INSERT INTO user” span I can see the SQL used, transaction ID, etc.:

elastic apm 5 Elastic

There is so much more to do with APM, in this article we are just looking at the out of the box traces with zero code changes. To add custom spans and more, see the documentation.

Elastic Common Schema

The Elastic Common Schema is a powerful way to correlate information from multiple sources. It specifies a common set of fields for logs and metrics that is used by all of the Elastic Beats, and it is extendable by you. By adopting the Elastic Common Schema in the Spring Boot app, the logs are both clear and consistent with the other data sources we are ingesting. This helps correlate a record from the application log with a database metric and an APM trace. You can read about logging best practices for Java apps and watch a video showing a Spring Boot application getting a logging overhaul.

Put it all together

In this article I covered several ways of collecting data from and about your application and the infrastructure. You can do the same and see logs, metrics, synthetic testing of your APIs, APM traces, and more. Earlier I said that you should not limit yourself in the type and amount of data that you bring in; decide what will help you prevent downtime and expand your list of sources. In future articles I will go into topics like alerting, machine learning, and bringing in custom data sources.

If you want to discuss what I have shown, head to discuss.elastic.co and start a conversation. Pop @DanRoscigno in the post and I will try to help.

More things to consider

I skipped over “When you page me in the middle of the night you will give me the info I need to find the problem” from the goals, and a few other things that should be part of every project. In a future article I will write about building service level dashboards, best practices for ingest with Beats, alerting, and real user monitoring. In the meantime, see the links.

If you’re ready to try it out yourself, the easiest way to get started is to take advantage of the free 14-day trial of the Elasticsearch Service on Elastic Cloud — the official hosted Elasticsearch offering from Elastic, which includes Kibana. If you prefer, you can also download Elasticsearch and Kibana to run on your laptop or deploy in a data center. Free hands-on training to build your skills is available at elastic.co/training/free. There is a group of observability related courses listed, and I would also suggest the Kibana course on that page as it walks you through creating your own visualizations.

Dan Roscigno is a customer success manager focused on bringing decades of operations experience to customers in webinars, videos, tutorials, and documentation. He has been an EMT, sysadmin, developer, tech lead in operations, consultant, learning content developer, and a product marketing manager. He loves to make ops less stressful. Contact him on Twitter or discuss.elastic.co: @DanRoscigno.

New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to newtechforum@infoworld.com.

Copyright © 2020 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2