MySQL Blog Archive
For the latest blogs go to blogs.oracle.com/mysql
Connector/J 8.0.11, the Face for Your Brand New Document-oriented Database
Posted on Apr 19, 2018 by Filipe Silva
Category: Connectors / Languages
Tags: connectors, mysql-development, connector-j, java, jdbc, x-devapi

This is exciting news: MySQL 8.0 is officially GA!

This time around we do not only offer you a new MySQL Server full of exciting new features, but also a brand new toolset that let you make the most out of MySQL, including the now GA Java database driver, MySQL Connector/J 8.0.11!

This story is about Connector/J but before even I get started I must say a few words about the new features that conducted the development of this new driver. If you prefer, you can jump directly to the Connector/J 8.0 topic below.

What’s new in MySQL Server 8.0?

There’s a new Client/Server communication protocol in MySQL: the X Protocol

The X Protocol introduced in MySQL 8.0 series is the backbone for the document-oriented paradigm in the MySQL ecosystem. The new X Protocol and the old MySQL Client/Server Protocol are both available in this new MySQL Server. It is up to the clients to choose which one to implement for connecting to MySQL servers, either through pure protocol implementations or via a MySQL C API library, for example. Clearly the easiest option is to use one of the MySQL connectors that already implement either one or both of the protocols that we offer for the programming language of your choice.

The X Protocol is implemented as a plugin for the MySQL Server—the X Plugin. By default the X Plugin is listening on TCP port 33060.

Among the main features in the X Protocol you’ll find data serialization via Protocol Buffers, Vectored I/O, Pipelining, SASL authentication, extensible messages via protobuf, and so on. We are also working on other interesting features so stay tuned. I invite you to go to the X Protocol documentation to learn all about it.

So, there’s also the new X Plugin

As mentioned before, the X Plugin implements the X Protocol on the MySQL Server side.

Besides implementing the X Protocol, the X Plugin is responsible for defining the message structure that provides the base for the X DevAPI – the API that exposes a MySQL server as a document-oriented database. It is also responsible for communicating with the internal components of the server: the storage engines, the query parser, the query optimizer, query executors, and so on. In the other end, it also manages all active X DevAPI sessions and handles clients’ requests.

As of MySQL 8.0 GA the X Plugin is loaded by default. Chances are that it is already available for you in your new server installation.

Knowledge of the X Plugin will provide you insights into the features of the X DevAPI as well as how MySQL handles those features. The X Plugin documentation is, again, your best companion for learning about the plugin.

MySQL as a document-oriented database: the X DevAPI

It is through the new X DevAPI that the document-oriented database features in MySQL are exposed. This is how you use MySQL as a document store, a schema-less, and therefore schema-flexible storage system for documents. When using MySQL as a document store you do not need to know and define all possible attributes of your data before storing and operating with it. This differs from working with a relational database and storing data in a table, where all columns of the table must be known and defined before adding data to the database.

Long ago MySQL added support for JSON data along with a large variety of functions and operations that allow creating, manipulating, and handling JSON structures and values. This new data type combined with the old and reliable InnoDB based tables, with the ability to create dynamically generated columns and indexes over them, constituted the right environment for spawning MySQL’s interpretation of our document store.

The X DevAPI provides you with a unifying API for creating and handling documents in JSON format in MySQL, which you can use to develop applications. It offers a modern programming interface with a simple yet powerful design that provides support for established industry-standard concepts. One of the major keynotes in the X DevAPI is harmony you get when working with different MySQL Connectors that implement it. It becomes so easy for a developer to switch between different programming languages and still get the same “look & feel” from different Connectors.

Although MySQL Connectors are the typical entry point for using the X DevAPI, you can actually start using it even without jumping into a programming language; just try it in the new command-line client: MySQL Shell.

MySQL Reference Manual contains an overview of MySQL as a Document Store which I recommend reading. The X DevAPI User Guide is the central source of information for all common aspects of this new and exiting API.

And here’s the new MySQL command-line client: MySQL Shell

MySQL Shell is an advanced command-line client and code editor for MySQL server. In addition to the usual SQL functionality provided by the mysql client, MySQL Shell provides scripting capabilities for JavaScript and Python and includes APIs for working with MySQL. When MySQL Shell is connected to the MySQL server through the X Protocol, the X DevAPI can be used to work with both relational tables and document data.

This is the perfect companion tool for your development work with any of the new Connectors supporting X DevAPI, as it allows viewing and manipulating documents and collections easily, so that you can prototype X DevAPI calls, expressions syntax, and a lot more before starting coding them into your projects.

As of MySQL Server 8.0.11, MySQL Shell is not included in the server’s default toolset, so you will have to install it separately. Get started with MySQL Shell by downloading it from MySQL Downloads and get more info from its User Guide.

TL;DR: After all, this post is about Connector/J!

Connector/J 8.0, the new GA JDBC/X DevAPI driver for MySQL databases

I am particularly proud to announce that Connector/J 8.0 is now GA. This is the officially recommended version to use in your projects from now on. And it doesn’t matter what server version you are using.

Note that Connector/J became GA with version 8.0.11, skipping version 8.0.10, which was expected to appear after the last Release Candidate. If you haven’t heard about it, just know that several MySQL products, including Connector/J, will be version aligned with MySQL Server releases from now on. This means that you can expect new releases of Connector/J every time there is a MySQL Server 8.0 release. This doesn’t mean, however, that you must also align the Connector/J library in your projects with the server version you are connecting to. As always, our recommendation is to use the latest Connector/J library, no matter what server version you use.

Connector/J 8.0 is the rightful successor of the widely used Connector/J 5.1. Now compiled in Java 8 only, it comes with huge refactoring. Connector/J 8.0 is better organized and follows better and more up-to-date coding practices, as well as many of the exciting features of Java 8. This is also the driver to use with the new X DevAPI in the Java ecosystem.

We know that, despite our best efforts, no code is ever perfect. This is why we are continuously working on it and we really appreciate your comments and suggestions for improving this product. Therefore, please continue to use the resources we provide to get in touch with us, including the MySQL bug system, the Connector/J forum, the GitHub repository or MySQL Community Slack, with which you can talk to us directly.

Connector/J 8.0 does not only implement the JDBC 4.2 API, but also the X DevAPI for new document store features, so you can use them interchangeably. As matter of fact, the two API implementations share a lot of core base code. This is so to offer you a better user experience and versatility in your projects.

The new X DevAPI implementation provides an up to date and more user-friendly programming method with its Fluent Interface API. It all starts with obtaining a Session object, which is similar to getting a JDBC Connection, and, from then on, you have a tool to work with with Schemas, Collections and even old-style Tables. Always remember to keep Connector/J Developer Guide and the Connector/J X DevAPI Reference handy for quick reference.

Getting started with Connector/J 8.0 and X DevAPI

Initial setup & getting Connector/J binaries

Your first step should be setting up a new Java project using whatever method you prefer, immediately followed by attaching the Connector/J library. You can do it either by setting up a Maven dependency in your pom.xml or by downloading the Connector/J bundle from the official MySQL Downloads page.

In order to setup a Maven dependency just add the following in the <dependencies> section of your project’s pom.xml file, then proceed with the usual Maven commands to build and run your project:

<dependency>
  <groupId>mysql</groupId>
  <artifactId>mysql-connector-java</artifactId>
  <version>8.0.11</version>
</dependency>

If you choose to download the Connector/J bundle from the MySQL repositories, you must extract its contents to some temporary place and copy the following files to your project’s libraries directory:

  • mysql-connector-java-8.0.11.jar
  • lib/protobuf-java-2.6.0.jar

Instead of using the provided Protocol Buffers Java API library, you could also get protobuf-java-2.6.0.jar from its official repositories. Note that there are a few other Java libraries in the libs directory and you are not required to add them to your project if you don’t use any of the features that need them.

If you are bold enough, you can even grab the Connector/J source code and compile it yourself. This is not a task for a Java novice but you will find some help in Connector/J Developer Guide – Installing from source.

I will not go into more details but you must make sure the classpath settings include the two libraries in all further Java commands or project configurations in your IDE.

Needless to say that you will also need a MySQL Server 8.0.11 (at least) in order to proceed. If you don’t have one, you can install it in few easy steps; just choose the right package for you and follow the instructions in the Reference Manual.

Hello World, using JSON documents and X DevAPI

It is time to create the a simple demo Java class. Lets call it HelloWorld.java.

public class HelloWorld {
    public static void main(String[] args) throws Exception {
        // your code goes here.
    }
}

You should implement proper exception handling in your application. For the sake of this demo I’ll just throw Exception  in the method main .

First thing to do is to establish a connection to the MySQL server, that is, obtaining a Session in X DevAPI terminology. Lets assume the X Plugin is listening on the default port 33060. To do this in Connector/J you first need to instantiate a SessionFactory  and then use it to create the Session  object for you:

// add to imports:
import com.mysql.cj.xdevapi.Session;
import com.mysql.cj.xdevapi.SessionFactory;

// put inside the main() method:
SessionFactory sFact = new SessionFactory();
Session sess = sFact.getSession("mysqlx://johndoe:password@localhost:33060");

Replace the user and password according to your database setup. The same goes for the host and port parts.

Note that jonhdoe is just a standard MySQL user. Since this user will be used to create objects in the database, please make sure that it has the required permissions to do so.

The connection string in this sample is pretty straightforward. It identifies the protocol to use (mysqlx), the user, password, hostname and port. Except for the part johndoe:password@ this is quite similar to the old JDBC connection strings. In fact, Connector/J 8.0 lets you use this same syntax for JDBC connections too.

The previous code returns a Session  object you can to use manage schemas. A Schema is essentially a MySQL database. Lets create one for this demo:

// add to imports:
import com.mysql.cj.xdevapi.Schema;

// put inside the main() method:
Schema schema = sess.createSchema("demo", true);

By now there should be a new database named demo in your MySQL server instance.

The Schema object you have got lets you manage collections and tables in the database. Just keep in mind that collections are just tables with a specific column structure, which you shouldn’t modify in any way other than through a X DevAPI interface.

Let’s now create a collection and add a few documents to it:

// add to imports:
import com.mysql.cj.xdevapi.Collection;

// put inside the main() method:
Collection col = schema.createCollection("greetings");
col.add("{\"language\":\"English\", \"greeting\":\"Hello World\"}")
        .add("{\"language\":\"Português\", \"greeting\":\"Olá Mundo\"}")
        .add("{\"language\":\"Français\", \"greeting\":\"Bonjour Monde\"}")
        .execute();

So, this creates a table named greetings in the demo database. This table has two columns: doc and _id:

> desc demo.greetings;
+-------+---------------+------+-----+---------+------------------+
| Field | Type          | Null | Key | Default | Extra            |
+-------+---------------+------+-----+---------+------------------+
| doc   | json          | YES  |     | NULL    |                  |
| _id   | varbinary(32) | NO   | PRI | NULL    | STORED GENERATED |
+-------+---------------+------+-----+---------+------------------+

The X DevAPI, through the X Plugin, manages this 1:1 mapping between the collection greetings and the table with the same name. Following X DevAPI code related to this collection operates over this relational table using a panoply of JSON functions available in MySQL server, however, all this happens internally in the server and it is transparent to clients.

Lets wrap this demo up with a couple of data retrieving operations:

// add to imports:
import com.mysql.cj.api.xdevapi.DocResult;
import com.mysql.cj.xdevapi.DbDoc;

// put inside the main() method:
DocResult res = col.find("$.language='Português'").execute();
for (DbDoc doc : res.fetchAll()) {
    System.out.println(doc);
}

res = col.find("greeting LIKE '%World%'").execute();
for (DbDoc doc : res.fetchAll()) {
    System.out.println(doc);
}

This time we work with the Collection instance in order to fetch its data using two simple filters, exactly as what you would do using standard SQL tables.

The returned object DocResult  can then be used to get the list of the documents filtered from the collection. Each document is exposed as a DbDoc  instance. DbDoc  is particularly important in Connector/J X DevAPI implementation, because it represents a JSON object, something that Java doesn’t support natively.

This should be the end result of this simple demo:

{
"_id" : "c8075d720b0cb5894eabec605229e304",
"greeting" : "Olá Mundo",
"language" : "Português"
}
{
"_id" : "a10ea22f459b8c974f10994d278c653d",
"greeting" : "Hello World",
"language" : "English"
}

With exception of the _id values, which are generated each time you add a document to a collection, you should get the same results as me, if you try out the demo.

What else?

This little demo was about a few basic CRUD operations on Collections, but there’s a lot more in the X DevAPI:

  • CRUD operations on Collections and Tables;
  • Parameter binding;
  • Document patching;
  • Various types of results and metadata;
  • Raw (SQL) statements execution;
  • Transaction save-points;
  • Row locking mechanisms;
  • Simplified expression syntax, etc.

Closing comments

MySQL Connector/J 8.0.11 is the new GA version of Connector/J for MySQL databases. It implements the X DevAPI, an exciting new feature that enables the use of MySQL Server 8.0 as a document-oriented database.

The 8.0 series of the Connector/J driver, fully re-factored from the previous GA version 5.1, requires Java 8 or above to work with. If that is a limitation for you, you can still use the Connector/J 5.1 series that supports Java 5 up to Java 8. We’ll keep maintaining the 5.1 series for the time being, but be mindful that you will only be able to use JDBC with it.

Although a feature preview version of the X DevAPI is also available in the MySQL Server 5.7 series, which you could use with Connector/J 8.0.11 and later, I recommend that you use the X DevAPI with MySQL 8.0. The X Plugin implementation in the  5.7 series has some known limitations and, most likely, no new features will be back ported to MySQL 5.7.

Exiting times are coming for all of us. Enjoy the new features we offer you and let us know what you think about them.

On behalf of the MySQL Connector/J development team, I wish you a pleasant programming experience with Connector/J 8.0.

 Search
Categories