Tag: mysql 5.7

NOT NULL all the things!

Different types of languages deal with this “value” in diverse ways. You can have a more comprehensive list of what NULL can mean on this website. What I like to think about NULL is along the lines of invalid, as if some sort of garbage is stored there. It doesn’t mean it’s empty, it’s just mean that something is there, and it has no value to you.

Databases deal when storing this type in a similar way, PostgreSQL treats it as “unknown” while MySQL treats it as “no data“.

Both databases recommend using \N to represent NULL values where import or exporting of data is necessary.

When to use it

You don’t. Particularly, I DON’T recommend using NULL.

NULL doesn’t mean empty

So if you want to represent lack of data or optional fields use a default value. It’s bad sign of architecture having NULLABLE fields, there is an extra case to test and to write for. It adds unnecessary complexity.

However, there is one case where I do think NULL is acceptable. And that is when working with MySQL date related fields. I will talk more about this further down.

How to Query it

MySQL doesn’t recognize field = NULL because, remember, NULL means invalid, not empty. Thus using it will not return any rows.

As much as NULL value will never be equal to another NULL, when using ORDER BY, GROUP BY and DISTINCT, the server interprets the values as equal. Aggregators functions such as MIN(), SUM() and COUNT() ignore NULL values, except for COUNT(*) that counts rows, and not columns.

When using ORDER BY a column is nullable the NULL values appear first if instructed as ASC and in the end if DESC is requested.

PostgreSQL on the other hand has an option to convert equal comparisons expressions to field IS NULL, if enabled (transform_num_equals).

The ordering for ORDER BY depends on indexing of the field, by default NULL comes first, but you can specify when creating an index where the NULL values should be: top or bottom.

For aggregators functions, PostgreSQL works the same way.

Performance

Having NOT NULL columns permits similar performance on MySQL as an column = 1 do. However that doesn’t happen in LEFT JOIN operations while a field could be NULL. But this is for the type of queries where IS NULL is used:

# Assuming that `active` column is NOT NULL
SELECT * FROM users WHERE active = 1 OR active IS NULL;

Summing up directly from MySQL documentation:

Declare columns to be NOT NULL if possible. It makes SQL operations faster, by enabling better use of indexes and eliminating overhead for testing whether each value is NULL. You also save some storage space, one bit per column. If you really need NULL values in your tables, use them. Just avoid the default setting that allows NULL values in every column.

The COALESCE() function

This function return the first non-null result of a column. Keep in mind to perform this operation on non-indexed columns. It is slow by its nature, just a friendly warning of when you are using to be mindful of its fallback.

The exception to the rule

I think this applies to MySQL databases. DATE/DATETIME should not be allowed to be NULL if, and only if the sql_mode directive NO_ZERO_DATE is disabled.

What does it mean? NO_ZERO_DATE doesn’t allow for 0000-00-00 to be inserted in a DATE/DATETIME. MySQL 5.7 sql_mode insures some restrictions into the database by default. If for instance you have DATE column that is NOT NULL and doesn’t pass a value to it, 0000-00-00 will be saved, because the column is NOT NULL, BUT will give a warning:

mysql> DESCRIBE users;
+------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+------------------+------+-----+---------+----------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| name | varchar(45) | NO | | NULL | |
| created_at | datetime | NO | | NULL | |
+------------+------------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)

mysql> INSERT INTO users (name) VALUES ("Gabi");
Query OK, 1 row affected, 1 warning (0.01 sec)

mysql> SHOW WARNINGS;
+---------+------+-------------------------------------------------+
| Level | Code | Message |
+---------+------+-------------------------------------------------+
| Warning | 1364 | Field "created_at" doesn't have a default value |
+---------+------+-------------------------------------------------+
1 row in set (0.00 sec)

Making this operation to make sure no zero date will be allowed as it is by default in MySQL 5.7:

SET @@GLOBAL.sql_mode = "STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE;

When trying to insert a similar query:

mysql> INSERT INTO users (name) VALUES ("Blossom");
ERROR 1364 (HY000): Field "created_at" doesn't have a default value

To sum it up

If you have MySQL and NO_ZERO_DATE is in your sql_mode, you should ALWAYS use NOT NULL. MySQL 5.7 brings the mode enabled by default among with other things, read more here.

If you don’t have it enabled for any other reason, then DATE/DATETIME MAY be NULL, because data integrity > performance in this case.

Again, 0000-00-00 IS NOT a valid date.

GROUP BY, are you sure you know it?

New MySQL version, YAY!

MySQL 5.7 is full of new features, like virtual columns, virtual indexes and JSON fields! But, it came with some changes to the default configuration. When running:

SELECT @@GLOBAL.sql_mode;

We get:

ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

What I want to talk about is the ONLY_FULL_GROUP_BY mode. This mode rejects queries where nonaggregated columns are expected, but aren’t on the GROUP BY or HAVING clause. Before MySQL 5.7.5, ONLY_FULL_GROUP_BY was disabled by default, now it is enabled.

You know the drill…

This is a simple statement, people use it everywhere, it shouldn’t be that hard to use, right?

Given the following schema:

Suppose I want to list all users that commented on post_id = 1, MySQL 5.6:

 SELECT * FROM comments c INNER JOIN users u ON c.user_id = u.id WHERE c.post_id = 1 GROUP BY c.user_id; 

And this is the result:

+----+---------+---------+---------+---------------------+----+---------------+----------------------+---------+---------------------+
| id | user_id | post_id | message | created | id | name | email | country | created |
+----+---------+---------+---------+---------------------+----+---------------+----------------------+---------+---------------------+
| 1 | 1 | 1 | NULL | 2016-03-03 21:20:29 | 1 | Lisa Simpson | lisa@simpsons.com | US | 2016-03-03 20:07:23 |
| 2 | 2 | 1 | NULL | 2016-03-03 21:20:39 | 2 | Bart Simpson | bart@simpsons.com | US | 2016-03-03 20:07:28 |
| 4 | 3 | 1 | NULL | 2016-03-03 21:20:56 | 3 | Homer Simpson | nobrain@simpsons.com | US | 2016-03-03 20:07:38 |
+----+---------+---------+---------+---------------------+----+---------------+----------------------+---------+---------------------+
3 rows in set (0.00 sec)
view raw result1.txt hosted with ❤ by GitHub

Same query running on 5.7.11 gives the following results:

[42000][1055] Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'blog.c.id' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by

What does it mean?

What MySQL is complaining about here is this: you grouped rows by c.user_id, but the problem is there are more than one result to be retrieved for the c.id column. Since you didn’t use any aggregators, as min(c.id) for instance, it doesn’t know which result to bring.

Previous versions of MySQL would solve this “magically”. This change is not MySQL being temperamental with you, it is them implementing long old industry standard specifications (SQL/92 and SQL/99) to the database. To rely on results brought in the previous versions of that query is not smart. Those results are unpredictable and totally arbitrary.

From the 5.6 documentation:

MySQL extends the standard SQL use of GROUP BY so that the select list can refer to nonaggregated columns not named in the GROUP BY clause. This means that the preceding query is legal in MySQL. You can use this feature to get better performance by avoiding unnecessary column sorting and grouping. However, this is useful primarily when all values in each nonaggregated column not named in the GROUP BY are the same for each group. The server is free to choose any value from each group, so unless they are the same, the values chosen are indeterminate. Furthermore, the selection of values from each group cannot be influenced by adding an ORDER BY clause. Result set sorting occurs after values have been chosen, and ORDER BY does not affect which values within each group the server chooses.

How do I fix it?

It will make your query more verbose, but it will make it right. There are two ways of doing this.

One way is using aggregators in the fields you need to retrieve and that will be grouped by the email field, for instance.

 SELECT any_value(u.id) AS user_id, any_value(u.name) AS name, u.email, any_value(u.country) AS country, any_value(u.created) AS registration_date, max(c.created) AS last_comment, count(*) AS total_comments FROM comments c INNER JOIN users u ON c.user_id = u.id WHERE c.post_id = 1; 

Another way is to name the fields that will be unique in the GROUP BY clause:

 SELECT u.id AS user_id, u.name, u.email, u.country, u.created AS registration_date, max(c.created) AS last_comment, count(*) AS total_comments FROM comments c INNER JOIN users u ON c.user_id = u.id WHERE c.post_id = 1 GROUP BY u.email, u.id, u.name, u.country, u.created; 

Result for both queries:

 +---------+---------------+----------------------+---------+---------------------+---------------------+----------------+ | user_id | name | email | country | registration_date | last_comment | total_comments | +---------+---------------+----------------------+---------+---------------------+---------------------+----------------+ | 2 | Bart Simpson | bart@simpsons.com | US | 2016-03-03 20:07:28 | 2016-03-03 21:21:08 | 2 | | 1 | Lisa Simpson | lisa@simpsons.com | US | 2016-03-03 20:07:23 | 2016-03-03 21:20:50 | 2 | | 3 | Homer Simpson | nobrain@simpsons.com | US | 2016-03-03 20:07:38 | 2016-03-03 21:20:56 | 1 | +---------+---------------+----------------------+---------+---------------------+---------------------+----------------+ 3 rows in set (0.00 sec) 

In another words, both queries follows SQL/92 specification:

The SQL/92 standard for GROUP BY requires the following:

  • A column used in an expression of the SELECT clause must be in the GROUP BY clause. Otherwise, the expression using that column is an aggregate function.
  • A GROUP BY expression can only contain column names from the select list, but not those used only as arguments for vector aggregates.

The results of a standard GROUP BY with vector aggregate functions produce one row with one value per group.

In the 5.7.5 version, MySQL also implemented SQL/99, which means that if such a relationship exists between name and id, the query is legal. This would be the case, for example, where you group by a primary key or foreign key:

 SELECT u.id AS user_id, u.name, u.email, u.country, u.created AS registration_date, max(c.created) AS last_comment, count(*) AS total_comments FROM comments c INNER JOIN users u ON c.user_id = u.id WHERE c.post_id = 1 GROUP BY u.id; 

You can read more details about how MySQL handles GROUP BY in their documentation.

TL;DR;

According to the documentation, this configuration is being enabled by default because GROUP BY processing has become more sophisticated to include detection of functional dependencies. It also brings MySQL closer to the best practices for SQL language with the bonus of removing the “magic” element when grouping. Having that, grouping fields are no longer arbitrary selected.

Disabling ONLY_FULL_GROUP_BY

If you are upgrading your database server and want to avoid any possible breaks you can disable by removing it from your sql_mode.

Changing in runtime

 SET @@GLOBAL.sql_mode = 'STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION' 

A restart is not necessary, but a reconnection is.

Change permanently

If you want to disable it permanently, add/edit the following in your my.cnf file:

sql_mode = "STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"

For this change a service restart is required:

 $ mysql service restart