Repository navigation
Templates for MySQL - #9
Merged
Merged
Conversation
CreateDatabaseFromTemplate now returns the name of the database it made. PostgreSQL returns the requested name; a provider that prepares clones in advance and cannot rename them (MySQL) returns the name it already has. New uses the returned name for the DSN, Name and cleanup.
MySQL has nothing that clones a database, so cloneDatabase copies a template into a new database: charset and collation, tables from SHOW CREATE TABLE (indexes, foreign keys, checks, AUTO_INCREMENT position), rows with INSERT ... SELECT leaving generated columns to be recomputed, then views with the schema name rewritten (in passes, since views may depend on views), triggers after the data so they do not fire during the copy, and procedures and functions, each under its own sql_mode and without its DEFINER clause. Tables named _testdb_* are testdb's own bookkeeping inside a template and are not copied. A failed clone drops what it created.
mysql.EnsureTemplate builds a template database once and gives each test a clone of it, with the guarantees postgres has: the first process builds, others wait and reuse, the last one out drops it, a crashed process drops out of the count, and a half-built template is never used. MySQL's means differ: GET_LOCK for the build step, bookkeeping tables inside the template (_testdb_info, _testdb_users, _testdb_ready) for the fingerprint, the use count and prepared databases, and liveness checked against the process list together with the server's start time, since connection ids restart with the server. MySQL cannot rename a database, so a template is built under its final name with a completeness marker written last. A clone costs hundreds of milliseconds for a real schema (every table is created again), so a Template keeps clones prepared ahead of demand: WithReadyDatabases (default GOMAXPROCS) sets how many, a refiller clones in the background, and a test takes one without ever waiting; when none is ready the test clones on demand. CreateDatabaseFromTemplate returns the prepared database's name. Release drops the prepared databases and sweeps those of dead processes. testdb.NewDatabaseName is exported so prepared databases are named like every other test database.
A clone's cost is the server's per-table DDL, which the server handles concurrently, so the pool uses several workers (WithReadyWorkers, default the pool size) with exact slot accounting: size = free slots + clones in flight + names ready. A worker records the database in _testdb_ready before cloning into it, so a clone cut short by Release or a crash is on the books and gets dropped or swept rather than orphaned; cloneDatabase rolls back through a fresh connection, since database/sql discards the one whose statement was cancelled. The clone reads every table's columns in one query. Template.Ready reports how many clones are waiting.
…MySQL mysql/internal/concurrencytest runs as several real processes sharing one template with ready pools: setup runs once, nothing is left behind, and a process killed while holding the template and prepared databases is swept by the survivor, which drops the template as the last one out. Benchmarks with a 50-table schema (MySQL 8.4 in Docker, laptop): an empty database 8 ms; fresh database plus migrations 498 ms; a clone on demand 768 ms, no faster than migrating since it creates the same tables; a prepared clone 6 ms at 100% pooled once tests take at least ~200 ms each, and 130-200 ms mean for 50 ms tests back to back, where the pool cannot refill as fast as tests consume. The README lists MySQL with those numbers and what the clone does and does not copy.
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 4, PR 2 of 2. With this, MySQL meets
PROVIDER_PARITY.mdand is listed in the README: isolation and speed, same API, its own numbers.What MySQL lacks, and what we do instead
CREATE DATABASE … TEMPLATE.cloneDatabasecopies a template object by object: charset/collation, tables fromSHOW CREATE TABLE(indexes, FKs, checks,AUTO_INCREMENTposition), rows viaINSERT … SELECTwith generated columns recomputed, views (schema name rewritten, created in passes), triggers after the data so they don't fire during the copy, procedures and functions under their ownsql_mode. Not copied: events, grants. A fidelity test checks each of these in the clone.RENAME DATABASE. A template is built under its final name with a completeness marker written last; checks run under the build lock, so the only way to meet an incomplete template is after a crashed build, and the rule is drop and rebuild. Tested.GET_LOCKfor the build step; three_testdb_*tables inside the template (never copied into clones) hold the fingerprint, the use count and the prepared databases. A user row is live only if its connection id is inPROCESSLISTand its recorded server start matches the current one, since connection ids restart with the server.The speed mechanism
A clone creates every table again, so it costs what the migrations cost (~0.5–0.8 s for 50 tables). The template therefore keeps clones prepared ahead of demand:
WithReadyDatabases(n)(defaultGOMAXPROCS), refilled byWithReadyWorkersparallel workers with exact slot accounting; a test takes one without ever waiting, and clones on demand when the pool is dry. A worker records a database before cloning into it, so an interrupted clone is always on the books and gets dropped or swept.The README states this plainly: for integration tests the pool keeps up and every test pays 6 ms; in a tight loop of trivial tests the floor is the server's clone throughput.
Core change
TemplateProvider.CreateDatabaseFromTemplatenow returns the database's actual name, since a prepared clone already has one and MySQL cannot rename it. PostgreSQL returns the requested name.testdb.NewDatabaseNameis exported so prepared databases are named like every other test database.Tests
mysql/internal/concurrencytestas real processes: four at once sharing a template (setup once, nothing left behind), and one killed with SIGKILL while holding the template and two prepared databases (the survivor reuses the template, sweeps the dead process's databases, and drops the template as the last one out). Plus lifecycle (last-out, fingerprint/keep, stale-in-use, half-built rebuild, dead-process sweep), the pool (fill, refill, fallback, nothing left after Release), and the clone's fidelity.Found along the way, fixed in a separate PR: the runnable
Example_*functions use a zerotesting.T, whoseCleanupnever runs, so everygo testleaks ~12 Postgres databases (219 were on the dev server).