diff --git a/README.md b/README.md index 00a0dc1..79aa90b 100644 --- a/README.md +++ b/README.md @@ -11,18 +11,19 @@ - **Zero Docker** - Uses native PostgreSQL CREATE DATABASE - **Auto Cleanup** - Databases are dropped automatically - **Migration Support** - goose, tern and golang-migrate through their Go libraries, or plain SQL files; nothing to install -- **PostgreSQL Support** - Production-ready with pgxpool +- **PostgreSQL and MySQL** - The same API and the same guarantees on both; pgxpool for PostgreSQL, database/sql for MySQL ## Quick Start ### Prerequisites -A running PostgreSQL server (local installation, managed instance, or remote). testdb connects to PostgreSQL - it doesn't manage the server itself. +A running PostgreSQL or MySQL server (local installation, managed instance, or remote). testdb connects to it - it doesn't manage the server itself. ### Install ```bash -go get github.com/bashhack/testdb/postgres +go get github.com/bashhack/testdb/postgres # PostgreSQL +go get github.com/bashhack/testdb/mysql # MySQL ``` ### Basic Usage @@ -108,7 +109,7 @@ export TEST_DATABASE_URL="postgres://user:pass@localhost:5432/postgres" Or use the default: `postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable` -Each database provider also has a variable of its own that wins over the shared ones, so a project that tests against more than one database server can point each provider at the right one: `TEST_POSTGRES_URL` for PostgreSQL. The full order is `WithAdminDSN`, the provider's variable, `TEST_DATABASE_URL`, `DATABASE_URL`, then the default. +Each database provider also has a variable of its own that wins over the shared ones, so a project that tests against more than one database server can point each provider at the right one: `TEST_POSTGRES_URL` for PostgreSQL, `TEST_MYSQL_URL` for MySQL. The full order is `WithAdminDSN`, the provider's variable, `TEST_DATABASE_URL`, `DATABASE_URL`, then the default. ### Options @@ -142,7 +143,7 @@ TESTDB_KEEP_ON_FAILURE=1 go test ./... -run TestCheckout psql "postgres://postgres:postgres@localhost:5432/test_1761100000_a1b2c3d4?sslmode=disable" ``` -Passing tests still clean up after themselves. The same switch exists in code as `testdb.WithKeepOnFailure()`, and with the low-level `testdb.New` you call `db.Keep()` yourself. Kept databases are yours to drop afterwards; a `WithDBPrefix` makes them easy to find. Leave this off in CI, where nobody will look and the databases would only pile up. +Passing tests still clean up after themselves. The same switch exists in code as `testdb.WithKeepOnFailure()`, and with the low-level `testdb.New` you call `db.Keep()` yourself. On MySQL the log line gives a `mysql` command line instead. Kept databases are yours to drop afterwards; a `WithDBPrefix` makes them easy to find. Leave this off in CI, where nobody will look and the databases would only pile up. ## Advanced Usage @@ -285,7 +286,23 @@ pool := postgres.Setup(t, ) ``` -Currently, only PostgreSQL is supported. Additional database support can be added by implementing the `testdb.Provider` interface. +### MySQL + +```go +import "github.com/bashhack/testdb/mysql" + +db := mysql.Setup(t, // *sql.DB + testdb.WithMigrator(goose.Dir("./migrations")), +) +``` + +The admin connection string may be in go-sql-driver's native form (`user:pass@tcp(host:port)/?parseTime=true`) or a `mysql://` URL, from `TEST_MYSQL_URL` or the shared variables; the default is `root:root@tcp(localhost:3306)/?parseTime=true`. The user needs `CREATE`/`DROP DATABASE` and the right to kill other sessions (`CONNECTION_ADMIN`), which `root` has. + +Two things about MySQL that testdb handles for you. `DROP DATABASE` does not refuse while other sessions are connected, it waits for the locks they hold (by default, for a year); cleanup kills the test's sessions first and bounds the wait at five seconds, so a leaked transaction can never hang a test run. And a MySQL database with 50 tables takes about half a second to create however you do it, so for speed see [Templates on MySQL](#templates-on-mysql). + +For the fastest possible test server, start it without durability, as `docker-compose.test.yml` does: `--innodb-flush-log-at-trx-commit=0 --sync-binlog=0 --skip-log-bin --innodb-doublewrite=0`. Tested against MySQL 8.4; MariaDB is likely to work but is not tested. + +Other databases can be added by implementing `testdb.Provider`, plus `testdb.TemplateProvider` for templates. ## Migrations @@ -309,7 +326,7 @@ pool := postgres.Setup(t, testdb.WithMigrator(postgres.SQLFiles("./schema"))) ### goose, tern, golang-migrate -Each adapter is its own Go module, so your project only depends on the tool it uses: +Each adapter is its own Go module, so your project only depends on the tool it uses. goose and golang-migrate work with PostgreSQL and MySQL; tern is PostgreSQL-only. ```bash go get github.com/bashhack/testdb/migrate/goose # github.com/pressly/goose @@ -447,6 +464,27 @@ postgres.EnsureTemplate(ctx, "full", )) ``` +### Templates on MySQL + +`mysql.EnsureTemplate`, `Release`, `WithFingerprint`, `WithKeepTemplate` and `testdb.WithTemplate` work exactly as above, with the same guarantees across processes. What differs is underneath, because MySQL has no way to clone a database: + +- A clone is made object by object: tables (indexes, foreign keys, checks, `AUTO_INCREMENT` positions), rows, views, triggers, procedures and functions. Not copied: events and grants. +- That clone costs the server's per-table DDL, about 0.5–0.8 s for a 50-table schema on a laptop, no faster than running the migrations. So a MySQL template **prepares clones ahead of demand**: `WithReadyDatabases(n)` (default `GOMAXPROCS`) keeps `n` databases cloned and ready, refilled in the background by `WithReadyWorkers` parallel workers, and a test takes one instead of waiting. When the pool runs dry, the test clones on demand rather than wait. + +Measured on MySQL 8.4 in Docker on a laptop, 50 tables with an index each, 5,000 rows: + +| | per test | +|---|---| +| empty database | 8 ms | +| fresh database + migrations | 498 ms | +| clone on demand | 768 ms | +| prepared clone, tests of 200 ms or longer | **6 ms**, 100% from the pool | +| prepared clone, 50 ms tests back to back | 130–200 ms, 83–92% from the pool | + +The pool keeps up as long as tests take clones no faster than the server makes them. For integration tests, which do real work, that holds comfortably and every test pays 6 ms. In a tight loop of trivial tests the floor is your server's clone throughput; more workers help until the server saturates. + +Bookkeeping lives in `_testdb_*` tables inside the template (never copied into clones). A process that dies leaves nothing permanent: its hold on the template and its prepared databases are swept by the next `EnsureTemplate` or `Release` on that template. + ### Multiple Templates Different templates for different scenarios: diff --git a/config.go b/config.go index cbd2e85..ea45356 100644 --- a/config.go +++ b/config.go @@ -270,6 +270,14 @@ func ResolveAdminDSN(cfg Config, defaultDSN string, providerEnv ...string) strin return defaultDSN } +// NewDatabaseName returns a unique database name with the given prefix, in +// the form every test database gets: {prefix}_{timestamp}_{random}, for +// example test_1699564231_a1b2c3d4. Providers that prepare databases ahead +// of a test use it so those look like any other test database. +func NewDatabaseName(prefix string) (string, error) { + return generateDatabaseName(prefix) +} + // generateDatabaseName creates a unique database name with the given prefix. // Format: {prefix}_{timestamp}_{random} // diff --git a/mysql/clone.go b/mysql/clone.go new file mode 100644 index 0000000..fb32480 --- /dev/null +++ b/mysql/clone.go @@ -0,0 +1,348 @@ +package mysql + +import ( + "context" + "database/sql" + "errors" + "fmt" + "regexp" + "strings" + + mysqldriver "github.com/go-sql-driver/mysql" +) + +// MySQL has nothing that clones a database, so a template is copied object +// by object: tables from SHOW CREATE TABLE, rows with INSERT ... SELECT, +// then views, triggers, procedures and functions from their SHOW CREATE +// forms. Tables named _testdb_* hold testdb's own bookkeeping inside a +// template and are not copied. + +// bookkeepingPrefix names the tables testdb keeps inside a template. +const bookkeepingPrefix = "_testdb_" + +// definerClause matches the DEFINER=`user`@`host` clause in SHOW CREATE +// output. It is dropped so the clone is owned by whoever runs it, which +// also avoids needing SET_USER_ID to impersonate the original definer. +var definerClause = regexp.MustCompile("DEFINER=`[^`]*`@`[^`]*` ") + +// cloneDatabase creates target as a copy of source. It opens its own +// connection and makes target the current database, because SHOW CREATE +// output names objects without a schema. On failure target is dropped. +func cloneDatabase(ctx context.Context, adminDSN, source, target string) (err error) { + config, err := mysqldriver.ParseDSN(adminDSN) + if err != nil { + return fmt.Errorf("parse admin DSN: %w", err) + } + config.DBName = "" + db, err := sql.Open("mysql", config.FormatDSN()) + if err != nil { + return fmt.Errorf("open connection: %w", err) + } + defer func() { err = errors.Join(err, db.Close()) }() + conn, err := db.Conn(ctx) + if err != nil { + return fmt.Errorf("connect: %w", err) + } + defer func() { err = errors.Join(err, conn.Close()) }() + + var charset, collation string + err = conn.QueryRowContext(ctx, ` + SELECT DEFAULT_CHARACTER_SET_NAME, DEFAULT_COLLATION_NAME + FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = ?`, source).Scan(&charset, &collation) + if errors.Is(err, sql.ErrNoRows) { + return fmt.Errorf("template database %q does not exist (create it with mysql.EnsureTemplate)", source) + } + if err != nil { + return fmt.Errorf("look up template: %w", err) + } + + if _, err := conn.ExecContext(ctx, fmt.Sprintf("CREATE DATABASE %s CHARACTER SET %s COLLATE %s", + quoteIdentifier(target), charset, collation)); err != nil { + return fmt.Errorf("create database: %w", err) + } + // From here, a failure must not leave a half-copied database behind. + // The drop goes through db, not conn: if the failure was a cancelled + // statement, database/sql has discarded conn, and db hands out a + // working connection instead. + defer func() { + if err != nil { + _, dropErr := db.ExecContext(context.WithoutCancel(ctx), "DROP DATABASE IF EXISTS "+quoteIdentifier(target)) + err = errors.Join(err, dropErr) + } + }() + + for _, stmt := range []string{ + "USE " + quoteIdentifier(target), + "SET FOREIGN_KEY_CHECKS = 0", + "SET UNIQUE_CHECKS = 0", + } { + if _, err := conn.ExecContext(ctx, stmt); err != nil { + return fmt.Errorf("%s: %w", stmt, err) + } + } + + tables, err := listNames(ctx, conn, ` + SELECT TABLE_NAME FROM information_schema.TABLES + WHERE TABLE_SCHEMA = ? AND TABLE_TYPE = 'BASE TABLE' ORDER BY TABLE_NAME`, source) + if err != nil { + return fmt.Errorf("list tables: %w", err) + } + tables = withoutBookkeeping(tables) + + for _, table := range tables { + create, err := showCreate(ctx, conn, "SHOW CREATE TABLE "+qualify(source, table), "Create Table") + if err != nil { + return err + } + if _, err := conn.ExecContext(ctx, create); err != nil { + return fmt.Errorf("create table %s: %w", table, err) + } + } + + // Generated columns cannot be inserted into; the clone recomputes them. + // One query for every table's columns rather than one per table. + columnsByTable, err := insertableColumns(ctx, conn, source) + if err != nil { + return err + } + for _, table := range tables { + columns := columnsByTable[table] + if len(columns) == 0 { + continue + } + quoted := make([]string, len(columns)) + for i, c := range columns { + quoted[i] = quoteIdentifier(c) + } + list := strings.Join(quoted, ", ") + if _, err := conn.ExecContext(ctx, fmt.Sprintf("INSERT INTO %s (%s) SELECT %s FROM %s", + quoteIdentifier(table), list, list, qualify(source, table))); err != nil { + return fmt.Errorf("copy rows of %s: %w", table, err) + } + } + + if err := cloneViews(ctx, conn, source, target); err != nil { + return err + } + if err := cloneTriggers(ctx, conn, source, target); err != nil { + return err + } + if err := cloneRoutines(ctx, conn, source, target); err != nil { + return err + } + return nil +} + +// cloneViews creates the source's views in the target. A view's body is +// stored with its tables schema-qualified, so the source's name is +// rewritten to the target's. Views may depend on other views, so creation +// runs in passes until every view exists or a pass makes no progress. +func cloneViews(ctx context.Context, conn *sql.Conn, source, target string) error { + views, err := listNames(ctx, conn, + "SELECT TABLE_NAME FROM information_schema.VIEWS WHERE TABLE_SCHEMA = ? ORDER BY TABLE_NAME", source) + if err != nil { + return fmt.Errorf("list views: %w", err) + } + pending := views + for len(pending) > 0 { + var failed []string + var lastErr error + for _, view := range pending { + create, err := showCreate(ctx, conn, "SHOW CREATE VIEW "+qualify(source, view), "Create View") + if err != nil { + return err + } + create = rewriteSchema(create, source, target) + if _, err := conn.ExecContext(ctx, create); err != nil { + failed = append(failed, view) + lastErr = fmt.Errorf("create view %s: %w", view, err) + } + } + if len(failed) == len(pending) { + return lastErr + } + pending = failed + } + return nil +} + +// cloneTriggers creates the source's triggers in the target, after the +// data so they do not fire during the copy. +func cloneTriggers(ctx context.Context, conn *sql.Conn, source, target string) error { + triggers, err := listNames(ctx, conn, + "SELECT TRIGGER_NAME FROM information_schema.TRIGGERS WHERE TRIGGER_SCHEMA = ? ORDER BY TRIGGER_NAME", source) + if err != nil { + return fmt.Errorf("list triggers: %w", err) + } + for _, trigger := range triggers { + if err := createWithSQLMode(ctx, conn, "SHOW CREATE TRIGGER "+qualify(source, trigger), + "SQL Original Statement", source, target); err != nil { + return fmt.Errorf("create trigger %s: %w", trigger, err) + } + } + return nil +} + +// cloneRoutines creates the source's stored procedures and functions in +// the target. +func cloneRoutines(ctx context.Context, conn *sql.Conn, source, target string) error { + rows, err := conn.QueryContext(ctx, + "SELECT ROUTINE_NAME, ROUTINE_TYPE FROM information_schema.ROUTINES WHERE ROUTINE_SCHEMA = ? ORDER BY ROUTINE_NAME", source) + if err != nil { + return fmt.Errorf("list routines: %w", err) + } + type routine struct{ name, kind string } + var routines []routine + for rows.Next() { + var r routine + if err := rows.Scan(&r.name, &r.kind); err != nil { + return errors.Join(fmt.Errorf("list routines: %w", err), rows.Close()) + } + routines = append(routines, r) + } + if err := errors.Join(rows.Err(), rows.Close()); err != nil { + return fmt.Errorf("list routines: %w", err) + } + for _, r := range routines { + // SHOW CREATE PROCEDURE / FUNCTION; the column is "Create Procedure" / "Create Function" + kind := strings.ToUpper(r.kind) + column := "Create " + strings.ToUpper(kind[:1]) + strings.ToLower(kind[1:]) + if err := createWithSQLMode(ctx, conn, fmt.Sprintf("SHOW CREATE %s %s", kind, qualify(source, r.name)), + column, source, target); err != nil { + return fmt.Errorf("create %s %s: %w", strings.ToLower(kind), r.name, err) + } + } + return nil +} + +// createWithSQLMode runs a SHOW CREATE statement and executes the +// definition it returns in the target, under the sql_mode the object was +// created with, without its DEFINER clause and with the source schema's +// name rewritten to the target's. +func createWithSQLMode(ctx context.Context, conn *sql.Conn, show, column, source, target string) error { + result, err := showCreateRow(ctx, conn, show) + if err != nil { + return err + } + create, ok := result[column] + if !ok || create == "" { + return fmt.Errorf("%s returned no %q (missing privilege?)", show, column) + } + create = rewriteSchema(definerClause.ReplaceAllString(create, ""), source, target) + + if mode, ok := result["sql_mode"]; ok { + if _, err := conn.ExecContext(ctx, "SET SESSION sql_mode = ?", mode); err != nil { + return fmt.Errorf("set sql_mode: %w", err) + } + } + _, err = conn.ExecContext(ctx, create) + return err +} + +// showCreate runs a SHOW CREATE statement and returns one named column. +func showCreate(ctx context.Context, conn *sql.Conn, show, column string) (string, error) { + result, err := showCreateRow(ctx, conn, show) + if err != nil { + return "", err + } + create, ok := result[column] + if !ok || create == "" { + return "", fmt.Errorf("%s returned no %q", show, column) + } + return create, nil +} + +// showCreateRow runs a SHOW CREATE statement, whose result has a different +// set of columns for each object kind, and returns the row by column name. +func showCreateRow(ctx context.Context, conn *sql.Conn, show string) (map[string]string, error) { + rows, err := conn.QueryContext(ctx, show) + if err != nil { + return nil, fmt.Errorf("%s: %w", show, err) + } + defer func() { _ = rows.Close() }() // read fully below; nothing left to report + columns, err := rows.Columns() + if err != nil { + return nil, fmt.Errorf("%s: %w", show, err) + } + if !rows.Next() { + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("%s: %w", show, err) + } + return nil, fmt.Errorf("%s returned no rows", show) + } + values := make([]sql.NullString, len(columns)) + ptrs := make([]any, len(columns)) + for i := range values { + ptrs[i] = &values[i] + } + if err := rows.Scan(ptrs...); err != nil { + return nil, fmt.Errorf("%s: %w", show, err) + } + result := make(map[string]string, len(columns)) + for i, c := range columns { + result[c] = values[i].String + } + return result, rows.Err() +} + +// insertableColumns returns, per table, the columns rows can be inserted +// into (everything but generated columns), in table order. +func insertableColumns(ctx context.Context, conn *sql.Conn, schema string) (map[string][]string, error) { + rows, err := conn.QueryContext(ctx, ` + SELECT TABLE_NAME, COLUMN_NAME FROM information_schema.COLUMNS + WHERE TABLE_SCHEMA = ? AND EXTRA NOT LIKE '%GENERATED%' + ORDER BY TABLE_NAME, ORDINAL_POSITION`, schema) + if err != nil { + return nil, fmt.Errorf("list columns: %w", err) + } + byTable := map[string][]string{} + for rows.Next() { + var table, column string + if err := rows.Scan(&table, &column); err != nil { + return nil, errors.Join(fmt.Errorf("list columns: %w", err), rows.Close()) + } + byTable[table] = append(byTable[table], column) + } + if err := errors.Join(rows.Err(), rows.Close()); err != nil { + return nil, fmt.Errorf("list columns: %w", err) + } + return byTable, nil +} + +// listNames runs a query whose rows each hold one name. +func listNames(ctx context.Context, conn *sql.Conn, query string, args ...any) ([]string, error) { + rows, err := conn.QueryContext(ctx, query, args...) + if err != nil { + return nil, err + } + var names []string + for rows.Next() { + var name string + if err := rows.Scan(&name); err != nil { + return nil, errors.Join(err, rows.Close()) + } + names = append(names, name) + } + return names, errors.Join(rows.Err(), rows.Close()) +} + +func withoutBookkeeping(tables []string) []string { + kept := tables[:0] + for _, t := range tables { + if !strings.HasPrefix(t, bookkeepingPrefix) { + kept = append(kept, t) + } + } + return kept +} + +// rewriteSchema points schema-qualified references in a definition at the +// target database instead of the source. SHOW CREATE output always quotes +// identifiers, so only the backticked form is rewritten. +func rewriteSchema(definition, source, target string) string { + return strings.ReplaceAll(definition, quoteIdentifier(source)+".", quoteIdentifier(target)+".") +} + +func qualify(schema, object string) string { + return quoteIdentifier(schema) + "." + quoteIdentifier(object) +} diff --git a/mysql/clone_test.go b/mysql/clone_test.go new file mode 100644 index 0000000..d96e57e --- /dev/null +++ b/mysql/clone_test.go @@ -0,0 +1,249 @@ +package mysql + +import ( + "context" + "crypto/rand" + "database/sql" + "encoding/hex" + "os" + "testing" + "time" + + "github.com/bashhack/testdb" +) + +// adminDSNOrSkip returns the admin DSN when a MySQL server answers; it +// skips otherwise (fails in CI, where the server must be there). +func adminDSNOrSkip(t *testing.T) string { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + p := &Provider{} + if err := p.Initialize(ctx, testdb.Config{}); err != nil { + if os.Getenv("CI") != "" { + t.Fatalf("MySQL is required in CI but not reachable: %v", err) + } + t.Skipf("MySQL not available: %v", err) + } + dsn := p.ResolvedAdminDSN() + if err := p.Cleanup(ctx); err != nil { + t.Fatal(err) + } + return dsn +} + +func uniqueName(t *testing.T, prefix string) string { + t.Helper() + var b [4]byte + if _, err := rand.Read(b[:]); err != nil { + t.Fatal(err) + } + return prefix + "_" + hex.EncodeToString(b[:]) +} + +func openAdmin(t *testing.T, dsn string) *sql.DB { + t.Helper() + db, err := sql.Open("mysql", dsn) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + if err := db.Close(); err != nil { + t.Error(err) + } + }) + return db +} + +func mustExec(t *testing.T, db *sql.DB, statements ...string) { + t.Helper() + for _, s := range statements { + if _, err := db.ExecContext(context.Background(), s); err != nil { + t.Fatalf("%s: %v", s, err) + } + } +} + +func dropOnCleanup(t *testing.T, db *sql.DB, names ...string) { + t.Helper() + t.Cleanup(func() { + for _, name := range names { + if _, err := db.ExecContext(context.Background(), "DROP DATABASE IF EXISTS "+quoteIdentifier(name)); err != nil { + t.Errorf("drop %s: %v", name, err) + } + } + }) +} + +func scanInt(t *testing.T, db *sql.DB, query string, args ...any) int64 { + t.Helper() + var n int64 + if err := db.QueryRowContext(context.Background(), query, args...).Scan(&n); err != nil { + t.Fatalf("%s: %v", query, err) + } + return n +} + +// TestCloneDatabaseFidelity builds a source database with the things a +// logical clone can get wrong, clones it, and checks each one in the copy. +func TestCloneDatabaseFidelity(t *testing.T) { + dsn := adminDSNOrSkip(t) + admin := openAdmin(t, dsn) + ctx := context.Background() + source := uniqueName(t, "clone_src") + target := uniqueName(t, "clone_dst") + dropOnCleanup(t, admin, source, target) + + q := func(table string) string { return qualify(source, table) } + mustExec(t, admin, + "CREATE DATABASE "+quoteIdentifier(source)+" CHARACTER SET utf8mb4 COLLATE utf8mb4_bin", + "CREATE TABLE "+q("authors")+" (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(100) NOT NULL)", + "CREATE TABLE "+q("books")+` ( + id INT AUTO_INCREMENT PRIMARY KEY, + author_id INT NOT NULL, + title VARCHAR(200) NOT NULL, + pages INT NOT NULL, + weight_kg DECIMAL(6,3) GENERATED ALWAYS AS (pages * 0.005) STORED, + CONSTRAINT fk_books_author FOREIGN KEY (author_id) REFERENCES `+q("authors")+`(id), + INDEX idx_books_title (title) + )`, + "CREATE TABLE "+q("audit")+" (id INT AUTO_INCREMENT PRIMARY KEY, note VARCHAR(200) NOT NULL)", + // Bookkeeping that must not be copied + "CREATE TABLE "+q(bookkeepingPrefix+"info")+" (fingerprint VARCHAR(64))", + "INSERT INTO "+q(bookkeepingPrefix+"info")+" VALUES ('abc')", + "INSERT INTO "+q("authors")+" (name) VALUES ('Ann'), ('Bo')", + "INSERT INTO "+q("books")+" (author_id, title, pages) VALUES (1, 'First', 100), (2, 'Second', 300)", + // Push the counter well past the rows, so a clone that only copied rows would differ + "ALTER TABLE "+q("authors")+" AUTO_INCREMENT = 1000", + "CREATE VIEW "+q("book_titles")+" AS SELECT b.title, a.name FROM "+q("books")+" b JOIN "+q("authors")+" a ON a.id = b.author_id", + "CREATE VIEW "+q("book_title_count")+" AS SELECT COUNT(*) AS n FROM "+q("book_titles"), + "CREATE TRIGGER "+q("books_audit")+" AFTER INSERT ON "+q("books")+" FOR EACH ROW INSERT INTO "+q("audit")+" (note) VALUES (CONCAT('added ', NEW.title))", + "CREATE PROCEDURE "+q("add_author")+" (IN p_name VARCHAR(100)) INSERT INTO "+q("authors")+" (name) VALUES (p_name)", + "CREATE FUNCTION "+q("book_count")+" () RETURNS INT READS SQL DATA RETURN (SELECT COUNT(*) FROM "+q("books")+")", + ) + + if err := cloneDatabase(ctx, dsn, source, target); err != nil { + t.Fatalf("cloneDatabase: %v", err) + } + + // Charset and collation + var charset, collation string + if err := admin.QueryRowContext(ctx, "SELECT DEFAULT_CHARACTER_SET_NAME, DEFAULT_COLLATION_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = ?", target).Scan(&charset, &collation); err != nil { + t.Fatal(err) + } + if charset != "utf8mb4" || collation != "utf8mb4_bin" { + t.Errorf("charset/collation = %s/%s, want utf8mb4/utf8mb4_bin", charset, collation) + } + + // Tables and rows, bookkeeping excluded + if n := scanInt(t, admin, "SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA = ? AND TABLE_TYPE = 'BASE TABLE'", target); n != 3 { + t.Errorf("clone has %d base tables, want 3 (authors, books, audit; no bookkeeping)", n) + } + if n := scanInt(t, admin, "SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA = ? AND TABLE_NAME LIKE '\\_testdb\\_%'", target); n != 0 { + t.Errorf("bookkeeping tables were copied") + } + if n := scanInt(t, admin, "SELECT COUNT(*) FROM "+qualify(target, "books")); n != 2 { + t.Errorf("books rows = %d, want 2", n) + } + + // Foreign key, index, generated column + if n := scanInt(t, admin, "SELECT COUNT(*) FROM information_schema.REFERENTIAL_CONSTRAINTS WHERE CONSTRAINT_SCHEMA = ? AND CONSTRAINT_NAME = 'fk_books_author'", target); n != 1 { + t.Error("foreign key was not copied") + } + if n := scanInt(t, admin, "SELECT COUNT(*) FROM information_schema.STATISTICS WHERE TABLE_SCHEMA = ? AND INDEX_NAME = 'idx_books_title'", target); n == 0 { + t.Error("index was not copied") + } + var weight string + if err := admin.QueryRowContext(ctx, "SELECT weight_kg FROM "+qualify(target, "books")+" WHERE title = 'Second'").Scan(&weight); err != nil { + t.Fatalf("generated column: %v", err) + } + if weight != "1.500" { + t.Errorf("generated column recomputed to %s, want 1.500", weight) + } + + // AUTO_INCREMENT position carried over + var nextID int64 + mustExec(t, admin, "INSERT INTO "+qualify(target, "authors")+" (name) VALUES ('Cy')") + if err := admin.QueryRowContext(ctx, "SELECT id FROM "+qualify(target, "authors")+" WHERE name = 'Cy'").Scan(&nextID); err != nil { + t.Fatal(err) + } + if nextID < 1000 { + t.Errorf("AUTO_INCREMENT did not carry over: new id %d, want >= 1000", nextID) + } + + // Views point at the clone, not the source, and a view over a view works + if n := scanInt(t, admin, "SELECT n FROM "+qualify(target, "book_title_count")); n != 2 { + t.Errorf("view over view returned %d, want 2", n) + } + mustExec(t, admin, "INSERT INTO "+qualify(target, "books")+" (author_id, title, pages) VALUES (1, 'Third', 50)") + if n := scanInt(t, admin, "SELECT n FROM "+qualify(target, "book_title_count")); n != 3 { + t.Errorf("view does not read the clone's tables: %d, want 3", n) + } + if n := scanInt(t, admin, "SELECT COUNT(*) FROM "+qualify(source, "books")); n != 2 { + t.Errorf("writing to the clone changed the source: %d books", n) + } + + // Trigger fires in the clone (for the insert above) and did not fire during the copy + if n := scanInt(t, admin, "SELECT COUNT(*) FROM "+qualify(target, "audit")+" WHERE note = 'added Third'"); n != 1 { + t.Errorf("trigger did not fire in the clone") + } + if n := scanInt(t, admin, "SELECT COUNT(*) FROM "+qualify(target, "audit")); n != 1 { + t.Errorf("audit has %d rows, want 1: the trigger must not fire while rows are copied", n) + } + + // Routines + mustExec(t, admin, "CALL "+qualify(target, "add_author")+"('Di')") + if n := scanInt(t, admin, "SELECT COUNT(*) FROM "+qualify(target, "authors")); n != 4 { + t.Errorf("procedure did not run in the clone: %d authors", n) + } + if n := scanInt(t, admin, "SELECT "+qualify(target, "book_count")+"()"); n != 3 { + t.Errorf("function returned %d, want 3", n) + } +} + +func TestCloneDatabaseMissingSource(t *testing.T) { + dsn := adminDSNOrSkip(t) + admin := openAdmin(t, dsn) + target := uniqueName(t, "clone_dst") + dropOnCleanup(t, admin, target) + + err := cloneDatabase(context.Background(), dsn, uniqueName(t, "no_such"), target) + if err == nil || !containsString(err.Error(), "does not exist") { + t.Fatalf("expected a 'does not exist' error, got %v", err) + } + if n := scanInt(t, admin, "SELECT COUNT(*) FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = ?", target); n != 0 { + t.Error("target was created although the source is missing") + } +} + +func TestCloneDatabaseFailureLeavesNothing(t *testing.T) { + dsn := adminDSNOrSkip(t) + admin := openAdmin(t, dsn) + source := uniqueName(t, "clone_src") + target := uniqueName(t, "clone_dst") + dropOnCleanup(t, admin, source, target) + + // A view whose table is gone cannot be recreated: SHOW CREATE VIEW works + // but the CREATE fails, which must roll the whole clone back + mustExec(t, admin, + "CREATE DATABASE "+quoteIdentifier(source), + "CREATE TABLE "+qualify(source, "t")+" (id INT)", + "CREATE VIEW "+qualify(source, "v")+" AS SELECT id FROM "+qualify(source, "t"), + "DROP TABLE "+qualify(source, "t"), + ) + if err := cloneDatabase(context.Background(), dsn, source, target); err == nil { + t.Fatal("expected the clone to fail") + } + if n := scanInt(t, admin, "SELECT COUNT(*) FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = ?", target); n != 0 { + t.Error("a failed clone left the target database behind") + } +} + +func containsString(s, sub string) bool { + for i := 0; i+len(sub) <= len(s); i++ { + if s[i:i+len(sub)] == sub { + return true + } + } + return false +} diff --git a/mysql/concurrency_test.go b/mysql/concurrency_test.go new file mode 100644 index 0000000..8c0a2b1 --- /dev/null +++ b/mysql/concurrency_test.go @@ -0,0 +1,214 @@ +package mysql + +// These tests prove the MySQL template protocol with real, separate +// processes: the internal/concurrencytest package is compiled to a test +// binary and run several times at once, or killed while holding the +// template and prepared databases, against one template on the shared +// server. + +import ( + "bytes" + "context" + "database/sql" + "os" + "os/exec" + "path/filepath" + "strings" + "sync" + "testing" + "time" +) + +// buildConcurrencyBinary compiles internal/concurrencytest to a test binary +// that can be run as independent processes. +func buildConcurrencyBinary(t *testing.T) string { + t.Helper() + if testing.Short() { + t.Skip("compiles and runs separate test processes") + } + goBin, err := exec.LookPath("go") + if err != nil { + t.Skip("go binary not on PATH") + } + bin := filepath.Join(t.TempDir(), "concurrencytest.test") + cmd := exec.Command(goBin, "test", "-c", "-o", bin, "./internal/concurrencytest") + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("compile concurrencytest: %v\n%s", err, out) + } + return bin +} + +// concurrencyProcess prepares one run of the test binary, in the package's +// directory so the fixture's relative path resolves. +func concurrencyProcess(bin string, env ...string) *exec.Cmd { + cmd := exec.Command(bin, "-test.count=1") + cmd.Dir = filepath.Join("internal", "concurrencytest") + cmd.Env = append(os.Environ(), env...) + return cmd +} + +func countLines(t *testing.T, path string) int { + t.Helper() + data, err := os.ReadFile(path) + if os.IsNotExist(err) { + return 0 + } + if err != nil { + t.Fatal(err) + } + return len(strings.Split(strings.TrimSpace(string(data)), "\n")) +} + +func waitUntil(t *testing.T, timeout time.Duration, what string, cond func() bool) { + t.Helper() + deadline := time.Now().Add(timeout) + for !cond() { + if time.Now().After(deadline) { + t.Fatalf("timed out waiting for %s", what) + } + time.Sleep(50 * time.Millisecond) + } +} + +func schemaExists(t *testing.T, admin *sql.DB, name string) bool { + t.Helper() + return scanInt(t, admin, "SELECT COUNT(*) FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = ?", name) == 1 +} + +func schemasLike(t *testing.T, admin *sql.DB, pattern string) int64 { + t.Helper() + return scanInt(t, admin, "SELECT COUNT(*) FROM information_schema.SCHEMATA WHERE SCHEMA_NAME LIKE ?", pattern) +} + +func TestTemplateConcurrentProcesses(t *testing.T) { + dsn := adminDSNOrSkip(t) + admin := openAdmin(t, dsn) + bin := buildConcurrencyBinary(t) + name := uniqueName(t, "mysql_conc") + readyPrefix := uniqueName(t, "conc_ready") + setupLog := filepath.Join(t.TempDir(), "setup.log") + + // Each process keeps the template for a while after ensuring it, so + // all of them overlap even if one is slow to start; without overlap a + // late process would legitimately rebuild and "setup ran once" would be + // down to timing. + const procs = 4 + outs := make([][]byte, procs) + errs := make([]error, procs) + var wg sync.WaitGroup + for i := range procs { + wg.Go(func() { + cmd := concurrencyProcess(bin, + "TESTDB_CONC_TEMPLATE="+name, + "TESTDB_CONC_SETUP_LOG="+setupLog, + "TESTDB_CONC_FINGERPRINT=v1", + "TESTDB_CONC_READY_PREFIX="+readyPrefix, + "TESTDB_CONC_HOLD=2s") + outs[i], errs[i] = cmd.CombinedOutput() + }) + } + wg.Wait() + + for i := range procs { + if errs[i] != nil { + t.Errorf("process %d failed: %v\n%s", i, errs[i], outs[i]) + } + } + if n := countLines(t, setupLog); n != 1 { + t.Errorf("setup ran %d times across %d processes, want 1", n, procs) + } + if schemaExists(t, admin, name) { + t.Errorf("template %s was left behind", name) + } + if n := schemasLike(t, admin, strings.ReplaceAll(readyPrefix, "_", `\_`)+`\_%`); n != 0 { + t.Errorf("%d prepared databases were left behind", n) + } +} + +func TestTemplateSurvivesKilledProcess(t *testing.T) { + dsn := adminDSNOrSkip(t) + admin := openAdmin(t, dsn) + bin := buildConcurrencyBinary(t) + name := uniqueName(t, "mysql_conc_kill") + readyPrefix := uniqueName(t, "kill_ready") + dir := t.TempDir() + setupLog := filepath.Join(dir, "setup.log") + ready := filepath.Join(dir, "ready") + t.Cleanup(func() { + // Whatever the test concludes, do not leave the server dirty + if schemaExists(t, admin, name) { + if _, err := admin.ExecContext(context.Background(), "DROP DATABASE "+quoteIdentifier(name)); err != nil { + t.Error(err) + } + } + }) + + // A process that takes the template, fills its pool, and then just sits + var holderOut bytes.Buffer + holder := concurrencyProcess(bin, + "TESTDB_CONC_TEMPLATE="+name, + "TESTDB_CONC_SETUP_LOG="+setupLog, + "TESTDB_CONC_FINGERPRINT=v1", + "TESTDB_CONC_READY_PREFIX="+readyPrefix, + "TESTDB_CONC_READY_FILE="+ready, + "TESTDB_CONC_HOLD=5m") + holder.Stdout = &holderOut + holder.Stderr = &holderOut + if err := holder.Start(); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = holder.Process.Kill() }) + + waitUntil(t, 60*time.Second, "holder to ensure the template", func() bool { + _, err := os.Stat(ready) + return err == nil + }) + users := qualify(name, bookkeepingPrefix+"users") + readyTable := qualify(name, bookkeepingPrefix+"ready") + waitUntil(t, 30*time.Second, "holder's pool to fill", func() bool { + return scanInt(t, admin, "SELECT COUNT(*) FROM "+readyTable) == 2 + }) + if n := scanInt(t, admin, "SELECT COUNT(*) FROM "+users); n != 1 { + t.Fatalf("%d template users registered while the holder runs, want 1\n%s", n, holderOut.String()) + } + var holderConn uint64 + if err := admin.QueryRowContext(context.Background(), "SELECT connection_id FROM "+users).Scan(&holderConn); err != nil { + t.Fatal(err) + } + + // Kill it outright: no Release, prepared databases left on the server + if err := holder.Process.Kill(); err != nil { + t.Fatal(err) + } + if err := holder.Wait(); err == nil { + t.Fatal("holder exited cleanly; the test needs it to die without releasing") + } + // The server notices the dead connection + waitUntil(t, 10*time.Second, "the killed process's session to end", func() bool { + return scanInt(t, admin, "SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE id = ?", holderConn) == 0 + }) + if n := schemasLike(t, admin, strings.ReplaceAll(readyPrefix, "_", `\_`)+`\_%`); n != 2 { + t.Fatalf("expected the dead process's 2 prepared databases to still exist before the sweep, found %d", n) + } + + // A later process reuses the template (setup does not run again), + // sweeps the dead process's prepared databases, and, being the last + // live user, drops the template when it is done + out, err := concurrencyProcess(bin, + "TESTDB_CONC_TEMPLATE="+name, + "TESTDB_CONC_SETUP_LOG="+setupLog, + "TESTDB_CONC_FINGERPRINT=v1", + "TESTDB_CONC_READY_PREFIX="+readyPrefix).CombinedOutput() + if err != nil { + t.Fatalf("process after the kill failed: %v\n%s", err, out) + } + if n := countLines(t, setupLog); n != 1 { + t.Errorf("setup ran %d times, want 1 (the survivor should reuse the template)", n) + } + if schemaExists(t, admin, name) { + t.Errorf("template %s was left behind: a dead process kept it alive", name) + } + if n := schemasLike(t, admin, strings.ReplaceAll(readyPrefix, "_", `\_`)+`\_%`); n != 0 { + t.Errorf("%d prepared databases of the dead process were not swept", n) + } +} diff --git a/mysql/internal/concurrencytest/concurrency_test.go b/mysql/internal/concurrencytest/concurrency_test.go new file mode 100644 index 0000000..65514ba --- /dev/null +++ b/mysql/internal/concurrencytest/concurrency_test.go @@ -0,0 +1,118 @@ +package concurrencytest + +import ( + "context" + "fmt" + "math/rand/v2" + "os" + "testing" + "time" + + "github.com/bashhack/testdb" + "github.com/bashhack/testdb/mysql" +) + +var templateName string + +func TestMain(m *testing.M) { + ctx := context.Background() + + // Without a server there is nothing to test here + probe := &mysql.Provider{} + probeCtx, cancel := context.WithTimeout(ctx, 5*time.Second) + err := probe.Initialize(probeCtx, testdb.Config{}) + cancel() + if err != nil { + if os.Getenv("CI") != "" { + fmt.Fprintln(os.Stderr, "MySQL is required in CI but not reachable:", err) + os.Exit(2) + } + fmt.Fprintln(os.Stderr, "skipping: MySQL not available:", err) + os.Exit(0) + } + if err := probe.Cleanup(ctx); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(2) + } + + templateName = os.Getenv("TESTDB_CONC_TEMPLATE") + if templateName == "" { + templateName = "testdb_mysql_conc_standalone" + } + + setup := func(ctx context.Context, dsn string) error { + if path := os.Getenv("TESTDB_CONC_SETUP_LOG"); path != "" { + f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644) + if err != nil { + return err + } + _, writeErr := fmt.Fprintf(f, "setup pid=%d\n", os.Getpid()) + closeErr := f.Close() + if writeErr != nil { + return writeErr + } + if closeErr != nil { + return closeErr + } + } + return mysql.SQLFiles("../../../testdata/mysql/schema")(ctx, dsn) + } + + opts := []mysql.TemplateOption{mysql.WithReadyDatabases(2)} + if fp := os.Getenv("TESTDB_CONC_FINGERPRINT"); fp != "" { + opts = append(opts, mysql.WithFingerprint(fp)) + } + if prefix := os.Getenv("TESTDB_CONC_READY_PREFIX"); prefix != "" { + opts = append(opts, mysql.WithReadyPrefix(prefix)) + } + + tpl, err := mysql.EnsureTemplate(ctx, templateName, setup, opts...) + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(2) + } + + if path := os.Getenv("TESTDB_CONC_READY_FILE"); path != "" { + if err := os.WriteFile(path, []byte("ready\n"), 0o644); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(2) + } + } + if hold := os.Getenv("TESTDB_CONC_HOLD"); hold != "" { + d, err := time.ParseDuration(hold) + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(2) + } + time.Sleep(d) + } + + code := m.Run() + + if err := tpl.Release(ctx); err != nil { + fmt.Fprintln(os.Stderr, err) + if code == 0 { + code = 2 + } + } + os.Exit(code) +} + +func TestClonesFromTemplate(t *testing.T) { + for i := range 3 { + t.Run(fmt.Sprintf("clone%d", i), func(t *testing.T) { + t.Parallel() + // Spread the clones out so processes interleave differently each run + time.Sleep(time.Duration(rand.IntN(50)) * time.Millisecond) + + db := mysql.Setup(t, testdb.WithTemplate(templateName)) + var n int + if err := db.QueryRowContext(context.Background(), "SELECT COUNT(*) FROM users").Scan(&n); err != nil { + t.Fatalf("query clone: %v", err) + } + if n == 0 { + t.Fatal("clone has no fixture rows") + } + }) + } +} diff --git a/mysql/internal/concurrencytest/doc.go b/mysql/internal/concurrencytest/doc.go new file mode 100644 index 0000000..70c8640 --- /dev/null +++ b/mysql/internal/concurrencytest/doc.go @@ -0,0 +1,20 @@ +// Package concurrencytest is a test package that mysql's multi-process +// template tests compile and run as separate processes. Each process +// ensures the same template with a small ready pool, clones from it in +// parallel subtests, and releases it. It has no non-test code of its own. +// +// Run on its own without a reachable MySQL server it exits successfully +// without running anything, so a Postgres-only checkout stays green; in CI +// (CI set) a missing server is a failure. +// +// Environment variables set by the driver tests: +// +// TESTDB_CONC_TEMPLATE template name to share (default: a standalone name) +// TESTDB_CONC_FINGERPRINT fingerprint passed to WithFingerprint, if set +// TESTDB_CONC_SETUP_LOG file to append a line to each time setup runs +// TESTDB_CONC_READY_FILE file to create once EnsureTemplate has returned +// TESTDB_CONC_READY_PREFIX name prefix for prepared databases +// TESTDB_CONC_HOLD duration to sleep before running tests, so the +// driver can kill the process while it holds the +// template and prepared databases +package concurrencytest diff --git a/mysql/mysql_test.go b/mysql/mysql_test.go index 505e138..6fd2591 100644 --- a/mysql/mysql_test.go +++ b/mysql/mysql_test.go @@ -258,10 +258,14 @@ func TestKeepOnFailureKeepsDatabaseAndSaysHowToOpenIt(t *testing.T) { } } -func TestWithTemplateIsRefused(t *testing.T) { - _, err := testdb.New(t, &mysql.Provider{}, nil, testdb.WithTemplate("anything")) - if !errors.Is(err, testdb.ErrTemplatesNotSupported) { - t.Fatalf("expected ErrTemplatesNotSupported, got %v", err) +func TestWithUnknownTemplate(t *testing.T) { + requireMySQL(t) + _, err := testdb.New(t, &mysql.Provider{}, nil, testdb.WithTemplate("no_such_template")) + if err == nil || !strings.Contains(err.Error(), `template database "no_such_template" does not exist (create it with mysql.EnsureTemplate)`) { + t.Fatalf("expected a does-not-exist error naming EnsureTemplate, got %v", err) + } + if errors.Is(err, testdb.ErrTemplatesNotSupported) { + t.Fatal("MySQL supports templates; the error must not say otherwise") } } diff --git a/mysql/pool.go b/mysql/pool.go new file mode 100644 index 0000000..884f1b4 --- /dev/null +++ b/mysql/pool.go @@ -0,0 +1,164 @@ +package mysql + +import ( + "context" + "database/sql" + "errors" + "fmt" + "log" + "sync" + + "github.com/bashhack/testdb" +) + +// readyPool keeps clones of a template prepared ahead of demand, so a test +// pays a channel receive instead of a clone. Several workers clone in +// parallel, because a clone's cost is the server's per-table DDL and the +// server handles concurrent clones well; take never waits for them. +// +// Accounting is exact: size = slots free + clones in flight + names ready. +// A worker takes a slot before cloning; take hands the slot back. +// +// Workers do their bookkeeping through a *sql.DB rather than a pinned +// connection: when Release cancels a statement in flight, database/sql +// discards that connection, and the clean-up that follows must still get +// a working one. +type readyPool struct { + tpl *Template + db *sql.DB + prefix string + ready chan string + slots chan struct{} + ctx context.Context + cancel context.CancelFunc + wg sync.WaitGroup + + mu sync.Mutex + failure error // the first refill error; after it, no more refills +} + +func startReadyPool(tpl *Template, size, workers int) (*readyPool, error) { + db, err := sql.Open("mysql", tpl.provider.adminDSN) + if err != nil { + return nil, fmt.Errorf("open pool connection: %w", err) + } + ctx, cancel := context.WithCancel(context.Background()) + pool := &readyPool{ + tpl: tpl, + db: db, + prefix: tpl.readyPrefix, + ready: make(chan string, size), + slots: make(chan struct{}, size), + ctx: ctx, + cancel: cancel, + } + for range size { + pool.slots <- struct{}{} + } + for range min(workers, size) { + pool.wg.Go(pool.work) + } + return pool, nil +} + +// take returns a prepared database, or false without waiting. +func (p *readyPool) take() (string, bool) { + select { + case name := <-p.ready: + p.slots <- struct{}{} // never blocks: a name taken is a slot freed + return name, true + default: + return "", false + } +} + +func (p *readyPool) work() { + for { + select { + case <-p.slots: + case <-p.ctx.Done(): + return + } + name, err := p.prepare() + if err != nil { + if p.ctx.Err() != nil { + return + } + // Tests keep working with on-demand clones; say so once + p.mu.Lock() + first := p.failure == nil + p.failure = err + p.mu.Unlock() + if first { + log.Printf("testdb: template %q: preparing databases ahead failed, tests will clone on demand: %v", p.tpl.name, err) + } + return + } + select { + case p.ready <- name: + case <-p.ctx.Done(): + // Nobody will take it; drop it before leaving + if err := p.drop(name); err != nil { + log.Printf("testdb: %v", err) + } + return + } + } +} + +// prepare records a new database in _testdb_ready first and clones the +// template into it second, so a clone cut short (by cancellation or a +// crash) is on the books and gets dropped by Release or swept as a dead +// process's leftover. +func (p *readyPool) prepare() (string, error) { + name, err := testdb.NewDatabaseName(p.prefix) + if err != nil { + return "", err + } + _, err = p.db.ExecContext(p.ctx, + fmt.Sprintf("INSERT INTO %s (db_name, connection_id, server_started_at) VALUES (?, ?, ?)", + qualify(p.tpl.name, bookkeepingPrefix+"ready")), name, p.tpl.connID, p.tpl.serverStart) + if err != nil { + return "", fmt.Errorf("record prepared database: %w", err) + } + if err := cloneDatabase(p.ctx, p.tpl.provider.adminDSN, p.tpl.name, name); err != nil { + return "", errors.Join(err, p.drop(name)) + } + return name, nil +} + +// drop removes a prepared database and its record, whichever of the two +// exist. It does not depend on the pool's context, so it works during +// shutdown. +func (p *readyPool) drop(name string) error { + ctx := context.Background() + if _, err := p.db.ExecContext(ctx, "DROP DATABASE IF EXISTS "+quoteIdentifier(name)); err != nil { + return fmt.Errorf("drop prepared database %s: %w", name, err) + } + if _, err := p.db.ExecContext(ctx, "DELETE FROM "+qualify(p.tpl.name, bookkeepingPrefix+"ready")+" WHERE db_name = ?", name); err != nil { + return fmt.Errorf("forget prepared database %s: %w", name, err) + } + return nil +} + +// stop ends the workers and drops the prepared databases nobody took. +// Anything a worker had recorded but not finished is dropped by the +// template's Release through the bookkeeping table. +func (p *readyPool) stop() error { + p.cancel() + p.wg.Wait() + + var errs []error + close(p.ready) + for name := range p.ready { + errs = append(errs, p.drop(name)) + } + errs = append(errs, p.db.Close()) + p.mu.Lock() + failure := p.failure + p.mu.Unlock() + if failure != nil { + errs = append(errs, fmt.Errorf("preparing databases ahead had failed: %w", failure)) + } + return errors.Join(errs...) +} diff --git a/mysql/template.go b/mysql/template.go new file mode 100644 index 0000000..61a8928 --- /dev/null +++ b/mysql/template.go @@ -0,0 +1,692 @@ +package mysql + +import ( + "context" + "crypto/sha256" + "database/sql" + "encoding/hex" + "errors" + "fmt" + "log" + "runtime" + "strconv" + "sync" + "time" + + "github.com/bashhack/testdb" +) + +// Templates for MySQL: a database built once and copied for each test. The +// guarantees are the ones postgres gives (first process builds, others wait +// and reuse, last one out drops, crashes release, a half-built template is +// never used), reached with MySQL's means: GET_LOCK for the build step, +// bookkeeping tables inside the template for the use count and ready +// databases, and liveness checked against the process list. +// See .internal/MYSQL_PROVIDER_DESIGN.md. + +// SetupFunc populates a template database; see postgres.SetupFunc. It is +// testdb.MigratorFunc, so a migration adapter's Migrate method is one. +type SetupFunc = testdb.MigratorFunc + +// Template is this process's handle on a template database obtained from +// EnsureTemplate. It holds one connection to the server for as long as it +// lives; that connection's id is how other processes know the template is +// in use. Call Release when the process is done. +type Template struct { + name string + provider *Provider // the long-lived session + connID uint64 + serverStart int64 + keep bool + readyPrefix string + pool *readyPool + + mu sync.Mutex + released bool +} + +// Name returns the template database's name, for use with testdb.WithTemplate. +func (t *Template) Name() string { return t.name } + +// Option returns the testdb.Option that makes a test database a clone of +// this template, so tests can write mysql.Setup(t, tpl.Option()). +func (t *Template) Option() testdb.Option { return testdb.WithTemplate(t.name) } + +// Ready reports how many prepared clones are waiting to be taken right now +// (see WithReadyDatabases); 0 when the pool is disabled or has run dry. +func (t *Template) Ready() int { + if t.pool == nil { + return 0 + } + return len(t.pool.ready) +} + +// TemplateOption configures EnsureTemplate. +type TemplateOption func(*templateConfig) + +type templateConfig struct { + adminDSN string + fingerprint string + keep bool + ready int + readyWorkers int + readyPrefix string +} + +// WithFingerprint identifies the template's content, typically a hash of the +// files it is built from. EnsureTemplate reuses an existing template whose +// fingerprint matches and rebuilds one whose fingerprint differs, so +// templates can safely outlive a test run (see WithKeepTemplate). Without a +// fingerprint an existing template is always reused as it is. +func WithFingerprint(fingerprint string) TemplateOption { + return func(c *templateConfig) { c.fingerprint = fingerprint } +} + +// WithKeepTemplate leaves the template in place when the last process +// releases it, so the next run can reuse it instead of rebuilding. Pair it +// with WithFingerprint, otherwise a kept template is never refreshed. +func WithKeepTemplate() TemplateOption { + return func(c *templateConfig) { c.keep = true } +} + +// WithTemplateAdminDSN overrides the admin connection string used to create +// and manage the template. By default it is discovered as for test +// databases (TEST_MYSQL_URL, then the shared variables, then the default). +func WithTemplateAdminDSN(dsn string) TemplateOption { + return func(c *templateConfig) { c.adminDSN = dsn } +} + +// WithReadyDatabases sets how many clones of the template this process +// keeps prepared ahead of demand. Cloning a MySQL database costs hundreds of +// milliseconds for a real schema (every table is created again); taking a +// prepared one costs nothing, and the pool refills while tests run. The +// default is runtime.GOMAXPROCS(0), which is also go test's default +// -parallel. 0 disables the pool: every test clones on demand. +// +// The pool keeps up as long as tests take clones no faster than the +// workers (see WithReadyWorkers) can make them. When it runs dry, a test +// clones on demand rather than wait. +func WithReadyDatabases(n int) TemplateOption { + return func(c *templateConfig) { c.ready = n } +} + +// WithReadyWorkers sets how many clones are made in parallel to keep the +// pool full. The server handles concurrent clones well, so the default is +// the pool size. +func WithReadyWorkers(n int) TemplateOption { + return func(c *templateConfig) { c.readyWorkers = n } +} + +// WithReadyPrefix sets the name prefix of prepared databases (default +// "test", the same as test databases get). +func WithReadyPrefix(prefix string) TemplateOption { + return func(c *templateConfig) { c.readyPrefix = prefix } +} + +// registry holds this process's templates by name, so the provider can +// hand a test a database the template prepared in advance. +var registry sync.Map // string -> *Template + +// EnsureTemplate makes sure the named template database exists and marks it +// as in use by this process, returning a handle to Release when done. +// Tests then clone it with testdb.WithTemplate(name) or tpl.Option(). +// +// It is safe to call from several test packages or CI jobs at once against +// the same server: the first process to arrive builds the template with +// setup, the others wait for it and reuse it, and the last one to Release +// drops it. A process that crashes drops out of the count because its +// connection is gone. A build that crashed half-way is recognised (its +// completeness marker is missing) and redone. +// +// func TestMain(m *testing.M) { +// ctx := context.Background() +// tpl, err := mysql.EnsureTemplate(ctx, "schema", goose.Dir("./migrations").Migrate) +// if err != nil { +// log.Fatal(err) +// } +// code := m.Run() +// if err := tpl.Release(ctx); err != nil { +// log.Print(err) +// } +// os.Exit(code) +// } +func EnsureTemplate(ctx context.Context, name string, setup SetupFunc, opts ...TemplateOption) (*Template, error) { + if err := validateTemplateName(name); err != nil { + return nil, fmt.Errorf("mysql.EnsureTemplate: %w", err) + } + cfg := templateConfig{ready: runtime.GOMAXPROCS(0), readyPrefix: "test"} + for _, opt := range opts { + opt(&cfg) + } + if cfg.readyWorkers <= 0 { + cfg.readyWorkers = cfg.ready + } + + p := &Provider{} + if err := p.Initialize(ctx, testdb.Config{AdminDSNOverride: cfg.adminDSN}); err != nil { + return nil, fmt.Errorf("mysql.EnsureTemplate %q: %w", name, err) + } + tpl := &Template{name: name, provider: p, keep: cfg.keep, readyPrefix: cfg.readyPrefix} + if err := tpl.ensure(ctx, setup, cfg); err != nil { + err = errors.Join(err, p.Cleanup(context.WithoutCancel(ctx))) + return nil, fmt.Errorf("mysql.EnsureTemplate %q: %w", name, err) + } + registry.Store(name, tpl) + if cfg.ready > 0 { + pool, err := startReadyPool(tpl, cfg.ready, cfg.readyWorkers) + if err != nil { + // The pool is an optimisation; without it tests clone on demand + log.Printf("testdb: template %q: cannot prepare databases ahead, tests will clone on demand: %v", name, err) + return tpl, nil + } + tpl.pool = pool + } + return tpl, nil +} + +func (t *Template) ensure(ctx context.Context, setup SetupFunc, cfg templateConfig) error { + conn := t.provider.conn + if err := conn.QueryRowContext(ctx, "SELECT CONNECTION_ID()").Scan(&t.connID); err != nil { + return fmt.Errorf("connection id: %w", err) + } + start, err := serverStartedAt(ctx, conn) + if err != nil { + return err + } + t.serverStart = start + + if err := lockBuild(ctx, conn, t.name); err != nil { + return err + } + err = t.ensureLocked(ctx, setup, cfg) + // Failing to release the build lock would stall every other process + // sharing this template; the caller then closes the session, which + // releases it. + return errors.Join(err, unlockBuild(context.WithoutCancel(ctx), conn, t.name)) +} + +func (t *Template) ensureLocked(ctx context.Context, setup SetupFunc, cfg templateConfig) error { + conn := t.provider.conn + info, err := t.info(ctx) + if err != nil { + return err + } + switch { + case !info.exists: + if err := t.build(ctx, setup, cfg.fingerprint); err != nil { + return err + } + case !info.complete: + // A build that crashed before writing its marker + if err := t.dropTemplate(ctx); err != nil { + return fmt.Errorf("remove half-built template: %w", err) + } + if err := t.build(ctx, setup, cfg.fingerprint); err != nil { + return err + } + case cfg.fingerprint != "" && info.fingerprint != cfg.fingerprint: + live, err := t.liveUsers(ctx) + if err != nil { + return err + } + if live > 0 { + return fmt.Errorf("template %q has a different fingerprint and is in use by %d other process(es); let that run finish first", t.name, live) + } + if err := t.dropTemplate(ctx); err != nil { + return fmt.Errorf("replace stale template: %w", err) + } + if err := t.build(ctx, setup, cfg.fingerprint); err != nil { + return err + } + default: + // Reusing: tidy up after processes that died with prepared databases + if err := t.sweepDeadReady(ctx); err != nil { + return err + } + } + + _, err = conn.ExecContext(ctx, fmt.Sprintf("INSERT INTO %s (connection_id, server_started_at) VALUES (?, ?)", + qualify(t.name, bookkeepingPrefix+"users")), t.connID, t.serverStart) + if err != nil { + return fmt.Errorf("register as a user of the template: %w", err) + } + return nil +} + +// build creates the template under its final name (MySQL cannot rename a +// database), runs setup, creates the bookkeeping tables, and writes the +// completeness marker last. On failure the database is dropped. +func (t *Template) build(ctx context.Context, setup SetupFunc, fingerprint string) (err error) { + conn := t.provider.conn + if err := t.provider.CreateDatabase(ctx, t.name); err != nil { + return fmt.Errorf("create template database: %w", err) + } + defer func() { + if err != nil { + err = errors.Join(err, t.provider.DropDatabase(context.WithoutCancel(ctx), t.name)) + } + }() + + if setup != nil { + dsn, err := t.provider.BuildDSN(t.name) + if err != nil { + return err + } + if err := setup(ctx, dsn); err != nil { + return fmt.Errorf("set up template: %w", err) + } + } + + for _, stmt := range []string{ + fmt.Sprintf(`CREATE TABLE %s ( + id TINYINT PRIMARY KEY, + fingerprint VARCHAR(255) NOT NULL, + complete TINYINT NOT NULL, + built_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP + )`, qualify(t.name, bookkeepingPrefix+"info")), + fmt.Sprintf(`CREATE TABLE %s ( + connection_id BIGINT UNSIGNED NOT NULL, + server_started_at BIGINT NOT NULL, + PRIMARY KEY (connection_id, server_started_at) + )`, qualify(t.name, bookkeepingPrefix+"users")), + fmt.Sprintf(`CREATE TABLE %s ( + db_name VARCHAR(64) PRIMARY KEY, + connection_id BIGINT UNSIGNED NOT NULL, + server_started_at BIGINT NOT NULL + )`, qualify(t.name, bookkeepingPrefix+"ready")), + } { + if _, err := conn.ExecContext(ctx, stmt); err != nil { + return fmt.Errorf("create bookkeeping table: %w", err) + } + } + // The marker: anything that sees this row sees a finished template + if _, err := conn.ExecContext(ctx, fmt.Sprintf("INSERT INTO %s (id, fingerprint, complete) VALUES (1, ?, 1)", + qualify(t.name, bookkeepingPrefix+"info")), fingerprint); err != nil { + return fmt.Errorf("mark template complete: %w", err) + } + return nil +} + +// Release ends this process's use of the template: prepared databases this +// process still holds are dropped, and if no other process is using the +// template (and WithKeepTemplate was not given) the template is dropped +// too, along with anything left by processes that died. It is safe to call +// more than once. +func (t *Template) Release(ctx context.Context) error { + t.mu.Lock() + defer t.mu.Unlock() + if t.released { + return nil + } + t.released = true + registry.CompareAndDelete(t.name, t) + + var errs []error + if t.pool != nil { + errs = append(errs, t.pool.stop()) + } + errs = append(errs, t.release(ctx)) + // Closing the session releases the GET_LOCK and ends this connection id, + // which is how other processes see this one leave + errs = append(errs, t.provider.Cleanup(context.WithoutCancel(ctx))) + if err := errors.Join(errs...); err != nil { + return fmt.Errorf("mysql: release template %q: %w", t.name, err) + } + return nil +} + +func (t *Template) release(ctx context.Context) error { + conn := t.provider.conn + if err := lockBuild(ctx, conn, t.name); err != nil { + return err + } + err := t.releaseLocked(ctx) + return errors.Join(err, unlockBuild(context.WithoutCancel(ctx), conn, t.name)) +} + +func (t *Template) releaseLocked(ctx context.Context) error { + conn := t.provider.conn + info, err := t.info(ctx) + if err != nil { + return err + } + if !info.exists { + // Already dropped by another last-out process; nothing to do + return nil + } + // Our prepared databases, whether or not the pool managed to drop them + if err := t.dropReadyOf(ctx, t.connID, t.serverStart); err != nil { + return err + } + if _, err := conn.ExecContext(ctx, fmt.Sprintf("DELETE FROM %s WHERE connection_id = ? AND server_started_at = ?", + qualify(t.name, bookkeepingPrefix+"users")), t.connID, t.serverStart); err != nil { + return fmt.Errorf("deregister: %w", err) + } + if t.keep { + return t.sweepDeadReady(ctx) + } + live, err := t.liveUsers(ctx) + if err != nil { + return err + } + if live > 0 { + return nil + } + return t.dropTemplate(ctx) +} + +// dropTemplate drops the template and every prepared database recorded in +// it, whoever made them. +func (t *Template) dropTemplate(ctx context.Context) error { + conn := t.provider.conn + names, err := listNames(ctx, conn, "SELECT db_name FROM "+qualify(t.name, bookkeepingPrefix+"ready")) + if err != nil && errNumber(err) != errNoSuchTable { + return fmt.Errorf("list prepared databases: %w", err) + } + for _, name := range names { + if err := t.provider.DropDatabase(ctx, name); err != nil { + return err + } + } + return t.provider.DropDatabase(ctx, t.name) +} + +// templateInfo is what one look at a template tells us. +type templateInfo struct { + exists bool + complete bool + fingerprint string +} + +func (t *Template) info(ctx context.Context) (templateInfo, error) { + conn := t.provider.conn + exists, err := t.provider.TemplateExists(ctx, t.name) + if err != nil || !exists { + return templateInfo{}, err + } + var info templateInfo + info.exists = true + var complete int + err = conn.QueryRowContext(ctx, "SELECT fingerprint, complete FROM "+qualify(t.name, bookkeepingPrefix+"info")+" WHERE id = 1"). + Scan(&info.fingerprint, &complete) + switch { + case errors.Is(err, sql.ErrNoRows), errNumber(err) == errNoSuchTable: + return info, nil // exists, not complete + case err != nil: + return templateInfo{}, fmt.Errorf("read template info: %w", err) + } + info.complete = complete == 1 + return info, nil +} + +// liveUsers counts processes using the template and removes the rows of +// processes that are gone. It must run under the build lock. +func (t *Template) liveUsers(ctx context.Context) (int, error) { + conn := t.provider.conn + alive, err := liveConnections(ctx, conn) + if err != nil { + return 0, err + } + rows, err := conn.QueryContext(ctx, "SELECT connection_id, server_started_at FROM "+qualify(t.name, bookkeepingPrefix+"users")) + if err != nil { + return 0, fmt.Errorf("list template users: %w", err) + } + type user struct { + id uint64 + start int64 + } + var users []user + for rows.Next() { + var u user + if err := rows.Scan(&u.id, &u.start); err != nil { + return 0, errors.Join(fmt.Errorf("list template users: %w", err), rows.Close()) + } + users = append(users, u) + } + if err := errors.Join(rows.Err(), rows.Close()); err != nil { + return 0, fmt.Errorf("list template users: %w", err) + } + + live := 0 + for _, u := range users { + if alive.has(u.id, u.start) { + live++ + continue + } + if _, err := conn.ExecContext(ctx, fmt.Sprintf("DELETE FROM %s WHERE connection_id = ? AND server_started_at = ?", + qualify(t.name, bookkeepingPrefix+"users")), u.id, u.start); err != nil { + return 0, fmt.Errorf("remove dead template user: %w", err) + } + } + return live, nil +} + +// sweepDeadReady drops prepared databases whose creating process is gone. +// It must run under the build lock. +func (t *Template) sweepDeadReady(ctx context.Context) error { + conn := t.provider.conn + alive, err := liveConnections(ctx, conn) + if err != nil { + return err + } + rows, err := conn.QueryContext(ctx, "SELECT db_name, connection_id, server_started_at FROM "+qualify(t.name, bookkeepingPrefix+"ready")) + if err != nil { + return fmt.Errorf("list prepared databases: %w", err) + } + var dead []string + for rows.Next() { + var name string + var id uint64 + var start int64 + if err := rows.Scan(&name, &id, &start); err != nil { + return errors.Join(fmt.Errorf("list prepared databases: %w", err), rows.Close()) + } + if !alive.has(id, start) { + dead = append(dead, name) + } + } + if err := errors.Join(rows.Err(), rows.Close()); err != nil { + return fmt.Errorf("list prepared databases: %w", err) + } + for _, name := range dead { + if err := t.forgetReady(ctx, name); err != nil { + return err + } + } + return nil +} + +// dropReadyOf drops the prepared databases one connection made. +func (t *Template) dropReadyOf(ctx context.Context, connID uint64, start int64) error { + names, err := listNames(ctx, t.provider.conn, + "SELECT db_name FROM "+qualify(t.name, bookkeepingPrefix+"ready")+" WHERE connection_id = ? AND server_started_at = ?", connID, start) + if err != nil { + return fmt.Errorf("list prepared databases: %w", err) + } + for _, name := range names { + if err := t.forgetReady(ctx, name); err != nil { + return err + } + } + return nil +} + +// forgetReady drops a prepared database and its record. +func (t *Template) forgetReady(ctx context.Context, name string) error { + if err := t.provider.DropDatabase(ctx, name); err != nil { + return err + } + if _, err := t.provider.conn.ExecContext(ctx, "DELETE FROM "+qualify(t.name, bookkeepingPrefix+"ready")+" WHERE db_name = ?", name); err != nil { + return fmt.Errorf("forget prepared database %s: %w", name, err) + } + return nil +} + +// Session-level named locks coordinate the processes sharing a template. +// GET_LOCK names are limited to 64 characters, so the template's name is +// hashed into the lock name. + +const errNoSuchTable = 1146 // ER_NO_SUCH_TABLE + +func lockName(template string) string { + sum := sha256.Sum256([]byte(template)) + return "testdb:template:" + hex.EncodeToString(sum[:12]) +} + +// lockBuild takes the template's build lock, waiting as long as ctx allows. +func lockBuild(ctx context.Context, conn *sql.Conn, template string) error { + timeout := -1 // forever, as far as MySQL is concerned; ctx still applies + if deadline, ok := ctx.Deadline(); ok { + timeout = max(int(time.Until(deadline).Seconds()), 1) + } + var got sql.NullInt64 + if err := conn.QueryRowContext(ctx, "SELECT GET_LOCK(?, ?)", lockName(template), timeout).Scan(&got); err != nil { + return fmt.Errorf("acquire build lock: %w", err) + } + if !got.Valid { + return errors.New("acquire build lock: GET_LOCK returned NULL") + } + if got.Int64 != 1 { + return fmt.Errorf("acquire build lock: another process held it for over %d seconds", timeout) + } + return nil +} + +// unlockBuild releases the build lock; not holding it is a sequencing bug. +func unlockBuild(ctx context.Context, conn *sql.Conn, template string) error { + var released sql.NullInt64 + if err := conn.QueryRowContext(ctx, "SELECT RELEASE_LOCK(?)", lockName(template)).Scan(&released); err != nil { + return fmt.Errorf("release build lock: %w", err) + } + if !released.Valid || released.Int64 != 1 { + return errors.New("release build lock: it was not held by this session") + } + return nil +} + +// serverStartedAt returns when the server started, as a Unix timestamp. +// Connection ids restart from 1 when the server does, so a user row is only +// trusted if it was written during the current run of the server. +func serverStartedAt(ctx context.Context, conn *sql.Conn) (int64, error) { + var now int64 + if err := conn.QueryRowContext(ctx, "SELECT UNIX_TIMESTAMP()").Scan(&now); err != nil { + return 0, fmt.Errorf("server time: %w", err) + } + var name, value string + if err := conn.QueryRowContext(ctx, "SHOW GLOBAL STATUS LIKE 'Uptime'").Scan(&name, &value); err != nil { + return 0, fmt.Errorf("server uptime: %w", err) + } + uptime, err := strconv.ParseInt(value, 10, 64) + if err != nil { + return 0, fmt.Errorf("server uptime %q: %w", value, err) + } + return now - uptime, nil +} + +// serverStartTolerance absorbs the second-level rounding of two processes +// computing the server's start time at different moments. +const serverStartTolerance = 5 + +// liveSet answers whether a (connection id, server start) pair belongs to a +// session that exists right now. +type liveSet struct { + ids map[uint64]struct{} + start int64 +} + +func (s liveSet) has(id uint64, start int64) bool { + if start < s.start-serverStartTolerance || start > s.start+serverStartTolerance { + return false // written during an earlier run of the server + } + _, ok := s.ids[id] + return ok +} + +func liveConnections(ctx context.Context, conn *sql.Conn) (liveSet, error) { + start, err := serverStartedAt(ctx, conn) + if err != nil { + return liveSet{}, err + } + rows, err := conn.QueryContext(ctx, "SELECT id FROM information_schema.PROCESSLIST") + if err != nil { + return liveSet{}, fmt.Errorf("list sessions: %w", err) + } + set := liveSet{ids: map[uint64]struct{}{}, start: start} + for rows.Next() { + var id uint64 + if err := rows.Scan(&id); err != nil { + return liveSet{}, errors.Join(fmt.Errorf("list sessions: %w", err), rows.Close()) + } + set.ids[id] = struct{}{} + } + if err := errors.Join(rows.Err(), rows.Close()); err != nil { + return liveSet{}, fmt.Errorf("list sessions: %w", err) + } + return set, nil +} + +// maxTemplateNameLen is MySQL's identifier limit. +const maxTemplateNameLen = 64 + +func validateTemplateName(name string) error { + if name == "" { + return errors.New("template name is empty") + } + if len(name) > maxTemplateNameLen { + return fmt.Errorf("template name %q is %d bytes; the maximum is %d", name, len(name), maxTemplateNameLen) + } + return nil +} + +// CreateDatabaseFromTemplate hands the test a database cloned from the +// template: one this process prepared in advance if there is one ready +// (see WithReadyDatabases), otherwise a clone made now under the requested +// name. It never waits for the pool. +func (p *Provider) CreateDatabaseFromTemplate(ctx context.Context, requestedName, templateName string) (string, error) { + if v, ok := registry.Load(templateName); ok { + if tpl := v.(*Template); tpl.pool != nil { + if name, ok := tpl.pool.take(); ok { + if _, err := p.conn.ExecContext(ctx, "DELETE FROM "+qualify(templateName, bookkeepingPrefix+"ready")+" WHERE db_name = ?", name); err != nil { + // Not ours after all; do not leave it unaccounted for + return "", errors.Join(fmt.Errorf("claim prepared database %s: %w", name, err), + p.DropDatabase(context.WithoutCancel(ctx), name)) + } + return name, nil + } + } + } + if err := cloneDatabase(ctx, p.adminDSN, templateName, requestedName); err != nil { + return "", err + } + return requestedName, nil +} + +// TemplateExists reports whether a database with the given name exists. +func (p *Provider) TemplateExists(ctx context.Context, name string) (bool, error) { + var n int + err := p.conn.QueryRowContext(ctx, "SELECT COUNT(*) FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = ?", name).Scan(&n) + if err != nil { + return false, fmt.Errorf("check template exists: %w", err) + } + return n == 1, nil +} + +// TemplateExists reports whether a database with the given name exists. +// This is a debugging aid; EnsureTemplate handles existence itself. +func TemplateExists(ctx context.Context, name string, opts ...TemplateOption) (bool, error) { + var cfg templateConfig + for _, opt := range opts { + opt(&cfg) + } + p := &Provider{} + if err := p.Initialize(ctx, testdb.Config{AdminDSNOverride: cfg.adminDSN}); err != nil { + return false, fmt.Errorf("mysql.TemplateExists: %w", err) + } + exists, err := p.TemplateExists(ctx, name) + if err := errors.Join(err, p.Cleanup(context.WithoutCancel(ctx))); err != nil { + return false, fmt.Errorf("mysql.TemplateExists: %w", err) + } + return exists, nil +} + +var _ testdb.TemplateProvider = (*Provider)(nil) diff --git a/mysql/template_benchmark_test.go b/mysql/template_benchmark_test.go new file mode 100644 index 0000000..b3c701e --- /dev/null +++ b/mysql/template_benchmark_test.go @@ -0,0 +1,145 @@ +package mysql_test + +import ( + "context" + "database/sql" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/bashhack/testdb" + "github.com/bashhack/testdb/mysql" +) + +// The benchmarks compare what a test pays for an isolated database with a +// realistic schema: creating it and running the migrations, cloning a +// template on demand, and taking a clone the template prepared ahead. +// These are the numbers the README quotes for MySQL. + +const benchTables = 50 + +// writeBenchSchema writes a schema of benchTables tables with an index each +// and some rows, as one multi-statement SQL file. +func writeBenchSchema(b *testing.B) string { + b.Helper() + var sql strings.Builder + for i := range benchTables { + fmt.Fprintf(&sql, `CREATE TABLE t%d ( + id INT AUTO_INCREMENT PRIMARY KEY, + name VARCHAR(100) NOT NULL, + price DECIMAL(10,2) NOT NULL, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + INDEX (name) + ); +`, i) + } + for i := range 5 { + fmt.Fprintf(&sql, "INSERT INTO t%d (name, price) WITH RECURSIVE n AS (SELECT 1 AS k UNION ALL SELECT k+1 FROM n WHERE k < 1000) SELECT CONCAT('Product ', k), 19.99 FROM n;\n", i) + } + dir := b.TempDir() + path := filepath.Join(dir, "001_schema.sql") + if err := os.WriteFile(path, []byte(sql.String()), 0o644); err != nil { + b.Fatal(err) + } + return dir +} + +func requireMySQLB(b *testing.B) { + b.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + p := &mysql.Provider{} + if err := p.Initialize(ctx, testdb.Config{}); err != nil { + b.Skipf("MySQL not available: %v", err) + } + if err := p.Cleanup(ctx); err != nil { + b.Fatal(err) + } +} + +func ensureBenchTemplate(b *testing.B, name string, schemaDir string, opts ...mysql.TemplateOption) *mysql.Template { + b.Helper() + ctx := context.Background() + tpl, err := mysql.EnsureTemplate(ctx, name, mysql.SQLFiles(schemaDir), opts...) + if err != nil { + b.Fatalf("EnsureTemplate: %v", err) + } + b.Cleanup(func() { + if err := tpl.Release(ctx); err != nil { + b.Errorf("Release: %v", err) + } + }) + return tpl +} + +// BenchmarkEmptyDatabase measures CREATE DATABASE and the drop, nothing else. +func BenchmarkEmptyDatabase(b *testing.B) { + requireMySQLB(b) + b.ReportAllocs() + for b.Loop() { + mysql.New(b, &mysql.SQLDBInitializer{}) + } +} + +// BenchmarkFreshWithMigrations measures what every test pays without a +// template: a new database plus the whole schema. +func BenchmarkFreshWithMigrations(b *testing.B) { + requireMySQLB(b) + schemaDir := writeBenchSchema(b) + b.ReportAllocs() + for b.Loop() { + mysql.New(b, &mysql.SQLDBInitializer{}, testdb.WithMigrator(mysql.SQLFiles(schemaDir))) + } +} + +// BenchmarkTemplateCloneOnDemand measures a clone made when the test asks, +// with no databases prepared ahead. +func BenchmarkTemplateCloneOnDemand(b *testing.B) { + requireMySQLB(b) + schemaDir := writeBenchSchema(b) + tpl := ensureBenchTemplate(b, "bench_mysql_tpl_ondemand", schemaDir, mysql.WithReadyDatabases(0)) + b.ReportAllocs() + for b.Loop() { + mysql.New(b, &mysql.SQLDBInitializer{}, tpl.Option()) + } +} + +// BenchmarkTemplateReady measures taking databases the template prepared +// ahead, with a pause between takes standing in for the test's own work. +// The pool keeps up only while tests take clones no faster than the +// workers make them, so the pause length is what decides the result; the +// sub-benchmarks cover a quick test and an integration-sized one, and +// report how many takes were served from the pool. +func BenchmarkTemplateReady(b *testing.B) { + for _, workers := range []int{2, 4, 8} { + for _, work := range []time.Duration{50 * time.Millisecond, 200 * time.Millisecond, 500 * time.Millisecond} { + b.Run(fmt.Sprintf("workers=%d/test=%s", workers, work), func(b *testing.B) { + requireMySQLB(b) + schemaDir := writeBenchSchema(b) + tpl := ensureBenchTemplate(b, "bench_mysql_tpl_ready", schemaDir, + mysql.WithReadyDatabases(8), mysql.WithReadyWorkers(workers), mysql.WithReadyPrefix("bench_ready")) + time.Sleep(6 * time.Second) // let the pool fill once + pooled := 0 + b.ReportAllocs() + for b.Loop() { + db := mysql.New(b, &mysql.SQLDBInitializer{}, tpl.Option()) + b.StopTimer() + if strings.HasPrefix(db.Name(), "bench_ready_") { + pooled++ + } + // The test's own work; not part of what we measure + var n int + if err := db.Entity().(*sql.DB).QueryRowContext(context.Background(), "SELECT COUNT(*) FROM t0").Scan(&n); err != nil { + b.Fatal(err) + } + time.Sleep(work) + b.StartTimer() + } + b.ReportMetric(float64(pooled)/float64(b.N)*100, "%pooled") + }) + } + } +} diff --git a/mysql/template_test.go b/mysql/template_test.go new file mode 100644 index 0000000..ae2ef37 --- /dev/null +++ b/mysql/template_test.go @@ -0,0 +1,344 @@ +package mysql_test + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "strings" + "testing" + "time" + + "github.com/bashhack/testdb/mysql" +) + +func uniqueTemplateName(t *testing.T, prefix string) string { + t.Helper() + var b [4]byte + if _, err := rand.Read(b[:]); err != nil { + t.Fatal(err) + } + return prefix + "_" + hex.EncodeToString(b[:]) +} + +func release(t *testing.T, tpl *mysql.Template) { + t.Helper() + if err := tpl.Release(context.Background()); err != nil { + t.Errorf("Release: %v", err) + } +} + +func waitFor(t *testing.T, what string, cond func() bool) { + t.Helper() + deadline := time.Now().Add(30 * time.Second) + for !cond() { + if time.Now().After(deadline) { + t.Fatalf("timed out waiting for %s", what) + } + time.Sleep(50 * time.Millisecond) + } +} + +func TestTemplateBasic(t *testing.T) { + requireMySQL(t) + admin := adminDB(t) + ctx := context.Background() + name := uniqueTemplateName(t, "tpl_basic") + + tpl, err := mysql.EnsureTemplate(ctx, name, mysql.SQLFiles(schemaDir), mysql.WithReadyDatabases(0)) + if err != nil { + t.Fatalf("EnsureTemplate: %v", err) + } + t.Cleanup(func() { release(t, tpl) }) + + if tpl.Name() != name || !databaseExists(t, admin, name) { + t.Fatal("template should exist under its name") + } + + db := mysql.Setup(t, tpl.Option()) + for _, table := range []string{"test_table", "users", "products"} { + if !tableExists(t, db, table) { + t.Errorf("clone lacks %s", table) + } + } + if tableExists(t, db, "_testdb_info") || tableExists(t, db, "_testdb_users") { + t.Error("bookkeeping tables leaked into the clone") + } + var n int + if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM users").Scan(&n); err != nil { + t.Fatal(err) + } + if n != 2 { + t.Errorf("fixture rows = %d, want 2", n) + } + if currentDatabase(t, db) == name { + t.Fatal("test got the template itself, not a clone") + } +} + +func TestTemplateReleaseDropsWhenLastOut(t *testing.T) { + requireMySQL(t) + admin := adminDB(t) + ctx := context.Background() + name := uniqueTemplateName(t, "tpl_lastout") + + setupRuns := 0 + setup := func(ctx context.Context, dsn string) error { setupRuns++; return nil } + + first, err := mysql.EnsureTemplate(ctx, name, setup, mysql.WithReadyDatabases(0)) + if err != nil { + t.Fatal(err) + } + second, err := mysql.EnsureTemplate(ctx, name, setup, mysql.WithReadyDatabases(0)) + if err != nil { + t.Fatal(err) + } + if setupRuns != 1 { + t.Errorf("setup ran %d times, want 1", setupRuns) + } + + release(t, first) + if !databaseExists(t, admin, name) { + t.Fatal("template dropped while another handle used it") + } + release(t, second) + if databaseExists(t, admin, name) { + t.Fatal("template not dropped by the last release") + } + release(t, second) // harmless +} + +func TestTemplateFingerprintAndKeep(t *testing.T) { + requireMySQL(t) + admin := adminDB(t) + ctx := context.Background() + name := uniqueTemplateName(t, "tpl_fp") + t.Cleanup(func() { + if databaseExists(t, admin, name) { + if _, err := admin.ExecContext(ctx, "DROP DATABASE `"+name+"`"); err != nil { + t.Error(err) + } + } + }) + + setupRuns := 0 + setup := func(ctx context.Context, dsn string) error { setupRuns++; return nil } + ensure := func(opts ...mysql.TemplateOption) *mysql.Template { + t.Helper() + tpl, err := mysql.EnsureTemplate(ctx, name, setup, append(opts, mysql.WithReadyDatabases(0))...) + if err != nil { + t.Fatalf("EnsureTemplate: %v", err) + } + return tpl + } + + tpl := ensure(mysql.WithFingerprint("v1"), mysql.WithKeepTemplate()) + release(t, tpl) + if !databaseExists(t, admin, name) { + t.Fatal("kept template was dropped") + } + tpl = ensure(mysql.WithFingerprint("v1"), mysql.WithKeepTemplate()) + release(t, tpl) + if setupRuns != 1 { + t.Fatalf("setup ran %d times after a matching fingerprint, want 1", setupRuns) + } + tpl = ensure(mysql.WithFingerprint("v2")) + if setupRuns != 2 { + t.Fatalf("setup ran %d times after a changed fingerprint, want 2", setupRuns) + } + release(t, tpl) + if databaseExists(t, admin, name) { + t.Fatal("template without WithKeepTemplate was not dropped") + } +} + +func TestTemplateStaleWhileInUse(t *testing.T) { + requireMySQL(t) + ctx := context.Background() + name := uniqueTemplateName(t, "tpl_stale") + + holder, err := mysql.EnsureTemplate(ctx, name, nil, mysql.WithFingerprint("v1"), mysql.WithReadyDatabases(0)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { release(t, holder) }) + + _, err = mysql.EnsureTemplate(ctx, name, nil, mysql.WithFingerprint("v2"), mysql.WithReadyDatabases(0)) + if err == nil || !strings.Contains(err.Error(), "in use by") { + t.Fatalf("expected an in-use error, got %v", err) + } +} + +func TestTemplateHalfBuiltIsRebuilt(t *testing.T) { + requireMySQL(t) + admin := adminDB(t) + ctx := context.Background() + name := uniqueTemplateName(t, "tpl_half") + + // A database under the template's name with no completeness marker is + // what a build that crashed leaves behind + if _, err := admin.ExecContext(ctx, "CREATE DATABASE `"+name+"`"); err != nil { + t.Fatal(err) + } + if _, err := admin.ExecContext(ctx, "CREATE TABLE `"+name+"`.leftover (id INT)"); err != nil { + t.Fatal(err) + } + + setupRuns := 0 + tpl, err := mysql.EnsureTemplate(ctx, name, func(ctx context.Context, dsn string) error { setupRuns++; return nil }, + mysql.WithReadyDatabases(0)) + if err != nil { + t.Fatalf("EnsureTemplate over a half-built template: %v", err) + } + t.Cleanup(func() { release(t, tpl) }) + if setupRuns != 1 { + t.Errorf("setup ran %d times, want 1 (rebuild)", setupRuns) + } + var n int + if err := admin.QueryRowContext(ctx, "SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA = ? AND TABLE_NAME = 'leftover'", name).Scan(&n); err != nil { + t.Fatal(err) + } + if n != 0 { + t.Error("the half-built template's contents survived the rebuild") + } +} + +func TestTemplateDeadProcessIsIgnoredAndSwept(t *testing.T) { + requireMySQL(t) + admin := adminDB(t) + ctx := context.Background() + name := uniqueTemplateName(t, "tpl_dead") + + tpl, err := mysql.EnsureTemplate(ctx, name, nil, mysql.WithReadyDatabases(0)) + if err != nil { + t.Fatal(err) + } + // Fake another process that died: a user row and a prepared database + // whose connection id no longer exists + var now int64 + if err := admin.QueryRowContext(ctx, "SELECT UNIX_TIMESTAMP()").Scan(&now); err != nil { + t.Fatal(err) + } + var uptime string + var ignored string + if err := admin.QueryRowContext(ctx, "SHOW GLOBAL STATUS LIKE 'Uptime'").Scan(&ignored, &uptime); err != nil { + t.Fatal(err) + } + var up int64 + if _, err := fmtSscan(uptime, &up); err != nil { + t.Fatal(err) + } + start := now - up + const deadID = 4000000000 // no live session has this id + orphan := uniqueTemplateName(t, "test_orphan") + if _, err := admin.ExecContext(ctx, "CREATE DATABASE `"+orphan+"`"); err != nil { + t.Fatal(err) + } + if _, err := admin.ExecContext(ctx, "INSERT INTO `"+name+"`.`_testdb_users` (connection_id, server_started_at) VALUES (?, ?)", deadID, start); err != nil { + t.Fatal(err) + } + if _, err := admin.ExecContext(ctx, "INSERT INTO `"+name+"`.`_testdb_ready` (db_name, connection_id, server_started_at) VALUES (?, ?, ?)", orphan, deadID, start); err != nil { + t.Fatal(err) + } + + // We are the only live user, so Release must drop the template and + // sweep the dead process's prepared database + release(t, tpl) + if databaseExists(t, admin, name) { + t.Error("template kept alive by a dead process's row") + } + if databaseExists(t, admin, orphan) { + t.Error("a dead process's prepared database was not swept") + _, _ = admin.ExecContext(ctx, "DROP DATABASE `"+orphan+"`") + } +} + +func TestTemplateReadyPool(t *testing.T) { + requireMySQL(t) + admin := adminDB(t) + ctx := context.Background() + name := uniqueTemplateName(t, "tpl_pool") + + tpl, err := mysql.EnsureTemplate(ctx, name, mysql.SQLFiles(schemaDir), + mysql.WithReadyDatabases(2), mysql.WithReadyPrefix("ready_pool")) + if err != nil { + t.Fatal(err) + } + // Registered first, so it runs after the test databases taken below + // have been dropped by their own cleanups: then nothing may remain + t.Cleanup(func() { + release(t, tpl) + var leftovers int + if err := admin.QueryRowContext(ctx, "SELECT COUNT(*) FROM information_schema.SCHEMATA WHERE SCHEMA_NAME LIKE 'ready\\_pool\\_%'").Scan(&leftovers); err != nil { + t.Fatal(err) + } + if leftovers != 0 { + t.Errorf("%d prepared databases left behind after Release", leftovers) + } + if databaseExists(t, admin, name) { + t.Error("template not dropped") + } + }) + + waitFor(t, "the pool to fill", func() bool { return tpl.Ready() == 2 }) + + // A test takes a prepared database: the name carries the pool's prefix, + // the clone is complete, and the pool refills + db := mysql.Setup(t, tpl.Option()) + got := currentDatabase(t, db) + if !strings.HasPrefix(got, "ready_pool_") { + t.Errorf("test got %s, want a prepared database", got) + } + if !tableExists(t, db, "users") { + t.Error("prepared database lacks the schema") + } + waitFor(t, "the pool to refill", func() bool { return tpl.Ready() == 2 }) + + // Taking faster than the pool refills never waits: whatever is not + // ready is cloned on demand, under a regular test name. Every database + // handed out has the schema either way. + onDemand := 0 + for range 5 { + fresh := mysql.Setup(t, tpl.Option()) + if !tableExists(t, fresh, "users") { + t.Fatal("clone lacks the schema") + } + if strings.HasPrefix(currentDatabase(t, fresh), "test_") { + onDemand++ + } + } + t.Logf("5 rapid takes: %d served on demand, %d from the pool", onDemand, 5-onDemand) +} + +func TestTemplateNameValidation(t *testing.T) { + ctx := context.Background() + if _, err := mysql.EnsureTemplate(ctx, "", nil); err == nil { + t.Error("expected an error for an empty name") + } + if _, err := mysql.EnsureTemplate(ctx, strings.Repeat("x", 65), nil); err == nil || !strings.Contains(err.Error(), "maximum is 64") { + t.Errorf("expected a length error, got %v", err) + } +} + +func TestTemplateContextCancelled(t *testing.T) { + requireMySQL(t) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, err := mysql.EnsureTemplate(ctx, uniqueTemplateName(t, "tpl_ctx"), nil, mysql.WithReadyDatabases(0)) + if !errors.Is(err, context.Canceled) { + t.Fatalf("expected context.Canceled, got %v", err) + } +} + +// fmtSscan parses a decimal string into n. +func fmtSscan(s string, n *int64) (int, error) { + var v int64 + for _, c := range s { + if c < '0' || c > '9' { + return 0, errors.New("not a number: " + s) + } + v = v*10 + int64(c-'0') + } + *n = v + return 1, nil +} diff --git a/postgres/postgres.go b/postgres/postgres.go index 67db32e..80aa4f8 100644 --- a/postgres/postgres.go +++ b/postgres/postgres.go @@ -194,7 +194,7 @@ func (p *PostgresProvider) CreateDatabase(ctx context.Context, name string) erro // // The template database must already exist; postgres.EnsureTemplate creates // it and keeps it alive while this process uses it. -func (p *PostgresProvider) CreateDatabaseFromTemplate(ctx context.Context, name, templateName string) error { +func (p *PostgresProvider) CreateDatabaseFromTemplate(ctx context.Context, name, templateName string) (string, error) { quotedName := pgx.Identifier{name}.Sanitize() quotedTemplate := pgx.Identifier{templateName}.Sanitize() @@ -203,12 +203,13 @@ func (p *PostgresProvider) CreateDatabaseFromTemplate(ctx context.Context, name, if err != nil { // Check if error is due to non-existent template and provide helpful message if strings.Contains(err.Error(), "does not exist") { - return fmt.Errorf("template database %q does not exist (create it with postgres.EnsureTemplate): %w", templateName, err) + return "", fmt.Errorf("template database %q does not exist (create it with postgres.EnsureTemplate): %w", templateName, err) } - return fmt.Errorf("create database from template: %w", err) + return "", fmt.Errorf("create database from template: %w", err) } - return nil + // PostgreSQL clones on demand, so the database is the one asked for + return name, nil } // TemplateExists checks if a template database exists. diff --git a/renamed_clone_test.go b/renamed_clone_test.go new file mode 100644 index 0000000..d331d54 --- /dev/null +++ b/renamed_clone_test.go @@ -0,0 +1,51 @@ +package testdb + +import ( + "context" + "testing" +) + +// renamingProvider hands over a database under its own name, as a provider +// with clones prepared in advance does. +type renamingProvider struct { + mockProvider + dropped []string +} + +func (p *renamingProvider) CreateDatabaseFromTemplate(ctx context.Context, requested, template string) (string, error) { + return "ready_" + requested, nil +} + +func (p *renamingProvider) DropDatabase(ctx context.Context, name string) error { + p.dropped = append(p.dropped, name) + return nil +} + +func TestNewUsesTheNameTheProviderReturns(t *testing.T) { + provider := &renamingProvider{} + db, err := New(t, provider, nil, WithTemplate("tpl")) + if err != nil { + t.Fatal(err) + } + if got := db.Name(); len(got) < 6 || got[:6] != "ready_" { + t.Errorf("Name() = %q, want the provider's name", got) + } + if dsn := db.DSN(); dsn == "" || !containsName(dsn, db.Name()) { + t.Errorf("DSN %q should be for database %q", dsn, db.Name()) + } + if err := db.Close(); err != nil { + t.Fatal(err) + } + if len(provider.dropped) != 1 || provider.dropped[0] != db.Name() { + t.Errorf("Close dropped %v, want exactly [%s]", provider.dropped, db.Name()) + } +} + +func containsName(dsn, name string) bool { + for i := 0; i+len(name) <= len(dsn); i++ { + if dsn[i:i+len(name)] == name { + return true + } + } + return false +} diff --git a/testdb.go b/testdb.go index fdf0cb2..df668cf 100644 --- a/testdb.go +++ b/testdb.go @@ -142,10 +142,11 @@ type Provider interface { type TemplateProvider interface { Provider - // CreateDatabaseFromTemplate creates a new database by cloning a template. - // This is significantly faster than CreateDatabase for databases with large schemas - // or pre-seeded data. The template database must already exist. - CreateDatabaseFromTemplate(ctx context.Context, name, templateName string) error + // CreateDatabaseFromTemplate makes a database cloned from the template + // and returns its name: requestedName, or a different one when the + // provider had a clone ready in advance and cannot rename it. The + // template database must already exist. + CreateDatabaseFromTemplate(ctx context.Context, requestedName, templateName string) (actualName string, err error) // TemplateExists checks if a template database exists. // This is useful for debugging and conditional template creation. @@ -344,24 +345,27 @@ func New(t testing.TB, provider Provider, initializer DBInitializer, opts ...Opt } } - if cfg.Verbose { - if cfg.TemplateName != "" { - t.Logf("testdb: creating database %s from template %s", dbName, cfg.TemplateName) - } else { - t.Logf("testdb: creating database %s", dbName) - } - } - // Use template if specified, otherwise create fresh database if cfg.TemplateName != "" { - // Checked above - if err := provider.(TemplateProvider).CreateDatabaseFromTemplate(ctx, dbName, cfg.TemplateName); err != nil { + // Checked above. The provider may hand over a database it cloned in + // advance, under a name of its own. + actual, err := provider.(TemplateProvider).CreateDatabaseFromTemplate(ctx, dbName, cfg.TemplateName) + if err != nil { return nil, &Error{ Op: "provider.CreateDatabaseFromTemplate", Err: err, } } + if actual != "" { + dbName = actual + } + if cfg.Verbose { + t.Logf("testdb: database %s cloned from template %s", dbName, cfg.TemplateName) + } } else { + if cfg.Verbose { + t.Logf("testdb: creating database %s", dbName) + } if err := provider.CreateDatabase(ctx, dbName); err != nil { return nil, &Error{ Op: "provider.CreateDatabase", diff --git a/testdb_test.go b/testdb_test.go index 8e41ebc..2699cd8 100644 --- a/testdb_test.go +++ b/testdb_test.go @@ -700,8 +700,8 @@ func (m *mockErrorProvider) Cleanup(ctx context.Context) error { return nil } -func (m *mockErrorProvider) CreateDatabaseFromTemplate(ctx context.Context, name, templateName string) error { - return errors.New("templates not supported in mock") +func (m *mockErrorProvider) CreateDatabaseFromTemplate(ctx context.Context, name, templateName string) (string, error) { + return "", errors.New("templates not supported in mock") } func (m *mockErrorProvider) TemplateExists(ctx context.Context, name string) (bool, error) { @@ -772,8 +772,8 @@ func (m *mockProvider) Cleanup(ctx context.Context) error { return nil } -func (m *mockProvider) CreateDatabaseFromTemplate(ctx context.Context, name, templateName string) error { - return nil +func (m *mockProvider) CreateDatabaseFromTemplate(ctx context.Context, name, templateName string) (string, error) { + return name, nil } func (m *mockProvider) TemplateExists(ctx context.Context, name string) (bool, error) {