Release Notes
Thursday, November 29, 2018
Quest Software is pleased to make available this Beta version of Benchmark Factory®. Benchmark Factory is a database performance and code scalability testing tool that simulates users and transactions on the database and replays production workload in non-production environments. With Benchmark Factory developers, DBAs, and QA teams can validate that their databases will scale as user loads increase, application changes are made, and platform changes are implemented.
Benchmark Factory is available for Oracle®, SQL Server®, IBM® DB2®, SAP®, MySQL, and other databases via ODBC connectivity. Benchmark Factory:
Upon completion of a test execution, all test results are collected and stored in the repository for data analysis and reporting. Benchmark Factory collects a variety of informative and detailed statistics that include:
This beta release of Benchmark Factory includes the following new features and enhancements.
This release includes an exciting new feature. Benchmark Factory now provides an agent which you can deploy in your Linux environment. After installing the Linux version of Benchmark Factory Agent in your Linux environment, use the Benchmark Factory Console to set up the new agent. After setup, you can select the new agent from the list of agents available for load testing. In addition, the Benchmark Factory REST API supports configuring and modifying the Benchmark Factory Agent for Linux.
Supported Databases. The Benchmark Factory Agent for Linux can be used to test workloads against PostgreSQL databases.
Note: The Benchmark Factory Agent for Linux supports the following platforms only: Red Hat 7.x (64-bit), CentOs 7.x (64-bit), and Oracle Linux 7.x (64-bit).
The Benchmark Factory Agent for Linux is available for download from the Benchmark Factory Community on Toad World.
This release of Benchmark Factory includes a PostgreSQL Native connection.
Then select PostgreSQL from the Connection Type drop-down list.
Enter connection information in this dialog to create a PostgreSQL native connection.
Alternatively, if you want to create a PostgreSQL ODBC connection, select the ODBC tab and enter connection information.
The following PostgreSQL database versions are supported: PostgreSQL 9.4, 9.5, 9.6 and 10, and EnterpriseDB Postgres 10.
This release includes an enhanced Licensing dialog which provides a simpler, easier-to-use interface.
To delete a license, select the license to delete and click Delete License.
Important: If you delete a license key, any features enabled by that license and not supported by another currently-active license are disabled.
The following is a list of issues addressed and enhancements implemented in this beta release of Benchmark Factory.
| Feature | Resolved Issue | Defect ID |
|---|---|---|
| Agents | When attempting to create a new, additional local agent through the Benchmark Factory console, user credentials cannot be used. A warning message now informs the user when this is the case. | BMF-2767 |
| Benchmark Testing | Corrected an issue that created a slight inaccuracy in the calculation of scale factor from database size in the Scale tab in the Test Options section. This issue was encountered when creating a TPC-C or TPC-E benchmark test. | BMF-2146 |
| Benchmark Testing | Corrected an issue in which the job fails during the Create Objects step if "Create/Load objects if objects don't exist" is selected and only some objects exist (for example, tables don't exist but procedures do). | BMF-3031 |
| Benchmark Testing | The CPU and memory usage statistics now display as expected when running a job using BMFAgent.exe. | BMF-3254 |
| Benchmark Testing | Oracle: Corrected an issue that caused an ORA-01406 error when attempting to run a TPC-E test against Oracle when the NLS_DATE_FORMAT variable was set to DD-MON-YYYY HH24:MI:SS. | BMF-3759 |
| BFScripts | Corrected an issue in which using BFURand generated duplicate values. | BMF-1011 |
| BMFServer | The BMFServer.ini file is now located in C:\ProgramData\Quest Software\BMF. | BMF-2704 |
| BMFServer | The default REST API port number for BMFServer.exe is now the same as the Benchmark Factory console, port 30100. | BMF-2702 |
| BMFServer | The BMFServer log file (BMFServer.log) is now located in: \My Benchmark Factory\<version>\<bitness>\Error Logs | BMF-2703 |
| Capture and Replay | SQL Server: When creating a job and adding the "Replay load from SQL Server trace table" test, Benchmark Factory no longer automatically searches the network for hosts To search for hosts, click "Refresh" on the SQL Server Authentication page. | BMF-3371 |
| Capture and Replay | Oracle: Simplified the replay test name for an imported Oracle capture. The replay test name is now "Replay Oracle Workload" regardless of the capture mode used. | BMF-3264 |
| Capture and Replay | Oracle: Corrected an issue in which a capture using FGAC failed when the capture included SQL longer than 4K. The following warning message displayed in the Status window when the capture process was running: ORA-02003: invalid USERENV parameter. | BMF-3707 |
| Capture and Replay | When importing from the Oracle Dynamic Performance View, transactions with a very low execution rate might not be imported. When this is the case, Benchmark Factory now displays a message informing the user. | BMF-2997 |
| Capture and Replay | Enhancement: When importing from an ODBC trace file, other statements in addition to Insert, Update, Delete, and Select statements are now included (e.g., Drop statements). | BMF-3067 |
| Capture and Replay | Corrected an issue in which the user was unable to specify a start time when scheduling a capture if the user's datetime format (Regional settings) was set to Polish (24h format). | BMF-4397 |
| Custom Mix Test | In a custom mix or goal test, if no results are obtained during the test run, it could be due to the fact that not all executions were completed in the specified pre-sampling time. The console now displays a message when this is possible. | BMF-2858 |
| General | The Support Bundle now includes information about BMFServer and BMFAgent and the BMFServer log file. | BMF-2705 |
| REST API | Corrected an issue in which attempting to GET testrun information using the zero-based index id or the test run, for example http://{server}:{port}/api/testruns/{index id}, resulted in a 404 error. This issue occurred with the following resource collections: TestRuns, UserLoadResults, TransactionResults, SubTransactionResults, and NodeResults. | BMF-2700 |
| REST API | Requesting latency for a transaction in a test now returns the latency information, as expected. | BMF-2745 |
| REST API | The time format is now the same in the current results returned by the console or the BMFServer (BMFServer.exe). | BMF-2857 |
| REST API | Corrected an issue that allowed a job to post successfully even though the request body contained invalid connection information. | BMF-2904 |
| REST API | The Database Flush property is now included in the Connection request/response body when using the REST API. | BMF-2918 |
| REST API | A request for TOTALRESPONSETIME from a test run now successfully returns data. An example URI is: http://{server}:{port}/api/TestRuns/1/UserloadResults/0/transactionresults/0/totalresponsetime. | BMF-3174 |
| REST API | Corrected an issue that caused an error when attempting to GET the Settings resource using the REST API (/api/settings). | BMF-2744 |
| REST API | Corrected an issue in which the ID number of a job created through the Benchmark Factory console unexpectedly changed (increased incrementally) when a subsequent job was created and added to the queue. | BMF-2856 |
| REST API | Added a read-only resource to the REST API which returns the BMF version number. | BMF-3706 |
| Run Reports | In a Transaction Mix graph, long transaction names are no longer truncated and are displayed correctly. | BMF-4660 |
| TPC-E Benchmark Test | IBM DB2: Corrected an issue which caused a TPC-E transaction (Trade-Result Transaction) to fail in a test using a DB2 native connection. | BMF-4603 |
| User Interface | Licensing dialog. The Licensing dialog has been redesigned to streamline the process of entering and modifying Benchmark Factory license keys. | BMF-1233 |
| User Interface | If one of the directories specified in Settings (Edit | Settings | General) does not have appropriate permissions applied and Benchmark Factory attempts to write to it, an informative message reminds the user to check directory permissions. | BMF-3715 |
| User Interface | Oracle: After importing an ODBC trace file, the path and file name are now displayed in full in the Trace File Summary tab of the New Job Wizard. | BMF-2973 |
| User Interface | Corrected text display issues found in the New Job Wizard in a Windows 10 environment when text size (DPI) was set to125% or greater. | BMF-1670, BMF-4895 |
The following is a list of issues known to exist at the time of this release.
| Feature | Known Issue | Defect ID |
|---|---|---|
| Capture and Replay | Oracle workload: Capturing using the trace file method is not available and is disabled when connected to an Oracle 12c pluggable database (PDB). |
N/A |
| Capture and Replay |
Oracle workload: In Benchmark Factory 7.2 or later, if you attempt to use Oracle capture files created in an earlier version of Benchmark Factory (6.9.3 or earlier) and .NET Framework 2.0 is not installed, Windows may prompt you to install it (for example, in Windows 8 or later). Workaround: Install .NET Framework 2.0 (3.5). |
N/A |
| Capture and Replay | Oracle workload: Capturing using the trace file method is not available and is disabled when connected to an Oracle RAC system. |
N/A |
| Capture and Replay |
Issue: If you start a trace file capture using the current version of Benchmark Factory and then start another trace file capture using version 7.2 (or earlier) on the same database, the earlier version of Benchmark Factory cannot detect the current version performing a capture. This can result in an invalid capture. Solution: For best results, do not perform trace file captures using both the current and an earlier version of Benchmark Factory concurrently on the same database. |
N/A |
| Capture and Replay |
SQL Server workload: After capturing activity from a user logged-in using Windows authentication, attempting to replay the capture using the ODBC profile results in the following error: "Could not obtain information about Windows NT group/user '<group/user>', error code 0x5." Workaround: Use a domain account to start SQL Server service and make sure the account can connect to the Active Directory of Domain. |
N/A |
| Capture and Replay |
Oracle workload: In an Oracle capture and replay, Benchmark Factory cannot generate the Statspack report if the local replay directory is mapped to drive "B." This is because Benchmark Factory must have drive B open to execute a background .bat file during oracle capture and replay. Workaround: Do not map your local replay directory to drive "B" when running an Oracle capture and replay in which you chose to generate the Statspack report. |
109221 |
| Capture and Replay | In an Oracle 12c container database (CDB), Benchmark Factory cannot replay an existing workload that was captured in an earlier version of an Oracle database. In an Oracle 12c container database, a user name (common user) is prefixed with "c##" by default. |
N/A |
|
Capture and Replay |
Oracle Capture Using FGAC: When performing an Oracle Capture using the FGAC method, if the application has a trigger with a multi-schema select, additional transactions will be recorded. | N/A |
|
Connections - Teradata ODBC |
When creating or modifying a Teradata ODBC connection, if the Teradata DSN has incorrect connection information and you click Test Connection in the Connection dialog , Benchmark Factory might close unexpectedly. | BMF-1497 |
|
Connections - Transaction Wizard and IBM CLI Driver
|
When using the 64-bit version of Benchmark Factory, the Transaction Wizard stops working when using the IBM DB2 CLI Driver profile to connect the database which created objects using the ODBC Driver. |
N/A |
|
Connections - Microsoft Oracle ODBC Driver |
When using Microsoft’s Oracle ODBC driver for either testing or for Benchmark Factory repository access, there is an issue with the ODBC driver. It does not work with CLOB data. | N/A |
|
Connections - Oracle ODBC driver |
When using the Oracle 11g R2 Client and Oracle 32-bit ODBC driver in a Windows 64-bit environment, the Oracle 32-bit ODBC driver requires an 8-byte int data type. Workaround: Use the Oracle native client or Oracle 32-bit ODBC driver. |
N/A |
|
Connections - Microsoft SQL Server (Obsolete DB-Library) Connection |
If you run a benchmark test using a Microsoft SQL Server (Obsolete DB-Library) connection to SQL Server 2014, and you have the "Recreate objects and reload data every execution" option selected, you might encounter the following error: "DB-Library Message - Insert bulk is not supported over this access protocol" Workaround: Use the Microsoft SQL Server ODBC driver. |
N/A |
|
Connections - IBM CLI Driver |
A profile created with a DB2 ODBC data source may crash Benchmark Factory if the IBM DB2 client catalog becomes broken/corrupted. |
N/A |
|
Connections |
In Windows 10, if you open a New Connection dialog, create an SAP data source and then attempt to create a SQL Server data source, Benchmark Factory might close unexpectedly. |
BMF-555 |
|
General - Benchmark Factory Console |
Benchmark Factory does not support data types that are created by the user. | N/A |
| Industry Standard Benchmark Testing |
Background: This issue applies to benchmark testing using a SQL Server or SAP native connection when the Refresh data retention option is selected in the Test Options tab. Issue Description: In a test where the connection is lost while tables are loading, if you reconnect and attempt to rerun the job, Benchmark Factory might error with the following messages: “The Expected Total rows are not the same as the Actual Total rows,” or “a duplicate key was found” (when creating an index). The test is then canceled due to errors. Resolution: Recreate and run a new job against your selected database using the native connection. |
N/A |
| Industry Standard Benchmark Testing |
If testing against a database other than Oracle, SQL Server, DB2, SAP, or MySQL, and using the Object/Data Retention option Refresh data using backup sets, backup sets might not be created resulting in a canceled test. |
114266 |
| Benchmark Test - Scalable Hardware Test |
Attempting to use a native connection to run a Scalable Hardware benchmark test in which the objects were created using an ODBC connection and the objects are not recreated will result in a failed test with errors. The reverse is also true. Solution: When running a Scalable Hardware test using a native connection and objects were created using an ODBC connection (or vice versa) always recreate the objects. |
N/A |
|
Installation (uninstall) |
Beginning with release 7.1.1 in Windows 7 or later, if you have multiple editions of Benchmark Factory installed side-by-side (e.g., a Beta and a Commercial edition), and then uninstall one edition, you might encounter the following error when attempting to run a job: " This object does not represent an OLE control" Workaround: Do one of the following:
|
ST115773 |
| Integration with other Toad products |
Applies in Toad for Oracle suites (12.5 or later): When the database connection or the SQL statement contains Chinese characters, attempting to send SQL from SQL Optimizer for Oracle to Benchmark Factory results in an error. This issue only occurs when Spotlight on Oracle is launched prior to the other applications mentioned here. Workaround: Always launch Benchmark Factory and SQL Optimizer for Oracle prior to launching Spotlight on Oracle if sending SQL from SQL Optimizer to Benchmark Factory. |
N/A |
| Licensing |
License keys compatible with Benchmark Factory 5.8.1 (or earlier) are no longer supported. See Product Licensing for instructions on how to upgrade an earlier-version license key. |
N/A |
| Licensing | If your current Benchmark Factory license key is a 21-digit format (x-xxxxx-xxxxx-xxxxx-xxxxx) and you want to use the 64-Bit installation, you will need to request a new license key. Please obtain the new key before installing the new release. To obtain a new key, please refer to the Licensing Assistance page on our support portal: https://support.quest.com/licensing-assistance. | N/A |
| Oracle Database Privileges | In Oracle 12c, the RESOURCE role no longer includes the UNLIMITED TABLESPACE system privilege. Therefore, Benchmark Factory users with the RESOURCE role must also be granted the UNLIMITED TABLESPACE privilege. |
N/A |
| Repository |
In a new installation (not an upgrade) of Benchmark Factory 7.2 or later, a Standard (Limited) user may encounter a Repository error when attempting to launch the application. The issue involves the SQLite database that is used as the default database in a new installation of Benchmark Factory 7.2 or later. Workaround: Do one of the following:
|
BMF-202 |
| Repository | If you created a Data Source Name for the BMF MySQL ODBC Driver in a previous release (6.9.3 or 7.0), attempting to edit this DSN results in an error. |
N/A |
| Feature | Known Issue | Defect ID |
|---|---|---|
|
Connections - Oracle |
If you have multiple Oracle clients installed, after connecting using one Oracle home, attempting another connection using a different Oracle home results in an error. Workaround: Close and then re-open Benchmark Factory. |
N/A |
| Connections - SAP |
When creating a new SAP connection, if you partially define an SAP ASE ODBC DSN, then click "Cancel," and then switch to the Native tab of the New Connection dialog (or select a different connection type), Benchmark Factory quits unexpectedly. This is a known issue with the SAP ASE ODBC driver. To create a native SAP connection, use the Native tab of the New Connection dialog. To create an SAP ODBC connection, finish defining the SAP ODBC DSN in the ODBC tab of the New Connection dialog. |
N/A |
Before installing Benchmark, ensure your system meets the following minimum hardware and software requirements:
| Requirement | Details |
|---|---|
| Repository |
Benchmark Factory cannot run without a repository, and during installation a SQLite file-based repository is installed. After installation, you can select a repository of your choice. Benchmark Factory supports the following databases:
|
| Memory |
512 MB of RAM recommended Note: The memory required may vary based on the following:
|
| Platform | 1.0 GHz x86 or x64 compatible CPU |
| Browser Requirements | Benchmark Factory requires Internet Explorer® 9 or later. |
| Hard Disk Space | 120 MB of disk space for installation Note: The disk space required may vary based on options selected for installation. |
| Operating System |
Microsoft® Windows Server® 2008 R2 Service Pack 1 Microsoft Windows® 7 Service Pack 1 Microsoft Windows 8 Microsoft Windows 8.1 (Update) Microsoft Windows Server 2012 Microsoft Windows Server 2016 Microsoft Windows 10 |
| .NET Framework |
.NET Framework 2.0 is required to use Oracle workloads captured using Benchmark Factory 6.9.3 or earlier. |
| Additional Software |
The Universal C Runtime component for Windows is required. See Universal C Runtime Update or Visual C++ Redistributable for Visual Studio 2015 to download this software. Note: If you encounter an error when installing this software, install the missing prerequisite software. For Windows 8.1 or Windows Server 2012 R2, install the April 2014 update: https://support.microsoft.com/en-us/kb/2919355. For other operating systems, see the Universal C Runtime Update Prerequisites section. |
| Database Server |
Oracle®: 10g R2, 11g, 12c, and 18c (requires Oracle 10 client libraries or higher.) Benchmark Factory has been tested on Oracle Exadata 2.0 running Oracle database 11g R2. Note: Oracle Capture/Replay of workloads does not support Oracle databases earlier than 9i. PostgreSQL 9.4, 9.5, 9.6, and 10, and EnterpriseDB Postgres 10 SQL Server®: 2005, 2008, 2008 R2, 2012, 2014, 2016, and 2017
Note: Beginning with Benchmark Factory 8.1, a SQL Server client is no longer distributed along with Benchmark Factory. To create a SQL Server native connection, ensure a SQL Server client is installed. SQL Azure™: Benchmark Factory provides limited support of SQL Azure. SAP® ASE: 15.5, 15.7, 16 IBM® DB2® for LUW: 9.x through 10.x, 11.1 IBM® DB2® for z/OS: 9.x through 10.x, 11.1 MySQL: 5.5 and later Note: The 32-bit version of Benchmark Factory requires the 32-bit versions of the client libraries (including ODBC drivers). The 64-bit version requires 64-bit client libraries. |
| ODBC Database Server | Benchmark Factory supports almost all databases that you can connect to using an ODBC 3.0 or later driver. |
| Benchmark Factory Agent for Linux - Requirements |
Supported Operating Systems: CentOS 7.x (64-bit), RHEL 7.x (64-bit), and Oracle Linux 7.x (64-bit) Supported Databases for Load Testing: PostgreSQL |
| Cloud Database Service | Oracle Database as a Service |
License keys compatible with Benchmark Factory 5.8.1 and earlier versions are no longer supported.
In your current version of Benchmark Factory, go to Help | Licensing to find your license number . If not explicitly listed, it may be included as part of the Site Message associated with your key.
Example: ACCOUNTNAME-nnn-nnn-nnn
Then go to the License Key Upgrade page at: http://license.quest.com/upgrade.
Enter your e-mail address and your existing license number, and follow the prompts.
If you need help finding your license number or an upgrade key, please contact the License Administration team at: https://support.quest.com/licensing-assistance.
This section contains information about installing and operating this product in non-English configurations, such as those needed by customers outside of North America. This section does not replace the materials about supported platforms and configurations found elsewhere in the product documentation.
This release is Unicode-enabled and supports any character set. In this release, all product components should be configured to use the same or compatible character encodings and should be installed to use the same locale and regional options. This release is targeted to support operations in the following regions: North America, Western Europe and Latin America, Central and Eastern Europe, Far-East Asia, Japan.
The Benchmark Factory release package contains the following products:
Get the latest product information, find helpful resources, test the product betas, and join a discussion with the Benchmark Factory team and other community members. Join the Benchmark Factory community at: http://www.toadworld.com/products/benchmark-factory/default.aspx.
We are on a quest to make your information technology work harder for you. That is why we build community-driven software solutions that help you spend less time on IT administration and more time on business innovation. We help you modernize your data center, get you to the cloud quicker and provide the expertise, security and accessibility you need to grow your data-driven business. Combined with Quest’s invitation to the global community to be a part of its innovation, and our firm commitment to ensuring customer satisfaction, we continue to deliver solutions that have a real impact on our customers today and leave a legacy we are proud of. We are challenging the status quo by transforming into a new software company. And as your partner, we work tirelessly to make sure your information technology is designed for you and by you. This is our mission, and we are in this together. Welcome to a new Quest. You are invited to Join the Innovation.
Our logo reflects our story: innovation, community and support. An important part of this story begins with the letter Q. It is a perfect circle, representing our commitment to technological precision and strength. The space in the Q itself symbolizes our need to add the missing piece—you—to the community, to the new Quest.
For sales or other inquiries, visit www.quest.com/contact.
For the Beta version of Benchmark Factory, technical support is provided through the Benchmark Factory community at: http://www.toadworld.com/products/benchmark-factory/benchmark_factory_for_databases_beta_program/default.aspx
© 2018 Quest Software Inc.
ALL RIGHTS RESERVED.
This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser’s personal use without the written permission of Quest Software Inc.
The information in this document is provided in connection with Quest Software products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest Software products. EXCEPT AS SET FORTH IN THE TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST SOFTWARE ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST SOFTWARE BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST SOFTWARE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest Software makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest Software does not make any commitment to update the information contained in this document.
If you have any questions regarding your potential use of this material, contact:
Quest Software Inc.
Attn: LEGAL Dept
4 Polaris Way
Aliso Viejo, CA 92656
Refer to our web site (www.quest.com) for regional and international office information.
Patents
This product includes patent pending technology. For the most current information about applicable patents for this product, please visit our website at www.quest.com/legal.
Trademarks
Quest, Quest Software, Benchmark Factory, Foglight, Spotlight, SQL Navigator, Toad, SharePlex, and the Quest logo are trademarks of Quest Software Inc. in the U.S.A. and other countries. For a complete list of Quest Software trademarks, please visit our website at www.quest.com/legal. Microsoft, Windows, Windows Server, Visual Studio, SQL Server, SharePoint, Access and Excel are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Oracle is a trademark or registered trademark of Oracle and/or its affiliates in the United States and other countries. Citrix® and XenApp™ are trademarks of Citrix Systems, Inc. and/or one or more of its subsidiaries, and may be registered in the United States Patent and Trademark Office and in other countries. SAP is the registered trademark of SAP AG in Germany and in several other countries. IBM and DB2 are registered trademarks of International Business Machines Corporation. All other trademarks, servicemarks, registered trademarks, and registered servicemarks are property of their respective owners.