Skip to main content

v3.0

StarRocks version 3.0

3.0.6​

Release date: September 12, 2023

New Features​

Improvements​

  • Supports implicit conversions for all compound predicates and for all expressions in the WHERE clause. You can enable or disable implicit conversions by using the session variable enable_strict_type. The default value of this session variable is false. #21870
  • Unifies the logic between FEs and BEs in converting strings to integers. #29969

Bug Fixes​

  • If enable_orc_late_materialization is set to true, an unexpected result is returned when a Hive catalog is used to query STRUCT-type data in ORC files. #27971
  • During data queries through Hive Catalog, if a partitioning column and an OR operator are specified in the WHERE clause, the query result is incorrect. #28876
  • The values returned by the RESTful API action show_data for cloud-native tables are incorrect. #29473
  • If the shared-data cluster stores data in Azure Blob Storage and a table is created, the FE fails to start after the cluster is rolled back to version 3.0. #29433
  • A user has no permission when querying a table in the Iceberg catalog even if the user is granted permission on that table. #29173
  • The Default field values returned by the SHOW FULL COLUMNS statement for columns of the BITMAP or HLL data type are incorrect. #29510
  • Modifying the FE dynamic parameter max_broker_load_job_concurrency using the ADMIN SET FRONTEND CONFIG command does not take effect.
  • The FE may fail to start when a materialized view is being refreshed while its refresh strategy is being modified. #29964 #29720
  • The error unknown error is returned when select count(distinct(int+double)) from table_name is executed. #29691
  • After a Primary Key table is restored, metadata errors occur and cause metadata inconsistencies occur if a BE is restarted. #30135

3.0.5​

Release date: August 16, 2023

New Features​

Improvements​

Bug Fixes​

Fixed the following issues:

  • Incorrect table-level scan statistics in FE cause inaccurate metrics for table queries and loading. #27779
  • The query result is not stable if the sort key is modified for a partitioned table. #27850
  • The version number for a tablet is inconsistent between the BE and FE after data is restored. #26518
  • If the bucket number is not specified when users create a Colocation table, the number will be inferred as 0, which causes failures in adding new partitions. #27086
  • When the SELECT result set of INSERT INTO SELECT is empty, the load job status returned by SHOW LOAD is CANCELED. #26913
  • BEs may crash when the input values of the sub_bitmap function are not of the BITMAP type. #27982
  • BEs may crash when the AUTO_INCREMENT column is being updated. #27199
  • Outer join and Anti join rewrite errors for materialized views. #28028
  • Inaccurate estimation of average row size causes Primary Key partial updates to occupy excessively large memory. #27485
  • Activating an inactive materialized view may cause a FE to crash. #27959
  • Queries can not be rewritten to materialized views created based on external tables in a Hudi catalog. #28023
  • The data of a Hive table can still be queried even after the table is dropped and the metadata cache is manually updated. #28223
  • Manually refreshing an asynchronous materialized view via a synchronous call results in multiple INSERT OVERWRITE records in the information_schema.task_runs table. #28060
  • FE memory leak caused by blocked LabelCleaner threads. #28311

3.0.4​

Release date: July 18, 2023

New Feature​

Queries can be rewritten even when the queries contain a different type of join than the materialized view. #25099

Improvements​

  • Optimized the manual refreshing of asynchronous materialized views. Supports using the REFRESH MATERIALIZED VIEW WITH SYNC MODE syntax to synchronously invoke materialized view refresh tasks. #25910
  • If the queried fields are not included in the output columns of a materialized view but are included in the predicate of the materialized view, the query can still be rewritten to benefit from the materialized view. #23028
  • When the SQL dialect (sql_dialect) is set to trino, table aliases are not case-sensitive. #26094 #25282
  • Added a new field table_id to the table Information_schema.tables_config. You can join the table tables_config with the table be_tablets on the column table_id in the database Information_schema to query the names of the database and table to which a tablet belongs. #24061

Bug Fixes​

Fixed the following issues:

  • If a query that contains the sum aggregate function is rewritten to directly obtain query results from a single-table materialized view, the values in sum() field may be incorrect due to type inference issues. #25512
  • An error occurs when SHOW PROC is used to view information about tablets in a StarRocks shared-data cluster.
  • The INSERT operation hangs when the length of CHAR data in a STRUCT to be inserted exceeds the maximum length. #25942
  • Some data rows queried fail to be returned for INSERT INTO SELECT with FULL JOIN. #26603
  • An error ERROR xxx: Unknown table property xxx occurs when the ALTER TABLE statement is used to modify the table's property default.storage_medium. #25870
  • An error occurs when Broker Load is used to load empty files. #26212
  • Decommissioning a BE sometimes hangs. #26509

3.0.3​

Release date: June 28, 2023

Improvements​

  • Metadata synchronization of StarRocks external tables has been changed to occur during data loading. #24739
  • Users can specify partitions when they run INSERT OVERWRITE on tables whose partitions are automatically created. For more information, see Automatic partitioning. #25005
  • Optimized the error message reported when partitions are added to a non-partitioned table. #25266

Bug Fixes​

Fixed the following issues:

  • The min/max filter gets the wrong Parquet field when the Parquet file contains complex data types. #23976
  • Load tasks are still queuing even when the related database or table has been dropped. #24801
  • There is a low probability that an FE restart may cause BEs to crash. #25037
  • Load and query jobs occasionally freeze when the variable enable_profile is set to true. #25060
  • Inaccurate error message is displayed when INSERT OVERWRITE is executed on a cluster with less than three alive BEs. #25314

3.0.2​

Release date: June 13, 2023

Improvements​

  • Predicates in a UNION query can be pushed down after the query is rewritten by an asynchronous materialized view. #23312
  • Optimized the auto tablet distribution policy for tables. #24543
  • Removed the dependency of NetworkTime on system clocks, which fixes incorrect NetworkTime caused by inconsistent system clocks across servers. #24858

Bug Fixes​

Fixed the following issues:

  • A schema change sometimes may be hung if data loading occurs simultaneously with the schema change. #23456
  • Queries encounter an error when the session variable pipeline_profile_level is set to 0. #23873
  • CREATE TABLE encounters an error when cloud_native_storage_type is set to S3.
  • LDAP authentication succeeds even when no password is used. #24862
  • CANCEL LOAD fails if the table involved in the load job does not exist. #24922

Upgrade Notes​

If your system has a database named starrocks, change it to another name using ALTER DATABASE RENAME before the upgrade. This is because starrocks is the name of a default system database that stores privilege information.

3.0.1​

Release date: June 1, 2023

New Features​

  • [Preview] Supports spilling intermediate computation results of large operators to disks to reduce the memory consumption of large operators. For more information, see Spill to disk.
  • Routine Load supports loading Avro data.
  • Supports Microsoft Azure Storage (including Azure Blob Storage and Azure Data Lake Storage).

Improvements​

  • Shared-data clusters support using StarRocks external tables to synchronize data with another StarRocks cluster.
  • Added load_tracking_logs to Information Schema to record recent loading errors.
  • Ignores special characters in CREATE TABLE statements. #23885

Bug Fixes​

Fixed the following issues:

  • Information returned by SHOW CREATE TABLE is incorrect for Primary Key tables. #24237
  • BEs may crash during a Routine Load job. #20677
  • Null pointer exception (NPE) occurs if you specify unsupported properties when creating a partitioned table. #21374
  • Information returned by SHOW TABLE STATUS is incomplete. #24279

Upgrade Notes​

If your system has a database named starrocks, change it to another name using ALTER DATABASE RENAME before the upgrade. This is because starrocks is the name of a default system database that stores privilege information.

3.0.0​

Release date: April 28, 2023

New Features​

System architecture​

  • Decouple storage and compute. StarRocks now supports data persistence into S3-compatible object storage, enhancing resource isolation, reducing storage costs, and making compute resources more scalable. Local disks are used as hot data cache for boosting query performance. The query performance of the new shared-data architecture is comparable to the classic architecture (shared-nothing) when local disk cache is hit. For more information, see Deploy and use shared-data StarRocks.

Storage engine and data ingestion​

  • The AUTO_INCREMENT attribute is supported to provide globally unique IDs, which simplifies data management.
  • Automatic partitioning and partitioning expressions are supported, which makes partition creation easier to use and more flexible.
  • Primary Key tables support more complete UPDATE and DELETE syntax, including the use of CTEs and references to multiple tables.
  • Added Load Profile for Broker Load and INSERT INTO jobs. You can view the details of a load job by querying the load profile. The usage is the same as Analyze query profile.

Data Lake Analytics​

  • [Preview] Supports Presto/Trino compatible dialect. Presto/Trino's SQL can be automatically rewritten into StarRocks' SQL pattern. For more information, see the system variable sql_dialect.
  • [Preview] Supports JDBC catalogs.
  • Supports using SET CATALOG to manually switch between catalogs in the current session.

Privileges and security​

  • Provides a new privilege system with full RBAC functionalities, supporting role inheritance and default roles. For more information, see Overview of privileges.
  • Provides more privilege management objects and more fine-grained privileges. For more information, see Privileges supported by StarRocks.

Query engine​

  • Allows more queries on joined tables to benefit from the query cache. For example, the query cache now supports Broadcast Join and Bucket Shuffle Join.
  • Supports Global UDFs.
  • Dynamic adaptive parallelism: StarRocks can automatically adjust the pipeline_dop parameter for query concurrency.

SQL reference​

Improvements​

Deployment​

Storage engine and data ingestion​

  • Supports more CSV parameters for data ingestion, including SKIP_HEADER, TRIM_SPACE, ENCLOSE, and ESCAPE. See STREAM LOAD, BROKER LOAD, and ROUTINE LOAD.
  • The primary key and sort key are decoupled in Primary Key tables. The sort key can be separately specified in ORDER BY when you create a table.
  • Optimized the memory usage of data ingestion into Primary Key tables in scenarios such as large-volume ingestion, partial updates, and persistent primary indexes.
  • Supports creating asynchronous INSERT tasks. For more information, see INSERT and SUBMIT TASK. #20609

Materialized view​

  • Optimized the rewriting capabilities of materialized views, including:
    • Supports rewrite of View Delta Join, Outer Join, and Cross Join.
    • Optimized SQL rewrite of Union with partition.
  • Improved materialized view building capabilities: supporting CTE, select *, and Union.
  • Optimized the information returned by SHOW MATERIALIZED VIEWS.
  • Supports adding MV partitions in batches, which improves the efficiency of partition addition during materialized view building. #21167

Query engine​

  • All operators are supported in the pipeline engine. Non-pipeline code will be removed in later versions.
  • Improved Big Query Positioning and added big query log. SHOW PROCESSLIST supports viewing CPU and memory information.
  • Optimized Outer Join Reorder.
  • Optimized error messages in the SQL parsing stage, providing more accurate error positioning and clearer error messages.

Data Lake Analytics​

  • Optimized metadata statistics collection.
  • Supports using SHOW CREATE TABLE to view the creation statements of the tables that are managed by an external catalog and are stored in Apache Hive™, Apache Iceberg, Apache Hudi, or Delta Lake.

Bug Fixes​

  • Some URLs in the license header of StarRocks' source file cannot be accessed. #2224
  • An unknown error is returned during SELECT queries. #19731
  • Supports SHOW/SET CHARACTER. #17480
  • When the loaded data exceeds the field length supported by StarRocks, the error message returned is not correct. #14
  • Supports show full fields from 'table'. #17233
  • Partition pruning causes MV rewrites to fail. #14641
  • MV rewrite fails when the CREATE MATERIALIZED VIEW statement contains count(distinct) and count(distinct) is applied to the DISTRIBUTED BY column. #16558
  • FEs fail to start when a VARCHAR column is used as the partitioning column of a materialized view. #19366
  • Window functions LEAD and LAG incorrectly handle IGNORE NULLS. #21001
  • Adding temporary partitions conflicts with automatic partition creation. #21222

Behavior Change​

  • The new role-based access control (RBAC) system supports the previous privileges and roles. However, the syntax of related statements such as GRANT and REVOKE is changed.
  • Renamed SHOW MATERIALIZED VIEW as SHOW MATERIALIZED VIEWS.
  • Added the following Reserved keywords: AUTO_INCREMENT, CURRENT_ROLE, DEFERRED, ENCLOSE, ESCAPE, IMMEDIATE, PRIVILEGES, SKIP_HEADER, TRIM_SPACE, VARBINARY.

Upgrade Notes​

You can upgrade from v2.5 to v3.0 or downgrade from v3.0 to v2.5.

In theory, an upgrade from a version earlier than v2.5 is also supported. To ensure system availability, we recommend that you first upgrade your cluster to v2.5 and then to v3.0.

Take note of the following points when you perform a downgrade from v3.0 to v2.5.

BDBJE​

StarRocks upgrades the BDB library in v3.0. However, BDBJE cannot be rolled back. You must use BDB library of v3.0 after a downgrade. Perform the following steps:

  1. After you replace the FE package with a v2.5 package, copy fe/lib/starrocks-bdb-je-18.3.13.jar of v3.0 to the fe/lib directory of v2.5.
  2. Delete fe/lib/je-7.*.jar.

Privilege system​

The new RBAC privilege system is used by default after you upgrade to v3.0. You can only downgrade to v2.5.

After a downgrade, run ALTER SYSTEM CREATE IMAGE to create a new image and wait for the new image to be synchronized to all follower FEs. If you do not run this command, some of the downgrade operations may fail. This command is supported from 2.5.3 and later.

For details about the differences between the privilege system of v2.5 and v3.0, see "Upgrade notes" in Privileges supported by StarRocks.