HIVE-29281: Make proactive cache eviction work with catalog by Neer393 · Pull Request #6379 · apache/hive

Neer393 · 2026-03-19T15:41:37Z

What changes were proposed in this pull request?

Made the proactive cache eviction catalog aware by making changes in ProactiveCacheEviction file and the CacheTag file.

Why are the changes needed?

The proactive cache eviction should be catalog aware otherwise same name tables under different catalogs may cause false cache hits/miss. To avoid this, the cache eviction should be aware of the catalog.

Does this PR introduce any user-facing change?

No user facing changes as user does not know about the proactive cache eviction.

How was this patch tested?

Added unit tests for with and without catalog and all of them passed. Not sure how to manually test proactive cache eviction so verified only via unit tests

Neer393 · 2026-03-20T08:51:12Z

@zhangbutao I need a review here. I looked at all the merged PRs under HIVE-22820 and have made changes accordingly for making it catalog aware. Please help me here if I missed anything. Thanks

Copilot

Pull request overview

This PR makes LLAP proactive cache eviction catalog-aware by propagating catalog names through cache tags and eviction requests, preventing collisions when identical db/table names exist across catalogs.

Changes:

Extend LLAP proactive eviction request structure to include catalog scoping (catalog → db → table → partitions).
Introduce catalog tracking on TableDesc/PartitionDesc and update cache-tag generation to include catalog-qualified names.
Update LLAP cache metadata serialization and unit tests to reflect catalog-qualified cache tags.

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
storage-api/src/java/org/apache/hadoop/hive/common/io/CacheTag.java	Updates cache tag semantics/docs and parent-tag derivation to preserve catalog prefix.
ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java	Adds `catalogName` field and updates constructors/clone to carry catalog without polluting EXPLAIN.
ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java	Exposes catalog name via `PartitionDesc` based on `TableDesc`, with default fallback.
ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java	Makes eviction requests catalog-scoped and includes catalog in proto requests and tag matching.
llap-common/src/protobuf/LlapDaemonProtocol.proto	Adds catalog name field to `EvictEntityRequestProto`.
ql/src/java/org/apache/hadoop/hive/llap/LlapHiveUtils.java	Prefixes cache tags with catalog when deriving metrics tags.
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java	Adjusts eviction debug logging for catalog+db structure.
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapCacheMetadataSerializer.java	Adds backward-ish handling for cache tags missing catalog during decode.
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java	Updates synthetic tag to include default catalog prefix.
ql/src/java/org/apache/hadoop/hive/ql/ddl/**	Ensures eviction builders are invoked with catalog where available.
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java	Ensures `TableDesc` created from `Table` carries catalog name.
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java	Updates `TableDesc` construction call sites for new signature.
Various test files	Update existing tests and add new coverage for catalog-aware eviction and proto round-trips.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

llap-common/src/protobuf/LlapDaemonProtocol.proto

Copilot · 2026-03-23T11:48:27Z

ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java

+      /**
+       * Add a partition of a table scoped to the given catalog.
+       */
+      public Builder addPartitionOfATable(String catalog, String db, String tableName,
+          LinkedHashMap<String, String> partSpec) {
+        ensureTable(catalog, db, tableName);
+        entities.get(catalog).get(db).get(tableName).add(partSpec);
        return this;


Request.Builder claims the catalog key defaults to Warehouse.DEFAULT_CATALOG_NAME, but the builder currently stores the catalog parameter as-is. If a caller passes null (or an empty string), this will create a null key and later NPE in toProtoRequests() when calling toLowerCase(). Normalize catalog (and arguably db/table) at the builder boundary, e.g. default null/blank catalog to the default catalog and enforce non-null keys.

All calls made to this will never be null.

llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapCacheMetadataSerializer.java

ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java

Neer393 · 2026-03-25T07:07:25Z

@zhangbutao resolved the copilot's reviews

zhangbutao · 2026-03-25T12:59:50Z

@zhangbutao resolved the copilot's reviews

Thanks for pinging me. I will do the code review later.

itests/hive-jmh/src/main/java/org/apache/hive/benchmark/ql/exec/KryoBench.java

llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapCacheMetadataSerializer.java

llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java

llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java

llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestCacheContentsTracker.java

ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java

deniskuzZ · 2026-03-27T14:23:35Z

ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java

+    // Holds a hierarchical structure of catalogs, DBs, tables and partitions such as:
+    // { hive : { testdb : { testtab0 : [], testtab1 : [ {pk0 : p0v0, pk1 : p0v1} ] }, testdb2 : {} } }
+    // The catalog key defaults to Warehouse.DEFAULT_CATALOG_NAME ("hive") when no explicit catalog is given.
+    private final Map<String, Map<String, Map<String, Set<LinkedHashMap<String, String>>>>> entities;


can we refactor

record PartitionSpec(Map<String,String> spec) {} Map<String, Map<String, Map<String, Set<PartitionSpec>>>> entities;

deniskuzZ · 2026-03-27T14:27:29Z

ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java

+     * @return the single DB name, null if the count of (catalog, DB) pairs present is not exactly 1.
+     */
+    public String getSingleDbName() {
+      String catalog = getSingleCatalogName();


shouldn't this be done for a specific catalog and not first?

As per the flow I understand, entities will have multiple catalogs, under each catalog there may be multiple dbs and under each db there may be multiple tables and their partitions

Now toProtoRequests() will separately create proto messages for each of the [catalogname, dbname and list of tables] and send it to the LLAP daemons. So the LLAP daemons will convert them to entities using fromProtoRequest() so ideally when isTagMatch() is called, it would essentially contain only one catalog, one db under it and multiple tables under it. So getSingleCatalog() and getSingleDb() would work correctly as expected and get us the first one.

This is my understanding. Correct me if I am wrong

ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java

deniskuzZ · 2026-03-27T14:31:22Z

ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java

     * @return true if cacheTag matches and the related buffer is eligible for proactive eviction, false otherwise.
     */
    public boolean isTagMatch(CacheTag cacheTag) {
+      String catalog = getSingleCatalogName();


should we use default catalog here? otherwise the whole thing seems useless

I don't get how am I using the default catalog here ?
As per the flow I understand, entities will have multiple catalogs, under each catalog there may be multiple dbs and under each db there may be multiple tables and their partitions

Now toProtoRequests() will separately create proto messages for each of the [catalogname, dbname and list of tables] and send it to the LLAP daemons. So the LLAP daemons will convert them to entities using fromProtoRequest() so ideally when isTagMatch() is called, it would essentially contain only one catalog, one db under it and multiple tables under it. So getSingleCatalog() and getSingleDb() would work correctly as expected.

And nowhere I see a hardcoding for the DEFAULT WAREHOUSE.
Please let me know if I am missing something here.

There is a key issue that needs to be further optimized in the future: not all catalogs support caching. Currently, I assume that only the native catalog supports caching, while some other catalogs, such as the JDBC catalog, may have difficulty supporting caching.

So I think we can create a ticket to record this issue.

Created - HIVE-29537

fromProtoRequest() could just store catalogName and dbName as direct fields instead of stuffing them into the nested map.
Then isTagMatch() wouldn't need the getSingle* methods at all.

ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java

ql/src/java/org/apache/hadoop/hive/ql/ddl/database/drop/DropDatabaseOperation.java

ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java

ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java

storage-api/src/java/org/apache/hadoop/hive/common/io/CacheTag.java

zhangbutao · 2026-03-31T02:44:29Z

llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapCacheMetadataSerializer.java

  }

  private static CacheTag decodeCacheTag(LlapDaemonProtocolProtos.CacheTag ct) {
-    return ct.getPartitionDescCount() == 0 ? CacheTag.build(ct.getTableName()) : CacheTag


What about handling old cached data? Will there be any compatibility issues here?
I'm not very familiar with LLAP caching—perhaps a cache decoding failure won't cause user sql jobs to fail? Maybe we just need to restart LLAP to update the cache?

I don't think it would be an issue. I think we should document that post upgrade LLAP daemons would need a restart to flush the cache. This however does not have any functional impact. Worst case if the user does not restarts, they would face cache miss but no SQL job failures as far as I think.
@deniskuzZ any idea on this ?

ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java

zhangbutao · 2026-03-31T03:17:15Z

ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java

     * @return true if cacheTag matches and the related buffer is eligible for proactive eviction, false otherwise.
     */
    public boolean isTagMatch(CacheTag cacheTag) {
+      String catalog = getSingleCatalogName();


There is a key issue that needs to be further optimized in the future: not all catalogs support caching. Currently, I assume that only the native catalog supports caching, while some other catalogs, such as the JDBC catalog, may have difficulty supporting caching.

So I think we can create a ticket to record this issue.

llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapCacheMetadataSerializer.java

sonarqubecloud · 2026-04-02T07:53:05Z

Quality Gate passed

Issues
18 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.3% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added tests pending tests failed and removed tests pending labels Mar 19, 2026

Neer393 force-pushed the HIVE-29281 branch from e1b34fa to 220bddf Compare March 19, 2026 19:27

asf-ci-hive added tests pending tests unstable and removed tests failed tests pending labels Mar 19, 2026

Neer393 force-pushed the HIVE-29281 branch from 220bddf to 642a058 Compare March 20, 2026 05:19

asf-ci-hive added tests pending tests passed and removed tests unstable tests pending labels Mar 20, 2026

zhangbutao requested a review from Copilot March 23, 2026 11:43

Copilot started reviewing on behalf of zhangbutao March 23, 2026 11:43 View session

Copilot AI reviewed Mar 23, 2026

View reviewed changes

Neer393 force-pushed the HIVE-29281 branch from 642a058 to 59a0b8f Compare March 23, 2026 13:15

asf-ci-hive added tests pending tests passed and removed tests passed tests pending labels Mar 23, 2026